Planning Report 02-3

The Economic

Impacts of Inadequate

Infrastructure for 

Software Testing

Prepared by:

RTI

for

National Institute of

Standards & Technology

Program Office

Strategic Planning and

Economic Analysis Group

May 2002

U.S Department of Commerce

Technology Administration
 RTI Project Number 

 7007.011 

The Economic Impacts 

of Inadequate 

Infrastructure for 

Software Testing

Final Report

May 2002

Prepared for 

Gregory Tassey, Ph.D. 

National Institute of Standards and Technology 

Acquisition and Assistance Division 

Building 101, Room A1000 

Gaithersburg, MD 20899-0001 

Prepared by 

RTI 

Health, Social, and Economics Research 

Research Triangle Park, NC 27709 
iii 

Contents 

 Executive Summary ES-1

 1. Introduction to Software Quality and Testing 1-1

1.1 Software Quality Attributes................................................ 1-3

1.2 Software Quality Metrics ................................................... 1-6

1.2.1 What Makes a Good Metric ................................... 1-7

1.2.2 What Can be Measured ......................................... 1-8

1.2.3 Choosing Among Metrics....................................... 1-8

1.3 Software Testing.............................................................. 1-10

1.4 The Impact of Inadequate Testing .................................... 1-11

1.4.1 Failures due to Poor Quality................................. 1-11

1.4.2 Increased Software Development Costs ................ 1-12

1.4.3 Increased Time to Market..................................... 1-12

1.4.4 Increased Market Transaction Costs...................... 1-13

 2. Software Testing Methods and Tools 2-1

2.1 Historical Approach to Software Development................... 2-1

2.2 Software Testing Infrastructure........................................... 2-4

2.2.1 Software Testing Stages.......................................... 2-4

2.2.2 Commercial Software Testing Tools........................ 2-7

2.3 Software Testing Types ...................................................... 2-9

2.3.1 Conformance Testing............................................. 2-9

2.3.2 Interoperability Testing ........................................ 2-10

2.3.4 Relationship between Software Stages, Testing 

Types, and Testing Tools...................................... 2-13

2.3.5 Standardized Software Testing Technologies ........ 2-15

 3. Inadequate Infrastructure for Software Testing: 

Overview and Conceptual Model 3-1

3.1 Software Testing Inadequacies........................................... 3-2
iv 

3.1.1 Integration and Interoperability Testing Issues......... 3-2

3.1.2 Automated Generation of Test Code....................... 3-3

3.1.3 Lack of a Rigorous Method for Determining 

When a Product Is Good Enough to Release........... 3-3

3.1.4 Lack of Readily Available Performance Metrics 

and Testing Procedures.......................................... 3-4

3.1.5 Approaches for Improving Software Testing 

Infrastructure ......................................................... 3-5

3.2 Conceptual Economic Model............................................. 3-6

3.3 Software Developers ......................................................... 3-7

3.3.1 Cost Framework..................................................... 3-7

3.3.2 Factors Influencing the Profit-Maximizing Level 

of R&D Expenditures.............................................. 3-9

3.4 End Users........................................................................ 3-12

3.4.1 Cost Framework................................................... 3-12

3.5 The Market for Software Products .................................... 3-14

3.5.1 Quality’s Impact on Market Prices........................ 3-14

3.6 Modeling an Inadequate Software Testing 

Infrastructure................................................................... 3-15

3.6.1 Inadequate Infrastructure’s Impact on the Cost 

of Quality............................................................ 3-17

3.6.2 Inadequate Infrastructure’s Impact on the Cost 

of After-Sales Service ........................................... 3-18

3.6.3 Inadequate Infrastructure’s Impact on EndUsers’ Demand.................................................... 3-19

3.6.4 Aggregate Impact................................................. 3-19

3.7 The Time Dimension....................................................... 3-20

3.8 Conclusion...................................................................... 3-21

 4. Taxonomy for Software Testing Costs 4-1

4.1 Principles that Drive Software Testing Objectives............... 4-1

4.1.1 Testing Activities.................................................... 4-2

4.1.2 Detecting Bugs Sooner........................................... 4-3

4.1.3 Locating the Source of Bugs Faster and with 

More Precision ...................................................... 4-3

4.2 Software Developers’ Cost Taxonomy................................ 4-3

4.2.1 Resource Categories .............................................. 4-4
v 

4.2.2 Summary of Developer Technical and 

Economic Metrics.................................................. 4-6

4.3 Software Users’ Cost Taxonomy......................................... 4-7

4.3.1 Pre-purchase Costs ................................................ 4-8

4.3.2 Installation Costs.................................................... 4-9

4.3.3 Post-purchase Costs ............................................. 4-11

 5. Measuring the Economic Impacts of an 

Inadequate Infrastructure for Software Testing 5-1

5.1 Defining the Counterfactual World .................................... 5-1

5.1.1 Developers’ Costs of Identifying and Correcting 

Errors..................................................................... 5-3

5.1.2 Counterfactual Scenario for Developers ................. 5-8

5.1.3 Counterfactual Scenario for Users .......................... 5-9

5.2 Custom Versus Commercial Software Products .................. 5-9

5.3 Estimating Software Developer Costs ............................... 5-11

5.4 Estimating Software User Costs ........................................ 5-13

5.5 Period of Analysis............................................................ 5-17

5.6 Industry-Specific User Costs ............................................ 5-19

 6. Transportation Manufacturing Sector 6-1

6.1 Overview of CAD/CAM/CAE and PDM Software in the 

Transportation Manufacturing Sector ................................. 6-2

6.1.1 Use of CAD/CAM/CAE and PDM Software ............. 6-3

6.1.2 Development of CAD/CAM/CAE and PDM 

Software ................................................................ 6-5

6.2 Software Developer Costs in the Transportation 

Manufacturing Sector ........................................................ 6-6

6.2.1 Estimation Approach.............................................. 6-8

6.2.2 Survey Findings ..................................................... 6-9

6.2.3 Cost Impacts Per Employee for Software 

Developers.......................................................... 6-13

6.2.4 Industry-Level Impact........................................... 6-14

6.3 End-User Costs in the Transportation Manufacturing 

Sector ............................................................................. 6-15

6.3.1 Survey Method .................................................... 6-15

6.3.2 Survey Response Rates and Industry Coverage...... 6-16

6.3.3 Survey Findings ................................................... 6-18
vi 

6.3.4 Costs of Bugs and Errors Per Employee ................. 6-23

6.3.5 Partial Reduction of Software Errors...................... 6-26

6.4 Users’ Industry-Level Impact Estimates............................. 6-27

 7. Financial Services Sector 7-1

7.1 Overview of the Use of Clearinghouse Software and 

Routers and Switches in the Financial Services Sector ........ 7-2

7.1.1 Overview of Electronic Transactions in the 

Financial Services Sector........................................ 7-3

7.1.2 Software Used by Financial Services Providers ....... 7-5

7.1.3 Software Embedded in Hardware Used to 

Support Financial Transactions............................... 7-6

7.2 Software Developer Costs in the Financial Services 

Sector ............................................................................... 7-8

7.2.1 Industry Surveys................................................... 7-10

7.2.2 Survey Findings ................................................... 7-10

7.2.3 Cost Impacts Per Employee for Software 

Developers.......................................................... 7-14

7.2.4 Industry-Level Impacts......................................... 7-15

7.3 Software User Costs in the Financial Services Sector ........ 7-16

7.3.1 Survey Method .................................................... 7-16

7.3.2 Survey Response Rates and Industry Coverage...... 7-17

7.3.3 Survey Findings .................................................. 7-19

7.3.4 Software User Costs Per Transaction..................... 7-24

7.3.5 Partial Reduction of Software Errors...................... 7-26

7.3.6 Users’ Industry-Level Impact Estimates ................. 7-28

 8. National Impact Estimates 8-1

8.1 Per-Employee Testing Costs: Software Developers............. 8-2

8.2 Per-Employee Costs: Software Users ................................. 8-4

8.4 National Impact Estimates ................................................. 8-5

8.5 Limitations and Caveats..................................................... 8-6

 References R-1
vii 

 Appendixes 

A: Glossary of Testing Stages and Tools..................................A-1 

B: CAD/CAM/CAE/PDM Use and Development in the 

Transportation Sector ........................................................ B-1 

C: CAD/CAM/CAE/PDM Developers and Users Survey 

Instruments .......................................................................C-1 

D: Financial Services Software Use and Development ............D-1 

E: Financial Services Survey Instruments.................................E-1 
viii 

Figures 

 Figure 2-1 Waterfall Model......................................................................... 2-3 

 Figure 2-2 Commercial Software Testing Infrastructure Hierarchy................ 2-5 

 Figure 3-1 Software Quality’s Role in Profit Maximization......................... 3-10

 Figure 3-2 Minimize Joint Costs of Pre-sales Testing and After-Sales 

Service (Holding Price and Quantity Constant) ......................... 3-11 

 Figure 3-3 Change in Quality’s Impact on Price, Quantity, and Net 

Revenue .................................................................................. 3-16

 Figure 3-4 Enhanced Testing Tool’s Impact on the Marginal Cost of 

Quality .................................................................................... 3-18 

 Figure 5-1 The Waterfall Process................................................................. 5-3 

 Figure 5-2 Typical Cumulative Distribution of Error Detection..................... 5-6 

 Figure 5-3 Software Testing Costs Shown by Where Bugs Are Detected 

(Example Only).......................................................................... 5-7 

 Figure 5-4 Cost Reductions of Detecting Bugs and Fixing Them Faster 

(Example Only).......................................................................... 5-8 

 Figure 5-5 Custom vs. Commercial Development Cost Allocation ............. 5-10 

 Figure 5-6 Relationship between Users Costs and Percentage Reduction 

in Bugs .................................................................................... 5-18 

 Figure 6-1 Economic Relationship Among CAD/CAM/CAE Producers 

and Consumers.......................................................................... 6-3 

 Figure 6-2 CAD/CAE/CAM and PDM in the Product Development 

Cycle......................................................................................... 6-4 
ix 

Tables 

 Table 1-1 McCall, Richards, and Walters’s Software Quality Attributes....... 1-4 

 Table 1-2 ISO Software Quality Attributes.................................................. 1-5 

 Table 1-3 List of Metrics Available............................................................. 1-7 

 Table 1-4 Recent Aerospace Losses due to Software Failures .................... 1-11 

 Table 1-5 Relative Costs to Repair Defects when Found at Different 

Stages of the Life-Cycle ............................................................ 1-13 

 Table 2-1 Allocation of Effort..................................................................... 2-4 

 Table 2-2 The Degree of Usage of the Different Testing Stages with the 

Various Testing Types .............................................................. 2-13 

 Table 2-3 Software Testing Types Associated with the Life Cycle .............. 2-14 

 Table 2-4 Tools Used by Type of Testing.................................................. 2-16 

 Table 2-5 Tools Used by Testing Stage..................................................... 2-17 

 Table 4-1 Labor Taxonomy........................................................................ 4-4 

 Table 4-2 Software Testing Capital Taxonomy............................................ 4-5 

 Table 4-3 Impact Cost Metrics for Software Developers.............................. 4-7 

 Table 4-4 Users’ Pre-Purchase Costs Associated with Bugs ......................... 4-9 

 Table 4-5 Users’ Implementation Costs Associated with Bugs................... 4-10 

 Table 4-6 Users’ Post-purchase Costs Associated with Bugs...................... 4-11 

 Table 5-1 Relative Cost to Repair Defects When Found at Different 

Stages of Software Development (Example Only)........................ 5-4 

 Table 5-2 Preliminary Estimates of Relative Cost Factors of Correcting 

Errors as a Function of Where Errors Are Introduced and 

Found (Example Only) ............................................................... 5-4 
x 

 Table 5-3 Example of the Frequency (%) of Where Errors Are Found, in 

Relationship to Where They Were Introduced ............................ 5-5 

 Table 5-4 Impact Cost Metrics for Software Developers............................ 5-12 

 Table 5-5 Cost Metrics for Users .............................................................. 5-16 

 Table 5-6 Importance of Quality Attributes in the Transportation 

Equipment and Financial Services Industries............................. 5-20 

 Table 6-1 Cost Impacts on U.S. Software Developers and Users in the 

Transportation Manufacturing Sector Due to an Inadequate 

Testing Infrastructure ($ millions)................................................ 6-2 

 Table 6-2 Distribution of Bugs Found Based on Introduction Point ........... 6-10 

 Table 6-3 Hours to Fix Bug Based on Introduction Point........................... 6-10 

 Table 6-4 Time to Fix a Bug Based on Discovery Point............................. 6-11 

 Table 6-5 Distribution of Bugs Based on Infrastructure ............................. 6-12 

 Table 6-6 Developer Testing Costs for a Typical Company of 10,000 

Employees............................................................................... 6-13 

 Table 6-7 Annual Impact on U.S. Software Developers of 

CAD/CAM/CAE/PDM Software................................................. 6-14 

 Table 6-8 Transportation Equipment Industry Survey Completion Rates 

............................................................................................... 6-17 

 Table 6-9 Industry Coverage by Employment ........................................... 6-17 

 Table 6-10 Reported Software Products...................................................... 6-19 

 Table 6-11 Incidence and Costs of Software Bugs....................................... 6-21 

 Table 6-12 Average Company-Level Costs of Search, Installation, and 

Maintenance (Life-Cycle Costs) ................................................ 6-22 

 Table 6-13 Costs Per Employee.................................................................. 6-24 

 Table 6-14 Company-Level Costs Associated with Bugs for Hypothetical 

Transportation Company at Different Employment Levels ......... 6-25 

 Table 6-15 Cost Reductions as a Function of Bug Reductions ..................... 6-27 

 Table 6-16 Annual Impacts’ Weighted Cost Per Deposits and Loans........... 6-28 

 Table 7-1 Cost Impacts on U.S. Software Developers and Users in the 

Financial Services Sector Due to an Inadequate Testing 

Infrastructure ($ millions) ........................................................... 7-2 

 Table 7-2 Characteristics of Firms in the Financial Services Sector, 

1997.......................................................................................... 7-4 

 Table 7-3 Router Market Shares of Major Firms.......................................... 7-6 

 Table 7-4 Distribution of Bugs Found Based on Introduction Point ........... 7-11 

 Table 7-5 Hours to Fix Bug based on Introduction Point........................... 7-12 

 Table 7-6 Time to Fix a Bug Based on Discovery Point............................. 7-13 
xi 

 Table 7-7 Shift in the Distribution of Where Bugs are Found Based on 

Infrastructure ........................................................................... 7-13 

 Table 7-8 Developer Testing Costs for a Typical Company of 10,000 

Employees............................................................................... 7-15 

 Table 7-9 Annual Impact on U.S. Software Developers Supporting the 

Financial Services Sector.......................................................... 7-16 

 Table 7-10 Financial Industry Survey Completion Rates ............................. 7-18 

 Table 7-11 Industry Coverage.................................................................... 7-19 

 Table 7-12 Reported Software Products...................................................... 7-20 

 Table 7-13 Incidence and Costs of Software Errors ..................................... 7-21 

 Table 7-14 Total Costs of Search, Installation, and Maintenance (LifeCycle Costs)............................................................................. 7-23 

 Table 7-15 Software Bug and Error Costs Per Million Dollars of Deposits 

and Loans................................................................................ 7-25 

 Table 7-16 Company Costs Associated with Bugs for Hypothetical 

Company Sizes ........................................................................ 7-26 

 Table 7-17 Cost Reductions as a Function of Error Reductions.................... 7-27 

 Table 7-18 Annual Impacts’ Weighted Cost Per Deposits and Loans........... 7-28 

 Table 8-1 National Economic Impact Estimates.......................................... 8-1 

 Table 8-2 FTEs Engaged in Software Testing (2000) .................................... 8-3 

 Table 8-3 Software Developer Costs Per Tester........................................... 8-4 

 Table 8-4 National Employment in the Service and Manufacturing 

Sectors....................................................................................... 8-5 

 Table 8-5 Per-Employee Cost Metrics......................................................... 8-5 

 Table 8-6 National Impact Estimates .......................................................... 8-6 

ES-1 

 

 Executive Summary 

Software has become an intrinsic part of business over the last 

decade. Virtually every business in the U.S. in every sector depends 

on it to aid in the development, production, marketing, and support 

of its products and services. Advances in computers and related 

technology have provided the building blocks on which new 

industries have evolved. Innovations in the fields of robotic 

manufacturing, nanotechnologies, and human genetics research all 

have been enabled by low cost computational and control 

capabilities supplied by computers and software. 

In 2000, total sales of software reached approximately $180 billion. 

Rapid growth has created a significant and high-paid workforce, 

with 697,000 employed as software engineers and an additional 

585,000 as computer programmers. 

Reducing the cost of software development and improving software 

quality are important objectives of the U.S. software industry. 

However, the complexity of the underlying software needed to 

support the U.S.’s computerized economy is increasing at an 

alarming rate. The size of software products is no longer measured 

in terms of thousands of lines of code, but millions of lines of code. 

This increasing complexity along with a decreasing average market 

life expectancy for many software products has heightened 

concerns over software quality. 

Software nonperformance and failure are expensive. The media is 

full of reports of the catastrophic impact of software failure. For 

example, a software failure interrupted the New York Mercantile 

Exchange and telephone service to several East Coast cities in 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

ES-2 

February 1998 (Washington Technology, 1998). Headlines 

frequently read, “If Microsoft made cars instead of computer 

programs, product-liability suits might now have driven them out of 

business.” Estimates of the economic costs of faulty software in the 

U.S. range in the tens of billions of dollars per year and have been 

estimated to represent approximately just under 1 percent of the 

nation’s gross domestic product (GDP). 

In actuality many factors contribute to the quality issues facing the 

software industry. These include marketing strategies, limited 

liability by software vendors, and decreasing returns to testing and 

debugging. 

At the core of these issues is the difficulty in defining and measuring 

software quality. Common attributes include functionality, 

reliability, usability, efficiency, maintainability, and portability. But 

these quality metrics are largely subjective and do not support 

rigorous quantification that could be used to design testing methods 

for software developers or support information dissemination to 

consumers. Information problems are further complicated by the 

fact that even with substantial testing, software developers do not 

truly know how their products will perform until they encounter real 

scenarios. 

The objective of this study is to investigate the economic impact of 

an inadequate infrastructure for software testing in the U.S. The 

National Institute of Standards and Technology (NIST) undertook 

this study as part of joint planning with industry to help identify and 

assess technical needs that would improve the industry’s software 

testing capabilities. The findings from this study are intended to 

identify the infrastructure needs that NIST can supply to industry 

through its research programs. 

To inform the study, RTI conducted surveys with both software 

developers and industry users of software. The data collected were 

used to develop quantitative estimates of the economic impact of 

inadequate software testing methods and tools. Two industry 

groups were selected for detailed analysis: automotive and 

aerospace equipment manufacturers and financial services providers 

and related electronic communications equipment manufacturers. 

The findings from these two industry groups were then used as the 

“In analyzing repair 

histories of 13 kinds of 

products gathered by 

Consumer Reports, PC 

World found that roughly 

22 percent [of PCs] break 

down every year—

compared to 9 percent of 

VCRs, 7 percent of bigscreen TVs, 7 percent of 

clothes dryers and 8 

percent of refrigerators” 

(Barron, 2000). 
Executive Summary 

ES-3 

basis for estimating the total economic impact for U.S. 

manufacturing and services sectors. 

Based on the software developer and user surveys, the national 

annual costs of an inadequate infrastructure for software testing is 

estimated to range from $22.2 to $59.5 billion.1 Over half of these 

costs are borne by software users in the form of error avoidance and 

mitigation activities. The remaining costs are borne by software 

developers and reflect the additional testing resources that are 

consumed due to inadequate testing tools and methods. 

 ES.1 ISSUES OF SOFTWARE QUALITY 

Quality is defined as the bundle of attributes present in a 

commodity and, where appropriate, the level of the attribute for 

which the consumer (software users) holds a positive value. 

Defining the attributes of software quality and determining the 

metrics to assess the relative value of each attribute are not 

formalized processes. Compounding the problem is that numerous 

metrics exist to test each quality attribute. 

Because users place different values on each attribute depending on 

the product’s use, it is important that quality attributes be 

observable to consumers. However, with software there exists not 

only asymmetric information problems (where a developer has more 

information about quality than the consumer), but also instances 

where the developer truly does not know the quality of his own 

product. It is not unusual for software to become technically 

obsolete before its performance attributes have been fully 

demonstrated under real-world operation conditions. 

As software has evolved over time so has the definition of software 

quality attributes. McCall, Richards, and Walters (1977) first 

attempted to assess quality attributes for software. His software 

quality model characterizes attributes in terms of three categories: 

product operation, product revision, and product transition. In 

1991, the International Organization for Standardization (ISO) 

adopted ISO 9126 as the standard for software quality (ISO, 1991). 

 

1Note that the impact estimates do not reflect “costs” associated with mission 

critical software where failure can lead to extremely high costs such as loss of 

life or catastrophic failure. Quantifying these costs was beyond the scope of the 

study. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

ES-4 

It is structured around six main attributes listed below 

(subcharacteristics are listed in parenthesis): 

Z functionality (suitability, accurateness, interoperability, 

compliance, security) 

Z reliability (maturity, fault tolerance, recoverability) 

Z usability (understandability, learnability, operability) 

Z efficiency (time behavior, resource behavior) 

Z maintainability (analyzability, changeability, stability, 

testability) 

Z portability (adaptability, installability, conformance, 

replaceability) 

Although a general set of standards has been agreed on, the 

appropriate metrics to test how well software meets those standards 

are still poorly defined. Publications by IEEE (1988, 1996) have 

presented numerous potential metrics that can be used to test each 

attribute. These metrics include 

Z fault density, 

Z requirements compliance, 

Z test coverage, and 

Z mean time to failure. 

The problem is that no one metric is able to unambiguously 

measure a particular quality attribute. Different metrics may give 

different rank orderings of the same attribute, making comparisons 

across products difficult and uncertain. 

 ES.2 SOFTWARE TESTING INADEQUACIES 

Software testing is the action of carrying out one or more tests, 

where a test is a technical operation that determines one or more 

characteristics of a given software element or system, according to a 

specified procedure. The means of software testing is the hardware 

and/or software and the procedures for its use, including the 

executable test suite used to carry out the testing (NIST, 1997). 

Historically, software development focused on writing code and 

testing specific lines of that code. Very little effort was spent on 

determining its fit within a larger system. Testing was seen as a 

necessary evil to prove to the final consumer that the product 

worked. As shown in Table ES-1, Andersson and Bergstrand (1995) 

estimate that 80 percent of the effort put into early software 
Executive Summary 

ES-5 

Table ES-1. Allocation of Effort 

Requirements 

Analysis 

Preliminary 

Design 

Detailed 

Design 

Coding and 

Unit Testing 

Integration 

and Test 

System 

Test 

1960s – 1970s 10% 80% 10% 

1980s 20% 60% 20% 

1990s 40% 30% 30% 

Source: Andersson, M., and J. Bergstrand. 1995. “Formalizing Use Cases with Message Sequence Charts.” 

Unpublished Master’s thesis. Lund Institute of Technology, Lund, Sweden. 

development was devoted to coding and unit testing. This 

percentage has changed over time. Starting in the 1970s, software 

developers began to increase their efforts on requirements analysis 

and preliminary design, spending 20 percent of their effort in these 

phases. 

More recently, software developers started to invest more time and 

resources in integrating the different pieces of software and testing 

the software as a unit rather than as independent entities. The 

amount of effort spent on determining the developmental 

requirements of a particular software solution has increased in 

importance. Forty percent of the software developer effort is now 

spent in the requirements analysis phase. 

Testing activities are conducted throughout all the development 

phases shown in Table ES-1. Formal testing conducted by 

independent test groups accounts for about 20 percent of labor 

costs. However, estimates of total labor resources spent testing by 

all parties range from 30 to 90 percent (Beizer, 1990). 

The worldwide market for software testing tools was $931 million in 

1999 and is projected to grow to more than $2.6 billion by 2004 

(Shea, 2000). However, such testing tools are still fairly primitive. 

The lack of quality metrics leads most companies to simply count 

the number of defects that emerge when testing occurs. Few 

organizations engage in other advanced testing techniques, such as 

forecasting field reliability based on test data and calculating defect 

density to benchmark the quality of their product against others. 

Numerous issues affect the software testing infrastructure and may 

lead to inadequacies. For example, competitive market pressures 

may encourage the use of a less than optimal amount of time, 

Software testing 

infrastructure 

improvements include 

enhanced 

Z integration and 

interoperability testing 

tools, 

Z automated generation 

of test code, 

Z methods for 

determining sufficient 

quality for release, and 

Z performance metrics 

and measurement 

procedures. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

ES-6 

resources, and training for the testing function (Rivers and Vouk, 

1998), and with current software testing tools developers have to 

determine whether applications and systems will interoperate. 

In addition, the need for certified standardized test technology is 

increasing. The development of these tools and the accompanying 

testing suites often lag behind the development of new software 

applications (ITToolbox, 1999). Standardized testing tools, suites, 

scripts, reference data, reference implementations, and metrics that 

have undergone a rigorous certification process would have a large 

impact on the inadequacies listed above. For example, the 

availability of standardized test data, metrics, and automated test 

suites for performance testing would make benchmarking tests less 

costly to perform. Standardized automated testing scripts along 

with standard metrics would also provide a more consistent method 

for determining when to stop testing. 

In some instances, developing conformance testing code can be 

more time consuming and expensive than developing the software 

product being tested. Addressing the high testing costs is currently 

the focus of several research initiatives in industry and academia. 

Many of these initiatives are based on modeling finite state 

machines, combinatorial logic, or other formal languages such as Z 

(Cohen et al., 1996; Tai and Carver, 1995; NIST, 1997; Apfelbaum 

and Doyle, 1997). 

 ES.3 SOFTWARE TESTING COUNTERFACTUAL 

SCENARIOS 

To estimate the costs attributed to an inadequate infrastructure for 

software testing, a precise definition of the counterfactual world is 

needed. Clearly defining what is meant by an “inadequate” 

infrastructure is essential for eliciting consistent information from 

industry respondents. 

In the counterfactual scenarios the intended design functionality of 

the software products released by developers is kept constant. In 

other words, the fundamental product design and intended product 

characteristics will not change. However, the realized level of 

functionality may be affected as the number of bugs (also referred to 

as defects or errors) present in released versions of the software 

decreases in the counterfactual scenarios. 
Executive Summary 

ES-7 

The driving technical factors that do change in the counterfactual 

scenarios are when bugs are discovered in the software 

development process and the cost of fixing them. An improved 

infrastructure for software testing has the potential to affect software 

developers and users by 

Z removing more bugs before the software product is released, 

Z detecting bugs earlier in the software development process, 

and 

Z locating the source of bugs faster and with more precision. 

Note that a key assumption is that the number of bugs introduced 

into software code is constant regardless of the types of tools 

available for software testing; bugs are errors entered by the 

software designer/programmer and the initial number of errors 

depends on the skill and techniques employed by the programmer. 

Because it may not be feasible or cost effective to remove all 

software errors prior to product release, the economic impact 

estimates were developed relative to two counterfactual scenarios. 

The first scenario investigates the cost reductions if all bugs and 

errors could be found in the same development stage in which they 

are introduced. This is referred to as the cost of an inadequate 

software testing infrastructure. The second scenario investigates the 

cost reductions associated with finding an increased percentage (but 

not 100 percent) of bugs and errors closer to the development stages 

where they are introduced. The second scenario is referred to as 

cost reduction from “feasible” infrastructure improvements. For the 

“feasible” infrastructure improvements scenario, developers were asked to

estimate the potential cost savings associated with enhanced testing 

tools and users were asked to estimate cost savings if the software 

they purchase had 50 percent fewer bugs and errors. 

 ES.4 ECONOMIC IMPACT OF AN INADEQUATE 

SOFTWARE TESTING INFRASTRUCTURE: 

AUTOMOTIVE AND AEROSPACE INDUSTRIES 

We conducted a case study with software developers and users in 

the transportation equipment manufacturing sector to estimate the 

economic impact of an inadequate infrastructure for software 

testing. The case study focused on the use of computer-aided 

design/computer-aided manufacturing/computer-aided engineering 

An improved software 

testing infrastructure would 

allow developers to find 

and correct more errors 

sooner with less cost. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

ES-8 

(CAD/CAM/CAE) and product data management (PDM) software. 

Interviews were conducted with 10 software developers (vendors) 

and 179 users of these products. 

Developers of CAD/CAM/CAE and PDM software indicated that in 

the current environment, software testing is still more of an art than 

a science, and testing methods and resources are selected based on 

the expert judgment of senior staff. Respondents agreed that finding 

the errors early in the development process greatly lowered the 

average cost of bugs and errors. Most also indicated that the lack of 

historic tracking data and inadequate tools and testing methods, 

such as standard protocols approved by management, available test 

cases, and conformance specification, limited their ability to obtain 

sufficient testing resources (from management) and to leverage these 

resources effectively. 

Users of CAD/CAM/CAE and PDM software indicated that they 

spend significant resources responding to software errors (mitigation 

costs) and lowering the probability and potential impact of software 

errors (avoidance costs). Approximately 60 percent of the 

automotive and aerospace manufacturers surveyed indicated that 

they had experienced significant software errors in the previous 

year. For these respondents who experienced errors, they reported 

an average of 40 major and 70 minor software bugs per year in their 

CAD/CAM/CAE or PDM software systems. 

Table ES-2 presents the economic impact estimates for the 

development and use of CAD/CAM/CAE and PDM software in the 

U.S. automotive and aerospace industries. The total cost impact on 

these manufacturing sectors from an inadequate software testing 

infrastructure is estimated to be $1.8 billion and the potential cost 

reduction from feasible infrastructure improvements is $0.6 billion. 

Users of CAD/CAM/CAE and PDM software account for 

approximately three-fourths of the total impact, with the automotive 

industry representing about 65 percent and the aerospace industry 

representing 10 percent. Developers account for the remaining 

one-fourth of the costs. 
Executive Summary 

ES-9 

Table ES-2. Cost Impacts on U.S. Software Developers and Users in the Transportation 

Manufacturing Sector Due to an Inadequate Testing Infrastructure ($ millions) 

The Cost of Inadequate Software 

Testing Infrastructure 

(billions) 

Potential Cost Reduction from 

Feasible Infrastructure Improvements 

(billions) 

Software Developers 

CAD/CAM/CAE and PDM $373.1 $157.7 

Software Users 

Automotive $1,229.7 $377.0 

Aerospace $237.4 $54.5 

Total $1,840.2 $589.2 

 ES.5 ECONOMIC IMPACT OF AN INADEQUATE 

SOFTWARE TESTING INFRASTRUCTURE: 

FINANCIAL SERVICES SECTOR 

We conducted a second case study with four software developers 

and 98 software users in the financial services sector to estimate the 

economic impact of an inadequate infrastructure for software 

testing. The case study focused on the development and use of 

Financial Electronic Data Interchange (FEDI) and clearinghouse 

software, as well as the software embedded in routers and switches 

that support electronic data exchange. 

All developers of financial services software agreed that an improved 

system for testing was needed. They said that an improved system 

would be able to track a bug back to the point where it was 

introduced and then determine how that bug influenced the rest of the 

production process. Their ideal testing infrastructure would consist of 

close to real time testing where testers could remedy problems that 

emerge right away rather than waiting until a product is fully 

assembled. The major benefits developers cited from an improved 

infrastructure were direct cost reduction in the development process 

and a decrease in post-purchase customer support. An additional 

benefit that respondents thought would emerge from an improved 

testing infrastructure is increased confidence in the quality of the 

product they produce and ship. The major selling characteristic of the 

products they create is the certainty that that product will accomplish 

a particular task. Because of the real time nature of their products, the 

reputation loss can be great. 

Financial service software 

developers said that better 

testing tools and methods 

used during software 

development could reduce 

installation expenditures by 

30 percent. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

ES-10 

Approximately two-thirds of the users of financial services software 

(respondents were primarily banks and credit unions) surveyed 

indicated that they had experienced major software errors in the 

previous year. For the respondents that did have major errors, they 

reported an average of 40 major and 49 minor software bugs per 

year in their FEDI or clearinghouse software systems. 

Approximately 16 percent of those bugs were attributed to router 

and switch problems, and 48 percent were attributed to transaction 

software problems. The source of the remaining 36 percent of 

errors was unknown. Typical problems encountered due to bugs 

were 

Z increased person-hours used to correct posting errors, 

Z temporary shut down leading to lost transactions, and 

Z delay of transaction processing. 

Table ES-3 presents the empirical findings. The total cost impact on 

the financial services sector from an inadequate software testing 

infrastructure is estimated to be $3.3 billion. Potential cost 

reduction from feasible infrastructure improvements is $1.5 billion. 

Table ES-3. Cost Impacts on U.S. Software Developers and Users in the Financial Services 

Sector Due to an Inadequate Testing Infrastructure ($ millions) 

The Cost of Inadequate 

Software Testing Infrastructure 

Potential Cost Reduction from 

Feasible Infrastructure 

Improvements 

Software Developers 

Router and switch $1,897.9 $975.0 

FEDI and clearinghouse $438.8 $225.4 

Software Users 

Banks and savings institutions $789.3 $244.0 

Credit unions $216.5 $68.1 

Total Financial Services Sector $3,342.5 $1,512.6 

Software developers account for about 75 percent of the economic 

impacts. Users represented the remaining 25 percent of costs, with 

banks accounting for the majority of user costs. 
Executive Summary 

ES-11 

 ES.6 NATIONAL IMPACT ESTIMATES 

The two case studies generated estimates of the costs of an inadequate 

software testing infrastructure for software developers and users in the 

transportation equipment manufacturing and financial services 

sectors. The per-employee impacts for these sectors were 

extrapolated to other manufacturing and service industries to develop 

an approximate estimate of the economic impacts of an inadequate 

infrastructure for software testing for the total U.S. economy. 

Table ES-4 shows the national annual cost estimates of an 

inadequate infrastructure for software testing are estimated to be 

$59.5 billion. The potential cost reduction from feasible 

infrastructure improvements is $22.2 billion. This represents about 

0.6 and 0.2 percent of the U.S.’s $10 trillion dollar GDP, 

respectively. Software developers accounted for about 40 percent 

of total impacts, and software users accounted for the about 

60 percent. 

Table ES-4. Costs of Inadequate Software Testing Infrastructure on the National Economy 

The Cost of Inadequate Software 

Testing Infrastructure 

(billions) 

Potential Cost Reduction from Feasible 

Infrastructure Improvements 

(billions) 

Software developers $21.2 $10.6 

Software users $38.3 $11.7 

Total $59.5 $22.2 

1-1 

 Introduction to 

 Software Quality 1 and Testing 

Software is an intrinsic part of business in the late 20th century. 

Virtually every business in the U.S. in every sector depends on it to 

aid in the development, production, marketing, and support of its 

products and services. This software may be written either by 

developers who offer the shrink-wrapped product for sale or 

developed by organizations for custom use. 

Integral to the development of software is the process of detecting, 

locating, and correcting bugs. 

In a typical commercial development organization, the cost of 

providing [the assurance that the program will perform 

satisfactorily in terms of its functional and nonfunctional 

specifications within the expected deployment environments] 

via appropriate debugging, testing, and verification activities 

can easily range from 50 to 75 percent of the total 

development cost. (Hailpern and Santhanam, 2002) 

In spite of these efforts some bugs will remain in the final product to 

be discovered by users. They may either develop “workarounds” to 

deal with the bug or return it to the developer for correction. 

Software’s failure to perform is also expensive. The media is full of 

reports of the catastrophic impact of software failure. For example, 

a software failure interrupted the New York Mercantile Exchange 

and telephone service to several East Coast cities in February 1998 

(Washington Technology, 1998). More common types of software 

nonperformance include the failure to 

Beizer (1990) reports that 

half the labor expended to 

develop a working program 

is typically spent on testing 

activities. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

1-2 

Z conform to specifications or standards, 

Z interoperate with other software and hardware, and 

Z meet minimum levels of performance as measured by 

specific metrics. 

Reducing the cost of software development and improving software 

quality are important objectives of the commercial U.S. software 

industry and of in-house developers. Improved testing and 

measurement can reduce the costs of developing software of a given 

quality and even improve performance. However, the lack of a 

commonly accepted measurement science for information 

technology hampers efforts to test software and evaluate the tests’ 

results. 

Software testing tools are available that incorporate proprietary 

testing algorithms and metrics that can be used to measure the 

performance and conformance of software. However, the value of 

these tools and the metrics they produce depend on the extent to 

which standard measurements are developed by consensus and 

accepted throughout the software development and user community 

(NIST, 1997). Thus, development of standard testing tools and 

metrics for software testing could go a long way toward addressing 

some of the testing problems that plague the software industry. 

Improved tools for software testing could increase the net value 

(value minus cost) of software in a number of ways: 

Z reduce the cost of software development and testing; 

Z reduce the time required to develop new software products; 

and 

Z improve the performance, interoperability, and conformance 

of software. 

However, to understand the extent to which improvements in 

software testing metrology could provide these benefits, we must 

first understand and quantify the costs imposed on industry by the 

lack of an adequate software testing infrastructure. The objective of 

this study is to develop detailed information about the costs 

associated with an inadequate software testing infrastructure for 

selected software products and industrial sectors. 

This section describes the commonly used software quality 

attributes and currently available metrics for measuring software 

“[A] study of personalcomputer failure rates by 

the Gartner Group 

discover[ed] that there was 

a failure rate of 25 percent 

for notebook computers 

used in large American 

corporations” (Barron, 

2000). 

“Gary Chapman, director 

of the 21st Century Project 

at the University of Texas, 

noted that ‘repeated 

experiences with software 

glitches tend to narrow 

one’s use of computers to 

familiar and routine. 

Studies have shown that 

most users rely on less than 

10 percent of the features 

of common programs as 

Microsoft Word or 

Netscape Communicator’” 

(Barron, 2000). 
Section 1 — Introduction to Software Quality and Testing 

1-3 

quality. It also provides an overview of software testing procedures 

and describes the impact of inadequate software testing. 

 1.1 SOFTWARE QUALITY ATTRIBUTES 

Software consumers choose which software product to purchase by 

maximizing a profit function that contains several parameters 

subject to a budget constraint. One of the parameters in that profit 

function is quality. Quality is defined as the bundle of attributes 

present in a commodity and, where appropriate, the level of the 

attribute for which the consumer holds a positive value. 

Defining the attributes of software quality and determining the 

metrics to assess the relative value of each attribute are not 

formalized processes. Not only is there a lack of commonly agreed 

upon definitions of software quality, different users place different 

values on each attribute depending on the product’s use. 

Compounding the problem is that numerous metrics exist to test 

each quality attribute. The different outcome scores for each metric 

may not give the same rank orderings of products, increasing the 

difficulty of interproduct comparisons. 

McCall, Richards, and Walters (1977) first attempted to assess 

quality attributes for software. His software quality model focused 

on 11 specific attributes. Table 1-1 lists those characteristics and 

briefly describes them. McCall, Richards, and Walters’s 

characteristics can be divided into three categories: product 

operation, product revision, and product transition. 

Z Product operation captures how effective the software is at 

accomplishing a specific set of tasks. The tasks range from 

the ease of inputting data to the ease and reliability of the 

output data. Product operation consists of correctness, 

reliability, integrity, usability, and efficiency attributes. 

Z Product revision measures how easy it is to update, change, 

or maintain performance of the software product. This 

category is especially important to this analysis because it is 

concerned with software testing and the cost of fixing any 

bugs that emerge from the testing process. Maintainability, 

flexibility, and testability are three subcharacteristics that fit 

into this category. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

1-4 

Table 1-1. McCall, Richards, and Walters’s Software Quality Attributes 

Attribute Description 

Product Operation 

Correctness How well the software performs its required function and meets customers’ needs 

Reliability How well the software can be expected to perform its function with required 

precision 

Integrity How well accidental and intentional attacks on the software can be withstood 

Usability How easy it is to learn, operate, prepare input of, and interpret output of the software 

Efficiency Amount of computing resources required by the software to perform its function 

Product Revision 

Maintainability How easy it is to locate and fix an error in the software 

Flexibility How easy it is to change the software 

Testability How easy it is to tell if the software performs its intended function 

Product Transition 

Interoperability How easy it is to integrate one system into another 

Reusability How easy it is to use the software or its parts in other applications 

Portability How easy it is to move the software from one platform to another 

Source: McCall, J., P. Richards, and G. Walters. 1977. Factors in Software Quality, NTIS AD-A049-014, 015, 055. 

November. 

Z Product transition focuses on software migration. The three 

main factors that make up this category are the software’s 

ability to interact with other pieces of software, the 

frequency with which the software can be used in other 

applications, and the ease of using the software on other 

platforms. Three subcharacteristics are interoperability, 

reusability, and portability. 

Following McCall, Richards, and Walters’s work, Boehm (1978) 

introduced several additional quality attributes. While the two 

models have some different individual attributes, the three 

categories—product operation, product revision, and product 

transition—are the same. 

As software changed and improved and the demands on software 

increased, a new set of software quality attributes was needed. In 

1991, the International Organization for Standardization (ISO) 

adopted ISO 9126 as the standard for software quality (ISO, 1991). 

The ISO 9126 standard moves from three main attributes to six and 

from 11 subcharacteristics to 21. These attributes are presented in 

Table 1-2. The ISO standard is based on functionality, reliability, 
Section 1 — Introduction to Software Quality and Testing 

1-5 

Table 1-2. ISO Software Quality Attributes 

Attributes Subcharacteristics Definition 

Functionality Suitability Attributes of software that bear on the presence and appropriateness 

of a set of functions for specified tasks 

 Accurateness Attributes of software that bear on the provision of right or agreed 

upon results or effects 

 Interoperability Attributes of software that bear on its ability to interact with 

specified systems 

 Compliance Attributes of software that make the software adhere to applicationrelated standards or conventions or regulations in laws and similar 

prescriptions 

 Security Attributes of software that bear on its ability to prevent unauthorized 

access, whether accidental or deliberate, to programs or data 

Reliability Maturity Attributes of software that bear on the frequency of failure by faults 

in the software 

 Fault tolerance Attributes of software that bear on its ability to maintain a specified 

level of performance in case of software faults or of infringement of 

its specified interface 

 Recoverability Attributes of software that bear on the capability to re-establish its 

level of performance and recover the data directly affected in case 

of a failure and on the time and effort needed for it 

Usability Understandability Attributes of software that bear on the users’ effort for recognizing 

the logical concept and its applicability 

 Learnability Attributes of software that bear on the users’ effort for learning its 

application 

 Operability Attributes of software that bear on the users’ effort for operation and 

operation control 

Efficiency Time behavior Attributes of software that bear on response and processing times 

and on throughput rates in performing its function 

 Resource behavior Attributes of software that bear on the amount of resources used and 

the duration of such use in performing its function 

Maintainability Analyzability Attributes of software that bear on the effort needed for diagnosis of 

deficiencies or causes of failures or for identification of parts to be 

modified 

 Changeability Attributes of software that bear on the effort needed for 

modification, fault removal, or environmental change 

 Stability Attributes of software that bear on the risk of unexpected effect of 

modifications 

 Testability Attributes of software that bear on the effort needed for validating 

the modified software 

(continued) 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

1-6 

Table 1-2. ISO Software Quality Attributes (continued) 

Attributes Subcharacteristics Definition 

Portability Adaptability Attributes of software that bear on the opportunity for its adaptation 

to different specified environments without applying other actions 

or means than those provided for this purpose for the software 

considered 

 Installability Attributes of software that bear on the effort needed to install the 

software in a specified environment 

 Conformance Attributes of software that make the software adhere to standards or 

conventions relating to portability 

 Replaceability Attributes of software that bear on opportunity and effort using it in 

the place of specified other software in the environment of that 

software 

Source: ISO Standard 9126, 1991. 

usability, efficiency, maintainability, and portability. The paradigms 

share several similarities; for example, maintainability in ISO maps 

fairly closely to product revision in the McCall paradigm, and 

product transition maps fairly closely to portability. There are also 

significant differences between the McCall and ISO paradigms. The 

attributes of product operation under McCall’s paradigm are 

specialized in the ISO model and constitute four major categories 

rather than just one. 

The ISO standard is now widely accepted. Other organizations that 

set industry standards (e.g., IEEE) have started to adjust their 

standards to comply with the ISO standards. 

 1.2 SOFTWARE QUALITY METRICS 

Although a general set of standards has been agreed upon, the 

appropriate metrics to test how well software meets those standards 

are still poorly defined. Publications by IEEE (1988, 1996) have 

presented numerous potential metrics that can be used to test each 

attribute. Table 1-3 contains a list of potential metrics. The 

problem is that no one metric is able to unambiguously measure a 

particular attribute. Different metrics may give different rank 

orderings of the same attribute, making comparisons across products 

difficult and uncertain. 
Section 1 — Introduction to Software Quality and Testing 

1-7 

Table 1-3. List of Metrics Available 

Metric Metric 

Fault density Software purity level 

Defect density Estimated number of faults remaining (by seeding) 

Cumulative failure profile Requirements compliance 

Fault-days number Test coverage 

Functional or modular test coverage Data or information flow complexity 

Cause and effect graphing Reliability growth function 

Requirements traceability Residual fault count 

Defect indices Failure analysis elapsed time 

Error distribution(s) Testing sufficiently 

Software maturity index Mean time to failure 

Person-hours per major defect detected Failure rate 

Number of conflicting requirements Software documentation and source listing 

Number of entries and exits per module Rely-required software reliability 

Software science measures Software release readiness 

Graph-theoretic complexity for architecture Completeness 

Cyclomatic complexity Test accuracy 

Minimal unit test case determination System performance reliability 

Run reliability Independent process reliability 

Design structure Combined hardware and software (system) availability 

Mean time to discover the next K-faults 

The lack of quality metrics leads most companies to simply count 

the number of defects that emerge when testing occurs. Few 

organizations engage in other advanced testing techniques, such as 

forecasting field reliability based on test data and calculating defect 

density to benchmark the quality of their product against others. 

This subsection describes the qualities of a good metric, the 

difficulty of measuring certain attributes, and criteria for selecting 

among metrics. 

 1.2.1 What Makes a Good Metric 

Several common characteristics emerge when devising metrics to 

measure product quality. Although we apply them to software 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

1-8 

development, these metrics are not exclusive to software; rather 

they are characteristics that all good metrics should have: 

Z Simple and computable: Learning the metric and applying 

the metric are straightforward and easy tasks. 

Z Persuasive: The metrics appear to be measuring the correct 

attribute. In other words, they display face validity. 

Z Consistent and objective: The results are reproducible. 

Z Consistent in units or dimensions: Units should be 

interpretable and obvious. 

Z Programming language independent: The metrics should 

not be based on specific tasks and should be based on the 

type of product being tested. 

Z Gives feedback: Results from the metrics give useful 

information back to the person performing the test 

(Pressman, 1992). 

 1.2.2 What Can be Measured 

Regardless of the metric’s quality, certain software attributes are 

more amenable to being measured than other attributes. Not 

surprisingly, the metrics that are easiest to measure are also the least 

important in eliminating the uncertainty the consumer faces over 

software quality. 

Pressman (1992) describes the attributes that can be measured 

reliably and consistently across various types of software programs: 

Z effort, time, and capital spent in each stage of the project; 

Z number of functionalities implemented; 

Z number and type of errors remediated; 

Z number and type of errors not remediated; 

Z meeting scheduled deliverables; and 

Z specific benchmarks. 

Interoperability, reliability, and maintainability are difficult to 

measure, but they are important when assessing the overall quality of 

the software product. The inability to provide reliable, consistent, and 

objective metrics for some of the most important attributes that a 

consumer values is a noticeable failure of software metrics. 

 1.2.3 Choosing Among Metrics 

Determining which metric to choose from the family of available 

metrics is a difficult process. No one unique measure exists that a 

developer can use or a user can apply to perfectly capture the 
Section 1 — Introduction to Software Quality and Testing 

1-9 

concept of quality. For example, a test of the “cyclomatic” 

complexity of a piece of software reveals a significant amount of 

information about some aspects of the software’s quality, but it does 

not reveal every aspect.1 In addition, there is the potential for 

measurement error when the metric is applied to a piece of 

software. For example, mean time to failure metrics are not 

measures of certainty; rather they are measures that create a 

distribution of outcomes. 

Determining which metric to use is further complicated because 

different users have different preferences for software attributes. 

Some users care about the complexity of the software; others may 

not. 

The uncertainty over which metric to use has created a need to test 

the validity of each metric. Essentially, a second, observable, 

comprehensive and comparable set of metrics is needed to test and 

compare across all of the software quality metrics. This approach 

helps to reduce the uncertainty consumers face by giving them 

better information about how each software product meets the 

quality standards they value. 

To decide on the appropriate metric, several potential tests of the 

validity of each metric are available (IEEE, 1998). For a metric to be 

considered reliable, it needs to have a strong association with the 

underlying quality construct that it is trying to measure. IEEE 

standard 1061-1998 provides five validity measures that software 

developers can apply to decide which metrics are most effective at 

capturing the latent quality measure: 

1. Linear correlation coefficients—Tests how well the 

variation in the metrics explains the variations in the 

underlying quality factors. This validity test can be used to 

determine whether the metric should be used when 

measuring or observing a particular quality factor is difficult. 

2. Rank correlation coefficients—Provides a second test for 

determining whether a particular metric can be used as a 

proxy for a quality factor. The advantage of using a rank 

order correlation is that it is able to track changes during the 

development of a software product and see if those changes 

affect software quality. Additionally, rank correlations can 

be used to test for consistency across products or processes. 

 

1Cyclomatic complexity is also referred to as program complexity or McCabe’s 

complexity and is intended to be a metric independent of language and 

language format (McCabe and Watson, 1994). 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

1-10 

3. Prediction error—Is used to determine the degree of 

accuracy that a metric has when it is assessing the quality of 

a particular piece of software. 

4. Discriminative power—Tests to see how well a particular 

metric is able to separate low quality software components 

from high quality software components. 

5. Reliability—If a metric is able to meet each of the four 

previous validity measures in a predetermined percentage of 

tests then the metric is considered reliable. 

 1.3 SOFTWARE TESTING 

Software testing is the process of applying metrics to determine 

product quality. Software testing is the dynamic execution of 

software and the comparison of the results of that execution against 

a set of pre-determined criteria. “Execution” is the process of 

running the software on a computer with or without any form of 

instrumentation or test control software being present. “Predetermined criteria” means that the software’s capabilities are 

known prior to its execution. What the software actually does can 

then be compared against the anticipated results to judge whether 

the software behaved correctly. 

The means of software testing is the hardware and/or software and 

the procedures for its use, including the executable test suite used to 

carry out the testing (NIST, 1997). Section 2 of this report examines 

in detail the various forms of software testing, the common types of 

software testing being conducted and the available tools for 

software testing activities. 

In many respects, software testing is an infrastructure technology or 

“infratechnology.” Infratechnologies are technical tools, including 

scientific and engineering data, measurement and test methods, and 

practices and techniques that are widely used in industry (Tassey, 

1997). Software testing infratechnologies provide the tools needed 

to measure conformance, performance, and interoperability during 

the software development. These tools aid in testing the relative 

performance of different software configurations and mitigate the 

expense of reengineering software after it is developed and released. 

Software testing infratechnologies also provide critical information 

to the software user regarding the quality of the software. By 

increasing quality, purchase decision costs for software are reduced. 

In many respects, 

software testing is an 

infrastructure 

technology or 

“infratechnology.” 
Section 1 — Introduction to Software Quality and Testing 

1-11 

 1.4 THE IMPACT OF INADEQUATE TESTING 

Currently, there is a lack of readily available performance metrics, 

procedures, and tools to support software testing. If these 

infratechnologies were available, the costs of performance 

certification programs would decline and the quality of software 

would increase. This would lead to not only better testing for 

existing products, but also to the testing of products that are not 

currently tested. 

The impact on the software industry due to lack of robust, 

standardized test technology can be grouped into four general 

categories: 

Z increased failures due to poor quality, 

Z increased software development costs, 

Z increased time to market due to inefficient testing, and 

Z increased market transaction costs. 

 1.4.1 Failures due to Poor Quality 

The most troublesome effect of a lack of standardized test technology 

is the increased incidence of avoidable product defects that emerge 

after the product has been shipped. As illustrated in Table 1-4, in the 

aerospace industry over a billion dollars has been lost in the last 

several years that might be attributed to problematic software. And 

these costs do not include the recent losses related to the ill-fated 

Mars Mission. Large failures tend to be very visible. They often result 

in loss of reputation and loss of future business for the company. 

Recently legal action has increased when failures are attributable to 

insufficient testing. 

Table 1-4. Recent Aerospace Losses due to Software Failures 

Airbus A320 

(1993) 

Ariane 5 Galileo 

Poseidon 

Flight 965 

(1996) 

Lewis 

Pathfinder 

USAF Step 

(1997) 

Zenit 2 Delta 3 

Near 

(1998) 

DS-1 Orion 3 

Galileo Titan 4B 

(1999) 

Aggregate cost $640 million $116.8 million $255 million $1.6 billion 

Loss of life 3 160 

Loss of data Yes Yes Yes Yes 

Note: These losses do not include those accrued due to recent problems with the Mars Mission. 

Source: NASA IV&V Center, Fairmount, West Virginia. 2000. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

1-12 

Software defects are typically classified by type, location 

introduced, when found, severity level, frequency, and associated 

cost. The individual defects can then be aggregated by cause 

according to the following approach: 

Z Lack of conformance to standards, where a problem occurs 

because the software functions and/or data representation, 

translation, or interpretation do not conform to the 

procedural process or format specified by a standard. 

Z Lack of interoperability with other products, where a 

problem is the result of a software product’s inability to 

exchange and share information (interoperate) with another 

product. 

Z Poor performance, where the application works but not as 

well as expected. 

 1.4.2 Increased Software Development Costs 

Historically, the process of identifying and correcting defects during 

the software development process represents over half of 

development costs. Depending on the accounting methods used, 

testing activities account for 30 to 90 percent of labor expended to 

produce a working program (Beizer, 1990). Early detection of defects 

can greatly reduce costs. Defects can be classified by where they 

were found or introduced along the stages of the software 

development life cycle, namely, requirements, design, coding, unit 

testing, integration testing, system testing, installation/acceptance 

testing, and operation and maintenance phases. Table 1-5 illustrates 

that the longer a defect stays in the program, the more costly it 

becomes to fix it. 

 1.4.3 Increased Time to Market 

The lack of standardized test technology also increases the time that 

it takes to bring a product to market. Increased time often results in 

lost opportunities. For instance, a late product could potentially 

represent a total loss of any chance to gain any revenue from that 

product. Lost opportunities can be just as damaging as post-release 

product failures. However, they are notoriously hard to measure. If 

standardized testing procedures were readily available, testers 

would expend less time developing custom test technology. 

Standardized test technology would accelerate development by 

decreasing the need to 
Section 1 — Introduction to Software Quality and Testing 

1-13 

Table 1-5. Relative Costs to Repair Defects when Found at Different Stages of the Life-Cycle 

Life Cycle Stage 

Baziuk (1995) Study 

Costs to Repair when Found 

Boehm (1976) Study 

Costs to Repair when Founda

Requirements 1Xb 0.2Y 

Design 0.5Y 

Coding 1.2Y 

Unit Testing 

Integration Testing 

System Testing 90X 5Y 

Installation Testing 90X-440X 15Y 

Acceptance Testing 440X 

Operation and Maintenance 470X-880Xc 

aAssuming cost of repair during requirements is approximately equivalent to cost of repair during analysis in the Boehm 

(1976) study. 

bAssuming cost to repair during requirements is approximately equivalent to cost of an HW line card return in Baziuk 

(1995) study. 

cPossibly as high as 2,900X if an engineering change order is required. 

Z develop specific test software for each implementation, 

Z develop specific test data for each implementation, and 

Z use the “trial and error” approach to figuring out how to use 

nonstandard automated testing tools. 

 1.4.4 Increased Market Transaction Costs 

Because of the lack of standardized test technology, purchasers of 

software incur difficulties in comparing and evaluating systems. 

This information problem is so common that manufacturers have 

warned purchasers to be cautious when using performance numbers 

(supplied by the manufacturer) for comparison and evaluation 

purposes. Standardized test technology would alleviate some of the 

uncertainty and risk associated with evaluating software choices for 

purchase by providing consistent approaches and metrics for 

comparison. 

2-1 

 Software Testing 2 Methods and Tools 

Software testing is the action of carrying out one or more tests, 

where a test is a technical operation that determines one or more 

characteristics of a given software element or system, according to a 

specified procedure. The means of software testing is the hardware 

and/or software and the procedures for its use, including the 

executable test suite used to carry out the testing (NIST, 1997). 

This section examines the various forms of software testing, the 

types of software testing, and the available tools for software testing. 

It also provides a technical description of the procedures involved 

with software testing. The section begins with a brief history of 

software development and an overview of the development process. 

 2.1 HISTORICAL APPROACH TO SOFTWARE 

DEVELOPMENT 

The watershed event in the development of the software industry 

can be traced to 1969, when the U.S. Justice Department forced 

IBM to “unbundle” its software from the related hardware and 

required that the firm sell or lease its software products. Prior to 

that time, nearly all operating system and applications software had 

been developed by hardware manufacturers, dominated by IBM, or 

by programmers in the using organizations. Software developers in 

the 1950s and 1960s worked independently or in small teams to 

tackle specific tasks, resulting in customized one-of-a-kind products. 

Since this landmark government action, a software development 

market has emerged, and software developers and engineers have 

moved through several development paradigms (Egan, 1999). 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-2 

During the 1970s, improvements in computing capabilities caused 

firms to expand their use of automated information-processing tasks, 

and the importance of programming to firms’ activities increased 

substantially. Simple tools to aid software development, such as 

programming languages and debugging tools, were introduced to 

increase the software programmer’s productivity. The introduction 

of the personal computer and its widespread adoption after 1980 

accelerated the demand for software and programming, rapidly 

outpacing these productivity improvements. Semiconductor power, 

roughly doubling every 18 months, has dramatically outpaced the 

rate of improvement in software, creating a “software bottleneck.” 

Although software is easily mass-produced, allowing for economies 

of scale, the entrenched customized approach to software 

development was so strong that economies of scale were never 

realized. 

The historic approach to the software development process, which 

focused on system specification and construction, is often based on 

the waterfall model (Andersson and Bergstrand, 1995). Figure 2-1 

shows how this process separates software development into several 

distinct phases with minimal feedback loops. First, the 

requirements and problem are analyzed; then systems are designed 

to address the problem. Testing occurs in two stages: the program 

itself is tested and then how that program works with other 

programs is tested. Finally, normal system operation and 

maintenance take place. Feedback loops only exist between the 

current stage and its antecedent and the following stage. This 

model can be used in a component-based world for describing the 

separate activities needed in software development. For example, 

the requirements and design phase can include identifying available 

reusable software. 

Feedback loops throughout the entire development process increase 

the ability to reuse components. Reuse is the key attribute in 

component-based software development (CBSD). When building a 

component-based program, developers need to examine the 

available products and how they will be integrated into not only the 

system they are developing, but also all other potential systems. 

Feedback loops exist throughout the process and each step is no 

longer an isolated event. 
Section 2 — Software Testing Methods and Tools 

2-3 

Requirements Analysis and

Definition

System and Software Design

Implementation and Unit

Testing

Integration and System

Testing

Operation and Maintenance

Adapted from Andersson and Bergstrand (1995), Table 2-1 

illustrates where software developers have placed their efforts 

through time. In the 1960s and 1970s, software development 

focused on writing code and testing specific lines of that code. Very 

little effort was spent on determining its fit within a larger system. 

Testing was seen as a necessary evil to prove to the final consumer 

that the product worked. Andersson and Bergstrand estimate that 

80 percent of the effort put into early software development was 

devoted to coding and unit testing. This percentage has changed 

over time. Starting in the 1970s, software developers began to 

increase their efforts on requirements analysis and preliminary 

design, spending 20 percent of their effort in these phases. 

Additionally, software developers started to invest more time and 

resources in integrating the different pieces of software and testing 

the software as a system rather than as independent entities (units). 

The amount of effort spent on determining the developmental 

requirements of a particular software solution has increased in 

importance. Forty percent of the software developer effort is now 

spent in the requirements analysis phase. Developers have also 

increased the time spent in the design phase to 30 percent, which 

Figure 2-1. Waterfall 

Model 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-4 

Table 2-1. Allocation of Effort 

Requirements 

Analysis 

Preliminary 

Design 

Detailed 

Design 

Coding and 

Unit Testing 

Integration 

and Test 

System 

Test 

1960s – 1970s 10% 80% 10% 

1980s 20% 60% 20% 

1990s 40% 30% 30% 

Source: Andersson, M., and J. Bergstrand. 1995. “Formalizing Use Cases with Message Sequence Charts.” 

Unpublished Master’s thesis. Lund Institute of Technology, Lund, Sweden. 

reflects its importance. Design phases in a CBSD world are 

extremely important because these phases determine the 

component’s reuse possibilities. 

 2.2 SOFTWARE TESTING INFRASTRUCTURE 

Figure 2-2 illustrates the hierarchical structure of software testing 

infratechnologies. The structure consists of three levels: 

Z software test stages, 

Z software testing tools, and 

Z standardized software testing technologies. 

Software testing is commonly described in terms of a series of 

testing stages. Within each testing stage, testing tools are used to 

conduct the analysis. Standardized testing technologies such as 

standard reference data, reference implementations, test procedures, 

and test cases (both manual and automated) provide the scientific 

foundation for commercial testing tools. 

This hierarchical structure of commercial software-testing 

infratechnologies illustrates the foundational role that standardized 

software testing technologies play. In the following subsections, we 

discuss software testing stages and tools. 

 2.2.1 Software Testing Stages 

Aggregated software testing activities are commonly referred to as 

software testing phases or stages (Jones, 1997). A software testing 

stage is a process for ensuring that some aspect of a software 

product, system, or unit functions properly. The number of software 

testing stages employed varies greatly across companies and 
Section 2 — Software Testing Methods and Tools 

2-5 

Figure 2-2. Commercial Software Testing Infrastructure Hierarchy 

General

Subroutine

New Function

Regression

Integration

System

Specialized

Stress

Error

Recovery

Security

Performance

Platform

Viral

User-Involved

Usability

Field

Lab

Acceptance

Stages

Procedure

Tests

Automated Scripts

Reference Data

Reference Value

Reference Implementation

Test Suites

Manual Scripts

Standardized Software Testing Technologies

Tools:

• Test Design

• Test Execution and Evaluation

• Accompanying and Support Tools

applications. The number of stages can range from as low as 1 to as 

high as 16 (Jones, 1997). 

For large software applications, firms typically use a 12-stage 

process that can be aggregated into three categories: 

Z General testing stages include subroutine testing, unit 

testing, new function testing, regression testing, integration, 

and system testing. 

Z Specialized testing stages consist of stress or capacity testing, 

performance testing, platform testing and viral protection 

testing. 

Z User-involved testing stages incorporate usability testing and 

field testing. 

After the software is put into operational use, a maintenance phase 

begins where enhancements and repairs are made to the software. 

During this phase, some or all of the stages of software testing will 

be repeated. Many of these stages are common and well 

understood by the commercial software industry, but not all 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-6 

companies use the same vocabulary to describe them. Therefore, as 

we define each software stage below, we identify other names by 

which that stage is known. 

General Testing Stages 

General testing stages are basic to software testing and occur for all 

software (Jones, 1997). The following stages are considered general 

software testing stages:1 

Z subroutine/unit testing 

Z new function testing 

Z regression testing 

Z integration testing 

Z system testing 

Specialized Testing Stages 

Specialized software testing stages occur less frequently than 

general software testing stages and are most common for software 

with well-specified criteria. The following stages are considered 

specialized software testing stages: 

Z stress, capacity, or load testing 

Z error-handling/survivability testing 

Z recovery testing 

Z security testing 

Z platform testing stage 

Z viral protection testing stage 

User-Involved Testing Stages 

For many software projects, the users and their information 

technology consultants are active participants at various stages 

along the software development process, including several stages of 

testing. Users generally participate in the following stages. 

Z usability testing 

Z field or beta testing 

Z lab or alpha testing 

Z acceptance testing 

 

1All bulleted terms listed in this section are defined in Appendix A. 
Section 2 — Software Testing Methods and Tools 

2-7 

 2.2.2 Commercial Software Testing Tools 

A software testing tool is a vehicle for facilitating the performance of 

a testing stage. The combination of testing types and testing tools 

enables the testing stage to be performed (Perry, 1995). Testing, like 

program development, generates large amounts of information, 

necessitates numerous computer executions, and requires 

coordination and communication between workers (Perry, 1995). 

Testing tools can ease the burden of test production, test execution, 

test generation, information handling, and communication. Thus, 

the proper testing tool increases the effectiveness and efficiency of 

the testing process (Perry, 1995). 

This section categorizes software testing tools under the following 

headings: 

Z test design and development tools; 

Z execution and evaluation tools; and 

Z accompanying and support tools (which includes tools for 

planning, reviews, inspections, and test support) (Kit, 1995). 

Many of the tools that have similar functions are known by different 

names. 

Test Design and Development Tools 

Test design is the process of detailing the overall test approach 

specified in the test plan for software features or combinations of 

features and identifying and prioritizing the associated test cases. 

Test development is the process of translating the test design into 

specific test cases. 

Tools used for test design and development are referred to as test 

data/case generator tools. As this name implies, test data/case 

generator tools are software systems that can be used to 

automatically generate test data/cases for test purposes. Frequently, 

these generators only require parameters of the data element values 

to generate large amounts of test transactions. Test cases can be 

generated based on a user-defined format, such as automatically 

generating all permutations of a specific, user-specified input 

transaction. The following are considered test data/case generator 

tools: 

Z data dictionary tools 

Z executable specification tools 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-8 

Z exhaustive path-based tools 

Z volume testing tools 

Z requirements-based test design 

Test Execution and Evaluation Tools 

Test execution and evaluation is the process of executing test cases 

and evaluating the results. This includes selecting test cases for 

execution, setting up the environment, running the selected tests, 

recording the execution activities, analyzing potential product 

failures, and measuring the effectiveness of the effort. 

Execution tools primarily are concerned with easing the burden of 

running tests. Execution tools typically include the following. 

Z capture/playback tools 

Z test harnesses and drivers tools 

Z memory testing tools 

Z instrumentation tools 

Z snapshot monitoring tools 

Z system log reporting tools 

Z coverage analysis tools 

Z mapping tools 

Simulation tools are also used to test execution. Simulation tools 

take the place of software or hardware that interacts with the 

software to be tested. Sometimes they are the only practical method 

available for certain tests, like when software interfaces with 

uncontrollable or unavailable hardware devices. These include the 

following tools: 

Z disaster testing tools 

Z modeling tools 

Z symbolic execution tools 

Z system exercisers 

Accompanying and Support Tools 

In addition to the traditional testing tools discussed above, 

accompanying and support tools are frequently used as part of the 

overall testing effort. In the strict sense, these support tools are not 

considered testing tools because no code is usually being executed 

as part of their use. However, these tools are included in this 

discussion because many organizations use them as part of their 
Section 2 — Software Testing Methods and Tools 

2-9 

quality assurance process, which is often intertwined with the 

testing process. 

Accompanying tools include tools for reviews, walkthroughs, and 

inspections of requirements; functional design, internal design, and 

code are also available. In addition, there are other support tools 

such as project management tools, database management software, 

spreadsheet software, and word processors. The latter tools, 

although important, are very general in nature and are implemented 

through a variety of approaches. We describe some of the more 

common testing support tools: 

Z code comprehension tools 

Z flowchart tools 

Z syntax and semantic analysis tools 

Z problem management tools 

 2.3 SOFTWARE TESTING TYPES 

Software testing activities can also be classified into three types: 

Z Conformance testing activities assess the conformance of a 

software product to a set of industry wide standards or 

customer specifications. 

Z Interoperability testing activities assess the ability of a 

software product to interoperate with other software. 

Z Performance testing activities assess the performance of a 

software product with respect to specified metrics, whose 

target values are typically determined internally by the 

software developer. 

In the following subsections, we define the roles played by each of 

the three types of software testing in the software development 

process. 

 2.3.1 Conformance Testing 

Conformance testing activities assess whether a software product 

meets the requirements of a particular specification or standard. 

These standards are in most cases set forth and agreed upon by a 

respected consortium or forum of companies within a specific 

sector, such as the Institute of Electrical and Electronics Engineers, 

Inc. (IEEE) or the American National Standards Institute (ANSI). 

They reflect a commonly accepted “reference system,” whose 

standards recommendations are sufficiently defined and tested by 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-10 

certifiable test methods. They are used to evaluate whether the 

software product implements each of the specific requirements of 

the standard or specification. 

One of the major benefits of conformance testing is that it facilitates 

interoperability between various software products by confirming 

that each software product meets an agreed-upon standard or 

specification. Because of its broad usefulness, conformance testing 

is used in most if not all of the software testing stages and by both 

software developers and software users. Conformance testing 

methodologies have been developed for operating system 

interfaces, computer graphics, document interchange formats, 

computer networks, and programming language processors. 

Conformance testing methodologies typically use the same concepts 

but not always the same nomenclature (NIST, 1997). Since the 

specifications in software standards are complex and often 

ambiguous, most testing methodologies use test case scenarios (e.g., 

abstract test suites, test assertions, test cases), which themselves 

must be tested. 

Standardization is an important component of conformance testing. 

It usually includes developing the functional description and 

language specification, creating the testing methodology, and 

“testing” the test case scenarios. Executable test codes, the code 

that tests the scenarios, have been developed by numerous 

organizations, resulting in multiple conformance testing products on 

the market. However, many rigorous testing methodology 

documents have the capability to measure quality across products. 

Sometimes an executable test code and the particular 

hardware/software platform it runs on are accepted as a reference 

implementation for conformance testing. Alternatively, a widely 

successful commercial software product becomes both the defacto 

standard and the reference implementation against which other 

commercial products are measured (NIST, 1997). 

 2.3.2 Interoperability Testing 

Interoperability testing activities, sometimes referred to as 

intersystems testing, assess whether a software product will 

exchange and share information (interoperate) with other products. 

Interoperability testing activities are used to determine whether the 

proper pieces of information are correctly passed between 

For router software 

development: 

Z Conformance testing 

verifies that the routers 

can accurately 

interpret header 

information and route 

data given standard 

ATM specification. 

Z Interoperability testing 

verifies that routers 

from different vendors 

operate properly in an 

integrated system. 

Z Performance testing 

measures routers’ 

efficiency and tests if 

they can handle the 

required capacity 

loading under real or 

simulated scenarios. 
Section 2 — Software Testing Methods and Tools 

2-11 

applications. Thus, a major benefit of interoperability testing is that 

it can detect interoperability problems between software products 

before these products are put into operation. Because 

interoperability testing often requires the majority of the software 

product to be completed before testing can occur, it is used 

primarily during the integration and system testing stages. It may 

also be used heavily during beta and specialized testing stages. 

Interoperability testing usually takes one of three approaches. The 

first is to test all pairs of products. Consumers are in a poor position 

to accomplish this because they are unaware of the interoperability 

characteristics across software products and across software firms. 

This leads to the second approach—testing only part of the 

combinations and assuming the untested combinations will also 

interoperate. The third approach is to establish a reference 

implementation and test all products against the reference 

implementation (NIST, 1997). For example, a typical procedure 

used to conduct interoperability testing includes developing a 

representative set of test transactions in one software product for 

passage to another software product for processing verification. 

Performance Testing 

Performance testing activities assess the performance of a software 

product with respect to specified metrics. The target metrics are 

usually determined within the company using industry reference 

values. Performance testing measures how well the software system 

executes according to its required response times, throughput, CPU 

usage, and other quantified features in operation by comparing the 

output of the software being tested to predetermined corresponding 

target and reference values. 

Performance testing is also commonly known by the other names 

and/or associated with other testing activities, such as stress testing, 

capacity testing, load testing, volume testing, and benchmark 

testing. These various performance testing activities all have 

approximately the same goal: “measuring the software product 

under a real or simulated load” (Beizer, 1984). 

Performance testing is usually performed as a separate testing stage, 

known as the performance testing stage. However, it is not 

uncommon for performance testing activities to be conducted as 

part of the integration or system testing stage. Typically, 

Throughput, delay, 

and load are typical 

performance testing 

parameters for large 

transaction systems, 

such as product data 

management (PDM). 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-12 

performance testing cannot be performed earlier in the life cycle 

because a fully or nearly fully developed software product is 

needed. In fact, proper performance testing may require that the 

software product be fully installed in a real or simulated operational 

environment. As result of its benefits, both users and developers 

engage in performance testing. The process is so valuable that large 

software developers, users, and system integrators frequently 

conduct benchmark comparisons (Michel, 1998). 

A major benefit of performance testing is that it is typically designed 

specifically for pushing the envelope on system limits over a long 

period of time. This form of testing has commonly been used to 

uncover unique failures not discovered during conformance or 

interoperability tests (Jones, 1997; Perry, 1995; Wilson, 1995). In 

addition, benchmarking is typically used to provide competitive 

baseline performance comparisons. For instance, these tests are 

used to characterize performance prior to manufacturing as well as 

to compare performance characteristics of other software products 

prior to purchase (Wilson, 1995). 

Performance testing procedures provide steps for determining the 

ability of software to function “properly,” particularly when near or 

beyond the boundaries of its specified capabilities or requirements. 

These “boundaries” are usually stated in terms of the volume of 

information used. The “specific metrics” are usually stated in terms 

of time to complete an operation. Ideally, performance testing is 

conducted by running a software element against standard datasets 

or scenarios, known as “reference data” (NIST, 1997). 

Performance measures and requirements are quantitative, which 

means that they consist of numbers that can be measured and 

confirmed by rational experiments. A performance specification 

consists of a set of specified numbers that can be reduced to 

measured numbers, often in the form of a probability distribution. 

The numbers measured for the software product are either less or 

more than or equal to the specified values. If less, the software 

product fails, if more than or equal to, the software product passes 

the tests. Every performance specification is a variation of these 

simple ideas (Beizer, 1984). 

Z The rate at which the 

system processes 

transactions is called 

the throughput. 

Z The time that it takes 

to process those 

transactions is called 

the processing delay. 

Z Processing delay is 

measured in seconds. 

Z The rate at which 

transactions are 

submitted to a 

software product is 

called the load. 

Z Load is measured in 

arriving transactions 

per second. 
Section 2 — Software Testing Methods and Tools 

2-13 

 2.3.4 Relationship between Software Stages, Testing 

Types, and Testing Tools 

Certain software testing types are associated with particular software 

testing stages. During these stages, different types of testing are 

performed by different parts of the software industry. Table 2-2 

illustrates the relationship between the software testing types and 

stages, while Table 2-3 maps the software testing types with the 

software development life cycle. Table 2-3 also indicates whether 

developers or end users are likely to conduct the activities. 

Table 2-2. The Degree of Usage of the Different Testing Stages with the Various Testing 

Types 

 Testing Types 

Testing Stages Conformance Interoperability Performance 

General 

Subroutine/unit H 

New function H L 

Regression H L 

Integration M H M 

System M H H 

Specialized 

Stress/capacity/load 

Error-handling/survivability 

Recovery 

Security H 

Performance H 

Platform H M 

Viral protection H 

User-involved 

Usability H M L 

Field (beta) M H H 

Lab (alpha) 

Acceptance 

Note: H = Heavy, M = Medium, L = Light: These descriptors illustrate the relative use of the testing types during the 

various testing stages. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-14 Table 2-3. Software Testing Types Associated with the Life Cycle 

General Specialized User Involved 

Unit 

New 

Function 

Regression Integra- tion System Stress Error Recov- ery Secu- rity Perform- ance Plat- form Viral Usa- bility Field Lab Accept- ance 

Conformance D D D D D D D B U Interoperability D D B B U Performance D D D B B U Note: D = Developers, U = Users, B = Both. Note: The information in Tables 2-2 and 2-3 was gathered from the literature and informal industry interviews. 
Section 3 — Software Testing Methods and Tools 

2-15 

Certain software testing tools are also associated with particular 

software testing types. In addition, certain tools are also associated 

with certain software testing stages. Table 2-4 illustrates the 

relationship between the software testing tools and types, and 

Table 2-5 maps the software testing tools to the software testing 

stages. 

 2.3.5 Standardized Software Testing Technologies 

Standardized software testing technologies such as standard 

reference data, reference implementations, test procedures, metrics, 

measures, test scripts, and test cases (both manual and automated) 

provide a scientific foundation for the commercial testing tools and 

the testing types used during the software testing stages. 

Although there are general standards for test documentation and 

various verification and validation activities and stages (IEEE/ANSI, 

1993), there appears to be a lack of specific standardized test 

technology (such as reference data and metrics) that is readily 

available for commercial software. The degree of standardization 

varies across software applications. In addition, even when 

software publishers provide testing tools, they still require 

customization and contain inconsistencies because the 

development of testing tools lags behind new software product 

releases (ITToolbox, 1999). 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

2-16 

Table 2-4. Tools Used by Type of Testing 

Test Tools Conformance Interoperability Performance 

Test Design and Development 

Test data/case generator M L H 

Data dictionary 

Executable specification 

Exhaustive path based 

Volume testing tool 

Requirements-based test design tool 

Execution Evaluation 

Execution tools H M H 

Capture/playback 

Test harness and drivers 

Analysis tools H L L 

Coverage analysis 

Mapping 

Evaluation tools L L H 

Memory testing 

Instrumentation 

Snapshot monitoring 

System log reporting 

Simulation tools M H H 

Performance 

Disaster testing 

Modeling tools 

Symbolic execution 

System exercisers 

Accompanying and Support Tools 

Code inspection tools L 

Code comprehension 

Flowchart 

Syntax and semantic analysis 

Problem management tools 

System control audit database 

L L L 

Scoring database tools 

Configuration management tools H H H 

Note: H = Heavy, M = Medium, L = Light: These descriptors illustrate the relative use of the testing tools with the 

various testing types. 
Section 3 — Software Testing Methods and Tools 

2-17 

Table 2-5. Tools Used by Testing Stage 

Test Tools General Specialty User-Involved 

Test Design and Development 

Test data/case generator H M L 

Data dictionary 

Executable specification 

Exhaustive path based 

Volume testing tool 

Requirements-based test design tool 

Execution Evaluation 

Execution tools H M H 

Capture/playback 

Test harness and drivers 

Analysis tools M M 

Coverage analysis 

Mapping 

Evaluation tools M M M 

Memory testing 

Instrumentation 

Snapshot monitoring 

System log reporting 

Simulation tools M H M 

Performance 

Disaster testing 

Modeling tools 

Symbolic execution 

System exercisers 

Accompanying and Support Tools 

Code inspection tools L 

Code comprehension 

Flowchart 

Syntax and semantic analysis 

Problem management tools 

System control audit database 

H H L 

Scoring database tools 

Configuration management tools H H L 

Note: H = Heavy, M = Medium, L = Light: These descriptors illustrate the relative use of the testing types during the 

various testing stages. 

Note: The information in Tables 2-4 and 2-5 is based on the literature and comments from industry participants. 

3-1 

 Inadequate 

 Infrastructure for 

 Software Testing: 

 Overview and 3 Conceptual Model 

An inadequate infrastructure for software testing means that software 

developers and users incur costs above levels with more efficient 

testing methods. For example, with the current infrastructure, 

developers spend extra resources on detecting, locating, and 

correcting bugs to produce a given level of product quality, but 

more bugs remain in the software to be discovered by users. Users 

who encounter bugs incur the costs associated with the reduced 

quality of the activities supported by the software and the costs of 

developing “workarounds” to deal with the bug or of returning the 

software to the developer for correction. 

Because bugs negatively affect perceived product quality, they can 

also be expected to negatively impact software sales. For example, 

bugs present in early (beta) versions of software releases increase the 

cost for early adopters, slowing the diffusion of new software 

products. Decreased software sales reduce developers’ revenues 

and mean that some potential users forego the benefits of new 

releases. Furthermore, such delays may mean that a firm or country 

will lose the early-mover advantage. When an entity is the first to 

introduce a product that changes the competitive position of the 

market, being first may give it an advantageous position for some 

time. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-2 

 3.1 SOFTWARE TESTING INADEQUACIES 

General standards for test documentation and various verification 

and validation activities and stages have been available for several 

years (IEEE/ANSI, 1993). Organizations such as the Carnegie 

Mellon Software Engineering Institute have promoted de facto 

standards for assessing and improving software processes.1 

Carnegie Mellon is also managing the Sustainable Computing 

Consortium that is investigating standards and methods to reduce 

software defects (InformationWeek.com, 2002). However, a 

specific standardized test technology (such as reference data and 

metrics) that is readily available for commercial software appears to 

be lacking. Even when the software publisher provides testing tools, 

they still require customization and contain inconsistencies because 

development of testing tools lags behind new software product 

releases (ITToolbox, 1999). 

Compounding this problem are competitive market pressures that 

have increased automation in business and manufacturing, 

increasing the amount of information that is shared between 

applications within and among companies. These forces are 

simultaneously pushing the complexity, reliability, interoperability, 

performance, and “speed of deployment” requirements of software. 

However, these forces have led several inadequacies in software 

testing infrastructure technology to emerge and become problematic 

for the software industry. For the discussion below, inadequacies 

are grouped into four categories: 

Z integration and interoperability testing issues, 

Z automated generation of test code, 

Z lack of a rigorous method for determining when a product is 

good enough to release, and 

Z lack of readily available performance metrics and testing 

measuring procedures. 

 3.1.1 Integration and Interoperability Testing Issues 

Initiatives such as real-time integrated supply chain management are 

driving the need to integrate PDM and computer-aided design 

(CAD), computer-aided manufacturing (CAM), and computer-aided 

 

1See Carnegie Mellon Software Engineering Institute’s Capability Maturity Model 

for Software (SW-CMM), <http://www.sei.cmu.edu/cmm/>. Last modified April 

24, 2002. 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-3 

engineering (CAE) with other systems that are part of the extended 

organization and supply chain. The integration of applications is a 

difficult and uncertain process. Meta Group estimates that 

application integration can account for up to one-third of the cost of 

systems implementation (Booker, 1999). Enterprise applications 

integration (EAI) is currently a huge expense, occupying 30 percent 

of company information technology budgets. Its importance is 

expected to increase in the future when it could occupy up to 56 

percent of company information technology budgets (Booker, 

1999). Estimated worldwide information technology expenditures 

were $270 billion in 1998. Given that 30 percent of the 

expenditures were on EAI, this translates to total expenditures of 

$81 billion in 1998. 

Developers rely heavily on interoperability testing during the 

integration testing stage. One of the major inadequacies within the 

software-testing infrastructure is the difficulty in determining 

whether applications and systems will interoperate. For example, if 

application A and application B interoperate and if application B 

and application C interoperate, what are the prospects of 

applications A and C interoperating (NIST, 1997)? 

 3.1.2 Automated Generation of Test Code 

Developing conformance testing code can be more time consuming 

and expensive than developing the standard or product that will be 

tested. Addressing the high testing costs is currently the focus of 

several research initiatives in industry and academia. Some of these 

initiatives are based on modeling finite state machines, combinatorial 

logic, or other formal languages such as Z (Cohen et al., 1996; Tai 

and Carver, 1995; NIST, 1997; Apfelbaum and Doyle, 1997). NIST 

has also been involved in developing formal methods for 

automatically generating tests for software products from formal 

specifications (Black, 2002; Gallagher, 1999). 

 3.1.3 Lack of a Rigorous Method for Determining When a 

Product Is Good Enough to Release 

The major problem for the software industry is deciding when a firm 

should stop testing (Vouk, 1992; Voas and Friedman, 1995; Voas, 

1998; Rivers and Vouk, 1998; Offlutt and Jeffery, 1997; NIST, 1997). 

In other words, how much testing is enough, or when is the quality 

“sufficient” for the product to be released. A more rigorous definition 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-4 

of the certainty of software quality is needed. The problem is 

exacerbated because there is disagreement not only on how to define 

enough, but also on what tests should be run to determine what is 

enough. For example, commercial software developers use a 

combination of the following nonanalytical methods to decide when 

a software element is “good enough” to release: 

Z A “sufficient” percentage of test cases run successfully. 

Z Developers execute a test suite while running a code 

coverage analyzer to gather statistics about what code has 

been exercised. 

Z Defects are classified into different severity categories and 

numbers and trends within each category are analyzed. 

Z Beta testing is conducted, allowing real users to run a 

product for a certain period of time and report problems; 

then developers analyze the severity and trends for reported 

problems. 

Z Developers analyze the number of reported problems in a 

period of time; when the number stabilizes or is below a 

certain threshold for a period of time, it is considered “good 

enough.” 

Although code coverage and trend analysis are initial steps towards 

a more rigorous definition of the certainty of software quality, 

mathematical foundations and methods for assessing the uncertainty 

in quality determinations still need to be defined. Analytically 

derived levels of confidence for software test results would give 

software developers and users a more consistent method of 

determining and comparing their estimates of the risk of deploying 

software products. 

 3.1.4 Lack of Readily Available Performance Metrics and 

Testing Procedures 

The larger software developers provide performance testing 

certification programs as well as performance benchmark metrics 

(Michel, 1998). However, performance-testing programs are 

expensive to develop and maintain and too costly for smaller 

software developers (Michel, 1998). Typically, hardware platform 

developers only conduct performance testing for the more popular 

or largest software systems. Small, new, or less popular systems 

often have no performance testing done by either the software or 

hardware developer. 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-5 

Currently, there is a lack of readily available performance metrics or 

testing procedures. If these metrics and procedures were available, 

the costs of performance certification programs would decline. This 

would lead to not only better testing for existing products, but also 

to the testing of products that are not currently tested. 

 3.1.5 Approaches for Improving Software Testing 

Infrastructure 

Numerous issues affect the software testing infrastructure and may 

lead to inadequacies. For example, competitive market pressures 

may encourage the use of a less than optimal amount of time, 

resources, and training for the testing function (Rivers and Vouk, 

1998). Improvements in standardized test technology can provide 

cascading improvements throughout the entire software testing 

infrastructure and as a result provide improvements throughout the 

software industry. As illustrated in Figure 2-2 standardized software 

testing technologies are the foundation of the entire software testing 

infrastructure, which in turn supports the software industry. 

There is a great need for certified standardized test technology. For 

example, some software publishers provide test tools. However, the 

development of these tools and the accompanying testing suites 

often lag behind the development of new software applications 

(ITToolbox, 1999). Even when commercial testing tools are 

available, testers complain that many of these tools are confusing 

and potentially harmful to the firm that uses them (ITToolbox, 

1999). Standardized testing tools, suites, scripts, reference data, 

reference implementations, and metrics that have undergone a 

rigorous certification process would have a large impact on the 

inadequacies listed in the previous section. For instance, 

integration issues could be reduced if standard test suites could give 

a certain level of confidence that if products A, B, and C pass these 

tests, then these products will interoperate with each other. Another 

example would be the availability of standardized test data, metrics, 

and automated test suites for performance testing. This would make 

benchmarking tests on less popular applications less costly to 

perform. Standardized automated testing scripts along with 

standard metrics would also provide a more consistent method for 

determining when to stop testing. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-6 

One of the main objectives of this study is to identify approaches to 

improve the software testing infrastructure. Based on findings from 

our surveys and case studies, this subsection will be expanded. 

 3.2 CONCEPTUAL ECONOMIC MODEL 

The cost of an inadequate infrastructure for software testing can also 

be expressed as the benefit of an improved infrastructure for 

software testing. These values (cost and benefit) are symmetrical. 

They are properly measured as either the minimum amount of 

money all members of society would collectively require to forego 

the improved infrastructure or as the maximum amount of money all 

members of society would collectively pay for the improved 

infrastructure. 

An appropriate measure of the economic impact of an inadequate 

infrastructure for software testing is the profit differences of 

developers and users between conditions with the current testing 

infrastructure and conditions with the counterfactual infrastructure. 

This can be expressed by summing over all developers and users as 

follows: 

∆ economic welfare = Σ ∆ developers’ profits + 

Σ ∆ end-users’ profits. 

An improved testing infrastructure could have several potential 

impacts on software developers and end users. Understanding the 

mechanism through which costs are incurred (or benefits foregone) 

is an important first step in developing a cost taxonomy (presented 

in Section 4) for estimating the economic impact of the failure to 

achieve these improvements. 

To model these impacts, we set up representative firms’ profit 

functions for developers and end users under the current and 

counterfactual conditions and investigated how changes in the 

software testing infrastructure affect firms’ costs and revenues. In 

addition, we are interested in the software developer’s selection of 

the “optimal” level of software testing resources dedicated to 

achieving software quality. The empirical analysis in Sections 6 

through 8 investigates not only a testing infrastructure’s cost impact 

associated with achieving a given level of quality, but also its 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-7 

impact on the level of quality embedded in software products, 

which is influenced by the market. 

 3.3 SOFTWARE DEVELOPERS 

In this section, we define the software developer’s profit function in 

terms of sales revenue, pre-sale software R&D expenditures, 

production costs, marketing, and after-sales service costs. We also 

graphically illustrate the developers’ selection of the profitmaximizing level of R&D expenditures and show how this level is 

affected by an inadequate testing infrastructure. 

 3.3.1 Cost Framework 

The appropriate measure of the value developers would place on an 

improved infrastructure for software testing is their profit difference 

between conditions with the current testing infrastructure and 

conditions with the counterfactual infrastructure (see Just, Hueth, 

and Schmitz [1982]). 

Profits are firm revenues minus costs. Suppose the firm produces a 

single software product (q) at a price (p). Total revenues are 

 TR = pq 

Taking a product life-cycle perspective (but ignoring the timing of 

activities to simplify the notation), costs are of two types: R&D and 

production. R&D costs are the one-time fixed costs of product 

development including testing activities. Production costs are the 

recurring costs of product production, distribution, and service. 

Suppose the developer uses n inputs or resources (x11, ..., x1n) in 

the R&D phase of software development and that the prices for the 

resources are w11, ..., w1n. The cost of R&D effort expended to 

develop and test the product is 

∑

i=1

n

 w1ix1i. 

The cost of production (i.e., of all activities after the successful 

development of the software) includes both the production, 

marketing, and distribution costs and the costs of dealing with useridentified bugs. Suppose the developer uses r resources (x11, ..., 

x1r) per product sold in software production and distribution and s 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-8 

resources (x21, ..., x2s) per product sold in after-sales service dealing 

with user-identified bugs. Developers’ production and 

distribution/service costs are 

w x w x q

s

i 1

3i 3i

r

i 1

2i 2i 















∑ + ∑ = =

. (3.1) 

The total costs over the product life-cycle of developing and 

producing the software are the sum of the R&D and production, and 

distribution/service costs: 

w x w x w x q

s

i 1

3i 3i

r

i 1

2i 2i

n

i 1

1i 1i 















∑ ∑ + ∑ = = =

. (3.2) 

The profit, π, the developer receives over the entire product lifecycle is 

































π = − ∑ ∑ + ∑ = = =

pq w x w x w x q

s

i 1

3i 3i

r

i 1

2i 2i

n

i 1

1i 1i (3.3) 

where the first term is the revenues, the second, costs. 

With improvements in testing infrastructure, resource use in the 

R&D phase (x11, ..., x1n) will change. Fewer bugs will be 

embodied in shipped products; thus, resource use for after-sales 

service (x31, ..., x3s) will also change. With improvements in 

product quality demand may increase, increasing sales of the 

software products (q) and thereby changing the resource use in 

software production and distribution (x1, ..., xr). Because 

developers are producing a (better) unique product and because 

production costs will change, product prices (p) will also change. 

Profit, π‘, under the counterfactual condition will be (where the 

prime symbol is used to indicate changed values for the variables): 





 



 



 



 π′ = ′ ′− ∑ ′ ′ + ∑ ′ ′ + ∑ ′ ′

= = =

p q w x w x w x q

s

i

i i

r

i

i i

n

i

i i

1

3 3

1

2 2

1

1 1 . (3.4) 

Thus, the benefit of an improved software testing infrastructure to a 

developer is the developer’s profit difference: π‘ – π. Alternatively, 

this profit difference can be viewed as the cost to the developer of 

failing to provide the improved infrastructure. Regardless of the 

perspective, the value can be thought of as having two components: 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-9 

the difference in the R&D and production costs plus the difference 

in the revenues received. The industry-level values are the sum of 

the firm-level profit differences for all firms in the industry. 

 3.3.2 Factors Influencing the Profit-Maximizing Level of 

R&D Expenditures 

Product quality is an integrating factor underlying the firm’s R&D 

expenditure decision, after-sales service costs, and revenue from the 

sale of software products. This subsection models R&D 

expenditures on software testing as an endogenous variable in the 

developer’s profit-maximizing decision and investigates the 

developer’s decision criteria for determining the level of quality it 

will provide in its products. The level of quality is modeled as a 

function of the R&D resources developers invest prior to shipping a 

software product. We present our model in terms of a shrinkwrapped product. However, it could be easily extended to custom 

software development by replacing the quality decision maximized 

at the time of shipping for a shrink-wrapped product with the 

quality decision maximized at the time of acceptance for a custom 

software product. 

Consider a software developer who is maximizing profits 

(represented by Eq. [3.3]) with respect to the level of R&D 

expenditures it will devote to product quality. The developer would 

prefer to maximize with respect to product quality; however, 

product quality is an unobservable attribute at the time of shipping. 

Thus, what the developer selects is the level of testing resources 

invested to produce a target level of quality. The software quality 

(Q) production function can be expressed as a function of R&D 

expenditures (i.e., labor and capital to support testing) (Σx1i) 

invested prior to shipping plus an error term (e): 

 Q = f(Σx1i) + e (3.5) 

where f’ > 0 and f’’ < 0. 

The level of quality can be thought of as the inverse in the number 

of bugs remaining in the product including its level of 

interoperability with complementary products or legacy systems. 

As shown in Figure 3-1, software quality potentially affects 

developers’ profits through changes in 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-10 

Profit = Revenue – [Testing + Production + After-Sales Service] 

































π = − ∑ + ∑ + ∑ = = =

pq w x w x w x q

s

i

i i

r

i

i i

n

i

i i

1

3 3

1

2 2

1

1 1

 Q = f(Σx1i) + e 

Z after-sales service costs, 

Z the market price of the software, and 

Z the quantity sold. 

The exact relationships determining the impact of quality on these 

three profit components depend on a variety of factors. For 

example, the extent to which quality affects developers’ after-sales 

service costs depends on the type of service agreements established 

between developers and end users. Also, the extent to which 

quality influences market price and quantity depends on end-users’ 

ability to evaluate software quality and on the search costs they are 

willing to expend to obtain information on quality. 

After-Sales Service Costs 

We begin evaluating software developers’ R&D expenditure 

decision by investigating the tradeoff between pre-sales testing and 

after-sales service costs (i.e., holding price and quantity of the 

software product constant—this assumption is relaxed in the 

following section). The profit-maximizing software developer will 

continue to invest in software testing as long as the marginal cost of 

obtaining an additional unit of quality is less than the marginal 

benefit of the additional unit of quality.2 As shown in Figure 3-2a, 

the marginal cost of pre-sales quality increases exponentially and 

the marginal benefit of avoided after-sales service is represented as 

 

2The MC curve represents the distribution of costs for a given level of testing 

technology. Additional testing resources move a developer along the curve. 

An improved testing infrastructure will shift the MC curve down. 

Figure 3-1. Software 

Quality’s Role in Profit 

Maximization 

Software quality not only affects 

price (p) and quantity (q), but 

also the resources per unit sold 

(x3i) needed, for after-sales 

service. 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-11 

Figure 3-2. Minimize Joint Costs of Pre-sales Testing and After-Sales Service (Holding Price 

and Quantity Constant) 

Marginal

Cost

(MC)

Marginal

Benefit

(MB)

"Quality Gap"

Share of After-Sales

Costs borne by End

Users

MC Testing

Prior to

Shipping

MB1

Q1 Quality

MB* of

Avoided AfterSales Service

Q*

Developer’s

Profits

π1

Quality

π*

(b)

(a)

flat. The flat marginal benefits curve reflects a constant avoided 

after-sales service cost per unit of quality.3

If the developer bears all the after-sales service costs (or if the 

developer and end user are the same entity such as in-house 

software development), as shown by MB*, the optimal level of 

 

3It is unclear if bugs found after a “large” amount of testing has already been done 

are more costly or less costly to fix. Thus, we assume a flat MB curve, implying 

that the average cost per after-sales bug is constant with respect to the level of 

quality. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-12 

quality is Q*. Q* also reflects the optimal social level of software 

quality. However, if the developer only bears part of the after-sales 

costs, the MB of quality to the developer is less. As a result, the 

developer will select a quality level of less than Q*, yielding a 

“quality gap” of (Q* – Q1). 

As shown in Figure 3-2b, the quality gap reflects instances where 

profit-maximizing software developers do not have the proper 

incentives to invest testing resources to achieve the socially optimal 

level of software testing. The quality gap illustrates that the greater 

the market power of developers, the more costs are shifted toward 

users, lowering developers’ incentives to invest in quality. 

 3.4 END USERS 

End users complete the market for software products. They 

influence R&D testing efforts through the share of after-sales costs 

they bear and through their valuation of perceived software quality. 

Restated, the end-users’ ability to observe software quality at the 

time of purchase and the contractual agreements determining who 

bears the after-sales costs of poor quality influence end-users’ 

demand for software quality. 

 3.4.1 Cost Framework 

As with software developers, the appropriate measure of the value 

end users would place on an improved infrastructure for software 

testing is their profit difference between conditions with the current 

testing infrastructure and conditions with the counterfactual 

infrastructure. 

End-users’ profits are modeled as a function of the difference in 

revenues and production costs. End-users’ total revenues are 

expressed as the price times quality for the product the firm 

produces: 

 TR = py. 

The key inputs to end-users’ production functions are divided into 

four components: pre-purchase software costs, software 

expenditures, after-purchase software costs, and “other” 

nonsoftware-related costs incurred by the end user. As with 

software developers, costs are viewed from a product life-cycle 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-13 

perspective (but again ignoring the timing of activities to simplify 

the notation). 

Suppose the end user expends n inputs or resources (x11, ..., x1n) 

prior to purchasing software and that the prices for the resources are 

w11, ..., w1n. These costs may include, for example, search costs or 

delay costs from uncertainty over the quality of available software. 

End users will then purchase up to r software products (x21, ..., x2r) 

at market prices (w21, ..., w2r). Purchase costs are one-time fixed 

costs covering software and implementation expenditures. 

In addition to the purchase cost of the software, end users may 

experience after-purchase (after-acceptance) costs comprising 

resources (x31, ..., x3s) at prices (w31, ..., w3s). After-purchase costs 

include activities, such as implementing patches and work arounds, 

idle labor, and capital resources due to software problems. Note 

that resources x1, x2, and x3 are modeled as fixed, one-time 

expenditures. 

Finally, end-user “other” production costs are included for 

completeness to capture all nonsoftware-related activities per unit 

produced. Other production costs are represented as V resources 

(x41, ..., x4v) at a price of (w41, ..., w4v), times y units produced. 

The end-user’s profit, π, can be expressed as its product life-cycle 

revenue minus its costs: 

















π = − ∑ + ∑ + ∑ + ∑ = = = =

py w x w x w x w x y

v

i 1

4i 4i

s

i 1

3i 3i

r

i 1

2i 2i

n

i 1

1i 1i (3.6) 

where the first term is revenues, the remaining terms are costs. 

With improvements in testing infrastructure, resource use in the prepurchase, purchase, and post-purchase phases of the software’s lifecycle will change. For example, certified testing procedures may 

facilitate the comparison of products across different software 

vendors, lowering search costs. Fewer bugs embodied in software 

products reduces after-sales purchase costs for end users. Finally, 

because better software may lead to better final products, the 

demand for the end-user’s final products may increase, leading to 

changes in final product prices and quantities. 

Profit, π‘, under the counterfactual condition will be 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-14 

















π′ = ′ ′ − ∑ ′ ′ + ∑ ′ ′ + ∑ ′ ′ + ∑ = = = =p y w x w x w x w x y

v

i 1

4i 4i

s

i 1

3i 3i

r

i 1

2i 2i

n

i 1

1i 1i (3.7) 

Thus, the benefit of an improved software testing infrastructure to a 

end user is the change in profit: π‘ – π. 

 3.5 THE MARKET FOR SOFTWARE PRODUCTS 

In this section we build on the insights from the developers’ and 

end-users’ profit-maximizing behavior to model the market for 

software products. We illustrate the determination of market price 

and quantity, along with consumer and producer surplus, assuming 

under a market structure of monopolistic competition. Section 3.6 

then shows the impact of an inadequate infrastructure for software 

testing on prices, quantities, and economic welfare. 

 3.5.1 Quality’s Impact on Market Prices 

If end users bear some share of the cost associated with the lack of 

software quality, this will influence the price (P) they are willing to 

pay for the product and the quantity purchased (q). To model the 

impact we assume that developers are maximizing profits with respect 

to selecting the level of pre-sale testing resources they will invest. In 

addition, we make the following modeling assumptions: 

Z Developers’ R&D expenditures, including software testing 

costs, are one-time fixed costs. 

Z After-sales service costs are variable costs and are a function 

of q (distribution of patches and customer service 

operations). 

Z End-user demand is a function of quality (Q). 

The distinction between fixed and variable costs is important in the 

software industry because the physical production of software 

products has close to zero marginal costs. In our model, per unit 

after-sales support is the primary variable cost and for simplicity is 

assumed to be constant with respect to the quantity produced.4

 

4There are likely to be some economies of scale in providing after-sales support; for 

example, maintaining service centers and developing and distributing patches 

will have decreasing per-unit costs. However, the more end users using a piece 

of software, the higher the probability a bug will be found or an interoperability 

problem will materialize. Relaxing the assumption of constant MC of after-sales 

service would add decreasing slope to the MC curve in Figure 3-3 but would 

not affect the analysis findings. 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-15 

Figure 3-3 illustrates the marginal benefits to users (referred to as the 

demand curve) and marginal cost as a function of the number of 

units sold (q) and shows how these curves shift as software quality 

changes. In a market with monopolistic competition, software 

developers will price their products where MR = MC. As quality 

improves, the software products’ value to end users increases, 

shifting out both the demand and marginal revenue curves. 

Increased quality also decreases the marginal cost of after-sales 

services, leading to a downward shift in the MC curve. The new 

intersection of the MC and marginal revenue (MR) curves results in 

increased price and quantity and increased net revenue for the 

developer. 

The profit-maximizing software developer will invest in product 

quality as long as the increased net revenue (change in total 

revenue [∆TR] minus change in total variable cost [∆TVC]), shown 

in Figure 3-3, is greater than the increased fixed costs (∆FC). It can 

be shown that the profit-maximizing level of R&D expenditures for 

the developer is where the marginal change in net revenue with 

respect to testing is equal to the marginal change in fixed costs. 

∂(TR – TVC) / ∂Σx1i = ∂FC / ∂Σx1i. (3.8) 

As mentioned earlier, key factors influencing the initial position of 

the curves in Figure 3-3 and the way they shift in response to 

changes in software quality are 

Z the share of after-sales costs borne by end users (this 

influences the initial demand and MC curves and how they 

respond to changes in quality), and 

Z end-users’ ability to determine the level of quality prior to 

purchasing the product (this influences the initial demand 

curve and its responsiveness to changes in quality). 

These factors are discussed in the following subsection. 

 3.6 MODELING AN INADEQUATE SOFTWARE 

TESTING INFRASTRUCTURE 

Inadequate software testing infrastructure affects both developers’ 

and end-users’ profit functions and hence affects their supply and 

demand for software quality, respectively. Enhanced testing tools 

and services will enable users to find bugs faster and fix them with 

fewer resources and allow users to better assess the quality of 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-16 

Quality 

 (TR TVC)0

Quantity q0

MR0 D0

MC0

Q↑

Q↑

Q↑

 (TR TVC)

Quantity q′

MR′

D′

MC′

P0

MC, MB

MC, MB

P′

a)

b)

software products. This in turn will affect developers’ and endusers’ behavior by changing the following underlying relationships 

embedded in the profit functions: 

Z cost of quality (prior to shipping), 

Z cost of after-sales service, and 

Z search costs for end users to determine quality. 

Figure 3-3. Change in 

Quality’s Impact on 

Price, Quantity, and Net 

Revenue 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-17 

The impact of these three items on developer and end-user’ profits, 

software quality, and economic welfare is described below. 

 3.6.1 Inadequate Infrastructure’s Impact on the Cost of 

Quality 

Improved software tools could decrease the testing resources 

needed to achieve a given level of quality. In effect an improved 

infrastructure would make R&D resources more productive and, as 

shown in Figure 3-4, shift the MC of testing prior to shipping down 

to the right closer to the asymptote of maximum quality (Qmax, i.e., 

no bugs in shipped software products). If f1 represents the 

relationship between R&D resources and quality (as shown in Eq. 

[3.5]) with an inadequate infrastructure and f2 represents the 

relationship with an improved infrastructure, then 

 f1(Σx1i) < f2(Σx1i). 

In terms of the cost minimization analysis illustrated in Figure 3-2, 

an improved testing infrastructure would decrease the MC of quality 

and increase the socially optimum and market level of quality (see 

Figure 3-4). In addition, an improved testing infrastructure might 

also narrow the “quality gap” by altering the shape of the MC 

testing function. For example, as the MC curve moves closer to the 

asymptote of “perfect” quality (i.e., no bugs) the MC curve may 

become steeper, leading to a smaller quality gap. 

In terms of the profit-maximizing developer shown in Figure 3-3, 

increased pre-sales quality due to enhanced testing tools will lead to 

decreased after-sales resources needed to fix bugs and develop and 

implement patches and will lead to increased demand for the higher 

quality software products. 

The overall impact on the level of R&D expenditures, however, is 

ambiguous. The shift in the quality function (Eq. [3.5]) means that 

fewer resources are required to achieve a target level of quality. But 

the lower cost of quality increases the demand for quality. The final 

change in R&D resources will depend greatly on who bears the 

costs of poor quality and end-users’ ability to evaluate the quality of 

software products at the time of purchase. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-18 

Figure 3-4. Enhanced Testing Tool’s Impact on the Marginal Cost of Quality 

Marginal Cost

(MC)

Marginal Benefit

(MB)

f

1

Quality

Gap1

Quality

f

2

Quality

Gap2

Share of After-Sales

Costs Borne by End

Users

MB

MB

Qmax

 3.6.2 Inadequate Infrastructure’s Impact on the Cost of 

After-Sales Service 

As mentioned above, fewer bugs lead to fewer resources required 

for after-sales service. In addition, an inadequate infrastructure also 

affects the cost of detecting and correcting bugs that are present in 

software after it is sold. By enhancing testing tools to detect and 

correct after-sales bugs and interoperability problems, the cost of 

after-sales service is lowered, leading to economic benefits to 

society. 

However, a counterintuitive effect of increasing the efficiency of 

after-sales services is that it could reduce the incentive for 

developers to build quality into their products. If it is less costly to 

fix errors after sales, then other factors, such as time-to-market, may 

dominate the quality determination. This in part may explain why 

software products have a lower quality compared to other consumer 

products such as appliances or automobiles. The cost of developing 

a software “patch” and distributing it to customers is relatively low 

for developers. Developers frequently e-mail patches out to 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-19 

customers at virtually zero marginal cost and the cost of installing 

the patch falls on the customers. In contrast, manufacturers of 

appliances or automobiles can incur significant per unit costs if their 

products need to be recalled to correct a defect. 

 3.6.3 Inadequate Infrastructure’s Impact on End-Users’ 

Demand

Changes in software quality will affect end-users’ demand functions 

only if end users are able to observe the changes in quality at the 

time of sales.5 An improved software testing infrastructure may 

include certification tests and metrics that would enable end users 

to compare quality across different venders’ products. These 

certification tests would increase the responsiveness (elasticity) of 

end-users’ demand to changes in software quality. Increasing the 

responsiveness of the end-users’ demand curve provides greater 

incentive for software developers to improve pre-sales quality 

through increased R&D resources. 

 3.6.4 Aggregate Impact

In every instance, an inadequate infrastructure for software testing 

leads to reductions in economic welfare as reflected in the 

combined profits of developers and end users. The magnitude and 

distribution of impacts between developers and end users depends 

on the underlying relationships in the R&D quality function, aftersales debugging function, and end-users’ demand function. 

The impact of an inadequate infrastructure on the level of quality 

provided by the market is less certain. In some instances enhanced 

testing and certification tools increase the optimal and market levels 

of software, such as in the cases of their impact on the R&D quality 

function and end-user demand function. On the other hand, aftersales testing tools lead to decreased levels of software quality at the 

time of sale. 

 

5Because, for simplicity, we have not incorporated time in our model, at this point 

we are not including reputational impacts from repeat buyers or word of mouth 

recommendations. It is true that the discovery of bugs and interoperability 

problems after sales do affect end-users’ perception of software quality and 

hence demand. However, for this discussion we are focusing on infrastructure 

technology that provides information or quality at the time of purchase or 

acceptance. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-20 

 3.7 THE TIME DIMENSION 

Because an inadequate software testing infrastructure delays when a 

new product can be introduced into the market, it decreases the 

probability of a supplier capturing the early-mover advantage. This 

can affect the timing and distribution of profits. 

The early-mover advantage is found in the superior profit position of 

the early mover compared to his position if he were not the early 

mover. The primacy of this position may be due to the following 

(Besanko, Dranove, and Shanley, 1996): 

Z Economies of learning give the innovator a cost advantage. 

Z Network externalities make a product more valuable as the 

number of consumers adopting the product increases. This 

may lead to a competitive advantage for the innovator. 

Z Reputation and buyer uncertainty over the expected 

performance of goods, especially experience goods, give the 

established supplier a competitive advantage. 

Z Buyer switching costs arise when product-specific 

knowledge is not fully transferable to new products, making 

it difficult for new suppliers to effectively compete with 

established suppliers. This is also referred to as “lock-in” or 

“installed-base” effects. 

Although the specific magnitude of benefits from the early-mover 

advantage is conditional on the specific context, the general 

consensus in the economics and strategy literature is that firms that 

move first and are able to establish a standard have the opportunity 

to economically benefit from their initiatives. In recent literature on 

the early-mover advantage, Robinson, Kalyanaram, and Urban 

(1994) find that firms first to market can develop advantages that 

can last for decades. Although the benefits vary across types of 

industry, the empirical evidence supports the belief that an earlymover advantage is greatest when brand name recognition for 

experience goods is involved. 

The literature does not, however, unambiguously find a competitive 

advantage for early movers. The highest risk for the early mover is 

the risk of backing the wrong technology or product. 

In addition, whereas early-mover advantage is of great interest to 

individual firms, it is primarily an issue of redistribution of sales. 

This can be important for U.S. market share, if U.S. companies 

adopt enhanced testing tools earlier than foreign competitors. 
Section 3 — Inadequate Infrastructure for Software Testing: Overview and Conceptual Model 

3-21 

However, if worldwide software developers all adopt enhanced 

testing tools together, then the primary benefit to the U.S. economy 

is the accelerated availability of higher quality products and not an 

early-mover advantage. 

 3.8 CONCLUSION 

Software testing infrastructure influences developers’ and end-users’ 

costs and hence the level of software quality provided in the market. 

Section 4 develops the resource cost taxonomy for developers and 

end users to inform the collection of the data needed to estimate the 

changes in the profits with an improved infrastructure. The cost 

taxonomy is built on the determinants of economic welfare 

described in this section. 

∆ economic welfare = Σ ∆ developers’ profits + 

Σ ∆ end-users’ profits 

where 

∆ developers’ profits = ∆ software revenues – ∆ R&D costs 

– ∆ software production costs 

– ∆ after-sales costs 

and 

∆ end-users’ profits = ∆ revenues 

– ∆ pre-purchase software costs 

– ∆ software expenditures 

– ∆ post-purchase software costs 

– ∆ nonsoftware production costs. 

But since 

∆ software revenues = ∆ software expenditures, 

and we assume no change in developers’ software production costs 

or end-users’ revenues and nonsoftware production costs, then 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

3-22 

∆ economic welfare = Σ [∆ developers’ R&D costs 

+ ∆ developers’ after-sales costs] 

+ Σ [∆ end users’ pre-purchase 

 software costs 

+ ∆ end-users’ post-purchase 

 software costs]. 

Technical and economic impact metrics for the components of 

economic welfare are defined in Sections 4 and 5. 
4-1 

 Taxonomy for 

 Software Testing 4 Costs 

Section 3 shows conceptually that an inadequate infrastructure for 

software testing affects the resources consumed by software 

developers to produce their products and the resources consumed 

by users to integrate and operate software in their business 

operations. This section provides a taxonomy to describe the 

resources employed by software developers and users that are 

linked to software testing activities. 

This section begins with a general discussion of the principles that 

drive software testing objectives. This discussion is followed by a 

taxonomy for measuring the labor and capital resources used by 

software developers to support software testing and by a taxonomy 

for the impact of errors (bugs) on users of software products. 

Section 5 builds on this taxonomy and describes our approach for 

estimating how an inadequate infrastructure for software testing 

affects these resources. 

 4.1 PRINCIPLES THAT DRIVE SOFTWARE 

TESTING OBJECTIVES 

Any code, no matter how accomplished the programmers, will have 

some bugs. Some bugs will be detected and removed during unit 

programming. Others will be found and removed during formal 

testing as units are combined into components and components into 

systems. However, all developers release products knowing that 

bugs still remain in the software and that some of them will have to 

be remedied later. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

4-2 

Determining the appropriate level of software testing is a subjective 

process. An infinite amount of testing will not prove the negative: 

that a bug is not in the software (see Myers [1979]). In addition, the 

more one tests software for bugs the more likely one is to find a bug 

(Beizer, 1990), and the number of feasible tests for a complex 

program is virtually infinite. 

If the primary reason why software is shipped with bugs is that it is 

impossible not to do so, the secondary reason is that it is seldom 

economically efficient to remove all bugs even if it were feasible. 

As shown in Section 3, testing consumes resources and, while it 

improves product quality, the efficient level of quality may well be 

short of perfection because, as the number of tests approaches 

infinity, the time and resource costs of such thorough testing would 

also become infinite. Thus, developers must identify the risk they 

are willing to accept and use it to identify when the product is good 

enough to ship (see Beizer [1990]). 

Identifying when the product is good enough is especially important 

in very competitive markets where being first to market offers 

economic returns to developers. In such cases where the pressure 

to meet delivery schedules and to remain competitive induces 

developers to release products before they are thoroughly vetted, 

early adopters become, in effect, beta test sites. 

 4.1.1 Testing Activities 

Testing requires planning, execution, and evaluation. Test planning 

requires selecting the specific test to be performed and organizing 

the tests. Test execution is the process of actually conducting the 

selected tests. It includes the pre-run setup, execution, and post-run 

analysis. In test evaluation, the test coverage is reviewed for 

thoroughness of the test cases, the product error is evaluated, and 

an assessment is made regarding the need for further tests or 

debugging before the software can be ready for the next stage in the 

production process (Kit, 1995). 

When users report bugs to the software developer, the developer 

has to first test the software to determine if a bug actually exists in 

the software or if the error is related to the user. If the developer 

confirms the bug’s existence, he re-develops the software and 

undertakes another round of testing. The re-development of the 

It is seldom 

economically 

efficient to remove 

all bugs even if it 

were feasible. 
Section 4 — Taxonomy for Software Testing Costs 

4-3 

product usually consists of building a software patch that is 

delivered to users. 

 4.1.2 Detecting Bugs Sooner 

“Test early, test often” is the mantra of experienced programmers. 

When defects are detected early in the software development 

process, before they are allowed to migrate to the next stage, fewer 

remain in the shipped product and they are less costly to correct 

than if they are discovered later in the process (Kit, 1995). 

For example, it is costlier to repair a bug that is created in the unit 

stage in the component or system development stage than it is to 

remedy the same bug in the unit stage when it was introduced. An 

important reason why it is more costly to correct bugs the longer 

they are left undetected is because additional code is written around 

the code containing the bug. The task of unraveling mounting 

layers of code becomes increasingly costly the further downstream 

the error is detected. 

 4.1.3 Locating the Source of Bugs Faster and with More 

Precision 

Modern software products typically contain millions of lines of 

code. Precisely locating the source of bugs in that code can be very 

resource consuming. If the location of bugs can be made more 

precise, both the calendar time and resource requirements of testing 

can be reduced. Most bugs are introduced at the unit stage. Thus, 

effective testing methods for finding such bugs before units are 

combined into components and components into systems would be 

especially valuable. 

 4.2 SOFTWARE DEVELOPERS’ COST TAXONOMY 

Every software developer provides at least some of their own 

software testing services. In some cases, however, commercial 

testing services supplement in-house services. When testing is 

outsourced, the costs are simply the expenditures made by the 

developer plus the implicit costs of contracting for these services. 

Implicit costs are the value of self-owned resources devoted to the 

activity. When testing services are self-provided, most costs are 

implicit, and we must identify, quantify, and value the self-owned 

resources developers allocate to testing. 

If the location of bugs 

can be made more 

precise, both the 

calendar time and 

resource requirements 

of testing can be 

reduced. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

4-4 

 4.2.1 Resource Categories 

The resources used in software testing can be broadly grouped into 

labor and capital services. The distinguishing feature of capital is 

that it is long-lived with an up-front payment, whereas labor costs 

are virtually a continuous expenditure by developers. 

Labor resources include all the labor-hours spent in testing the 

software, locating the source of the errors, and modifying the code. 

Because different types of labor have different opportunities, it is 

appropriate to subdivide labor into the skill levels used in testing. 

Table 4-1 describes the skills of three major types of programming 

expertise used in testing software. 

Table 4-1. Labor Taxonomy 

Labor Type Skills 

Annual Salary 

(median in 

2000) 

Computer programmers Write, test, and maintain the detailed instructions, called 

programs, that computers must follow to perform their 

functions. They also conceive design and test logical 

structures for solving problems by computer. 

$57,590 

Computer software 

engineers: applications 

Analyze users’ needs and design, create, modify, and test 

general computer applications software or specialized 

utility programs. They develop both packaged systems and 

systems software or create customized applications. 

$67,670 

Computer software 

engineers: systems 

software 

Coordinate the construction and maintenance of a 

company’s computer systems, and plan their future 

growth. Software systems engineers work for companies 

that configure, implement, and install complete computer 

systems. 

$69,530 

Source: Bureau of Labor Statistics, Occupational Outlook Handbook, 2002. 

The annual costs for labor, computers, and testware do not fully 

capture the costs to developers of these resources because overhead 

is not included in the estimates. To estimate the labor cost 

associated with software testing, a fully loaded wage rate should be 

used that includes benefits and other employee-related costs 

incurred by software developers. It is impractical to quantify all of 

these individual resources. Thus, a simple loading factor of two is 

used to scale the hourly wages obtained from the BLS. 
Section 4 — Taxonomy for Software Testing Costs 

4-5 

One of the two primary capital resources used in software testing is 

the computer. It includes the hardware systems (including 

peripherals), software (e.g., operating system, compilers), and 

network configuration equipment (Wilson, 1995). Typically, these 

items are considered part of the test facility. Computer resources 

used in testing are further described in Table 4-2. Typically, 

computers are replaced not because they are physically incapable 

of performing their original purpose but because of technological 

obsolescence as new computers are introduced that have more 

desirable attributes (e.g., processing speed, memory). 

Table 4-2. Software Testing Capital Taxonomy 

Capital Type Description 

Computer Resources 

Hardware systems Clients, servers, simulator hardware (such as fault injectors, test harnesses, 

and drivers) plus operating systems or compilers (if necessary) 

Network infrastructure Routers, cabling, data storage devices, etc. 

Testing Resources (CAST)a 

Tools for test planning Project management tools, database management software, spreadsheet 

software, and word processors 

Tools for test design and 

development 

Test data/case generator tools include executable specification tools, 

exhaustive path-based tools, volume testing tools, data dictionary tools, 

and requirements-based test design tools 

Tools for test execution and 

evaluation 

Execution tools include capture/playback tools, test harnesses, and 

drivers. Analysis tools include coverage analysis tools and mapping tools. 

Evaluation tools include memory testing tools, instrumentation tools, 

snapshot monitoring tools, and system log reporting tools. Simulation 

tools include performance tools, disaster-testing tools, modeling tools, 

symbolic execution tools, and system exercisers 

aSource: Kit, Edward. 1995. Software Testing in the Real World. Essex, England: Addison-Wesley. 

The second main software testing capital resource is the software 

that runs the tests. Programmers may develop their own software 

testing capabilities or they may purchase computer-aided software 

testing (CAST) tools. Testware (software purchased or developed for 

testing applications) may be designed for a single application and 

then discarded, or more commonly, it is purchased or developed 

with the intent to be used in several projects. Other more generalpurpose software such as spreadsheets and word processors may 

also be used in testing. Testware is a product that does not wear out 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

4-6 

with repeated use; however, it is subject to technological 

obsolescence as testware and the software become more advanced. 

Testware is used for test planning, test design and development, and 

test execution and evaluation. Test planning tools assist a company 

in defining the scope, approach, resources, and scheduling of 

testing activities. Test design tools specify the test plan and identify 

and prioritize the test cases. Test execution and evaluation tools 

run the selected test, record the results, and analyze them. These 

tools may be supplemented with testing support tools that are used 

to assist with problem management and configuration management. 

Other more general-purpose software, such as spreadsheets and 

word processors, may also be used in testing (Kit, 1995). 

Testing resources may be shared with software development 

activities or dedicated to testing. The most obvious and important 

resource subject to such sharing of responsibilities is labor. In small 

organizations, testing may be each developer’s responsibility. 

Usually with growth in size come opportunities for division and 

specialization of labor. In the extreme case, software developers 

will have a centralized test organization that is independent of the 

development effort. Students of organizational theory argue that 

such independence is essential to provide the unbiased and 

complete examination needed to thoroughly evaluate the product. 

Computer resources have the potential to be used in both software 

development and in testing. Testware, however, is specific to the 

testing activity. 

In addition to the resources directly employed in software testing, 

any organization will have an infrastructure (overhead) needed to 

support testing. Because it is not practical to enumerate all the 

resources and estimate their quantities, we use a multiplier of 1.2 to 

capture the associated overhead costs associated with software and 

hardware expenditures. 

 4.2.2 Summary of Developer Technical and Economic 

Metrics 

Software developers’ costs include both pre-release costs and postrelease costs. Pre-release costs include testing costs absorbed by the 

developer of the software at each individual stage of the testing 

process. Technical and economic metrics are shown in Table 4-3. 

The worldwide market for 

automated software quality 

tools reached $931 million 

in 1999 and is projected to 

grow to $2.6 billion by 

2004 (Shea, 2000). 
Section 4 — Taxonomy for Software Testing Costs 

4-7 

Table 4-3. Impact Cost Metrics for Software Developers 

Specific Cost Technical Metric Economic Metric 

Pre-release costs 

Pre-release labor costs Labor hours to support testing 

to find bugs 

Labor costs of detecting bugs 

 Labor hours for locating and 

correcting bugs 

Labor costs for fixing bugs 

Hardware costs Total hardware used to support 

testing activities and support 

services 

Total hardware costs to support detecting 

and fixing bugs in the software 

development process and support activities 

Software costs Total software used to support 

testing activities and support 

services 

Total software costs to support detecting 

and fixing bugs in the software 

development process and support activities 

External Testing costs Testing services provided by 

specialized companies and 

consultants 

Total expenditures on external testing 

Post-release costs 

After-sales service costs Labor hours for support services Total labor costs for support services 

Post-release costs emerge after the user has accepted the custom 

product or after the developer has released the commercial product. 

In both custom and commercial applications, the developer 

frequently supplies some type of customer support services. This 

support can range from technical service hot lines that answer 

questions for commercial products, to developing patches to correct 

bugs that remain in the post-purchase versions of the software, to 

full-service support contracts to continually maintain and enhance 

custom products. 

 4.3 SOFTWARE USERS’ COST TAXONOMY 

Software testing activities affect users primarily through the bugs that 

remain in the software programs they purchase and operate. The 

degree to which bugs in software products affect users’ business 

operations varies across the types of software product purchased 

and their role in the user’s business operations. Bugs present in 

software integral to the real-time business operations of companies 

can significantly affect profits through installation delays and system 

failures. For other software applications that are more involved in 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

4-8 

batch or offline business operations, bugs may be problematic but 

less costly. 

To investigate the impact of bugs, we group user costs associated 

with software into three categories: 

Z pre-purchase costs—time and resources users invest to 

investigate different software companies and different 

software products; 

Z installation costs—time and resources users invest in 

installing and verifying operation of the new software 

products; and 

Z post-purchase costs—costs that emerge because of software 

failures and the corresponding maintenance and upkeep 

expenditures needed to repair the software bugs and 

damaged data. 

The following subsections provide more detail on these three 

categories and provide a taxonomy for measuring the cost of bugs to 

users. 

 4.3.1 Pre-purchase Costs 

Bugs in software products affect users even before they purchase the 

product. Because the number and severity of bugs remaining in a 

software product upon purchase are unobservable, users may be 

uncertain about the product’s quality. As a result, users must invest 

additional time and resources to learn about the commercial 

product they are purchasing or the company they are hiring to 

develop their custom software. Pre-purchase costs associated with 

bugs in software are shown in Table 4-4 and emerge in three ways: 

Z First, users must spend additional labor hours investigating 

products, learning about products, and gaining additional 

information. Senior scientists and upper management are 

typically involved in these purchase decisions, and labor 

costs can be generated using their typical hourly labor rates. 

Z Second, the time users spend investigating new software 

products delays the profits that firms could have received if 

they were to install the product earlier. This leads to the 

continued use of products with lower quality attributes. 
Section 4 — Taxonomy for Software Testing Costs 

4-9 

Table 4-4. Users’ Pre-Purchase Costs Associated with Bugs 

Cost Category Specific Cost Technical Matrix Economic Matrix 

Purchase decision 

costs 

Labor costs Labor hours spent on 

information gathering and 

purchase decision process 

Fully loaded labor rates times 

labor hours 

 Increase 

information 

gathering time 

Purchase time is delayed 

because of informationgathering activities 

Additional operating cost or 

lost revenue due to continued 

operation of lower-quality 

system 

Delayed adoption 

costs 

Delayed 

adoption 

Purchase time is postponed 

because of uncertainty over 

bugs 

Additional operating cost or 

lost revenue due to continued 

operation of lower-quality 

system 

Z Third, and related to the first two items, even after users 

gather all available information, they may choose to delay 

adoption of a new software product until the uncertainty is 

reduced when historical information is available about the 

product’s quality. By delaying their purchase, users 

decrease the probability of purchasing a product that has an 

unexpectedly large number of bugs. Most users do not want 

to be the “early adopters” or “beta testers,” so they wait until 

the product has been well established in the marketplace 

and several versions have been released. 

The economic impacts of the second and third categories are 

basically the same. They both delay the adoption of higher-quality 

software and hence increase the cost of operation or delay the 

introduction of new products and services. However, the source of 

the delay is slightly different—one lengthens the decision-making 

process, and the other delays the adoption decision. 

 4.3.2 Installation Costs 

Bugs remaining in software after its release can significantly increase 

the cost of installation. Installations of new software technologies 

often fail or generate unforeseen problems as they are integrated 

with existing (legacy) software products. When this occurs, users 

must spend additional resources on installing and repairing the 

software system. These expenditures can emerge as additional labor 

hours, expenditures on consultants, or time spent on support calls 

with software developers. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

4-10 

However, the magnitude of installation costs due to bugs and who 

bears these costs differ between commercial products and custom 

products. When a commercial product is purchased, installation is 

generally straightforward and relatively bug free. Many commercial 

software products are designed to interoperate with other 

technologies, lowering the installation costs. However, if installation 

problems do occur, the user typically bears most of the costs. 

In contrast, custom product installation can be a very complicated 

process, and users often work with the software developer or a 

third-party integrator to install the new software. Contractual 

arrangements determine which parties bear the bulk of 

implementation costs. If third-party developers are hired to aid with 

installation, then users typically bear the cost of bugs. If the 

contract with software developers includes installation support, then 

these costs will be captured in the total costs that the software 

developers incur during the development stage. As shown in 

Table 4-5, users’ labor costs can be estimated using the fully loaded 

labor costs presented in the previous section and the estimated 

number of additional labor hours due to software bugs. 

Table 4-5. Users’ Implementation Costs Associated with Bugs 

Cost Category Specific Cost Technical Matrix Economic Matrix 

Installation costs Labor costs Labor hours of company 

employees 

Fully loaded labor rates 

times labor hours 

 Third-party integrator Labor hours of consultants Consultants’ hourly rate 

times labor hours charged 

 Lost sales Company downtime due to 

extended installation 

Cost of foregone profits 

In addition to labor costs, bugs encountered during installation lead 

to lost sales due to company downtime while the product is being 

installed. In some cases, firms will be able to install software 

outside of traditional business hours. In these cases no sales are 

forfeited. However, other users may have to suspend business 

operations to install software. If part of this downtime is due to bugs 

in the software or increased post-installation testing due to 

uncertainty over bugs, then this will lead to increased lost profits. 
Section 4 — Taxonomy for Software Testing Costs 

4-11 

 4.3.3 Post-purchase Costs 

Once the decision to purchase the software has been made and the 

new software is installed, additional costs due to bugs may continue 

to emerge. Because of bugs, software may not have the desired 

functionality anticipated by users. This can lead to lower 

performance or total failure of the new and/or existing software 

systems. For example, bugs may lead to interoperability problems 

between the new software and existing software, leading to 

inefficient operations, system downtime, or lost data. Table 4-6 

describes post-purchase costs associated with software bugs. 

Table 4-6. Users’ Post-purchase Costs Associated with Bugs 

Cost Category Specific Cost Technical Matrix Economic Matrix 

Product failure 

and repair costs 

Labor costs Labor time of employees spent 

repairing bugs and reentering lost 

data 

Fully loaded labor rates 

times labor hours 

 Capital costs Early retirement or “scrapping” of 

ineffective systems 

Expenditures on 

new/replacement system 

 Consultants’ 

costs 

Hiring consultants to repair data 

archives 

Expenditures on outside 

consultants 

 Sales forfeited Company downtime attributable 

to lost data 

Lost profit from foregone 

transactions during this time 

period 

Inability to fully 

accomplish tasks 

Labor costs Labor time of employees to 

implement “second best” 

operating practices 

Fully loaded labor rates 

times labor hours 

 Sales forfeited Lost sales due to “second best” 

operating practices 

Lost profit from foregone 

transactions 

Redundant 

systems 

Hardware costs Multiple hardware systems 

maintained in case of system 

failure 

Expenditures on hardware 

systems 

 Software costs Licensing or updating old 

software after shift to new 

software system 

Expenditures to license or 

update old software 

 Labor costs Labor time of employees 

maintaining a redundant 

hardware and software system 

Fully loaded labor rates 

times labor hours for 

maintaining old system 

Software failures are the most publicized user impact associated 

with bugs. These failures typically stem from interoperability 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

4-12 

problems between new and existing software products. The result 

of failures is frequently a shutdown in part or all of the firm’s 

operations. However, not all catastrophic software failures are 

associated with bugs. Some failures are due to inadequate 

parameter specifications (by users) or unanticipated changes in the 

operating environment. Thus, when estimating the costs associated 

with software failure due to inadequate software testing one cannot 

simply quantify all failure costs. 

In addition to catastrophic failures, software bugs can also lead to 

efficiency problems for users. Although less dramatic, when 

software does not operate as promised, users can experience 

increased operating costs due to second-best work-arounds or 

patches and lost or delayed sales. User impacts can become sizable 

if these bugs lead to ongoing problems that impose costs over the 

life of the software product. 

The final post-purchase cost that emerges because of bugs is the 

cost of redundant systems. Because of uncertainty about bugs, 

software users often keep their old software system in place for a 

period of time after they have purchased and installed a new 

software system. If bugs are continually emerging in the new 

system, users may maintain their old system for significantly longer 

than they would have if they were more confident about the quality 

of the new software product that they purchased. 
5-1 

 Measuring the 

 Economic Impacts of 

 an Inadequate 

 Infrastructure for 5 Software Testing 

This section describes the counterfactual scenario associated with 

an inadequate infrastructure for software testing and outlines our 

approach for estimating the economic impacts for software 

developers and users. It also provides an introduction for the case 

studies that follow in Sections 6 and 7, describing how the impacts 

of inadequate software testing may differ between CAD/CAM/CAE 

users in the transportation equipment manufacturing sector and 

FEDI/clearinghouse software users in the financial services sector. 

 5.1 DEFINING THE COUNTERFACTUAL WORLD 

To estimate the costs attributed to an inadequate infrastructure for 

software testing, a precise definition of the counterfactual world is 

needed. Clearly defining what is meant by an “inadequate” 

infrastructure is essential for eliciting consistent information from 

industry respondents. 

In the counterfactual scenario we keep the intended functionality of 

the software products released by developers constant. In other 

words, the fundamental product design and intended product 

characteristics will not change. However, the realized level of 

functionality may be affected as the number of bugs (also referred to 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-2 

as defects or errors) present in released versions of the software 

decreases in the counterfactual scenario. 

The driving technical factors that do change in the counterfactual 

scenario are when bugs are discovered in the software development 

process and the cost of fixing them. An improved infrastructure for 

software testing has the potential to affect software developers and 

users by 

Z removing more bugs before the software product is released, 

Z detecting bugs earlier in the software development process, 

and 

Z locating the source of bugs faster and with more precision. 

A key assumption is that the number of bugs introduced into 

software code is constant regardless of the types of tools available 

for software testing; they are errors entered by the software 

designer/programmer and the initial number of errors depends on 

the skill and techniques employed by the programmer.1 

Figure 5-1 (re-illustrated from Section 2) provides an illustration of 

the software development process. The development of software 

starts with the system software design, moves to implementation and 

unit testing, and then ends with integration testing as the 

subcomponents of the software product are assembled and then the 

product is released. 

Errors are generated (or introduced) at each stage of the software 

development process. An improved infrastructure would find the 

bugs within (or closer to) the stage in which they were introduced 

rather than later in the production process or by the end user of the 

software product. As described in Section 4, the later in the 

production process that a software error is discovered the more 

costly it is to repair the bug. 

 

1We make the distinction between inadequate software testing and inadequate 

programming skills or techniques. For example, Carnegie Mellon Software 

Engineering Institute has developed the Personal Software Process (PSP) and the 

Team Software Process (TSP) that are designed to reduce the number of errors in 

the program when it is first compiled. In general, the PSP and TSP involve 

individual programmers tracking their errors to improve their programming 

skills and team members thoroughly reviewing code to identify errors prior to 

compiling and run time testing. For this study, we define these programming 

activities as up stream and not part of the software testing process. Thus, the 

number of errors generated as part of initial software coding does not change in 

the counterfactual scenario. It is the process of identifying and correcting these 

“exogenous” errors that changes. 

A key assumption is 

that the number of 

bugs introduced into 

software code is 

constant regardless 

of the types of tools 

available for 

software testing. 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-3 

Figure 5-1. The Waterfall Process 

In the waterfall process, testing occurs at multiple stages during the software development process. 

HLD: High-Level Design

I0: HLD Inspection

LLD: High-Level Design

I1: LLD Inspection

I2: Code Inspection

UT: Unit Test

RAISE: Reliability, Availability, Install

Serviceability, and Ease of Use

Requirements

Gathering and

Analysis

Architectural

Design

HLD/I0

Component Test

RAISE System Test

Release

Early Customer

Feedback and Beta

Test Programs

LLD/I1 CODE/I2 UT

• • •• • •• • •

• • •

Integration

 5.1.1 Developers’ Costs of Identifying and Correcting 

Errors 

The relative cost (also referred to as cost factors) of repairing defects 

found at different stages of software development increases the longer 

it takes to find a bug. Table 5-1 illustrates this with an example 

showing the relative differences in the cost of repairing bugs that are 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-4 

Table 5-1. Relative Cost to Repair Defects When Found at Different Stages of Software 

Development (Example Only) 

X is a normalized unit of cost and can be expressed terms of person-hours, dollars, etc. 

Requirements Gathering 

and Analysis/ 

Architectural Design 

Coding/Unit 

Test 

Integration and 

Component/RAISE 

System Test 

Early Customer 

Feedback/Beta Test 

Programs 

Post-product 

Release 

1X 5X 10X 15X 30X 

introduced in the requirements gathering and analysis/architectural 

design stage as a function of when they are detected. For example, 

errors introduced during this stage and found in the same stage cost 

1X to fix. But if the same error is not found until the integration and 

component/RAISE system test stage, it costs 10 times more to fix. This 

is due to the reengineering process that needs to happen because the 

software developed to date has to be unraveled and rewritten to fix 

the error that was introduced earlier in the production process. 

However, bugs are also introduced in the coding and integration 

stages of software design. 

A complete set of relative cost factors is shown in Table 5-2 and 

shows that regardless of when an error is introduced it is always 

more costly to fix it downstream in the development process. 

Table 5-2. Preliminary Estimates of Relative Cost Factors of Correcting Errors as a Function of 

Where Errors Are Introduced and Found (Example Only) 

 Where Errors are Found 

Where Errors are 

Introduced 

Requirements 

Gathering and 

Analysis/ 

Architectural Design 

Coding/ 

Unit Test 

Integration and 

Component/ 

RAISE System 

Test 

Early 

Customer 

Feedback/Beta 

Test Programs 

Postproduct 

Release 

Requirements Gathering 

and Analysis/ 

Architectural Design 

1.0 5.0 10.0 15.0 30.0 

Coding/Unit Test 1.0 10.0 20.0 30.0 

Integration and 

Component/ 

RAISE System Test 

 1.0 10.0 20.0 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-5 

In addition, as part of our analysis we investigate the difference in 

the cost of introducing errors in the same stage throughout the 

software development process. Conceptually there is no need to 

restrict the diagonal elements in Table 5-2 to be all 1.0. Each 

column has its own unique base multiplier. This could capture, for 

example, that errors introduced during integration are harder to find 

and correct than coding or design errors. 

The relative cost factors for developers shown in Table 5-2 also 

illustrate that errors are found by users in the beta testing and postproduct release stages because typically not all of the errors are 

caught before the software is distributed to customers. When users 

identify an error, developers bear costs related to locating and 

correcting the error, developing and distributing patches, and 

providing other support services. Users bear costs in the form of 

lost data, foregone transactions, and product failures; however, 

these costs are not included in developers’ relative cost factors and 

were estimated separately, as described Section 5.3. 

The total cost of errors can be calculated by combining the relative 

cost factors with the number and distribution of errors. Table 5-3 

shows an example of the frequency distribution of where errors may 

be found, in relationship to where they may be introduced. 

Table 5-3. Example of the Frequency (%) of Where Errors Are Found, in Relationship to Where 

They Were Introduced 

 Where Errors Are Found 

Where Errors are 

Introduced (%) 

Requirements 

Gathering and 

Analysis/ 

Architectural 

Design 

Coding/ 

Unit Test 

Integration 

and 

Component/ 

RAISE 

System Test 

Early 

Customer 

Feedback/ 

Beta Test 

Programs 

Postproduct 

Release Total 

Requirements Gathering 

and Analysis/Architectural 

Design 

3.5 10.5 35 6 15 70 

Coding/Unit Test 6 9 2 3 20 

Integration and 

Component/RAISE System 

Test 

 6.5 1 2.5 10 

Total 3.5 16.5 50.5 9 20.5 100% 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-6 

The “smoothed” cumulative distribution of error detection is 

depicted in Figure 5-2. The data in this figure exhibit the classic S 

shape of the cumulative distribution of the discovery of errors with 

respect to life-cycle stages as published by several researchers 

(Vouk, 1992; Beizer, 1984). This is important because it (along with 

Table 5-3) most clearly illustrates the problem plaguing the software 

development industry for years: “Most software errors are found 

during the middle to later stages of development (namely integration 

through primary release), which happen to be the most expensive 

stages to fix errors” (Rivers and Vouk, 1998). 

Figure 5-2. Typical Cumulative Distribution of Error Detection 

100

90

80

70

60

50

40

30

20

10

0

R-D C-U I-S E-R

Cumulative Distribution (%)

Errors Detected in Each Stage

P-R

Legend: 

R-D: Requirements Gathering and Analysis/Architectural Design 

C-U: Coding/Unit Test 

I-S: Integration and Component/RAISE System Test 

E-R: Early Customer Feedback/Beta Test Programs 

P-R: Post-product Release 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-7 

Combining the distribution of where errors are found with the 

relational cost factors to correct the errors provides a graphical 

depiction of developers’ costs. In Figure 5-3, the area below the 

step-wise graph represents the costs associated with errors detected 

in the various stages of the software life cycle. Thus, if we knew the 

total expenditures software developers spend on testing and 

correction activities, we can solve for the average cost per bug and 

the individual step-wise areas shown in Figure 5-3. 

Figure 5-3. Software Testing Costs Shown by Where Bugs Are Detected (Example Only) 

“Costs” can be expressed in terms of expenditures or hours of testing time. 

Average

Cost per

Bug

X1

X2

X3

X4

X5

I-S

R-D

C-U

15% 20% 40% 15% 10%

Distribution Where

Bugs Are Detected

100%

E-R P-R

R-D C-U E-R P-R I-S

Legend: 

R-D: Requirements Gathering and Analysis/Architectural Design 

C-U: Coding/Unit Test 

I-S: Integration and Component/RAISE System Test 

E-R: Early Customer Feedback/Beta Test Programs 

P-R: Post-product Release 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-8 

 5.1.2 Counterfactual Scenario for Developers 

The core of our counterfactual scenario for developers can then be 

described in terms of the introduction–found categories as shown in 

Figure 5-4. The impact of an inadequate infrastructure for software 

testing on fixing errors can be calculated from 

Z changes in the relative cost factors in the introduction–found 

error categories (Table 5-2) and 

Z changes in the distribution of where errors are detected 

(Table 5-3). 

Figure 5-4. Cost Reductions of Detecting Bugs and Fixing Them Faster (Example Only) 

Shaded area represents the developers’ costs due to an inadequate infrastructure for software testing. 

Average

Cost per

Bug

Lower Cost of

Fixing Bugs

Detecting

Bugs

Earlier

Distribution Where

Bugs Are Detected

C1

C0

15%

20%

For example, the cost to fix a bug that occurred during the coding 

stage that is not discovered until the integration phase may decrease 

from C0 to C1 if enhanced software testing tools decrease the time 

needed to locate the error’s source. Alternatively, with better testing 

tools, more bugs introduced in the requirements stage might be 

found during that stage, increasing the percentage of bugs found in 

this stage from 15 to 20 percent. 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-9 

Note again that the total number of errors introduced into the 

software is assumed to be unchanged in the counterfactual. These 

bugs are a normal and expected part of the software production 

process. The distribution of the bug’s location and the cost of fixing 

the errors change. 

In addition to changes in correction costs and detection distribution 

described in Tables 5-2 and 5-3, we also investigated changes in 

fixed costs such as hardware and software used to support software 

testing. With enhanced testing tools developers may change their 

annual expenditures on these capital inputs. However, changes in 

labor costs associated with locating and correcting errors are the 

dominant economic impact for developers. 

 5.1.3 Counterfactual Scenario for Users 

The primary impact for users associated with the counterfactual of 

an improved infrastructure for software testing is that few bugs 

would make it to the software operations stage. This would lead to 

lower user maintenance costs and lower software failure costs. In 

Section 5.4 we discuss the behavior changes users may undertake in 

response to fewer bugs. For example, changes in avoidance 

activities such as backup data storage and redundant operating 

systems may represent significant annualized cost savings. 

A key assumption in the counterfactual scenario is the “level” of 

reduction in the number of bugs encountered by users during 

business operations. In some instances it may be unrealistic to 

assume that an improved infrastructure will lead to the detection of 

all bugs during software testing. As part of the developers’ surveys, 

we asked developers to estimate cost impacts under different 

percentage error reduction scenarios. 

 5.2 CUSTOM VERSUS COMMERCIAL SOFTWARE 

PRODUCTS 

To quantify the economic costs attributable to an inadequate 

infrastructure for software testing, we distinguish between costs 

borne by the developer of the software product and costs borne by 

the users of the software product. This distinction is necessary to 

facilitate data collection activities and prevent double counting. To 

support this partitioning of costs, we will need to be cognizant of 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-10 

the difference between custom and commercial software product 

development, as presented in Figure 5-5. 

Custom Development Commercial Products

Software Development

(Developer Cost)

Testing

(Developer Cost)

Implementation

(Developer and User Cost)

Performance/Operation

(User Cost)

Software Development

(Developer Cost)

Testing

(Developer Cost)

Implementation

(User Cost)

Performance/Operation

 (User Cost)

Release

Figure 5-5. Custom vs. 

Commercial 

Development Cost 

Allocation 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-11 

Both custom and commercial (prepackaged) software products have 

similar production processes. As shown in Figure 5-5, they both start 

with software design and coding, move to software unit and 

integration testing, then implementation, and finally to operation and 

product support. 

The primary difference between custom and commercial software 

products is that there is no formal release for custom products and the 

implementation may require significant resources compared to 

commercial products. As a result, the developer plays a much larger 

role in the implementation and post-purchase service of custom 

software, compared to commercial software. Finally, third-party 

integrators are frequently involved in implementing custom software. 

Because third-party integrators are typically hired by users, we 

collected this cost information as part of the user surveys. 

 5.3 ESTIMATING SOFTWARE DEVELOPER COSTS 

An inadequate infrastructure for software testing will lead to errors 

being identified later in the development process and more 

resources being needed to locate and correct the source of the error. 

These consequences affect developer costs throughout the 

software’s life cycle through changes in the following: 

Z labor costs—additional employee and contract labor 

expenditures for pre-purchase testing and error correction, 

installation, and post-purchase repair; 

Z software costs—additional or redundant testing software 

purchases; 

Z hardware costs—additional expenditures on equipment, 

computers, and other physical technologies used in the 

testing process; 

Z after-sales service costs—additional nontesting and 

debugging activities such as fielding an increased number of 

service calls and the distribution of patches; 

Z delay costs—discounted value of lost profits due to time 

delays in product releases and delayed adoption by users 

due to large numbers of bugs in early software versions; and 

Z reputation costs—lost sales or market share due to highly 

publicized product failures. 

The impact cost metrics that guided the development of the survey 

instruments for survey developers are discussed in Section 4 and are 

summarized in Table 5-4. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-12 

Table 5-4. Impact Cost Metrics for Software Developers 

Cost Category Specific Cost Technical Metric Economic Metric 

Pre-release 

labor costs 

Labor hours to support testing 

to find bugs 

Pre-release Labor costs of detecting bugs 

costs 

 Labor hours for location and 

correction of bugs 

Labor costs for fixing bugs 

 Hardware 

costs 

Total hardware used to support 

testing activities and support 

services 

Total Hardware costs to support 

detecting and fixing bugs in the 

software development process and 

support activities 

 Software costs Total software used to support 

testing activities and support 

services 

Total software costs to support 

detecting and fixing bugs in the 

software development process and 

support activities 

 External 

Testing costs 

Testing services provided by 

specialized companies and 

consultants 

Total expenditures on external testing 

Post-release 

costs 

After sales 

service costs 

Labor hours for support services Total labor costs for support services 

Current 

Distribution of 

Cost and Errors

Relative cost 

factors 

Relative cost factors relating the 

cost of correcting errors for 

each introduction-detection 

category (Table 5-2) 

Area under graph in Figure 5-2 shows 

the distribution of costs by the stage 

detected 

 Distribution of 

bugs 

Distribution of detected bugs 

over the introduction-detection 

space (Table 5-3) 

 Relative cost 

factors 

Change relative cost factor for 

introduction-detection 

categories (Table 5-2) 

Change in labor costs locating and 

correcting errors once they have been 

identified 

Counterfactual 

Scenario 

(improved 

testing 

infrastructure) Distribution 

of bugs 

Change in distribution of bug 

introduction-detection 

(Table 5-3) 

 Hardware Change in hardware needed to 

support error detection, 

location and correction 

Change in annual hardware 

expenditures 

 Software Change in software needed to 

support error detection, 

location and correction 

Change in annual software 

expenditures 

Impact on sales Delayed 

market 

introduction 

Length of delay and the number 

of units that would have been 

sold per period of delay 

Delayed benefits to users 

 Delayed user 

adoption 

Decreased market penetration Delayed benefits to users 

 Reputation Lost market share NA—transfer payments 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-13 

To quantify developer costs, we began by asking for the company’s 

total pre-release testing costs and post-release (after-sales) service 

costs. We asked them to break the pre-release testing costs into 

total labor costs, software expenditures, hardware expenditures, and 

external testing services. 

The remaining developer metrics in Table 5-4 address the 

incremental impact of an inadequate software infrastructure. The 

information represented in Tables 5-2 and 5-3 first was developed 

for current development practices, referred to as the baseline 

scenario, and second for the counterfactual scenario of improved 

testing capabilities. During the case studies, we asked developers to 

focus on changes in labor costs captured by the relative cost factors 

when filling out the cost tables. We anticipated that labor costs 

account for most of the impact of an inadequate software testing 

infrastructure on software developers. However, we also asked 

developers to estimate the impact of improved testing capabilities 

on hardware and software expenditures. 

Finally we asked developers about the impact of market delay and 

reputation on revenues. As shown in the economic welfare 

equations in Section 3.6, these developer revenues do not directly 

enter into the calculation of economic impacts because they 

represent transfer payments between consumers and producers. 

However, the delay in introducing new products indirectly creates 

economic impacts by delaying the benefits realized by users from 

adopting new software products. Thus, in this light, developers 

delaying product introduction and users delaying adoption have a 

similar impact. 

 5.4 ESTIMATING SOFTWARE USER COSTS 

Inadequate software testing affects users through the uncertainty and 

number of bugs remaining in software that is released. Users are at 

the end of the supply chain and are the source of benefits and costs 

realized from software quality. For example, if there is a software 

failure that prevents a transaction from occurring or delays the 

release of a new product, these costs originate with the users. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-14 

User costs associated with software errors begin with the software 

purchase decision. Users evaluate the set of potential software 

products that are available to them and compare price and quality. 

This search process is costly and requires time because users do not 

have complete information about the quality of all of the software 

products that they could purchase. This lack of an ability to 

compare across products based on price and quality is magnified by 

an inadequate software testing infrastructure because uncertainty 

about bugs and interoperability increases. As a result, users must 

spend additional time and resources to determine which product to 

buy and in some instances may delay purchasing new software 

products until more information about software quality is revealed 

by early adopters. Delays in adoption reduce the benefits from the 

new software and in turn lead to reductions in economic welfare. 

Once users have decided to purchase a product, they must install 

and incorporate it into their business operations. If the product is a 

custom product, implementation can be potentially costly and may 

involve significant effort by both users and developers. Custom 

products must frequently be integrated with legacy systems, and 

errors leading to interoperability problems may exist in both the 

new software and the legacy software. Bugs encountered while 

implementing a custom product can lead to delays in bringing the 

system on line and the need for special patches and interface 

programs. The potential for excess costs due to an inadequate 

software testing infrastructure may be great at this point. To a lesser 

extent, these problems also potentially exist when implementing 

commercial software products. However, typically implementation 

problems such errors leading to improper or incomplete installation 

are minimal with commercial software. 

The final stage of the process for users occurs after the product has 

been implemented and business operations begin. At this point, 

additional bugs that cause the system to fail may emerge that were 

not captured during development and implementation. Costs 

associated with bugs in this stage can be catastrophic and include 

loss of production data and customer information, lost sales, 

production delays, and lost reputation and market share. 

The general costs categories for software users are described below: 

This search process 

is costly and 

requires time 

because users do 

not have complete 

information about 

the quality of all of 

the software 

products that they 

could purchase. 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-15 

Z labor costs—additional employee and contract labor (thirdparty integrators) expenditures for testing, installation, and 

repair of new software due to an inadequate infrastructure 

for testing the software before it is purchased; 

Z failure costs—costs associated with catastrophic failure of 

software products; 

Z performance cost—impact on users’ operating costs when 

software does not perform as expected. These include the 

cost of “work arounds” and loss of productivity when 

purchased software does not perform as anticipated; 

Z redundant systems—additional hardware or software 

systems that users maintain to support operations and back 

up data in case of a software failure attributable to an 

inadequate infrastructure for software testing; 

Z delayed profits—discounted value of time delays in 

production and transactions attributable to an inadequate 

software product; and 

Z sales forfeited—discounted value of foregone transactions 

due to an inadequate software product. 

Redundant systems resulting from inadequate software testing 

represent a significant, but less publicized, economic impact. 

Companies commonly maintain parallel systems for up to a year or 

more as a security measure against catastrophic failures. If an 

improved software testing infrastructure could reduce the 

probability and severity of bugs remaining in products after 

purchase, the time window for redundant systems could be greatly 

reduced. 

The number of bugs still remaining in software products with an 

improved software testing infrastructure is a key assumption that 

must be clearly addressed in the counterfactual scenario and related 

data collection efforts. Because assuming that all bugs can be 

removed is not realistic, users were asked how different cost 

categories will be affected by a partial reduction in bugs (say a 75 

percent reduction). Our approach to quantifying the impact of 

removing most but not all of the bugs users encounter is to 

Z estimate the total cost of bugs to users and 

Z determine which costs are linearly related to the number of 

bugs encountered and which costs are nonlinearly related. 

Table 5-5 summarizes cost categories and metrics for measuring the 

total costs bugs impose on users. We began our user surveys by 

asking respondents to estimate the total cost of bugs in each 

category. It is simpler for software users to provide information on 

Companies 

commonly maintain 

parallel systems for 

up to a year or more 

as a security 

measure against 

catastrophic failures. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-16 

Table 5-5. Cost Metrics for Users 

Cost Category Specific Cost Technical Matrix Economic Matrix 

Pre-purchase Costs 

Purchase decision 

costs 

Labor costs Additional effort spent searching 

for a new CAD/CAM/CAE and 

PDM software product 

Labor costs of 

employees 

Delayed installation 

costs 

Delay 

associated with 

search 

Additional time spent searching 

for a new CAD/CAM/CAE and 

PDM software product. 

Delayed benefits from 

adoption of new 

software products 

 Delayed 

adoption due to 

uncertainty 

Delayed adoption time associated 

with uncertainty over quality of 

CAD/CAM/ CAE and PDM 

software 

Delayed benefits from 

adoption of new 

software products 

Post-purchase costs 

Installation costs Labor costs User labor hours required for 

installation and testing 

Fully loaded wage rate 

times number of Labor 

hours 

 Labor costs Consultant labor hours required 

for installation and testing 

Fully loaded wage rate 

times number of Labor 

hours 

 Delay costs Delays due to new software 

causes old software to fail, or old 

software prevents new software 

from working 

Lost benefits associated 

with new software 

product 

Product failure costs Delayed profits Time required to reenter lost data Time delay attributable 

to reentering data 

 Repair costs Labor time of employees and 

consultants reentering lost data or 

repair data archives 

Labor costs of 

employees and 

consultants 

 Replacement 

costs 

Early retirement or “scrapping” of 

ineffective systems 

Expenditures on new/ 

replacement systems 

 Lost sales Company downtime attributable 

to software failure 

Lost profit from foregone 

transactions during this 

time period 

 Reputation 

costs 

Future impact on market share Expenditures on outside 

consultants 

Suboptimal 

performance 

Inability to fully 

accomplish 

tasks 

Resources expended for patches 

and work arounds—may be onetime cost or ongoing activity 

Increased labor and 

hardware expenditures 

that would be needed to 

accomplish the same 

task 

Redundant systems Hardware costs Multiple hardware systems 

maintained in case of system 

failure 

Expenditures on 

hardware systems 

 Software costs Maintaining old or redundant 

software system after shift to new 

software system 

Maintenance and labor 

expenditures on old 

software 

 Labor costs Labor time of employees 

maintaining a redundant system 

Labor costs of 

maintaining old system 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-17 

their total costs associated with bugs as opposed to marginal 

changes in costs associated with an incremental decrease in bugs. 

Users were then asked to assess general trends in how the total costs 

they provide would change as the number of bugs is reduced. For 

example, how would each cost category change if bugs were cut in 

half or reduced by 75 percent? For product failure or installation, 

the cost of bugs may be linearly related to the number of bugs (i.e., 

if product failures are reduced by 75 percent, then repair and lost 

sales would be reduced by 75 percent). However, for other cost 

categories, such as redundant system costs, a 75 percent reduction 

in the probability of bugs may not significantly reduce the need for 

backup systems. 

Figure 5-6 illustrates the relationship between user costs and the 

percentage reduction in bugs. The case studies investigate the 

shape of these curves for each cost category listed in Table 5-5. 

These relationships are useful for conducting sensitivity tests. The 

relationships in Figure 5-6 also allow us to estimate the upper and 

lower bounds for economic impacts associated with ranges, such as 

50 to 100 percent, of reductions in bugs. 

In addition, as described in the Section 5.6, the total costs and the 

relationship between total costs and the percentage reduction in 

bugs will be different for different sectors of the economy. A 

separate set of curves were developed for each of the two case 

studies in Sections 6 and 7. 

 5.5 PERIOD OF ANALYSIS 

Two conventions are available for developing the costs of an 

inadequate infrastructure for software testing. They are to express 

them either for 

Z a specific historical period (e.g., 2000) or 

Z a specific product or set of products. 

The empirical analysis that follows uses the first approach. The 

advantage of the historical period is that it directly provides the cost 

information in dollars per year where it can be readily compared to 

other annual flows. With this approach, impacts can be expressed 

as annual spending on software testing. The drawback with this 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-18 

Figure 5-6. Relationship between Users Costs and Percentage Reduction in Bugs 

Percent

Reduction

in Total

Costs

100%

Current

Total

Costs

(100%)

Failure Costs

Redundant

System Costs

Percent

Reduction in

Bugs

approach is that, for any set of developers, the period for which the 

information is collected may be unrepresentative of the costs for a 

typical year. For example, simply by historical accident, one may 

collect data for a year during which new projects were atypically 

few or frequent. 

By developing estimates of the testing costs for a product, we can be 

sure that the costs are comprehensive, not subject to a sampling 

convention. With this approach, one would be able to say 

something like “the testing cost of a typical software development 

project is about $y.” However, we would have no indication of 

how often that cost is incurred. All products would have to be 

enumerated, both commercial and in-house, and putting them on 

an equal footing to calculate an annual cost estimate would be 

difficult. Further, this approach requires greater recall by 

respondents than the first approach. For these reasons, we selected 

the first approach and collected testing resource usage and cost data 

from developers for 2000. 
Section 5 — Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing 

5-19 

 5.6 INDUSTRY-SPECIFIC USER COSTS 

Different industries experience different types of costs from an 

inadequate infrastructure for software testing. The individual 

industry studies that follow in Sections 6 and 7 describe how user 

costs differ between CAD/CAM/CAE users in the transportation 

equipment manufacturing sector and FEDI/clearinghouse software 

users in the financial services sector.

The transportation equipment manufacturing and financial services 

sectors differ in several important ways. The most important 

difference may be in the timing of business-to-business (B2B) 

interactions. The design of transportation equipment is generally a 

batch process where different subunits of the machine are designed 

and then assembled. On the other hand, the financial services 

sector relies on real-time processing to reconcile transactions 

between two entities. 

A second major difference between the two industries is in the 

nature of their B2B relationships. The transportation equipment 

manufacturing industry has traditionally interacted with a welldefined set of customers; buyer–supplier relationships are well 

established and frequently characterized by long-term business 

agreements. Knowledge of the users’ customers and repeat business 

may be used to mitigate some software shortcomings. In contrast, in 

the financial services sector, transactions can occur with anyone at 

any point in time. This creates a different set of needs and potential 

impacts within the financial services sector. However, it should be 

noted that the production process in the transportation equipment 

manufacturing sector is becoming more similar to the financial 

services sector as concurrent engineering and B2B commerce 

networks are established. 

The different roles software plays in the business operations of these 

two industry sectors lead to different impacts associated with an 

inadequate infrastructure for software testing. Based on the ISO 

standards’ quality categories presented in Section 1, Table 5-6 

indicates the quality issues associated with using software in the two 

industries. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

5-20 

Table 5-6. Importance of Quality Attributes in the Transportation Equipment and Financial 

Services Industries 

Quality Category Main Issues 

Transportation 

Equipment Financial Services 

Functionality Attributes of software that focus on the 

set of functions, the results of those 

functions, including security, timeliness, 

and adherence to common standards 

Less important 

because of fewer 

outside interactions 

More important 

because of security, 

timeliness, and 

interaction issues 

Reliability Attributes of software that bear on the 

frequency of failure by faults in the 

software, its specified level of 

performance, and its ability to recover 

lost data 

Important because 

of use in product 

design 

More important 

because of need to 

recover lost data if 

failure occurs 

Usability Attributes of software that bear on the 

users’ ability to understand, use, learn, 

and control the software 

More important 

because of 

manipulation of 

software to design 

product 

Less important 

because of minimal 

accounting 

knowledge required 

to engage in a 

transaction 

Efficiency Attributes of software that bear on 

response and processing times of the 

software 

Less important 

because of batch 

processing 

Very important 

because of real time 

processing 

Maintainability Attributes of software that bear on the 

effort needed for diagnosing failures, 

removing failures, updating the software, 

and validating changes to the software 

More important as 

errors become more 

costly to repair the 

longer they stay in 

the production 

process 

More important as 

errors become more 

costly to discover 

the longer they stay 

in the production 

process 

Portability Attributes of software that bear on the 

opportunity for its adaptation to different 

environments, ease of installation, and 

interaction with other software 

Less important 

because of 

commonly agreed 

upon 

interoperability 

standards (STEP) 

Very important 

because of potential 

interactions with 

numerous types of 

users 
6-1 

 Transportation 

 Manufacturing 6 Sector 

This section investigates the excess costs incurred by software 

developers and users in the transportation equipment manufacturing 

sector due to an inadequate infrastructure for software testing. The 

impact estimates are based on interviews with developers and users 

of CAD/CAM/CAE and PDM software. 

Impact estimates were developed relative to two counterfactual 

scenarios. The first scenario investigates the cost reductions if all 

bugs and errors could be found in the same development stage in 

which they are introduced. This is referred to as the cost of an 

inadequate software testing infrastructure. The second scenario 

investigates the cost reductions associated with finding an increased 

percentage (but not 100 percent) of bugs and errors closer to the 

development stages where they are introduced. The second 

scenario is referred to as a cost reduction from feasible infrastructure 

improvements. 

Table 6-1 presents an overview of the economic impact estimates 

for the development and use of CAD/CAM/CAE and PDM software 

in the U.S. automotive and aerospace industries. The total impact 

on these transportation equipment manufacturing sectors from an 

inadequate software testing infrastructure is estimated to be $1.8 

billion. The potential cost reduction from feasible infrastructure 

improvement is $0.6 billion. Developers of CAD/CAM/CAE and 

PDM software account for approximately 25 percent of the total 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-2 

Table 6-1. Cost Impacts on U.S. Software Developers and Users in the Transportation 

Manufacturing Sector Due to an Inadequate Testing Infrastructure ($ millions) 

The Cost of Inadequate Software 

Testing Infrastructure 

Potential Cost Reduction from 

Feasible Infrastructure Improvements 

Software Developers 

CAD/CAM/CAE and PDM $373.1 $157.7 

Software Users 

Automotive $1,229.7 $377.0 

Aerospace $237.4 $54.5 

Total $1,840.2 $589.2 

impact. Users account for the remaining share: the automotive 

industry accounts for about 65 percent and the aerospace industry 

accounts for about 10 percent. 

This section begins with an overview of the use of CAD/CAM/CAE 

and PDM software in the transportation manufacturing sector. A 

more detailed industry profile of CAD/CAM/CAE/PDM software 

developers and users is provided in Appendix B. We then describe 

the analysis approach and survey findings used to estimate the 

economic impacts of an inadequate infrastructure for software 

developers and software users in the automotive and aerospace 

industries in Sections 6.2 and 6.3. 

 6.1 OVERVIEW OF CAD/CAM/CAE AND PDM 

SOFTWARE IN THE TRANSPORTATION 

MANUFACTURING SECTOR 

Transportation equipment manufacturing consists of the production 

of products used for road, rail, water, and air transportation. It is 

one of the largest sectors in the economy, with total sales of over 

$639 billion in 2000 and employment of more than 1.8 million 

people (U.S. Department of Commerce, 2002). 

Software use within the transportation sector has steadily increased 

in recent years. It has now reached the point where transportation 

equipment is designed and production is managed almost 

exclusively with computers. 
Section 6 — Transportation Manufacturing Sector 

6-3 

This section provides a framework for understanding the interactions 

between CAD/CAM/CAE and PDM software developers and users in 

the transportation equipment manufacturing sector. The 

interrelationship of these sectors is shown in Figure 6-1. 

Figure 6-1. Economic Relationship Among CAD/CAM/CAE Producers and Consumers 

Several information technology and service industries provide CAD/CAM/CAE software and services to manufacturers. 

Software

Publishing

(511210)

Computer

Systems Design &

Related Services

(5415)

Testing

Services Modified &

Tested CAD/

CAM/CAE

Software

Transportation

Equipment

Manufacturing (336)

Manufactured

Products

CAD/CAM/CAE

Software

 6.1.1 Use of CAD/CAM/CAE and PDM Software 

The development and manufacturing of transportation equipment, 

like all products, goes through a product development cycle. 

Products move from a planning phase through design and 

engineering phases and end with the manufacturing and production 

phase. Figure 6-2 illustrates both the production process and points 

at which CAD/CAE/CAM and PDM are used. 

Engineers use two key types of software tools: “point tools” and 

“life-cycle” tools. CAD, CAE, and CAM are point tools because 

they are applied to one part of the production process. PDM is a 

life-cycle tool used to manage the flow of information throughout 

the product development cycle and the manufacturing organization. 

CAD, CAM, and CAE refer to functions that a computer and 

peripheral equipment may perform for a user with the aid of 

application software. 

CAD software functions enable users to design products and 

structures with the aid of computer hardware and peripherals more 

efficiently than with traditional drafting technologies. The user 

CAD, CAM, and 

CAE refer to 

functions that a 

computer and 

peripheral 

equipment may 

perform for a user 

with the aid of 

application 

software. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-4 

Product Planning

Product Development Cycle Software Uses

Product Design

Product Engineering

Production

CAD

CAE

CAM

PDM

creates a computer image of a two-dimensional or threedimensional design using a light pen, mouse, or tablet connected to 

a workstation or personal computer. The design can be easily 

modified. It can be viewed on a high-quality graphics monitor from 

any angle and at various levels of detail, allowing the user to readily 

explore its physical features. Designers can use CAD software to 

integrate drawings in such a way that adjusting one component 

alters every attached component as necessary. 

CAM software functions allow a manufacturer to automate 

production processes. CAM software includes programs that create 

instructions for manufacturing equipment that produces the product. 

In addition, the software provides instructions to other computers 

performing real-time control of processes, in using robots to 

assemble products, and in providing materials requirements 

associated with a product design (P.C. Webopaedia, 1996). 

CAE software functions allow users to conduct engineering analyses 

of designs produced using CAD applications to determine whether a 

product will function as desired. The engineering analysis may 

involve simulating the eventual operating conditions and 

performance of a designed product or structure. Or users can 

analyze the relationships between components of a product system. 

Figure 6-2. 

CAD/CAE/CAM and PDM 

in the Product 

Development Cycle 
Section 6 — Transportation Manufacturing Sector 

6-5 

PDM software supports concurrent engineering by managing all of 

the product-related information generated throughout the product 

life-cycle. PDM creates a master document that can be logged out 

and held in a secure location. Other engineers working on the 

project can access a duplicate copy that they can use in their work. 

Whenever changes are made to the master copy, all users are 

notified and the copy that they are using is updated to reflect any 

changes. PDM tools focus on automating existing processes and 

managing electronic documentation, files, and images. PDM is 

used almost exclusively in CAD/CAM/CAE systems. 

 6.1.2 Development of CAD/CAM/CAE and PDM Software 

The CAD/CAM/CAE and PDM software industry that supplies the 

transportation sector is a complex and changing landscape of 

information technology products, publishers, designers, consultants, 

and product users. Underlying this industry is a set of production 

relationships characterized by substantial resource requirements for 

product development and relatively few resources to reproduce and 

distribute the product. 

The total CAD/CAM/CAE industry comprises a small set of 

publishers who sold an estimated $9.3 billion worth of software 

products in 1999 and a very large number of potential users 

(Daratech, Inc., 1999). The industry also consists of a number of 

firms that make modifications to the basic CAD/CAM/CAE software 

products, tailoring them to specific applications; firms that provide 

design and related services; and consulting firms that primarily assist 

users in selecting and installing the software. The PDM industry is 

smaller than the CAD/CAM/CAE industry, with total sales estimated 

at $1.76 billion in 1999, but it is expected to grow rapidly with total 

sales expected to reach $4.4 billion by 2004 (CIMdata, 2000). 

The CAD/CAM/CAE and PDM software industries are built around 

the software’s capability to store, search, retrieve, copy, filter, 

manipulate, view, transmit, and receive digital representations of 

product design and operation information. Digitized information is 

anything that can be digitized (encoded as a stream of bits). 

Information products such as CAD/CAM/CAE software are defined 

by unique characteristics: 

Z a lack of tangible attributes, 

Z association with multiple forms of presentation, 

PDM software 

supports concurrent 

engineering by 

managing all of the 

product-related 

information 

generated 

throughout the 

product life-cycle. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-6 

Z the possibility of delivering the product with no direct 

contact between the supplier and consumer, 

Z protection by copyright laws, and 

Z the ease of adding value to the product (Executive Office of 

the President, 1998). 

The difficulty of potential users determining the precise 

characteristics of software a priori makes it an experience good: its 

characteristics must be learned through use; they cannot be 

determined by simple observation. This characteristic introduces a 

source of uncertainty in the purchase decision. 

Software is also an investment good. It is used by manufacturers 

over a period of time, usually years, and has several features 

common to all investment goods (Dixit and Pindyck, 1994): 

Z Irreversibility: The up-front costs of product purchase, 

evaluation, installation, testing, and worker training required 

to use the product are, once incurred, sunk costs that are 

unretrievable if the consumer changes her mind regarding 

the product’s utility. Furthermore, once users create designs 

using the new software, the designs generally do not 

translate easily into other design formats, which makes 

switching to a different software package additionally costly. 

Z Uncertainty: The future market demand for the 

manufacturer that will use the software product is unknown 

to the consumer. In addition, there is uncertainty over 

interest rates and the quality of the software product that the 

manufacturer purchases. Prior to purchasing and using the 

software, the consumer will have priors on the capability, 

usability, performance, reliability, installability, 

maintainability, documentation, and availability of the 

product but not until it is used will she be able to determine 

the accuracy of those priors. 

Z Postponability: There is leeway in the timing of most 

investment opportunities. Investors can delay their purchase 

of the software to gather additional information on the 

market conditions and characteristics but at the cost of 

foregoing the product’s expected benefits. 

 6.2 SOFTWARE DEVELOPER COSTS IN THE 

TRANSPORTATION MANUFACTURING 

SECTOR 

To investigate software testing costs, we conducted interviews with 

10 developers of CAD/CAM/CAE and PDM software products. 

Companies were typically forthcoming in their discussions of 

inadequate software tools and methods. All agreed that improved 
Section 6 — Transportation Manufacturing Sector 

6-7 

infrastructure could reduce testing costs and accelerate the time to 

market for their products. 

However, not all companies completed the entire survey that was 

used to collect information to quantify the costs of an inadequate 

software testing infrastructure for CAD/CAM/CAE/PDM developers. 

In several instances vendors said that information on testing 

expenditures and errors discovered was confidential because they 

reflected detailed information about their product development 

process. But the most common reason for firms not providing data 

was the simple fact that they did not track these metrics and the 

data were not available.1 

Several companies agreed that tracking metrics targeted in the 

survey instrument, such as the types of bugs found, in what stage of 

development they were introduced, and where they were found, 

would be very useful for developing better testing methods and 

guidelines. One software tester said that statistics on where errors 

are introduced and where they are found is “exactly the type of 

information they need to improve testing efficiency.” However, 

typically time and resource constraints prevented them from 

tracking this information. Companies indicated that in the current 

environment, software testing is still more of an art than a science, 

and testing methods and resource are allocated based on the expert 

judgment of senior staff. 

Error-tracking procedures and the resulting resource estimates 

would be particularly useful in the initial product development 

planning stages. Firms indicated that a lack of detailed timelines 

based on accurate estimates of testing needs frequently leads to 

limited resources in the early stages of development, resulting in 

errors propagating through the R&D process and not found until the 

later stages of commercialization. Respondents agreed that finding 

the errors early in the development process greatly lowered the 

average cost of bugs and errors. Most also indicated that the lack of 

historic tracking data and inadequate tools and testing methods, 

such as standard protocols approved by management, available test 

cases, and conformance specification, limited their ability to obtain 

 

1In the absence of actual data on errors in the software development process, 

vendors were asked to estimate the distributions of where errors were found and 

introduced. However, in almost all instances respondents were uncomfortable 

speculating about their error distributions and declined to do so. 

Companies 

indicated that in the 

current 

environment, 

software testing is 

still more of an art 

than a science, and 

testing methods and 

resource are 

allocated based on 

the expert judgment 

of senior staff. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-8 

sufficient testing resources (from management) and to leverage these 

resources efficiency. 

The remainder of this subsection quantifies the cost savings due to 

finding bugs and errors closer to when they are introduced based on 

four completed interviews with three CAD/CAD/CAE/PDM vendors. 

We used the empirical results from the developer surveys to 

quantify the economic impacts for the counterfactual scenarios 

described below. 

 6.2.1 Estimation Approach 

To estimate the costs associated with an inadequate infrastructure, 

we made two key assumptions/clarifications to make the analysis 

tractable: 

Z The same number of bugs still occurs regardless of the 

infrastructure used or the quality of that infrastructure (i.e., 

bugs are attributed to human error and will continue to 

occur). 

Z An improved infrastructure does not change where bugs are 

introduced because this again is assumed to be a function of 

human error. 

With these assumptions in mind, the primary impact of an improved 

infrastructure is to lower the cost of testing and fixing bugs and 

errors and find the bugs closer to the time they were introduced. 

Developers were asked questions to support the evaluation of two 

counterfactual scenarios for which economic impacts are estimated. 

The first scenario estimates the cost savings developers would 

realize if all bugs and error were found in the same development 

stage that they were introduced. This is referred to as the cost of an 

inadequate infrastructure for software testing. In addition to finding 

all errors sooner, this scenario includes the impact an improved 

software testing infrastructure has on lowering the costs of finding 

and repairing bugs and errors that are introduced and found in the 

same stage. 

The second scenario reflects that it may not be possible to develop a 

testing infrastructure that would support “perfect” software testing 

and that some errors are still likely be found in later development 

stages. This is referred to as an “feasible” infrastructure for software 

testing. To define this scenario, we asked software testers how the 

distribution of where errors are found as a function of where errors 
Section 6 — Transportation Manufacturing Sector 

6-9 

are introduced would change with enhanced testing tools and 

methods. The costs are then treated as a function of the time it 

takes to find and fix them and how much sooner the bugs that are 

introduced are found. 

 6.2.2 Survey Findings 

The software developer survey instrument is presented in 

Appendix C. We contacted developers by telephone and asked 

them to complete the questionnaire as part of an informal interview. 

Four developers of CAD/CAM/CAE and PDM software products 

completed substantial portions of the entire survey. The remaining 

six developers returned partially completed surveys due to the 

confidentiality and lack of data tracking systems discussed above. 

As part of the survey, developers were asked to estimate the current 

distribution of bugs (where they are introduced and where they are 

found), the time required to fix a bug given the stage where it was 

found, and the stage where it was introduced. In the final sections 

of the survey developers were then asked their expectations of how 

an improved infrastructure would affect these distributions and 

costs. 

Table 6-2 presents the first key pieces of information needed to 

calculate the impact estimates of an inadequate infrastructure for 

software testing. The table shows the distribution of where software 

bugs are found and their introduction point. For example, 

40 percent of bugs are found in the coding/unit testing stage. Of the 

bugs found in this stage, one-fifth (8 percent of 40) were introduced 

in the requirements stage and the other four-fifths (32 percent of 40) 

were introduced in the coding/unit testing stage. 

As shown in Table 6-2, over 80 percent of errors are introduced in 

the coding/unit testing stage, but well over half of these errors are 

not found until downstream in the development process.2

 

2Note that we are investigating only bugs and errors introduced in the software 

product development process. Errors introduced during beta testing or 

implementation are not included in the distributions in Table 6-2. However, 

developers said that it is often difficult for the testers and software engineers to 

determine where the bug was introduced by the user or as part of the 

development process. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-10 

Table 6-2. Distribution of Bugs Found Based on Introduction Point 

The diagonal elements in bold represent the occurrences where software errors are found in the same development stage 

where they are introduced. Occurrences to the right of the bold diagonal indicate errors found “downstream” in the 

product development process. 

Stage Found 

Stage Introduced Requirements 

Coding/Unit 

Testing Integration 

Beta 

Testing 

Post-product 

Release 

Row 

Percentage 

Requirements 5.0% 8.0% 2.3% 0.2% 0.2% 15.6% 

Coding/unit testing NA 32.0% 40.5% 4.5% 4.5% 81.5% 

Integration NA NA 2.3% 0.4% 0.4% 3.0% 

Column 

percentage 

5.0% 40.0% 45.0% 5.0% 5.0% 100.0% 

NA = Not applicable because a bug cannot be found before it is introduced. 

Once the distribution of bugs is determined, the next step is to 

determine the costs of fixing a bug based on the point of 

introduction. As discussed above, the costs of fixing a bug are 

greater the farther away from the point of introduction that the bug 

is found. This occurs for several reasons. First, it is more difficult to 

find a bug the farther away from the point of introduction. Second, 

more code has to be rewritten the farther away from the point of 

introduction that the bug is found. 

Table 6-3 shows resources (costs) in terms of the average number of 

tester hours required to investigate and fix a bug based on the 

survey responses. 

Table 6-3. Hours to Fix Bug Based on Introduction Point 

For errors introduced in the coding/unit testing stage, respondents indicated that it was twice as costly to fix the error if it 

was not found until the integration phase and five times as costly if it was not detected until post-product release. 

Stage Found 

Stage Introduced Requirements 

Coding/Unit 

Testing Integration Beta Testing 

Post-product 

Release 

Requirements 2 4 6 8 10 

Coding/unit testing NA 2 4 6 10 

Integration NA NA 4 8 16 

NA = Not applicable because a bug cannot be found before it is introduced. 
Section 6 — Transportation Manufacturing Sector 

6-11 

Using the distribution of bugs (introduced and found) in Table 6-2 

and the hours to fixed each type of bug in Table 6-3, we calculated 

the average hours per bug as a function of where the bug was found 

(see Table 6-4). For example, on average a bug found in 

coding/unit testing takes 2.4 hours to fix, whereas an average bug 

found in post-product release takes 13.1 hours to fix. In addition, 

using the distribution of where bugs are found we calculated that 

the weighted average time to investigate and fix a bug is 3.9 hours. 

The average is relatively small because 85 percent of the errors are 

found during the coding and integration stages of development, and 

relatively few are found in beta testing and post-product release. 

Table 6-4. Time to Fix a Bug Based on Discovery Point 

Respondents indicated that 45 percent of errors are found in the integration stage of development and it takes an average 

of 4.1 hours to correct the errors found in this stage of development. 

Location Hours 

Distribution of Where 

Bugs are Founda Weighted Average Hours 

Requirements 2.0 5% 

Coding/unit testing 2.4 40% 

Integration 4.1 45% 

Beta testing 6.2 5% 

Post-product release 13.1 5% 

Total 3.9 

aFrom bottom row in Table 6-2. 

Based on the cost-per-bug calculations presented above, we 

estimated the national costs of an inadequate infrastructure for 

software testing for each of the two counterfactual scenarios 

described in Section 6.2.1. For the first testing scenario, all bugs are 

found in the stage where they are introduced. For the “feasible” 

scenario, more bugs are found closer to the stage they were 

introduced because of improved testing methods and tools. The 

distributions of where bugs are found associated with each 

counterfactual scenario are shown in Table 6-5, along with the 

current distribution copied from Table 6-4. 

The current distribution reflects where bugs are discovered under 

the existing inadequate infrastructure for software testing. The 

second column shows the distribution if all bugs are discovered in 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-12 

Table 6-5. Distribution of Bugs Based on Infrastructure 

Finding errors earlier leads to a decrease in the total cost of finding and fixing errors. 

Location 

Current 

Infrastructure 

All Bugs Found in Same 

Stage Introduced 

Feasible Infrastructure 

Improvements 

Requirements 5% 15.6% 5% 

Coding/unit testing 40% 81.5% 60% 

Integration 45% 3.0% 30% 

Beta testing 5% 0 3% 

Post-product release 5% 0 2% 

Average hours per average bug 3.9 2.4 3.2 

Percentage reduction from current 

infrastructure 

 38.3% 16.9% 

the development stage where they occur. Note that this distribution 

is simply the row percentage shown in Table 6-2. The “feasible” 

infrastructure is based on survey data. Respondents were asked 

what the distribution of the discovery of bugs would look like with 

better tools. Under this scenario, some of the bugs are found sooner 

in the production process. 

As shown in Table 6-5 both testing scenarios shift the distribution of 

when bugs are found toward the early stages of development. The 

next to last row of Table 6-5 gives the weighted average number of 

hours required to find and fix an average bug under each scenario. 

This average was calculated by multiplying the distribution of bug 

discovery by the average number of hours spent finding and fixing a 

bug, as presented in Table 6-4. 

The final row gives the percentage change in total time spent per 

bug for each of the scenarios relative to the baseline scenario. This 

can be interpreted as the percentage of testing resources saved as a 

result of an improved infrastructure for software testing. 

The percentage reduction in testing resources presented in 

Table 6-5 results from shifting the distribution of when bugs are 

found forward. Software developers were also asked if feasible 

infrastructure improvements would decrease the time spent 

correcting the error (hours presented in Table 6-4). Most thought 

that the hours per bug would decrease; however, they were not able 
Section 6 — Transportation Manufacturing Sector 

6-13 

to quantify this impact. As a result, this potential cost savings is not 

included in the following developer impact estimates. 

 6.2.3 Cost Impacts Per Employee for Software Developers 

Once the average percentage change in testing resources was 

determined, we normalized cost impacts by company employee to 

develop a cost-per-employee metric associated with an inadequate 

infrastructure. We then used the cost per employee, in conjunction 

with total industry employment, to estimate the total cost impact on 

CAD/CAM/CAE and PDM software developers. 

A breakdown of testing costs based on information collected during 

developer surveys is presented in Table 6-6. The second column 

provides current labor and capital expenses for software testing for a 

typical company of 10,000 employees. The third and fourth 

columns show the cost associated with an inadequate infrastructure 

and potential cost reductions associated with feasible 

improvements. For a typical company of 10,000 employees the 

annual change in testing costs ranged from $9.3 to $21.1 million. 

Table 6-6. Developer Testing Costs for a Typical Company of 10,000 Employees 

Current 

Infrastructure 

Testing Costs 

The Cost of 

Inadequate 

Software Testing 

Infrastructure 

Potential Cost 

Reduction from 

Feasible Infrastructure 

Improvements 

Software testers $54,512,640 $20,884,777 $9,190,119 

Number of testers 400 153 67 

Fully loaded wage rate ($/hour) $67.60 $67.60 $67.60 

Hardware for testing $40,000 $15,325 $6,743 

External testing services $100,000 $38,312 $16,859 

After-sale service costs $545,126 $208,848 $91,901 

Total annual testing costs $55,198,234 

Annual change in testing costs $21,147,440 $9,305,701 

Percentage reduction from current 

infrastructure 

 38.3% 16.9% 

Cost savings as a percentage of sales 1.8% 0.8% 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-14 

Labor costs for software testers account for the overwhelming 

majority of total testing expenditures. We calculated labor costs for 

software testers using company employment (10,000), the average 

ratio of testers to total employees (4 percent), and the average fully 

loaded wage rate for software testers ($68 per hour). To this, 

external testing services, hardware costs, and after-sale service costs 

were added to estimate the total testing costs. 

The cost associated with an inadequate infrastructure for software 

testing is approximately 1.8 percent of the developers’ annual sales 

and the feasible cost reductions are 0.8 percent. 

 6.2.4 Industry-Level Impact 

To extrapolate the cost impacts to reflect all developers of 

CAD/CAM/CAE and PDM software, we multiplied the cost per 

employee by the total employment of companies supplying software 

to the transportation manufacturing sector. Industry employment 

was estimated to be approximately 85,000 and is based on the 

employment information shown in Table A-3 (CAD/CAM/CAE 

developers) and Table A-4 (PDM developers).3 

National costs impacts for CAD/CAM/CAE/PDM developers due to 

an inadequate software testing infrastructure are $373.1 million (see 

Table 6-7). The potential cost reductions from feasible 

infrastructure improvements are $157.7 million. These estimates 

represent 6.0 percent and 2.5 percent of CAD/CAM/CAE/PDM 

software sales, respectively.4

Table 6-7. Annual Impact on U.S. Software Developers of CAD/CAM/CAE/PDM Software 

The Cost of Inadequate 

Software Testing Infrastructure 

Potential Cost Reduction from 

Feasible Infrastructure Improvements 

Change in cost per employment $4,390 $1,856 

Total industry employment 85,000 85,000 

Industry-level savings (millions) $373.1 $157.7 

 

3Employment for IBM and Oracle Corporation were not included in the PDM 

employment totals because the majority of their operations involve non-PDM 

products, and using their total employment would have incorrectly inflated the 

impact estimates. 

4Based on U.S sales of $6.2 billion in 1997 for CAD/CAM/CAE/PDM software (U.S. 

Department of Commerce, 1998). 
Section 6 — Transportation Manufacturing Sector 

6-15 

 6.3 END-USER COSTS IN THE TRANSPORTATION 

MANUFACTURING SECTOR 

RTI collected data directly from users of CAD/CAM/CAE and PDM 

software products to estimate the costs due to an inadequate 

infrastructure for software testing. We conducted telephone surveys 

of 182 firms in the automotive and aerospace industries. This 

subsection provides an overview of the survey process, descriptive 

statistics from data collected, and the economic impact estimates of 

software errors and bugs for users in the automotive and aerospace 

industries. 

 6.3.1 Survey Method 

For the end-user survey of automotive and aerospace manufacturing 

firms, we used a telephone-Internet-telephone method in which the 

respondents were recruited via telephone, instructed to complete an 

Internet survey, and telephoned again if clarification was needed or 

if the respondents did not complete the survey in a timely manner. 

The survey was pre-tested by two automotive companies. The 

electronic instruments and resulting database were housed on RTI’s 

web site within RTI’s firewall to ensure security and confidentiality 

of the information provided by respondents. 

The final survey instrument is presented in Appendix C. Harris 

Interactive recruited the users using scripts prepared by RTI. Up to 

eight calls were made to locate the appropriate individual at each 

company, recruit participants, and follow up if surveys were not 

completed within 2 weeks. 

The goal of the survey effort was to capture as large a share of the 

impacts as possible while ensuring that our survey population is 

representative of the industry as a whole. To this end, the total 

sampling points were segmented, by industry, into a census of the 

original equipment manufacturers (OEMs), a purposeful sample of 

the “largest” software users, and a random sample of “medium to 

small” size software users. The sample was divided as follows: 

two-thirds surveys for the automotive industry and one-third for the 

aerospace industry because of the larger number of firms in the 

automotive industry relative to the aerospace industry. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-16 

We used the dollar value of sales for each of the companies as the 

size metric and stratified the sample into three components for each 

industry: 

Z We selected the major OEMs from each sector to ensure 

representation of the largest firms in the sector. If a random 

sample had been used, possibly none of the OEMs would 

have been included in the analysis simply because of the 

research design. 

Z We used a purposeful survey of the 50 largest companies in 

automotive manufacturing and the 20 largest in aerospace. 

We instructed Harris Interactive to recruit as many of these 

large companies as possible to capture as many of the firsttier suppliers as possible. 

Z We then rounded out the survey with a random survey of 

approximately mid- to small-sized automotive institutions 

and mid- to small-sized aerospace institutions. This group 

provided a representative sample of all other suppliers in the 

industries. 

 6.3.2 Survey Response Rates and Industry Coverage 

RTI contacted 752 companies in the automotive industry and 224 

aerospace companies for a total of 976 contacts. Out of the 976 

companies contacted, appropriate contacts were identified at 644 

(68 percent) companies, and slightly over 50 percent of these 

contacts agreed to fill out the survey. From the recruited 

participants, 179 completed the surveys and returned them to RTI. 

Table 6-8 provides a full description of the number of firms 

contacted, the recruitment rates, and completion rates of the survey 

within each of the two industries. 

Table 6-9 shows the extent of industry coverage from the 179 

completed surveys based on domestic employment within the 

automotive and aerospace industries.5 The automotive industry 

includes manufacturers of motor vehicles (NAICS 3361), motor 

vehicle bodies and trailers (NAICS 3362), and motor vehicle parts 

(NAICS 3363). Based on these NAICS codes, the automotive sector 

consists of 8,385 firms with combined revenues of $420.6 billion. 

As Table 6-9 shows, the survey conducted by RTI captures slightly 

over 33 percent of the total domestic industry employment. 

 

5The ideal weighting mechanism would have been the number of engineers that 

use the CAD/CAM/CAE software in each industry. However, these data were 

not available, so total employment was chosen as the closest proxy. 
Section 6 — Transportation Manufacturing Sector 

6-17 

Table 6-8. Transportation Equipment Industry Survey Completion Rates

Sample Type 

Companies 

Contacted 

Identified 

Appropriate 

Contacts 

Successful 

Recruits 

(Recruitment 

Rate) 

Completed 

Surveys 

(Completion 

Rate per Recruit) 

Automotive 

OEMs 3 3 1 1 

Large institutions 131 108 76 72 

Small and medium institutions 618 378 201 74 

Aerospace 

OEMs 6 6 2 1 

Large institutions 48 36 19 17 

Small and medium institutions 170 116 68 14 

Total 976 644 367 179 

Table 6-9. Industry Coverage by Employment 

Sample Type 

Total Industry 

Employmenta

(thousands) 

Completed Surveys 

Employment 

(thousands) Percentage of Industry 

Automotive 

Small: less than 500b 473.9 16.0 3.4% 

Large: greater than 500 1,925.6 775.9 40.3% 

Total 2,399.47 791.9 33.0% 

Aerospace 

Small: less than 500b 66.7 3.8 5.7% 

Large: greater than 500 733.4 301.5 41.1% 

Total 800.1 305.3 38.2% 

aDomestic employment of automotive/aerospace design and manufacturing activities. 

bShare of employment at companies with fewer than 500 employees is based on Small Business Administration (SBA) 

census. 

The aerospace industry includes aerospace product and parts 

manufacturers (NAICS 3364). The population consists of 1,810 

firms with combined revenues of $25.2 billion. The survey captures 

slightly over 38 percent of total industry employment. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-18 

The total employment shown in Table 6-9 provides the nationallevel weights used to extrapolate the per-employee impact estimates 

provided in Section 6.3.4. 

 6.3.3 Survey Findings 

For the 179 survey respondents in the automotive and aerospace 

industry, companies averaged approximately 6,500 employees per 

firm with average sales of almost $1.4 billion. Not surprising, the 

mean was much higher than the median because of the skewing of 

the data by several large OEMs and first-tier suppliers. 

Table 6-10 lists the various software products that the survey 

respondents reported using for CAD/CAM/CAE or PDM activities. 

The most commonly reported software products were AutoCAD, 

CATIA, ProEngineers, Unagraphics, and IDEAS. The average life 

expectancy for these software products was 7 years, and the 

majority of them were installed between 1995 and 2001. 

Companies responded that they maintained an average of 67 

employees (full-time equivalents [FTEs]) involved in operating and 

supporting CAD/CAM/CAE systems and an average of 125 

employees supporting PDM systems. However, one of the largest 

companies indicated that it had 800 CAD/CAM/CAE staff and 3,000 

PDM staff members. These figures include only the engineers using 

the CAD/CAM/CAE and PDM software and do not include the 

information technology and software support staff who provide 

maintenance and upkeep. 

Incidence and Costs of Software Errors and Bugs 

Several respondents indicated that they conduct all of the job tasks 

using the software; hence, when a failure occurs, the potential 

ramifications are significant because an entire firm or division might 

have to shut down while the problem is remedied. 

Approximately 60 percent of the companies providing information 

on software errors and bugs indicated that they had experienced 

major software errors in the previous year. The remaining 

40 percent of the companies said they did not experience any major 

software errors over the past year and that minor errors were quickly 

corrected with little to no cost. 
Section 6 — Transportation Manufacturing Sector 

6-19 

Table 6-10. Reported Software Products 

Software Product Vendor/Provider Frequency 

7.0.7 1 

Abaqus/STD Hibbit, Karlsson & Sorensen, Inc. 3 

ACAD 2 

Advantage 1 

Alias Wavefront Studio 9.6 Alias Wavefront 1 

ANSYS ANSYS, Inc. 1 

Anvil Express Manufacturing and Consulting Services, Inc. 2 

AutoCAD Autodesk, Inc. 48 

Autodesk Inventor Autodesk, Inc. 1 

AutoManager Workflow Cyco Software 1 

CADDS5 PTC 2 

Cadkey Cadkey Corp. 7 

Cadra SofTech 1 

Cam 1 

CATIA Dassault Systemes 33 

CENTRA Centra Software 1 

Desktop 1 

Edge 1 

ESPRIT DP Technology Corp. 1 

HyperMesh Altair Engineering 1 

ICEM/Surf ICEM Technologies 1 

IDEAS SDRC 14 

Intralink DSQ Software, Ltd. 1 

Inventor Autodesk, Inc. 1 

IPD IPD Software Systems 1 

IronCAD IronCAD 2 

LS_DYNA Livermore Software Technology Corp. 1 

MARC MARC Analysis Research Co. 1 

Master Cam CNC Software, Inc. 4 

MathCAD Math Soft Engineering & Education, Inc. 1 

Matrix 3 

Mechanical Desktop (Autodesk) Autodesk, Inc. 6 

Mechanica PTC 1 

Medina Debis Systemhaus 1 

(continued) 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-20 

Table 6.10. Reported Software Products (continued) 

Software Product Vendor/Provider Frequency 

Metaphase SDRC 1 

MicroCADAM MicroCADAM, Inc. 2 

MicroStation Bentley Systems, Inc. 1 

One 3 

Optimation Mentum Group 1 

Orcad Cadence Design Systems, Inc. 1 

Parametric Technology PTC 1 

Patran/Nastran Noran Engineering, Inc. 4 

PDGS 4 

PRO ENGINEER PTC 29 

Pro-Intralink PTC 1 

SDRC SDRC 4 

Shop Data Systems 1 

SmarTeam SmarTeam Design Group 1 

Solid 1 

Solid Edge UGS 2 

SolidWorks UGS 7 

STAR-CD CD Adaptco Group 1 

SurfCAM Surfware, Inc. 1 

UGS UGS 24 

VeriBest VeriBest ISD 1 

VersaCad Archway Systems, Inc. 1 

Visual 1 

An unexpected finding was that approximately two-fifths of the 

companies reported no major software errors and that minor errors 

were quickly corrected with little to no cost. This finding could be 

a result of several factors. First, the companies truly did not 

encounter any software errors using CAD/CAM/CAE/PDM software. 

Second, the companies had software errors but did not recall them 

or the respondent was not aware of them. Third, the companies 

had errors but did not feel comfortable reviewing this information. 

Because of the potential underestimation of the true incidence of 

errors, the economic impacts provided below should be considered 

a conservative estimate of the total cost of software errors and bugs. 
Section 6 — Transportation Manufacturing Sector 

6-21 

For the respondents that did have errors, they reported an average of 

40 major and 70 minor software bugs per year in their 

CAD/CAM/CAE or PDM software systems (see Table 6-11). Most 

respondents indicated that the software problems they experienced 

in 2000 were typical of other years. 

Table 6-11. Incidence and Costs of Software Bugs 

Firms Experiencing Errors Firms Experiencing No Errors 

Impact Categories 

Percentage of 

Firms Reporting 

Errors 

Average of Firms 

Responding 

Percentage of Firms 

Reporting No Errors

Number of major errors 61% 39.7 39% 

Repair cost per bug (labor hrs) 268.4 

Lost data per bug ($) $604,900 

Delayed new service 

introduction (months) 

 1.5 

Number of minor errors 78% 70.2 22% 

Costs per bug $4,018,588 

Typical problems encountered due to bugs were 

Z production and shipment delays, 

Z system down time, 

Z loss of customer confidence, 

Z customer dissatisfaction in regards to timing, and 

Z lost clients. 

Most respondents reported that the software bugs only temporarily 

delayed transactions. Five companies indicated that they had lost 

reputations and two companies indicated that they lost market share 

as a result of a software error. Forty-two respondents said that they 

experienced delayed product or service introduction as the result of 

a software error. The remaining 20 respondents said that they had 

no market share or reputation loss. Thirteen firms reported an 

average loss of sales of $105,100 as a result of software errors. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-22 

Software Life-Cycle Costs 

Companies in the automotive and aerospace industries were asked 

about the life-cycle costs of CAD/CAM/CAE and PDM software. 

Table 6-12 summarizes the total costs of life-cycle activities, 

including software purchase decisions, installation and acceptance 

testing, annual maintenance, and redundant system costs. The last 

column in Table 6-12 indicates the percentage of these 

expenditures that is due to software errors and bugs. This 

percentage reflects the average cost savings that a typical firm 

would receive if the developer found all software bugs prior to 

release of the software product. This percentage reduction 

represents an upper bound of the benefits from an improved 

software testing infrastructure. 

Table 6-12. Average Company-Level Costsof Search, Installation, and Maintenance 

(Life-Cycle Costs) 

 Average Cost of Activities ($) 

Average Cost Reduction Associated 

with Software Errorsa

Purchase decision $511,907 41.7% 

Installation and acceptance $163,115 26.7% 

Maintenance $77,896 14.4% 

Redundant system costs $17,202.6 100% 

aReflects percentage of cost savings from eliminating all software bugs and errors. 

Purchase Decision 

On average, the companies indicated that they spend 4.9 months 

and 1,399 staff hours researching new CAD/CAM/CAE or PDM 

software packages before they make a purchase decision. This 

represents an expenditure of approximately $511,908. 

Fifty-eight percent of respondents said that they could reduce their 

search costs if they had better information about the quality of the 

software products. These respondents indicated they could reduce 

search time by approximately 1.5 months and 582 staff hours. This 

leads to an average savings of about $218,250 per company. 

Installation and Acceptance Testing. Companies on average spend 

about 564 in-house staff hours and $8,574 in external consulting 

services for installation and acceptance testing, representing about 

$63,115 per installation. The level of effort varied greatly, ranging 
Section 6 — Transportation Manufacturing Sector 

6-23 

from 1 to 10,000 hours of staff time. Respondents indicated that 

errors encountered during installation were responsible for about 

one-fourth of their costs. 

Annual Maintenance Costs. Maintenance expenditures on 

CAD/CAM/CAE or PDM software also varied greatly, ranging from 

$1,250 to $2,600,000 in annual expenditures. Most expenditures 

were for standard maintenance contracts with the provider of the 

software. 

Respondents said that maintenance expenditures could be reduced 

by about 14.4 percent if software errors and bugs were eliminated, 

reflecting an average cost savings of approximately $10,905 per 

year. 

Redundant System Costs. Approximately half of the companies 

indicated that they maintain redundant backup systems after the 

installation of new software. On average these systems were 

maintained for about 5.6 months at a cost of $3,972 per month. 

Thus, the elimination of bugs would represent a savings of about 

$17,203 per new system installed for the 50 percent of the 

population that maintains redundant systems. 

 6.3.4 Costs of Bugs and Errors Per Employee 

Table 6-13 shows the costs of bugs and errors normalized by 

company employment for the cost subcomponents discussed above. 

Cost-per-employee impacts were calculated individually for large 

and small automotive firms and large and small aerospace firms to 

allow for variation by size and industry.6

For automotive firms with more than 500 employees, the total cost 

of software bugs and errors is $241.1 per employee. Minor and 

major errors account for 84 percent of the costs. Additional 

installation costs associated with bugs accounted for most of the 

remaining impacts. 

 

6Because not all respondents were able to provide information for each cost 

subcomponent (e.g., major errors, minor errors, purchase costs), we calculated 

an average cost-to-transaction ratio individually for each subcomponent. The 

average cost per employee for all subcomponents was then summed to obtain 

the total average cost per employee for large and small automotive and 

aerospace companies. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-24 

Table 6-13. Costs Per Employee 

Company Size 

(employees) 

Major 

Errors 

Minor 

Errors 

Purchase 

Decision 

Costs Due 

to Bugs 

Installation 

Costs Due 

to Bugs 

Maintenance 

Costs Due to 

Bugs 

Redundant 

Systems 

Costs Due 

to Bugs 

Total Cost 

Due to Bugs 

per 

Employee 

Automotive 

Size 1: fewer 

than 500 

$1,280.8 $81.9 $1.3 $51.6 $49.9 $0.8 $1,466.1 

% of costs 87% 6% 0% 4% 3% 0% 

Size 2: greater 

than 500 

$99.3 $121.0 $0.1 $41.6 $15.8 $0.0 $277.8 

% of costs 36% 44% 0% 15% 6% 0% 

Aerospace 

Size 1: fewer 

than 500 

$649.9 $0.9 $0.2 $48.1 $1,442.3 $0.0 $2,141.4 

% of costs 30% 0% 0% 2% 67% 0% 

Size 2: greater 

than 500 

$85.1 $26.9 $0.1 $13.1 $3.7 $0.1 $128.9 

% of costs 66% 21% 0% 10% 3% 0% 

For automotive firms with fewer than 500 employees, the total cost 

increases to $876.2 per employee. Major errors account for close to 

three-fourths of these costs. 

Aerospace costs per employee were similar in distribution to the 

automotive industry. Major and minor errors accounted for the 

large majority of costs for large companies. Small companies had 

higher total costs per employee, relative to large companies, with 

most of the costs resulting from major errors. 

It is of interest to note that major errors have a much larger impact 

on smaller firms compared to larger firms. Small automotive firms 

have a higher major error-per-employee cost compared to large 

firms, and major errors account for a much larger share of total costs 

per employee. 

The differences in the cost-per-employee estimates for large and 

small companies are driven by a couple of factors: 
Section 6 — Transportation Manufacturing Sector 

6-25 

Z Smaller firms are less likely to have the in-house staff to 

trouble shoot and correct errors as they occur. As a result, 

the error typically affects business operations for a longer 

period of time and may not be fully corrected the first time. 

Z Large companies get higher priority customer support from 

software vendors. It is not unusual for a software vendor to 

have two to three support staff predominantly assigned to 

their major clients. In contrast, smaller customers typically 

receive support through call-in help lines where response 

time may not be as fast. 

These differences imply that smaller firms are more likely to benefit 

from an improved infrastructure for software testing. 

Typical Company-Level Impacts 

Typical company-level impacts were calculated for representative 

firms of various sizes to assess whether estimated costs were 

“reasonable.” As Table 6-14 shows, an automotive company that 

has 100 employees experiences an economic cost of $87,620 per 

year due to software bugs and errors. As a company gets larger, its 

total cost attributable to software bugs and errors increases (but not 

linearly). For an automotive company that has 10,000 employees, 

its total cost attributable to software bugs and errors is just under 

$2.5 million per year. These cost calculations, build up from 

subcomponent costs per employee, are consistent with “top down” 

estimates provided by several companies in the automotive industry.

Table 6-14. Company-Level Costs Associated with Bugs for Hypothetical Transportation 

Company at Different Employment Levels 

Hypothetical Firm Size 

(Employment) Total Company Costs Associated with Software Errors and Bugs 

Automotive 

100 $146,614 

10,000 $2,777,868 

Aerospace 

100 $214,138 

10,000 $1,289,167 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-26 

 6.3.5 Partial Reduction of Software Errors 

The costs in the previous sections reflect the total cost associated 

with software errors. Although Table 6-14 generates an estimate of 

the total costs attributable to software bugs for different firm sizes, 

there is a difference between the total costs of software bugs and the 

amount of that cost that can be eliminated with improved tools. In 

addition to the feasibility of eliminating all bugs, there could also be 

an increasing marginal cost associated with eliminating bugs from 

the software development process. 

The survey of CAD/CAM/CAE/PDM software users also investigated 

how the cost savings associated with an improved infrastructure for 

software testing would change with the partial removal of bugs and 

errors. Many of our discussions with industry indicate that it is not 

feasible or economical for software developers to produce “bugfree” software. Thus, respondents were asked what the cost savings 

would be if their company encountered a 25, 50, or 75 percent 

reduction in software errors. 

It was anticipated that the rate at which the cost of bugs decreases 

as the number of bugs decreases will not be the same for all of the 

cost categories. For example, some cost–bug relationships may be 

linear (i.e., a 50 percent reduction in bugs leads to a 50 percent 

reduction is costs), and some may be nonlinear (i.e., a 50 percent 

reduction in bugs may lead to less than a 50 percent reduction in 

costs because even a small number of bugs requires testing, backup 

systems, etc.). 

Table 6-15 presents respondents’ estimates of the percentage cost 

reduction associated with different percentage reductions in bugs 

for each of the major cost categories discussed above. For major 

and minor software bugs, respondents indicated that the costs 

generally decline proportionally as the percentage of bugs is 

reduced. This implies that the cost per bug is relatively constant. 

These costs may be classified mostly as mitigation costs and are 

activities in response to errors. 

In comparison, the other categories—purchase decision costs, 

installation costs, maintenance costs, and redundant system costs—

are mostly avoidance costs. The benefits from reduced bugs for 

these categories are relatively flat until a substantial share (i.e., 
Section 6 — Transportation Manufacturing Sector 

6-27 

Table 6-15. Cost Reductions as a Function of Bug Reductions 

Average Percentage Cost Reduction in CAD/CAM/CAE or PDM 

Software for a Given Reduction in Software Bugs 

Cost Categories 25% 50% 75% 

Major failure costs 18 33 46 

Minor failure costs 20 33 48 

Purchase decision costs 9 14 20 

Installation costs 10 17 23 

Maintenance costs 7 11 14 

Redundant system costs 4 9 12 

75 percent) of the bugs are reduced. In these instances, a small 

number of bugs (or threat of bugs leading to failures) still lead to 

significant “avoidance” costs. 

A 50 percent reduction in bugs and errors is used in the analysis 

below to capture the “feasible” testing scenario. This is consistent 

with the decrease in the share of errors found in post product 

release shown in Table 6-5.7 As presented in Table 6-15, users 

indicated that a 50 percent reduction in errors would correspond to 

a 33 percent reduction in major and minor failure costs and 

between a 9 to 17 percent reduction in purchase, installation, 

maintenance, and redundant systems costs. 

 6.4 USERS’ INDUSTRY-LEVEL IMPACT 

ESTIMATES 

Industry-level impacts for the automotive and aerospace industry 

were estimated by weighting employment-level impacts provided in 

Table 6-9 by the domestic industry employment. As shown in 

Table 6-16, the industry-level impacts of an inadequate software 

testing infrastructure for the automotive and aerospace industries are 

estimated to be $1,467.1 million. Potential cost reductions from 

feasible infrastructure improvements are $431.5 million. Small 

 

7Post-product release errors decreased from 5 percent under the current 

infrastructure to 2 percent under the improved infrastructure. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

6-28 

Table 6-16. Annual Impacts’ Weighted Cost Per Deposits and Loans 

Company Size in 

Transactions 

Bug and Error 

Costs per 

Employee 

Weight 

(000s 

employees) 

The Cost of Inadequate 

Software Testing 

Infrastructure 

($millions) 

Potential Cost Reduction 

from Feasible 

Infrastructure 

Improvementsa

($millions) 

Automotive 

Small $1,466.1 474 $694.8 $220.0 

Large $277.8 1,926 $534.9 $157.0 

Total automotive $1,229.7 $377.0 

Aerospace 

Small $2,141.4 67 $142.9 $25.5 

Large $128.9 733 $94.5 $29.0 

Total aerospace $237.4 $54.5 

Total $1,467.1 $431.5 

aBased on a 50 percent reduction of errors. 

companies account for the majority of cost impacts. In both the 

automotive and aerospace industries they represent over half of the 

costs. 

The “feasible” infrastructure cost savings are less than 50 percent of 

the total infrastructure costs because there is not a one-to-one 

correlation between the share of bugs removed and the percentage 

cost reduction. As discussed in the previous section, a 50 percent 

reduction in bugs leads to less than a 50 percent reduction in costs. 
7-1 

 Financial 7 Services Sector 

This section investigates the excess costs incurred by software 

developers and users in the financial services sector due to an 

inadequate infrastructure for software testing. RTI conducted 

several case studies of software developers and an Internet survey of 

software users to quantify the cost impacts. 

Consistent with the transport ion analysis presented in Section 6, 

impact estimates were developed relative to two counterfactual 

scenarios. The first scenario investigates the cost reductions if all 

bugs and errors could be found in the same development stage in 

which they are introduced. This is referred to as the cost of an 

inadequate software testing infrastructure. The second scenario 

investigates the cost reductions associated with finding an increased 

percentage of bugs and errors closer to the development stages 

where they are introduced. The second scenario is referred to as 

cost reduction from feasible infrastructure improvements. 

Table 7-1 presents an overview of the empirical findings. The total 

impact on the financial services sector from an inadequate software 

testing infrastructure is estimated to be $3.3 billion. The potential 

cost reduction from feasible infrastructure improvements is 

$1.5 billion. Software developers account for about 75 percent of 

the total impact and users account for the remaining 25 percent of 

costs. 

This section begins with an overview of developers and users of 

software in the financial services sector. A more detailed industry 

profile is provided in Appendix D. We then present the analysis 

approach and survey findings used to estimate cost impacts for 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-2 

Table 7-1. Cost Impacts on U.S. Software Developers and Users in the Financial Services 

Sector Due to an Inadequate Testing Infrastructure ($ millions) 

The Cost of Inadequate Software 

Testing Infrastructure 

Potential Cost Reduction from 

Feasible Infrastructure 

Improvements 

Software Developers 

Router and switch $1,897.9 $975.0 

FEDI and clearinghouse $438.8 $225.4 

Software Users 

Banks and savings institutions $789.3 $244.0 

Credit unions $216.5 $68.1 

Total Financial Services Sector $3,342.5 $1,512.6 

software developers and users in Section 7.2 and Section 7.3, 

respectively. 

 7.1 OVERVIEW OF THE USE OF 

CLEARINGHOUSE SOFTWARE AND ROUTERS 

AND SWITCHES IN THE FINANCIAL 

SERVICES SECTOR 

The financial services sector (NAICS 52) consists of monetary 

authorities; credit intermediation, securities and commodity 

contracts organizations; and insurance carriers. In 1997 total 

revenue for this sector exceeded $2.1 trillion with employment of 

approximately 5.8 million. 

An increasing share of financial communications are occurring 

electronically. In 1999, over $19.5 trillion dollars worth of 

transactions occurred electronically, representing a 282 percent 

increase since 1989 (NACHA, 2000). 

The generic term used to describe the transfer of information 

electronically in the financial services sector is Financial Electronic 

Data Interchange (FEDI). FEDI transactions not only contain the 

information for the transaction that is being processed, but they also 

include the transfer of the financial resources. The reconciliation of 

accounts requires using a clearinghouse that adds a step to the FEDI 
Section 7 — Financial Services Sector 

7-3 

process that does not exist in generic Electronic Data Interchange 

(EDI) transactions. 

Computer software and hardware play two important roles in 

transferring information in the financial services sector. First, FEDI 

and clearinghouse software are used to manage the information 

content once it has arrived at its appropriate location. Second, 

routers and switches (a combination of software and hardware) are 

used to manage the flow of information from one entity to the next 

via the Internet and company intranets. This section provides an 

overview of electronic transactions in the financial services sector 

and describes the software that facilitates the process. 

 7.1.1 Overview of Electronic Transactions in the Financial 

Services Sector 

Financial transaction management is the overarching term used to 

describe the flow, monitoring, and control of data across and within 

banking institutions. It is defined as the firm’s ability to control and 

manage a range of transactions—from foreign exchange to securities 

deals—to their reconciliation and successful resolution. Financial 

transactions management can be subdivided into three general 

activities: financial transactions reconciliation, financial 

transactions services, and financial transactions control. 

Z Financial Transaction Reconciliation—The financial 

transaction reconciliation software allows the automated 

reconciliation of payments, securities, and foreign 

transactions. A flexible matching algorithm within each 

reconciliation module allows users to set up matching 

criteria to optimally meet the needs of partner banks or 

brokers, which increases matching rates. 

Z Financial Transaction Services—Financial transaction 

services include on-line transactions, archiving and retrieval 

functionality, and other services to aid the end user. 

Z Financial Transaction Control—Financial transactions 

control is software used to develop profiles and govern 

access to all functions. Roles and users can be defined 

individually or in groups, and user IDs can be assigned to all 

actions, providing a full audit trail. Several institutions can 

work with the same system independently of each other, and 

firms also have the ability to outsource matching services, if 

required. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-4 

Firms in the Financial Services Sector 

The Census Bureau aggregates firms engaged in financial 

transactions into four broad categories by NAICS code.1 Table 7-2 

provides establishment, revenue, payroll, and employment 

information for each category. 

Table 7-2. Characteristics of Firms in the Financial Services Sector, 1997

 Establishments 

Revenue 

(millions) 

Payroll 

(millions) Employees 

521 Monetary Authorities 42 24,581 903 21,674 

522 Credit Intermediation and Related 

Activities 

166,882 808,810 98,723 2,774,910 

523 Securities, Commodity Contracts, and 

Other Financial Investments and Related 

Activities 

54,491 274,986 71,281 706,053 

524 Insurance Carriers and Related 

Activities 

172,299 1,072,784 92,230 2,327,306 

Source: 1997 Economic Census, Finance and Insurance Subject Series. 

Firms within the Credit Intermediation and Related Activities sector 

(522) are the most dependent on software and hardware to support 

financial transactions. Sector 522 comprises firms engaged in 

financial transactions processing, reserve activities, and 

clearinghouse activities. Firms conducting clearinghouse activities 

(subsector 52232) are primarily engaged in financial transaction 

processing, reserve activities, and liquidity services or other 

financial instrument clearinghouse services. Firms in this sector are 

engaged in both automated and manual clearinghouse activities. In 

1997, the clearinghouse subsector included over 1,200 firms with 

over 60,000 employees. 

The finance and insurance sector of the economy (sectors 523 and 

524) comprises firms whose dominant line of business is either 

financial transactions or facilitating those transactions. Transactions 

are broadly defined to include the creation, liquidation, or change 

of ownership of a financial asset. 

 

1The appendix provides descriptions for each of the NAICS codes in sector 52. 
Section 7 — Financial Services Sector 

7-5 

 7.1.2 Software Used by Financial Services Providers 

Two main types of software are used to manage the exchange of 

information in the financial services sector: FEDI software and 

clearinghouse software. FEDI software manages the flow of 

information across firms, and clearinghouse software manages the 

flow of funds between financial institutions. Clearinghouse software 

balances interfirm transactions such as payrolls, travel and expense 

reimbursements, pensions, and dividends. Appendix D provides 

details on the characteristics and attributes of these transactions. 

Major Producers of FEDI and Clearinghouse Software 

When a firm is deciding on what FEDI or clearinghouse software to 

implement, it can either develop its own software, have the software 

custom built, or purchase a commercial application. Although 

some FEDI and clearinghouse software applications are 

commercially available, they often have to be adapted and altered 

to fit with the firm’s existing legacy system. 

The FEDI and clearinghouse software market has a large number of 

both large and small producers. The most significant role in the 

FEDI and clearinghouse software market is played by the Federal 

Reserve. The Federal Reserve Financial Services provides a version 

of FEDI (FEDEDI) at no additional cost for use by financial 

institutions, service bureaus, or other entities that have an electronic 

connection to the Federal Reserve. However, many large banks and 

credit unions purchase monolithic or highly customized FEDI and 

clearinghouse software specifically designed for their institution. 

This provides a niche for companies focused on customized 

software services. Other FEDI and clearinghouse software 

producers provide more generic, out of the box software. Some of 

the companies that play a significant role in this market are Check 

Free Corporation, Software Dynamics, Inc., and Fundtech 

Corporation. 

Impacts of Inadequate Testing 

The economic cost associated with inadequate FEDI and 

clearinghouse software can be substantial (System Transformation, 

2000). In some cases, software failures prevent transactions from 

occurring; in other cases, expensive work-arounds for failures need 

to be implemented. Examples of the problems and associated costs 

resulting from FEDI and clearinghouse software failures include: 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-6 

Z data interchange interruptions or errors, 

Z credit card processing failure in the banking system, and 

Z trading system failure. 

 7.1.3 Software Embedded in Hardware Used to Support 

Financial Transactions 

In addition to software used to support FEDI and clearinghouse 

transactions, software is also embedded in hardware that is used to 

facilitate the physical transfer of electronic information. The 

process of passing information from one user to another is called 

routing. The two key pieces of technology involved in routing are 

routers and switches, both of which are combination of hardware 

and software that manage the flow of information. However, the 

software used to manage the flow of information is often inoperable 

across firms, routers, and area networks. Different products use 

different languages and different algorithms when making decisions 

about the passage of information. These differing decision-making 

processes create an interoperability problem. 

Appendix D describes how information is passed through an 

internetwork to get from one user to another, including how 

software is used to route information. 

Major Producers of Routers and Switches 

Four major companies dominate the market for routers that are used 

to transfer information: Cisco, Nortel, Lucent, and 3Com. Each 

major company uses its proprietary software to write switching and 

routing algorithms for use in its routers. Table 7-3 presents a list of 

companies and the proprietary software they use. 

Table 7-3. Router Market Shares of Major Firms 

Company 

Number of 

Router Types 

Total Sales 

(millions in 3rd quarter, 1999) Market Share Software Product 

Cisco 16 $1,360 72% IOS, ConFig Maker 

Nortel 8 $51 3% Preside 

Lucent $278 15% Hybrid Access 

3Com 5 $196 10% Enterprise OS Software 

Source: The Dell’Oro Group. 2001. <www.delloro.com>. 
Section 7 — Financial Services Sector 

7-7 

The measure of the number of router types that each company has is 

a broad measure of product categories. Numerous potential 

configurations and upgrades are available to the end user within 

each broad router type, effectively increasing the number of 

available products. We used total sales in the third quarter of 1999 

to get a common metric for the relative share of the market for 

routers and switches held by each firm. 

Current Testing Inefficiencies 

The rapid growth in the sales of switches and routers and the 

significant technological improvements that have occurred in the 

second half of the 1990s have created routers and switches that may 

not interoperate. Insufficient testing of the software algorithms used 

in operating the routers and switches is contributing to the lack of 

interoperability. 

Failures in the software used to run internetworks, which can be 

attributed to inadequate testing, can cause serious information 

delivery problems. Attributes of the software used to run 

internetworks that are of concern to developers are connectivity, 

reliability, network management, and flexibility. Connectivity is a 

challenge because various sites use different types of technology 

that may operate at different speeds. Reliability is a concern 

because individual users need information from other users in a 

timely manner. Network management ensures that centralized 

support is available to all users. Flexibility deals with the ability to 

adapt, add on to, and improve the network. 

Failure on any of these measures leads to several potential impacts, 

including the following: 

Z decreased speed of information delivery, 

Z failure of information delivery, 

Z inefficient router algorithms, 

Z lack of robust routers, 

Z reduced security of Internet and intranet traffic, and 

Z inability to run specific programs. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-8 

 7.2 SOFTWARE DEVELOPER COSTS IN THE 

FINANCIAL SERVICES SECTOR 

We conducted interviews with four developers of router and switch, 

FEDI, and clearinghouse software. Companies eagerly admitted that 

the current set of tools was inadequate for finding all of the bugs 

that exist in an efficient manner before a new product is shipped to 

a customer. All agreed that an improved testing infrastructure could 

reduce testing costs and accelerate the time to market for their 

products. Additionally, they said that improved testing products 

would decrease the amount of customer support required and 

increase the value of the product they produce. 

Clearinghouse software developers were the most reluctant to 

provide information on their testing procedures or the level of 

resources devoted to finding and fixing software errors. In most 

instances developers said that information on testing expenditures 

and errors discovered were confidential because they reflected 

detailed information about their product development process. In 

addition, whereas most companies track the number and location of 

bugs that emerge, few companies track their expenditures on testing 

and system costs. 

All companies agreed an improved system for testing was needed 

that would be able to track a bug back to the point where it was 

introduced and then determine how that bug influenced the rest of 

the production process. Respondents said that they knew about 

bugs when they emerged but had the most difficulty in tracking 

them down to their inception point. Respondents noted that the 

technology they were working with lacked the ability to accomplish 

this. 

Respondents thought that an improved infrastructure would consist 

of tools that are able to spot an error as close to when it is 

introduced as possible. Their ideal testing infrastructure would 

support close to real time testing where testers could remedy 

problems that emerge right away rather than waiting until a product 

is fully assembled. Respondents also indicated that they would be 

willing to purchase and install new products that accomplished this. 

They said that they waste valuable resources later in the production 

process because of missed software bugs and that any improved 

infrastructure would be effective at reducing testing costs. The 

Their ideal testing 

infrastructure would 

support close to real 

time testing where 

testers could remedy 

problems that 

emerge right away 

rather than waiting 

until a product is 

fully assembled. 
Section 7 — Financial Services Sector 

7-9 

major benefit that they saw from an improved infrastructure was 

direct cost reduction in the development process and a decrease in 

post-purchase customer support. An additional benefit that 

respondents thought would emerge from an improved testing 

infrastructure is increased confidence in the quality of the product 

they produce and ship. The major selling characteristic of the 

products they create is the certainty of that product to accomplish a 

particular task. Because of the real time nature of their products, the 

reputational loss can be great. 

In addition to FEDI and clearinghouse software developers, we 

spoke with three router and switch producers who develop a 

significant amount of software that is embedded in the infrastructure 

to support financial services transactions. These companies 

indicated that testing costs would decrease dramatically if improved 

software testing tools could find more bugs prior to product release. 

The primary testing need for these companies is the ability to costeffectively generate more traffic (e.g., calls per second, requests for 

data per second) in a timely manner to simulate realistic operating 

scenarios during testing and debugging the traffic levels experienced 

at customers’ facilities. This would lead to more bugs being 

detected during integration versus at the customer’s site. 

Installation support is an important service provided by router and 

switch companies. Installation support typically involves having the 

developer’s employees at the customer’s site, providing assistance 

over the telephone, and remotely manipulating products (using data 

communication lines) at the customer’s site. Companies said that 

better testing tools and methods used during software development 

could reduce installation expenditures by 30 percent. 

Forty percent of these companies’ after-sales service costs are 

related to bugs found by customers during business operations.2 

Developers said that better software testing tools could reduce this 

percentage to 10 percent. 

The remainder of this subsection quantifies the developer cost 

savings due to finding bugs and errors closer to when they are 

introduced for the financial services sector based on the empirical 

results from the router and switch developer surveys. We used the 

 

2The remaining 60 percent of after-sales service costs are related to user error or 

other problems not related to defective software. 

Software developers said 

that better software testing 

tools could reduce aftersales service costs by 

30 percent. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-10 

estimated costs per employee as representative of the economic 

impact of an inadequate infrastructure for software testing on all 

software developers supporting the financial services sector. 

 7.2.1 Industry Surveys 

As with the surveys of software developers supporting the 

transportation sector, in determining the costs associated with an 

inadequate infrastructure for the financial services sector we made 

two key assumptions: 

Z The same number of bugs still occurs regardless of the 

infrastructure used or the quality of that infrastructure. Bugs 

are attributed to human error and will continue to occur. 

Z An improved infrastructure does not change where bugs are 

introduced. This again is assumed to be a function of 

human error. 

Data collection focused on the impact an improved infrastructure 

would have on lowering the cost of testing and fixing bugs and 

errors and finding the bugs closer to the time they were introduced. 

We collected information to support the evaluation of two 

counterfactuals scenarios. The first scenario investigates the cost 

savings developers would realized if all bugs and errors were found 

in the same development stage that they were introduced. The 

second scenario investigates the impact of a partial reduction in 

software bugs and errors.3 

 7.2.2 Survey Findings 

The metrics for quantifying the impact of inadequate software 

testing methods and tools are discusses in Section 5. Following this 

approach, the key pieces of information collected from the surveys 

were 

Z the current distribution of where bugs are introduced and 

found in software, 

Z the time required to fix a bug given this distribution, and 

Z the expectations of how an improved infrastructure would 

testing activities. 

To collect the information to estimates cost impacts RTI conducted 

on-site, telephone and internet interviews with software testers at 

companies that manufacture routers, switches and gateways that 

 

3See Section 6.2.1 for a more detailed discussion of the two counterfactual 

scenarios. 
Section 7 — Financial Services Sector 

7-11 

support financial transactions. The questionnaire used to collect the 

information is presented in Appendix E. 

Based on the survey findings, Table 7-4 shows where software bugs 

are found based on the introduction point. For example, about 

7 percent of bugs are introduced and found in the requirements 

stage. However, 3 percent of bugs are introduced in the 

requirements stage and not found until post-product release. As 

shown in Table 7-4, 58 percent of errors are introduced in the 

coding/unit testing stage with many of these errors not found until 

latter stages (integration stage for example).4

Table 7-4. Distribution of Bugs Found Based on Introduction Point 

Stage Found 

Stage Introduced Requirements 

Coding/Unit 

Testing Integration 

Beta 

Testing 

Post-product 

Release 

Row 

Percentage 

Requirements 6.7% 9.5% 6.1% 5.3% 2.8% 30.3% 

Coding/unit testing NA 32.2% 14.3% 6.3% 5.0% 57.8% 

Integration NA NA 7.9% 1.8% 2.3% 11.9% 

Column 

percentage 6.7% 41.7% 28.3% 13.3% 10.0% 100.0% 

NA = Not applicable because a bug cannot be found before it is introduced. 

Once the distribution of bugs is determined, the next step is to 

determine the costs of fixing a bug based on the point of 

introduction. As discussed above, the costs of fixing a bug are 

greater the farther away from the point of introduction is the point at 

which the bug is discovered. This occurs for several reasons. First, 

it is more difficult to find a bug the farther away from the point of 

introduction. Second, more code has to be rewritten the farther 

away from the point of introduction that the bug is found. 

 

4Note that we are investigating only bugs and errors introduced in the software 

product development process. Errors introduced during beta testing or 

implementation are not included in the distributions in Table 7-4. However, 

developers said that it is often difficult for the testers and software engineers to 

determine where the user introduced the bug or as part of the development 

process. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-12 

Table 7-5 shows resources (costs) in terms of the average number of 

tester hours required to investigate and fix a bug based on the 

survey responses. The first row of Table 7-5 shows that for bugs 

introduced in the requirement stage, it is increasing costly to find 

and fix them the longer they remain undetected. For example, to 

correct a requirements error not found until the post production 

stage it is approximately 15 time more costly than if the error would 

have been found back in the requirements stage where it was 

introduced. 

Table 7-5. Hours to Fix Bug based on Introduction Point 

Stage Found 

Stage Introduced Requirements 

Coding/Unit 

Testing Integration Beta Testing 

Post-product 

Release 

Requirements 1.2 8.8 14.8 15.0 18.7 

Coding/unit testing NA 3.2 9.7 12.2 14.8 

Integration NA NA 6.7 12.0 17.3 

NA = Not applicable because cannot find a bug before it is introduced 

Using the distribution of bugs (introduced and found) in Table 7-4 

and the hours to fixed each type of bug in Table 7-5, we are able to 

calculate the average hours per bug as a function of where the bug 

was found (see Table 7-6). For example, on average a bug found in 

coding/unit testing takes 4.9 hours to fix, whereas an average bug 

found in post-product release takes 15.3 hours to fix. In addition, 

using the distribution of where bugs are found we calculate that the 

overall average time to investigate and fix a bug is 17.4 hours. 

Based on the cost-per-bug calculations presented above, the 

national costs of an inadequate infrastructure for software testing are 

estimated for each of the two counterfactual scenarios described in 

Section 7.2.1. For the first scenario all bugs are found in the stage 

where they are introduced. For the “feasible” scenario, more bugs 

are found closer to the stage they were introduced because of 

improved testing methods and tools. The distributions of where 

bugs are found associated with each counterfactual scenario are 

shown in Table 7-7, along with the current distribution copied from 

Table 7-6. 
Section 7 — Financial Services Sector 

7-13 

Table 7-6. Time to Fix a Bug Based on Discovery Point 

Location Hours 

Current Distribution of 

Where Bugs are Founda Weighted Average Hours 

Requirements 1.2 7% 

Coding/unit testing 4.9 42% 

Integration 9.5 28% 

Beta testing 12.1 13% 

Post-product release 15.3 10% 

Total 17.4 

aFrom bottom row in Table 7-11. 

Table 7-7. Shift in the Distribution of Where Bugs are Found Based on Infrastructure 

Location 

Current 

Infrastructure 

All Bugs Found in Same 

Stage as Introduced 

Feasible Infrastructure 

Improvements 

Requirements 7% 30% 7% 

Coding/unit testing 42% 58% 57% 

Integration 28% 12% 27% 

Beta testing 13% 0% 5% 

Post-product release 10% 0% 5% 

Average hours per average bug 17.4 8.5 13.3 

Percentage reduction from current 

infrastructure 

 45.6% 24.3% 

The current distribution reflects where bugs are discovered under 

the existing inadequate infrastructure for software testing. Under 

the first scenario, all bugs are discovered in the development stage 

where they occur. Note that this distribution is simply the row 

percentage shown in Table 7-4. The “feasible” infrastructure 

scenario is based on survey data. Respondents were asked what the 

distribution of the discovery of bugs would look like with better 

tools. Under this scenario, some of the bugs are found sooner in the 

production process. 

As shown in Table 7-7 both scenarios shift the distribution of when 

bugs are found toward the early stages of development. In addition, 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-14 

respondents said that with feasible infrastructure improvements it 

would take approximate 15 percent less time to fix bugs (holding 

the distribution constant) because they would have more 

information as to the location of the error in the source code. Both 

of these effects are included in the change in the average number of 

hours required to find and fix an average bug under each scenario 

(next to last row of Table 7-7). For the feasible scenario, the 

average time to find and fix a bug dropped from 17.4 to 13.3 hours. 

If all bugs are found in the same stage as introduced, the average 

time dropped to 8.5 hours. 

The final row in Table 7-7 gives the percentage change in total time 

spent per bug for each of the scenarios relative to the baseline 

scenario. This can be interpreted as the amount of testing resources 

saved under the two counterfactual scenarios. 

 7.2.3 Cost Impacts Per Employee for Software Developers 

Once the average percentage change in testing resource is 

determined, we normalized cost impacts by company employee to 

develop a cost-per-employee metric associated with an inadequate 

infrastructure. We then used the cost per employee used in 

conjunction with total industry employment to estimate the total 

cost impact on the software developers of FEDI, clearinghouse, and 

router and switch software. 

Table 7-8 presents a breakdown of testing costs based on 

information collected during the case study. The second column 

provides current labor and capital expenses for software testing for a 

company of 10,000 employees. The third and fourth columns show 

the total cost of an inadequate infrastructure and the cost savings 

associated with feasible infrastructure improvements. We 

calculated the cost savings using the 45.6 percent and 24.3 percent 

reductions in testing resources calculated presented in Table 7-7. 

Labor costs for software testers account for the overwhelming 

majority of total testing expenditures. We calculated labor costs for 

software testers using company employment (10,000), the ratio of 

testers to total employees (10.5 percent), and the average fully 

loaded wage rate for software testers ($68 per hour). To this, 

external testing services, hardware costs, and after-sale service costs 

were added to estimate the total testing costs. 
Section 7 — Financial Services Sector 

7-15 

Table 7-8. Developer Testing Costs for a Typical Company of 10,000 Employees 

Current Testing 

Costs 

The Cost of 

Inadequate Software 

Testing 

Infrastructure 

Potential Cost 

Reduction from 

Feasible 

Infrastructure 

Improvements 

Software testers $104,400,839 $49,038,698 $25,121,963 

Number of testers 766 360 184 

Fully loaded wage rate ($/hour) $68 $68 $68 

Software and hardware for testing $13,230,335 $5,755,124 $3,271,988 

External testing services $3,527,337 $1,858,837 $809,923 

After-sale service costs $2,403,556 $1,266,627 $551,888 

Total annual testing costs $123,562,900 

Annual change in testing costs $57,919,713 $29,756,015 

Cost savings as a percentage of sales 1.8% 0.9% 

The cost associated with an inadequate infrastructure for software 

testing are approximately 2 percent of the developers’ annual sales 

and potential cost reductions from feasible improvements are about 

1 percent of sales. 

 7.2.4 Industry-Level Impacts 

To extrapolate the cost impacts to reflect all developers of financial 

services software, we multiplied the cost per employee by the total 

employment of companies supplying software to this industry 

segment. Industry employment for router/switch software producers 

and for FEDI/clearinghouse software developers was obtained from 

publicly available databases (Standard and Poor's Net Advantage 

and Reference USA) and individual company 10K reports. 

Table 7-9 shows that the weighted industry-level impacts for an 

inadequate software testing infrastructure are approximately 

$1.9 billion for router/switch software developers and $0.4 billion 

for FEDI/Clearinghouse software developers. The potential cost 

reductions from feasible infrastructure improvements are $1.0 and 

$0.2 billion, respectively. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-16 

Table 7-9. Annual Impact on U.S. Software Developers Supporting the Financial Services 

Sector 

The Cost of Inadequate Software 

Testing Infrastructure 

Potential Cost Reduction from 

Feasible Infrastructure 

Improvements 

Routers and Switches Software 

Change in cost per employment $5,792 $2,976 

Total industry employment 327,676 327,676 

Industry-level savings (millions) $1,897.9 $975.0 

FEDI and Clearinghouse Software 

Change in cost per employment $5,792 $2,976 

Total industry employment 75,760 75,760 

Industry-level savings (millions) $438.8 $225.4 

 7.3 SOFTWARE USER COSTS IN THE FINANCIAL 

SERVICES SECTOR 

To estimate the costs due to an inadequate testing infrastructure for 

software end users, RTI collected data directly from banks and credit 

unions that use FEDI and clearinghouse software products. This 

subsection presents an overview of the survey process, descriptive 

statistics from data collected, and the economic impact estimates of 

software errors and bugs for users in the financial services sector. 

 7.3.1 Survey Method 

The end-user survey employed a telephone-Internet-telephone 

survey method in which the respondents were recruited via 

telephone, instructed to complete an Internet survey, and 

telephoned again if clarification was needed or if the respondents 

did not complete the survey in a timely manner. The survey was 

pre-tested by the project consultants and two financial service 

companies. The electronic instruments and resulting database were 

housed on RTI’s web site within RTI’s firewall to ensure security and 

confidentiality of the information provided by respondents. 

RTI developed the survey instrument and samples. Appendix E 

includes the final survey instrument. Harris Interactive recruited the 

users using scripts prepared by RTI. Up to eight calls were made to 
Section 7 — Financial Services Sector 

7-17 

locate the appropriate individual at each company, recruit 

participants, and follow up if surveys were not completed within 

2 weeks. 

Thousands of firms may be significantly affected by an inadequate 

infrastructure for software testing. The goal of the survey effort was 

to capture as large a share of the impacts as possible while ensuring 

that our survey population is representative of the industry as a 

whole. To this end, the objective of the survey was to complete 

interviews with of the 50 “largest” software users and 100 “medium 

to small” size software users. Size was defined by either volume of 

electronic transactions or by the sum of depository and loan 

transactions.5

 7.3.2 Survey Response Rates and Industry Coverage 

Over 1,400 end users were contacted to fill out the RTI end-user 

survey for the financial services sector. Table 7-10 provides the 

number of firms that were contacted and recruited and the number 

of completed surveys. For slightly over 50 percent of company 

contacts we were able to identify and speak with the individual in 

charge of maintaining their FEDI or clearinghouse software. Of 

these, 37 percent were successfully recruited to participate in the 

survey. One-third of the recruited participants returned completed 

survey instruments to RTI.6 

 

5Volume of electronic transactions was the preferred method for identifying “large” 

companies because this metric is closely correlated with the impact of 

inadequate software testing. The top 50 companies by electronic transaction 

volume ($$) were obtained from American Banker.com. For companies where 

total electronic transaction volume was not available, we used the sum of 

depository and loan transactions obtained from Federal Deposit Insurance 

Corporation public filings as the measure to stratify the sample by company 

size. 

6The relatively low recruitment and completion rates for the survey of companies 

in the financial services sector are the result of several issues. First, the direct 

impact that software errors have on this sector’s final products and services. 

Within the financial services sector, transactions occur in real time. Once a 

bug occurs, customers of that particular financial services sector are directly 

affected through loss of service. Because software failures are highly 

publicized, companies in the financial services sector are reluctant to discuss 

these issues, even if the source of the error is inadequate testing by software 

vendors. Second, all of the firms in the financial services industry provide 

almost identical services. What gives the firm its competitive advantage is not 

the activities that it conducts, but rather the software tool it uses to conduct 

them. Because the software that they use is so instrumental to defining their 

competitive advantage, they are reluctant to discuss any potential failures of 

that product. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-18 

Table 7-10. Financial Industry Survey Completion Rates

Sample Type 

Companies 

Contacted 

Identified Appropriate 

Contact 

Successful 

Recruits 

Completed 

Surveys 

Financial top tier 40 26 8 2 

Financial random 1,375 722 273 96 

Total 1,415 758 281 98 

We successfully contacted 40 of the 50 largest companies. Out of 

the 40 large companies contacted, the appropriate individual was 

identified for 26 companies. Of the 26 companies, eight agreed to 

fill out the survey and two returned completed surveys. 

In addition to the large companies, from a random stratified sample, 

we contacted 1,375 medium to small companies. For 722 the 

appropriate IT contact was identified. We recruited 273 of these 

companies to participate in the study, and 96 completed surveys 

were returned to RTI. 

Table 7-11 provides information on the extent of the industry 

coverage from the survey. The financial services sector population 

from which the survey sample was drawn is defined as commercial 

banks, saving institutions, credit unions, and other entities included 

in NAICS codes 5221. The population consists of 19,308 firms with 

a combined depository and loan transaction amount of $8,718 

trillion. Approximately 92 percent of those transactions are 

associated with commercial banks and saving institutions. This 

population excludes firms that solely provide securities brokerage 

services, commodity contracts brokerage, and securities commodity 

exchanges services. 

Industry coverage is determined by comparing by the sum of 

depository and loan transactions from surveyed respondents to 

industry totals. In addition, the survey respondents and industry are 

separated into banks and credit unions. Table 7-11 shows the 

coverage of the financial services sector represented by the 

completed surveys. Companies completing the survey represent 

14 percent of the financial services sector in terms of transaction 
Section 7 — Financial Services Sector 

7-19 

Sample Type 

Total Industry 

Transactions 

($ millions) 

Completed Surveys Transactions 

($ millions) 

(% of industry) 

Deposits 

Banks 4,244,733 491,348 

(12%) 

Credit unions 379,200 7,698 

(2%) 

Loans 

Banks 3,793,310.7 754,590 

(20%) 

Credit unions 301,300 2,258 

(1%) 

Total transactions 8,718,543.7 1,255,888 

(14%) 

amounts. The percentage covered is primarily due to the completed 

surveys of large banks and savings institutions that account for a 

large share of the industry depository and loan transactions. 

The sum of depository and loan transactions in Table 7-11 also 

provides the appropriate weights to extrapolate the sample 

responses to the industry-level impact estimates. 

 7.3.3 Survey Findings 

Survey respondents have an average employment of 3,970 

employees and average sales of approximately $29 million. Most 

respondents provide a variety of services. Forty percent of firms 

reported providing credit intermediation services; 63 percent 

provide securities, commodity contracts, and other financial 

services; and 22 percent sell insurance. An additional 33 percent of 

firms reported providing other financial services or products. 

Table 7-12 lists various software products that the sample 

participants reported using for electronic data exchange. The most 

commonly reported products were software products provided by 

the Federal Reserve Financial Services. The average life expectancy 

for these software products was 1.5 years, and the majority of them 

were installed between 1983 and 2001. Most users of the software 

say that they have been using the same system for 1 to 10 years. 

Table 7-11. Industry 

Coverage 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-20 

Table 7-12. Reported Software Products 

Software Product Vendor/Provider Frequency 

ACH Federal Reserve Financial Services 2 

Bank on It Transact CTX Option 1 

Bulkdata Federal Reserve Financial Services 1 

CBS Origination Control 1 

Digital Insight Digital Insight 2 

Digital Unix Compaq 1 

ECS 1 

EPN PC Aims Electronic Payments Network 1 

FEDEDI Federal Reserve Financial Services 11 

FEDI Federal Reserve Financial Services 6 

Fedline Federal Reserve Financial Services 14 

FedPlu$ Fundtech Corporation 1 

FiServ Galaxy 2000 Technical Programming Services Inc. 2 

Fundtech Fundtech Corporation 2 

GMI Software GMI Software 1 

International Cash Management IBOS 1 

ITI Premier Bank Application Software Dynamics, Inc. 3 

Jack Henry & Associates Jack Henry & Associates 1 

Kirchman Financial Software Kirchman Corporation 1 

MISER Miser Software 1 

Mercator for EC Mercator 1 

Open Solutions Open Solutions, Inc. 1 

Modern Banking Systems Modern Banking Systems, Inc. 1 

Pay Systems International Credit 1 

PEP Check Free Corporation 7 

PC AIMS 1 

Pershing Net Xchange Pro Advantage Capital Corporation 1 

Shazam Vector 1 

Sterling Bankers ACH 1 

Sterling Commerce Connection SBC Communications 1 

Trading Partners 1 

Xp Software 1 

VISA Direct Exchange Open File Delivery VISA Corporation 1 

Federal Reserve FEDI Federal Reserve Financial Services 1 
Section 7 — Financial Services Sector 

7-21 

Most companies responded that they had only two employees (fulltime equivalents [FTEs]) involved in operating and supporting FEDI 

transactions and eight FTEs supporting clearinghouse transactions. 

However, one of the largest companies indicated that they had five 

FEDI staff and 200 clearinghouse staff supporting electronic 

transactions. Almost all of respondents said that their information 

reflected FEDI and clearinghouse transaction activity for the entire 

company. 

Incidence and Costs of Software Errors and Bugs 

Approximately two-thirds of the companies providing information 

on software errors and bugs indicated that they had experienced 

major software errors in the previous year. The remaining one-third 

of the companies said they did not experience any major software 

errors over the past year and that minor errors were quickly 

corrected at little to no cost. 

For the respondents that did have major errors, they reported an 

average of 40 major and 49 minor software bugs per year in their 

FEDI or clearinghouse software systems (see Table 7-13). 

Approximately 16 percent of those bugs were attributed to router 

and switch problems, and 48 percent were attributed to transaction 

software problems. The source of the remaining 36 percent of 

errors was unknown. All members of the sample reported that the 

software problems they experienced in 2000 were typical of other 

years. 

Table 7-13. Incidence and Costs of Software Errors 

Firms Experiencing Errors 

Impact Categories 

Percent of Firms 

Reporting 

Average of Firms 

Reporting Errors 

Percentage of Firms 

With No Errors 

Number of major errors 61% 40 39% 

 Repair cost per error (labor hrs) 18.4 hour 

 Lost data per error ($) $1,425 

 Delayed new service 

introduction (months) 

 1.5 months 

Number of minor errors 71% 49.4 29% 

 Costs per error $3,292.9 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-22 

Typical problems encountered due to bugs were 

Z increased person-hours used to correct posting errors, 

Z temporary shut down leading to lost transactions, and 

Z delay of transaction processing. 

Most respondents reported that the software errors only temporarily 

delayed transactions. One respondent reported transactions being 

shut down for 30 to 60 minutes. Approximately 15 percent of 

respondent companies indicated that they had lost reputation as a 

result of a software error, 5 percent reported lost market share, and 

10 percent said that they experienced delayed product or service 

introduction. The other respondents said that they had no market 

share or reputation loss. 

For the respondents who did have major software errors, they 

estimated that an average of 18.4 labor hours is spent to repair each 

error or bug. In addition, several firms indicated that they had lost 

information as a result of software errors and that the average value 

of information loss was about $1,425 per software error. 

Eight-two percent of minor errors experienced by the companies 

increased operating costs as a result of developing patches and 

work-arounds for their software. On average, companies spend 

approximately $3,293 per year on solutions for minor errors. 

However, responses varied greatly with one respondent saying that 

minor errors cost his company over $12,000 per year. 

Software Life-Cycle Costs 

Respondents were asked about the life-cycle costs of FEDI and 

clearinghouse software. Table 7-14 presents the total costs of lifecycle activities, including software purchase decisions, installation 

and acceptance testing, annual maintenance, and redundant system 

costs. The last column in Table 7-14 indicates the percentage of 

these expenditures that are due to software errors and bugs. This 

percentage reflects the average cost savings that a typical firm 

would receive if all software bugs were found by the developer 

prior to release of the software product. This percentage reduction 

represents an upper bound of the benefits from an improved 

software testing infrastructure. 
Section 7 — Financial Services Sector 

7-23 

Table 7-14. Total Costsof Search, Installation, and Maintenance (Life-Cycle Costs) 

Average Annual Cost of 

Activities ($) 

Average Cost Reduction Associated with 

Software Errors (%)a

Purchase decision $481.6 20% 

Installation and acceptance $393,500 16% 

Maintenance $1,578.3 11% 

Redundant system costs $3,466.7 46% 

aReflects cost savings from eliminating all software bugs and errors. 

Purchase Decision 

On average, the companies indicated that they spend approximately 

4 months and one to two FTEs researching new FEDI or 

clearinghouse software packages before they purchase a package. 

For this sample, the average expenditure was $482, which we 

calculated by multiplying the cost of each company’s reported FTEs 

by the amount of time the company reported expending for 

purchasing new FEDI or clearinghouse software times an hourly rate 

of $75 per hour. 

Sixty-seven percent of respondents said that they could reduce their 

search costs if they had better information about the quality of the 

software products. These respondents indicated they could reduce 

search time by approximately 1 month, reflecting an average 

savings of about 20 percent, or $12,000 per company for this 

percentage of the population. 

Installation and Acceptance Testing. Companies on average spend 

about 65 hours per month for 2 months on installation and 

acceptance testing, representing about $393,500 per installation. 

The level of effort varied greatly, ranging from 1 to 480 hours of staff 

time. 

Respondents said that about 16 percent of installation costs were 

associated with software errors and bugs. This reflects an average 

savings of about $62,960 per firm. Two respondents said that they 

used external consultants for installation and acceptance testing. 

Annual Maintenance Costs. Maintenance expenditures on FEDI 

and clearinghouse software averaged $1,578 per year. Most 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-24 

expenditures were for standard maintenance contracts with the 

provider of the software. 

Respondents said that maintenance expenditures could be reduced 

by about 11 percent if software errors and bugs were eliminated, 

reflecting an average cost savings of approximately $174 per year. 

Redundant System Costs. Approximately half of the companies 

indicated that they maintain redundant backup systems after 

installing new software. On average these systems were maintained 

about 3 months at a cost of $400 per month. Thus, the elimination 

of bugs would represent a savings of about $1,595 per new system 

installed for the 50 percent of the population maintaining redundant 

systems. 

 7.3.4 Software User Costs Per Transaction 

The total costs of software bugs and errors for a firm is the sum of 

the mitigation costs associated with major and minor errors when 

they occur (Table 7-13) and the avoidance costs incurred 

throughout the life-cycle of the software product (Table 7-14). We 

divided total firm cost by firm transactions to get a cost per 

transaction metric that we later used to weight the impact estimates. 

We developed separate impacts per deposit/loan transaction 

estimates for banks and credit unions. Banks and savings 

institutions are more likely to be diversified, engaging in many 

different business activities and hence may have low cost-to-sales 

and cost-to-employee ratios. In contrast, credit unions tend to be 

smaller companies where software costs are likely to be a much 

larger share of their deposit/loan transactions. Stratifying the 

population and using separate company-type cost-to-transaction 

ratios provide a more accurate estimate of national impacts 

Table 7-15 presents costs-to-transactions ratios for subcomponents 

for both banks and credit unions. Because not all respondents were 

able to provide information for each subcomponent (e.g., major 

errors, minor errors, purchase costs) an average costs-to-transaction 

ratio was calculated individually for each subcomponent. The 

average cost-to-transaction ratios for all subcomponents were then 

summed to obtain the total average cost-to-transaction ratio for each 

company type. In addition to giving the dollar cost per impact 

subcategory, we also present the percentage distribution of costs. It 

Separate impact estimates 

per deposit/loan were 

developed for banks and 

credit unions. 
Section 7 — Financial Services Sector 

7-25 

is of interest to note that the costs of an inadequate infrastructure are 

distributed across numerous types of bugs. 

Table 7-15 shows that the major error subcategory represents the 

largest share of total costs associated with software bugs and errors. 

This subcategory includes labor expenditures to fix major errors and 

the value of information lost as a result of major errors. The average 

cost per million dollars in transactions is $55 for major errors and 

$2 for minor errors. The second and third largest impact 

subcategories are additional expenditures (due to bugs) for software 

purchase decisions and installation costs associated with bugs. 

Table 7-15. Software Bug and Error Costs Per Million Dollars of Deposits and Loans 

Major 

Errors 

Minor 

Errors 

Purchase 

Decision 

Costs Due 

to Bugs 

Installation 

Costs Due 

to Bugs 

Maintenance 

Costs Due to 

Bugs 

Redundant 

Systems 

Costs Due 

to Bugs 

Total Cost 

Due to 

Bugs 

Banks and 

savings 

institutions 

$54.66 $2.13 $12.14 $28.73 $0.43 $0.11 $98.20 

Percentage of 

costs 

55.7% 2.2% 12.4% 29.3% 0.4% 0.1% 100.0% 

Credit unions $282.93 $7.43 $16.51 $10.71 $0.43 $0.11 $318.11 

Percentage of 

costs 

88.9% 2.3% 5.2% 3.4% 0.1% 0.0% 100.0% 

Table 7-16 illustrates the costs associated with software bugs of 

representative banks and credit unions of various sizes. The table 

indicates that the costs are significant. For a bank that has $100 

million in transactions, it experiences an economic cost of $10,000 

per year due to software bugs and errors. It is interesting to note 

that companies with less than $100 million dollars in depository 

and loan transactions are affected proportionally much more than 

companies with larger transaction amounts. For a bank with 

transactions of $10 billion, its total cost attributable to software bugs 

and errors is just under $1 million per year. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-26 

Table 7-16. Company Costs Associated with Bugs for Hypothetical Company Sizes 

Hypothetical Firm Size 

(millions of deposits and loans) Total Company Costs Associated with Software Errors and Bugs 

Banks and Savings Institutions 

$100 $9,820 

$10,000 $982,003 

Credit Unions 

$100 $31,811 

$10,000 $3,181,141 

Based on interviews with industry experts, we believe the increasing 

proportional impact for smaller companies is due two factors: 

Z Smaller firms are less likely to have the in-house capabilities 

to trouble shoot and correct errors as they occur. As a 

result, the error typically affects business operations for a 

longer period of time and may not be fully corrected the first 

time. 

Z Large companies get higher priority customer support from 

software vendors. It is not unusual for a software vendor to 

have two to three support staff permanently assigned to their 

major clients. In contrast, smaller customers typically 

receive support through call-in help lines where response 

time may not be as fast. 

 7.3.5 Partial Reduction of Software Errors 

The costs in the previous sections reflect the total cost associated 

with software errors and reflects an infrastructure where all bugs and 

errors are found and corrected prior to product release. However, 

our discussions with industry indicated that it is not feasible or 

economical for software developers to produce “bug-free” software. 

To estimate the impact of an improved testing infrastructure on end 

users, as part of the end-user survey we also investigated how the 

costs associated with bugs and errors in FEDI and clearinghouse 

software would change if the number of bugs and errors embedded 

in these software products were partially reduced. To this end, 

respondents were asked what the cost savings would be if their 

company encountered a 25, 50, or 75 percent reduction in software 

errors. 
Section 7 — Financial Services Sector 

7-27 

It was anticipated that the rate at which the cost of bugs decreases 

as the number of bugs decreases will not be the same for all of the 

cost categories. For example, some cost–bug relationships may be 

linear (i.e., a 50 percent reduction in bugs leads to a 50 percent 

reduction is costs), and some may be nonlinear (i.e., a 50 percent 

reduction in bugs may lead to less than a 50 percent reduction in 

costs because even a small number of bugs requires testing, backup, 

systems, etc.). 

Table 7-17 presents respondents’ estimates of the percentage cost 

reduction associated with different percentage reductions in bugs 

for each of the major cost categories discussed above. Table 7-17 

indicates that a 25 percent reduction in errors would lead to a 

17 percent reduction in major failure costs; 9 percent reduction in 

minor failure costs; and corresponding reductions in purchase, 

installation, maintenance, and redundant systems costs. 

Table 7-17. Cost Reductions as a Function of Error Reductions 

This table shows the average percentage reduction in costs for a given percent reduction in software errors. The rate at 

which costs decrease (as errors decrease) varies for different types of software costs. 

Cost Categories 25% Reduction in Errors 50% 75% 

Major failure costs 17 32 46 

Minor failure costs 9 19 36 

Purchase decision costs 26 28 32 

Installation costs 29 31 35 

Maintenance costs 30 32 32 

Redundant system costs 19 19 25 

For major and minor software bugs, respondents indicated that the 

costs generally decline proportionally as the percentage of bugs is 

reduced. This implies that the cost per bug is relatively constant. 

These costs may be classified mostly as mitigation costs and are 

activities in response to errors. 

In comparison, the other categories—purchase decisions, 

installation costs, maintenance costs, and redundant system costs—

are mostly avoidance costs. The benefits from reduced bugs for 

these categories are relatively flat until a substantial share (i.e., 

75 percent) of the bugs are reduced. In these instances, a small 

A 50 percent reduction in 

errors was used in the 

improved scenario. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

7-28 

number of bugs (or threat of bugs leading to failures) still lead to 

significant “avoidance” costs. This indicates that companies would 

continue to experience these costs even though the quality of the 

software product that they are producing is improving. In other 

words, these fixed costs may continue to exist until software quality 

nears the point of zero errors. 

Based on the developer case study, we estimate an improved 

infrastructure would lead to a 50 percent reduction in errors found 

in the post-product release stage. The 50 percent reduction 

estimate, along with the relationship between percentage error 

reduction and cost reduction presented in Table 7-17, is used to 

calculate cost saving for the users’ “feasible” infrastructure scenario 

presented below. 

 7.3.6 Users’ Industry-Level Impact Estimates 

We weighted cost per transaction impact estimates to obtain the 

industry-level economic impact of an inadequate software testing 

infrastructure for the financial services sector. We normalized and 

weighted the economic impact estimates by company depository 

and loan transaction data because the costs of errors and bugs are a 

function of the volume of transactions; this method leads to an 

estimate that reflects the total transactions within the industry. 

Multiplying the weight by the cost per transaction generates the 

total costs attributable to software bugs. As shown in Table 7-18, 

the total cost attributable to software bugs using this approach is 

$1 billion. The potential cost reduction from feasible infrastructure 

improvements is $312 million. Banks account for over 80 percent 

of the total impacts in both scenarios. 

Table 7-18. Annual Impacts’ Weighted Cost Per Deposits and Loans 

Company Size 

in Transactions 

Bug and Error 

Costs per 

$Million of 

Transactions 

Weight 

($Millions)a

The Cost of 

Inadequate Software 

Testing 

Infrastructure 

Potential Cost 

Reduction from 

Feasible Infrastructure 

Improvementsb

Banks $98.20 $8,038,044 $789,338,629 $244,027,852 

Credit unions $318.11 $680,500 $216,476,621 $68,083,419 

Total $8,718,544 $1,005,815,250 $312,111,271 

aTotal deposits and loans in financial services sector. 

bBased on a 50 percent reduction of errors. 
Section 7 — Financial Services Sector 

7-29 

The “feasible” infrastructure cost savings are less than 50 percent of 

the total infrastructure cost because there is not a one-to-one 

correlation between the share of bugs removed and the percentage 

cost reduction. As discussed in the previous section, a 50 percent 

reduction in bugs leads to less than a 50 percent reduction in costs. 

The impact estimates presented in Table 7-18 are conservative 

estimates because they only include the avoidance and mitigation 

costs for financial service companies. These estimates do not 

include the delay costs imposed on the consumers of financial 

services due to system downtime or costs associated with errors in 

financial transactions. 

8-1 

 National Impact 8 Estimates 

The analysis presented in the previous sections generated estimates 

of the costs of an inadequate software testing infrastructure for 

software developers and users in two representative sectors of the 

economy: transportation equipment manufacturing and financial 

services. This section extrapolates the per-employee costs for these 

two sectors to other manufacturing and service sectors to develop 

an approximate estimate of the economic impacts of an inadequate 

infrastructure for software testing for the total U.S. economy. 

Table 8-1 shows that the national cost estimate of an inadequate 

infrastructure for software testing is $59.5 billion. The potential cost 

reduction from feasible infrastructure improvements is $22.2 billion. 

This represents about 0.6 to 0.2 percent of the U.S.’s $10 trillion 

dollar gross domestic product (GDP). Software developers 

accounted for about 40 percent of impacts, and software users 

accounted for the remaining 60 percent. 

Table 8-1. National Economic Impact Estimates 

The Cost of Inadequate 

Software Testing Infrastructure 

(billions) 

Potential Cost Reduction from 

Feasible Infrastructure Improvements 

(billions) 

Software developers $21.2 $10.6 

Software users $38.3 $11.7 

Total $59.5 $22.2 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

8-2 

This section begins with a review of the per-employee impact 

estimates for the transportation equipment manufacturing and 

financial service sectors.1 Section 8.1 and Section 8.2 present the 

per employee cost metrics for software developers and software 

users that were estimated through the industry surveys. Section 8.3 

uses these impact metrics to extrapolate the survey findings to other 

industries to get an approximate measure of the total economic 

costs of software bugs and errors. The limitations of this approach 

are discussed in Section 8.4. 

 8.1 PER-EMPLOYEE TESTING COSTS: 

SOFTWARE DEVELOPERS 

To extrapolate cost impact estimates obtained from the developer 

surveys to national estimates, a proper weighting mechanism is 

needed. Typically weighting procedures are conducted using either 

employment or sales. For software testing, RTI elected to weight the 

results by an employee metric—specifically the number of software 

testers. 

Results are not weighted by sales because of the economics of 

software production. Software is a high-fixed cost, low (near zero) 

marginal cost industry. Software sales can often be large when very 

little effort is involved in the testing process. Alternatively, for some 

software products a significant amount of testing may have 

occurred, but sales could be limited because of the stage in the 

product life-cycle. 

The total number of computer programmers and computer software 

engineers is published by the Bureau of Labor Statistics (BLS) and is 

listed in Table 8-2. A portion of these programmers and software 

engineers are engaged in testing activities. The BLS categories listed 

in Table 8-2 are used to estimate the total number of FTEs engaged 

in testing and debugging activities. Based on interviews with 

industry, we estimate that approximately 10 percent of computer 

programmers’ and 35 percent of computer software engineers’ time 

is spent debugging and correcting errors. This yields a total of 

 

1Note that in Section 6 impacts for the financial services sector were weighted by 

transactions. However, transactions is not an appropriate weight for leveraging 

the impact estimates from this sector to other service sectors in the economy. 

For this reason, impacts per employee are calculated in this section and used to 

develop national service-sector impacts. 
Section 8 — National Impact Estimates 

8-3 

Table 8-2. FTEs Engaged in Software Testing (2000) 

BLS Categories 

National 

Employment 

Percentage 

Involved in Testing 

Number of 

FTEs 

Computer programmers 585,000 10% 58,500 

Computer software engineers: applications 380,000 35% 133,000 

Computer software engineers: systems software 317,000 35% 110,950 

National total 1,282,000 302,450 

302,450 FTEs engaged in testing and debugging activities and 

represents approximately one-fourth of all computer programmers 

and software engineers. 

Based on the findings from the software developers’ surveys 

presented in Section 6 and Section 7, total testing costs per software 

tester are about $138,000 for CAD/CAM/CAE/PDM software 

developers and $161,000 for FEDI/clearinghouse/router/switch 

developers.2 These costs include testing labor, hardware, external 

testing services, and related after-sales services. The labor costs are 

based on the average computer software engineers’ fully loaded 

annual wage obtained from the BLS (2002). As shown in Table 8-3, 

the cost of an inadequate infrastructure is $53,000 and $76,000 per 

tester for the transportation and financial services sectors. This 

represents the reduced level of testing resources if all errors were 

found in the stage they were introduced. Similarly, the potential 

cost reductions from feasible infrastructure improvements are 

$23,000 and $38,000 per tester for the transportation and financial 

services sectors. 

Because the BLS does not break out tester employment by industry 

sector, we used a weighted average of the automotive/aerospace 

and financial services cost savings in conjunction with national 

tester employment to calculate cost savings presented in Table 8-3. 

The weight is based on the total employment of the manufacturing 

and service sectors. The weighted average cost of an inadequate 

 

2The cost per employee estimates are based on the survey findings that are 

presented in Section 6 and Section 7 and are calculated as total testing costs 

(including labor, software, hardware, etc.) divided by the number of FTE testers. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

8-4 

Table 8-3. Software Developer Costs Per Tester 

Sector/Cost Category Cost Per Tester 

The Cost of Inadequate 

Software Testing 

Infrastructure 

Potential Cost Reduction 

from Feasible Infrastructure 

Improvements 

Transportation 

Labor expenditures $136,282 $52,212 $22,975 

External testing services $100 $38 $17 

Hardware $250 $96 $42 

After-sales services $1,363 $522 $230 

Total $137,996 $52,869 $23,264 

Financial Services 

Labor expenditures $136,282 $64,014 $32,793 

External testing services $17,270 $7,513 $4,271 

Hardware $4,604 $2,426 $1,057 

After-sales services $3,138 $1,653 $720 

Total $161,295 $75,607 $38,843 

Weighted Average Cost per Tester $155,493 $69,945 $34,964 

infrastructure is $70,000 per tester and the weighted average cost 

reduction from feasible improvements is $35,000 per tester. 

 8.2 PER-EMPLOYEE COSTS: SOFTWARE USERS 

As with the software developers, a proper weighting method is 

needed to extrapolate the impacts generated in Section 6 and 

Section 7 to national-level user cost impacts. Similar to above, we 

used employment as the weight to estimates national costs 

attributable to an inadequate infrastructure for software testing. 

Ideally the number of employees involved with operating and 

maintaining software products would be used as the weighting 

metric. However, because computer use increasingly cuts across all 

aspects of business operations, estimating a total FTE for computer 

user and support is difficult. For this reason total employment in the 

service and manufacturing sectors was used as the weighting 

metrics. This information is readily available from BLS and is 

presented in Table 8-4 (BLS, 2002). Software companies have been 
Section 8 — National Impact Estimates 

8-5 

Table 8-4. National Employment in the Service and Manufacturing Sectors 

Employment 

(millions) 

Service sectors: include services; retail trade; finance, insurance, and real estate; 

and wholesale trade 

74.1 

Manufacturing: includes manufacturing and construction 25.0 

Note: Excluded are the government, transportation and utilities, and mining sectors (27.2 million) because their 

computer use (intensity of use) was deemed significantly different from either the manufacturing or service sectors’ 

use. Also, excluded are computer programmers and software engineers (1.3 million) because their impacts are 

captured under developer costs. 

Source: U.S. Department of Labor, Bureau of Labor Statistics (BLS). 2002. Occupational Outlook Handbook, 2002-03 

Edition. Available at <http://www.bls.gov/oco/home.htm>. 

excluded from the service sector employment because they are 

weighted separately. 

Table 8-5 provides the per-employee cost metrics derived from the 

survey findings presented in Section 6 and Section 7. The third and 

fifth columns of Table 8-5 replicate the total user cost impacts for 

the automotive/aerospace sectors and the financial services sector. 

The sector-level impacts are then divided by their associated sector 

employment to develop cost impacts per employee. 

Table 8-5. Per-Employee Cost Metrics 

The Cost of Inadequate 

Software Testing 

Infrastructure 

Potential Cost Reduction 

from Feasible Infrastructure 

Improvements Number of 

Employees 

(thousands) 

Sector Costs 

(millions) 

Cost per 

Employee 

Sector Costs 

(millions) 

Cost per 

Employee 

Automotive and aerospace 3,199.6 $1,467.1 $459 $431.5 $135 

Financial services 2,774.9 $1,005.8 $362 $312.1 $112 

 8.4 NATIONAL IMPACT ESTIMATES 

To estimate the national cost of an inadequate infrastructure for 

software testing, the per-employee cost metrics for the financial 

services and transportation sectors are weighted to calculate the 

total costs for the U.S. manufacturing and service sectors. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

8-6 

Table 8-6. National Impact Estimates 

The Cost of Inadequate 

Software Testing Infrastructure 

Potential Cost Reduction from 

Feasible Infrastructure 

Improvementsa

Number of 

Testers/Employees 

(millions) Cost per 

Total Cost 

(millions) Cost per 

Total Cost 

(millions) 

Software developers 0.302 $69,945 $21,155 $34,964 $10,575 

Software users 

Manufacturing 25.0 $459 $11,463 $135 $3,375 

Service sector 74.1 $362 $26,858 $112 $8,299 

Total $59,477 $22,249 

aBased on a 50 percent reduction of errors. 

As shown in Table 8-6, the national impact estimate from an 

inadequate infrastructure for software testing is $59 billion and the 

potential cost reduction from feasible improvements is $22 billion. 

Software users account for a larger share of total inadequate 

infrastructure costs (64 percent) compared to “feasible” cost 

reductions (52 percent) because a large portion of users’ costs are 

due to avoidance activities. Whereas mitigation activities decrease 

proportionally to the decrease in the number of bugs and errors, 

avoidance costs (such as redundant systems and investigation of 

purchase decisions) are likely to persist even if only a few errors are 

expected. 

For software developers, the feasible cost savings are approximately 

50 percent of the total inadequate infrastructure costs. This reflects 

a more proportional decrease in testing effort as testing resources 

and tools improve. 

 8.5 LIMITATIONS AND CAVEATS 

We want to emphasis that because the national impact estimates 

presented in this section were developed from interviews with two 

sectors (transportation equipment manufacturers and financial 

service providers) representing 5 percent of the U.S. economy, these 

estimates should be considered approximations only. They are 

presented primarily to illustrate the magnitude of potential national 

impacts. 
Section 8 — National Impact Estimates 

8-7 

The following factors should be considered when interpreting the 

national estimates: 

Z The two industry sectors selected may not be representative 

of all the industries included in the manufacturing and 

service sectors. User costs per employee are likely to vary 

by industry. Thus, the user cost estimates should be 

considered to have a relatively high degree of uncertainty. 

For example, if costs per employee are greater in the 

automotive/aerospace and financial services sectors than the 

national averages, this would lead to an overestimate of the 

user impacts. 

Z Cost per software tester are more likely to be relatively 

constant across software companies serving different 

industries. Thus, we are more confident in the national 

impact estimates for software developers. And because 

tester costs represent between one-third to one-half the total 

national costs, this supports the robustness of our results. 

Z Several user sectors of the economy were excluded from the 

national employment figures used to weight impact 

estimates. In particular, we excluded the government sector 

with 19.9 million employees, which would lead to an 

underestimate of national costs. 

Z Quantifying the impact of inadequate testing on mission 

critical software was beyond the scope of this report. 

Mission critical software refers to software where there is 

extremely high cost to failure, such as loss of life. Including 

software failures associated with airbags or antilock brakes 

would increase the national impact estimates. 

Z Finally, the costs of software errors and bugs to residential 

households is not included in the national cost estimates. As 

the use of computers in residential households to facilitate 

transactions and provide services and entertainment 

increases, software bugs and errors will increasingly affect 

household production and leisure. Whereas these software 

problems do not directly affect economic metrics such as 

GDP, they do affect social welfare and continue to limit the 

adoption of new computer applications. 

R-1 

 References 

AmericanBanker.com. 1999. <http://www.americanbanker.com/ 

PSUser/ABO_Display.htm?type=RankingBanks& 

master=1999/Holding/ACHYE1999.html>. 

Andersson, M., and J. Bergstrand. 1995. “Formalizing Use Cases 

with Message Sequence Charts.” Unpublished Master’s 

thesis. Lund Institute of Technology, Lund, Sweden. 

Apfelbaum, L., and J. Doyle. 1997. “Model Based Testing.” 

Presented at the Software Quality Week Conference, May. 

Bank for International Settlements (BIS). 2000. Statistics on 

Payment Systems in the Group of Ten Countries. February. 

Barron, Cheryll Aimee. December 6, 2000. “High Tech’s 

Missionaries of Sloppiness.” <http://www.salon.com/tech/ 

feature/2000/12/06/bad_companies/print.html>. 

Baziuk, W. 1995. “BNR/NORTEL Path to Improve Product Quality, 

Reliability, and Customer Satisfaction.” Presented at the 

Sixth International Symposium on Software Reliability 

Engineering, Toulouse, France, October. 

Beizer, B. 1984. Software System Testing and Quality Assurance. 

New York: Van Nostrand Reinhold Company, Inc. 

Beizer, B. 1990. Software Testing Techniques. Boston: 

International Thomson Computer Press. 

Bentley Systems Incorporated. Corporate Backgrounder obtained 

November 1999. <http://www.bentley.com/bentley/ 

backgrnd.htm>. 

Besanko, David, David Dranove, and Mark Shanley. 1996. The 

Economics of Strategy. New York: John Wiley & Sons. 

Black, B.E. 2002. “Automatic Test Generation from Formal 

Specifications.” <http://hissa.nist.gov/~Black/ 

FTG/autotest.html>. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

R-2 

Boehm, B.W. 1976. “Software Engineering.” IEEE Transactions on 

Computer SE-1(4):1226-1241. 

Boehm, B.W. 1978. Characteristics of Software Quality. New 

York: American Elsevier. 

Booker, E. 1999. “In Focus: Enterprise Application 

IntegrationMiddleware Apps Scale Firewalls.” 

Internetweek 765. 

Carnegie Mellon Software Engineering Institute. Capability Maturity 

Model for Software (SW-CMM). 

<http://www.sei.cmu.edu/cmm/>. Last modified April 24, 

2002. 

Census. 2000. 1997 Finance and Insurance Economic Census. 

CIMdata. 2000. <http://www.cimdata.com/PR000307B.htm>. 

Cisco Systems. Cisco 2001 Annual Report. 2001. 

<www.cisco.com/warp/public/749/ar2001/online/financial_r

eview/mda.html>. 

Clarke, R. 1998. Electronic Data Interchange (EDI): An 

Introduction. <http://www.anu.edu.au/pepple/ 

Roger.Clarke/EC/EDIIntro.html>. 

Cohen, D.M., S.R. Dalal, J. Parelius, and G.C. Patton. 1996. “The 

Combinatorial Design Approach to Automatic Test 

Generation.” IEEE Software 13(5). 

Daratech, Inc. 1999. CAD/CAM/CAE Market Trends and Statistics, 

Vol. 1. Cambridge, MA. 

The Dell’Oro Group. 2001. <www.delloro.com>. 

Dixit, Avinash K., and Robert S. Pindyck. 1994. Investment under 

Uncertainty. Princeton, New Jersey: Princeton University 

Press. 

Economist. May 20, 2000. Survey: Online Finance, Paying 

Respects. <http://www.economist.com/ 

displayStory.cfm?Story_ID=3009189>. 

EDI Aware. 1994. The ABC of EDI. 

<http://www.edi.wales.org/feature4.htm>. 

Executive Office of the President, OMB, 1998. North American 

Industry Classification System, United States, 1997. 

Lanham, MD: Bernan Press. 

Freeman, Chris, and Luc Soete. 1999. The Economics of Industrial 

Innovation, 3rd edition. London: Cassell. 
References 

R-3 

Gallagher, L. 1999. Conformance Testing of Object-Oriented 

Components Specified by State/Transition Classes. 

<ftp://xsum/sdct.itl.nist.gov/sysm/NISTIR6592.pdf>. 

Gascoigne, Bill. 1995. ”PDM: The Essential Technology for 

Concurrent Engineering.” <http://www.PDMIC.com/ 

articles/index.html>. 

Hailpern, B., and P. Santhanam. 2002. “Software Debugging, 

Testing, and Verification.” IBM Systems Journal 41(1). 

International Business Machines (IBM). 1998. Annual Report: 

Management Discussion. <<http://www.ibm.com/ 

annualreport/1998/discussion/ibm98armd04.html>. 

IDC. 2000. Gigabit Router, Switch Purchases Soar During 1Q00. 

<http://www.idc.com/communications/press/pr/ 

CM061200PR.stm>. 

InformationWeek.com. 2002. “New Consortium to Target Software 

Quality.” May 16, 2002. 

Institute of Electrical and Electronics Engineers (IEEE). 1988. “IEEE 

Guide for the Use of IEEE Standard Dictionary of Measures 

to Produce Reliable Software.” New York: Institute of 

Electrical and Electronics Engineers. 

Institute of Electrical and Electronics Engineers (IEEE). 1996. “IEEE 

Software Engineering Collection: Standard Dictionary of 

Measures to Produce Reliable Software (IEEE Computer 

Society Document).” New York: Institute of Electrical and 

Electronics Engineers, Inc. 

Institute of Electrical and Electronics Engineers (IEEE). 1998. “IEEE 

Standard for Software Quality Metrics Methodology.” New 

York: Institute of Electrical and Electronics Engineers, Inc. 

Institute of Electrical and Electronics Engineers/American National 

Standards Institute (IEEE/ANSI). 1993. “Software 

Reliability.” Washington, DC: American Institute of 

Aeronautics and Astronautics.

ISO-9126 International Organization for Standardization. 1991. 

Information Technology Software Product 

EvaluationQuality Characteristics and Guidelines for their 

Use. Geneva, Switzerland: International Organization for 

Standardization. 

ITToolbox. 1999. “ITToolbox Knowledge Bank Forums. RE: SAP 

4.5B Test Scripts.” <http://www.sapassist.com/forum/ 

message.asp?i=3346&mo=&yr=&h1=306&h2=355>. 

Jones, C. 1997. Software Quality-Analysis and Guidelines for 

Success. Boston: International Thompson Computer Press. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

R-4 

Just, Richard E., Darrell L. Hueth, and Andrew Schmitz. 1982. 

Applied Welfare Economics and Public Policy. Englewood 

Cliffs, NJ: Prentice-Hall, Inc.

Kit, E. 1995. Software Testing in the Real World: Improving the 

Process. Reading, MA: ACM Press Addison Wesley 

Longman, Inc. 

Liebowitz, Stan J., and Stephen E. Margolis. 1999. “Causes and 

Consequences of Market Leadership in Application 

Software.” Paper presented at the conference Competition 

and Innovation in the Personal Computer Industry. April 24, 

1999. 

The MacNeal-Schwendler Corporation. 1998. Annual Report. 

<http://www.mscsoftware.com/ir/annual.html>. 

McCabe, T., and A. Watson. December 1994. “Software 

Complexity.” Cross talk, Journal of Defense Software 

Engineering 7(12):5-9. 

McCall, J., P. Richards, and G. Walters. 1977. Factors in Software 

Quality, NTIS AD-A049-014, 015, 055. November. 

Mentor Graphics Corporation. 1998. Annual Report. 

<http://www.mentorg.com/investor_relations/ 

MentorAnnual.pdf>. 

Michel, R. 1998. “Putting NT to the Test.” Manufacturing Systems

16(3):A18-A20. 

Myers, G.J. 1979. The Art of Software Testing. London: John 

Wiley & Sons. 

NACHA: The Electronic Payment Association. 2000. 

<www.nacha.org>. 

NASA IV&V Center, Fairmount, West Virginia. 2000. 

National Institute of Standards and Technology (NIST). 1997. 

Metrology for Information Technology (IT). 

<http://www.nist.gov/itl/lab/nistirs/ir6025.htm>. 

National Science Foundation (NSF). 1999. Research and 

Development in Industry: 1997. 

Offlutt, R.J., and R. Jeffery. 1997. “Establishing Software 

Measurement Programs.” IEEE Software 14(2):45-53. 

P.C. Webopaedia. 1996. <http://webopedia.internet.com>. 

Parametric Technology Corporation. 1998. Annual Report. 

<http://www.ptc.com/company/pmtc/1998/index.htm> As 

obtained October 1999. 
References 

R-5 

Perry, W.E. 1995. Effective Methods for Software Testing. New 

York: John Wiley & Sons, Inc. 

Pressman, R.S. 1992. Software Engineering: A Practitioner’s 

Approach, Third Edition. New York: McGraw-Hill. 

Pro/ENGINEER. 1999. <http://www.ptc.com/proe/ 

overview/index.html>. 

Product Data Management Information Center (PDMIC). 

<http://www.pdmic.com>. As obtained on March 13, 2000. 

Rivers, A.T., and M.A. Vouk. 1998. “Resource, Constrained Nonoperational Testing of Software.” Presented at the Ninth 

International Symposium on Software Reliability 

Engineering, Paderborn, Germany, November 4-7.

Robinson, William, Gurumurthy Kalyanaram, and Glen L. Urban. 

1994. “First-Mover Advantages from Pioneering New 

Markets: A Survey of Empirical Evidence.” Review of 

Industrial Organization 9(1):1-23.

Shea, Billie. July 3, 2000. “Software Testing Gets New Respect.” 

InformationWeek. 

Sullivan, Bob. 1999. “Online Banking Systems Crash.” 

MSNBC.com. <http://www.zdnet.com/zdnn/stories/news/ 

0%2C4586%2C2249362%2 C00.html?chkpt=zdnnsmsa>. 

System Transformation. 2000. Contingency Planning 

Methodology. Appendix D: Contingency Planning Option. 

<http://www.systemtransformation.com/cpgappdxd.htm>. 

Tai, K.C., and R.H. Carver. 1995. “A Specification-Based 

Methodology for Testing Concurrent Programs.” In 1995 

Europe Software Engineering Conference, Lecture Notes in 

Computer Science, W. Schafer and P. Botella, eds., pp. 

154-172. 

Tassey, G. 1997. The Economics of R&D Policy. Westport, CT: 

Quorum Books. 

U.S. Census Bureau. 1999av. “Automobile Manufacturing” from 

Manufacturing—Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3361a.pdf>. 

U.S. Census Bureau. 1999aw. “Light Truck and Utility Vehicle 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3361b.pdf>. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

R-6 

U.S. Census Bureau. 1999ax. “Heavy Duty Truck Manufacturing” 

from Manufacturing—Industry Series, 1997 Economic 

Census. 

<http://www.census.gov/prod/ec97/97m3361c.pdf>. 

U.S. Census Bureau. 1999ay. “Motor Vehicle Body 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3362a.pdf>. 

U.S. Census Bureau. 1999az. “Truck Trailer Manufacturing” from 

Manufacturing—Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3362b.pdf>. 

U.S. Census Bureau. 1999ba. “Motor Home Manufacturing” from 

Manufacturing—Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3362c.pdf>. 

U.S. Census Bureau. 1999bc. “Travel Trailer & Camper 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3362d.pdf>. 

U.S. Census Bureau. 1999bd. “Carburetor, Piston, Piston Ring & 

Valve Manufacturing” from Manufacturing—Industry Series, 

1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3363a.pdf>. 

U.S. Census Bureau. 1999be. “Gasoline Engine & Engine Parts 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3363b.pdf>. 

U.S. Census Bureau. 1999bf. “Vehicular Lighting Equipment 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3363c.pdf>. 

U.S. Census Bureau. 1999bg. “Other Motor Vehicle Electrical & 

Electronic Equipment Manufacturing” from Manufacturing—

Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3363d.pdf>. 

U.S. Census Bureau. 1999bh. “Motor Vehicle Steering & 

Suspension Component Manufacturing” from 

Manufacturing—Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3363e.pdf>. 
References 

R-7 

U.S. Census Bureau. 1999bi. “Motor Vehicle Brake System 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3363f.pdf>. 

U.S. Census Bureau. 1999bj. “Motor Vehicle Transmission & 

Power Train Parts Manufacturing” from Manufacturing—

Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3363g.pdf>. 

U.S. Census Bureau. 1999bk. “Motor Vehicle Seating and Interior 

Trim Manufacturing ” from Manufacturing—Industry Series, 

1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3363h.pdf>. 

U.S. Census Bureau. 1999bl. “Motor Vehicle Metal Stamping” 

from Manufacturing—Industry Series, 1997 Economic 

Census. 

<http://www.census.gov/prod/ec97/97m3363i.pdf>. 

U.S. Census Bureau. 1999bm. “Motor Vehicle Air-Conditioning 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3363j.pdf>. 

U.S. Census Bureau. 1999bn. “All Other Motor Vehicle Parts 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3363k.pdf>. 

U.S. Census Bureau. 1999bo. “Aircraft Manufacturing” from 

Manufacturing—Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3364a.pdf>. 

U.S. Census Bureau. 1999bp. “Aircraft Engine & Engine Parts 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3364b.pdf>. 

U.S. Census Bureau. 1999bq. “Railroad Rolling Stock 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3365a.pdf>. 

U.S. Census Bureau. 1999br. “Ship Building and Repairing” from 

Manufacturing—Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3366a.pdf>. 

U.S. Census Bureau. 1999bs. “Boat Building” from 

Manufacturing—Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3366b.pdf>. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

R-8 

U.S. Census Bureau. 1999bt. “Motorcycle, Bicycle & Parts 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3369a.pdf>. 

U.S. Census Bureau. 1999bu. “Military Armored Vehicle, Tank & 

Tank Component Manufacturing” from Manufacturing—

Industry Series, 1997 Economic Census. 

<http://www.census.gov/prod/ec97/97m3369b.pdf>. 

U.S. Census Bureau. 1999bv. “All Other Transportation Equipment 

Manufacturing” from Manufacturing—Industry Series, 1997 

Economic Census. 

<http://www.census.gov/prod/ec97/97m3369c.pdf>. 

U.S. Census Bureau. December 1999bx. “1997 Economic Census, 

Professional, Scientific, and Technical Services.” 

Geographic Area Series. 

U.S. Department of Commerce, International Trade Administration. 

1998. U.S. Industry & Trade Outlook ’98. New York: 

McGraw-Hill Companies. 

U.S. Department of Commerce, Economics and Statistics 

Administration, U.S. Census Bureau. February 2002. 

Annual Survey of Manufacturers: Statistics for Industry 

Groups and Industries: 2000. MOO(AS)-1. 

U.S. Department of Labor, Bureau of Labor Statistics (BLS). 2002. 

Occupational Outlook Handbook, 2002-03 Edition. 

Available at <http://www.bls.gov/oco/home.htm>. 

Unigraphics Solutions Incorporated website. 

<http://www.ug.eds.com>. 

Visio Corporation. 1998. Annual Report. 

<http://www.visio.com/files/ar98/Visar98.pdf>. 

Voas, J.M., and M. Friedman. 1995. Software Assessment: 

Reliability, Safety, and Testability. New York: Wiley & 

Sons, Inc. 

Voas, J.M. 1998. Software Fault Injection Inoculating Programs 

Against Errors. New York: John Wiley & Sons, Inc. 

Vouk, M.A. 1992. “Using Reliability Models During Testing with 

Non-Operational Profiles.” Presented at the Second 

Workshop on Issues in Software Reliability Estimation, 

Livingston, NJ, October 12-13. 

Washington Technology. 1998. Selected Security Events in the 

1990s. <http://wtonline.com/vol13_no18/ 

special_report/277-1.html>. 
References 

R-9 

Wilson, R.C. 1995. UNIX Test Tools and Benchmarks. New 

Jersey: Prentice Hall, Inc. 

A-1 

 Appendix A: 

 Glossary of Testing 

 Stages and Tools 

 A.1 GENERAL TESTING STAGES 

Subroutine/Unit Testing. This stage includes subroutine and unit 

testing. Software developers perform subroutine testing, the lowest 

form of testing, as they write the program. Programmers test a 

completed subroutine to see if it performs as expected. Unit testing 

is the testing of a complete module or small program that will 

normally range from perhaps 100 to 1,000 source lines of code. 

Although unit testing may often be performed informally, it is the 

stage where test planning and test case construction begins. 

New Function Testing. Developers use this stage to validate new 

features that are being added to a software package. Often used in 

conjunction with regression testing, new function testing is 

commonly used when existing applications are being updated or 

modified. 

Regression Testing. Regression testing is used to ensure that 

existing software functions of the product have not been 

accidentally damaged as an unintended by-product of adding new 

software features. As software evolves, regression testing becomes 

one of the most important and extensive forms of testing because 

the library of available test cases from prior releases continues to 

grow. 

Integration Testing. This stage focuses on testing groups of 

modules, programs, applications, or systems that developers 

combine to form a larger system. Integration testing focuses on 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

A-2 

testing for interoperability among the integrated elements of the 

software product. 

System Testing. This stage involves testing the system as a whole. 

System testing is typically performed by the software developer’s test 

personnel and is usually the last form of internal testing performed 

by the software developer before customers get involved with field 

testing (beta testing). 

 A.2 SPECIALIZED TESTING STAGES 

Stress, Capacity, or Load Testing. These stages judge the ability of 

an application or system to function when near or beyond the 

boundaries of its specified capabilities or requirements in terms of 

the volume of information used. The stress, load, or capacity testing 

stage is often considered synonymous with the performance testing 

stage. Stress testing attempts to break the system by overloading it 

with large volumes. It is usually performed by the software 

developer after, or in conjunction with, integration or system 

testing. Typically stress testing cannot be performed earlier because 

the full application is usually necessary. Although the following 

specialized testing stages are not considered stress testing, they also 

test how the system will perform under adverse conditions. 

Error-Handling/Survivability Testing. This stage assesses the 

software product’s ability to properly process incorrect transactions 

and survive from reasonably expected (or unexpected) error 

conditions. 

Recovery Testing. This stage assesses the software product’s ability 

to restart operations after integrity of the application has been lost. 

Security Testing. Security testing is used to evaluate whether a 

software product can adequately prevent improper access to 

information. Security testing is usually performed before and after 

the product has been released by testing personnel or by highly 

specialized consultants employed by the user (Perry, 1995). 

Performance Testing. This stage is used to determine whether an 

application can meet its performance goals (Jones, 1997). Typically 

the performance testing stage is executed by the software developer 

during, or in conjunction with, system testing. Benchmarks are 

standards against which other measurements may be referred and 
Appendix A — Glossary of Testing Stages and Tools 

A-3 

are used to provide competitive analysis information that marketing 

and sales personnel can use to give consumers measures of the 

software’s quality relative to other products (Wilson, 1995). 

Customers use marketing benchmarks to compare performance 

prior to purchase, whereas system architects and designers use 

technical benchmarks to characterize performance prior to 

manufacturing (Wilson, 1995). 

Platform Testing Stage. Sometimes known as the compatibility 

testing stage, platform testing evaluates the software’s ability to 

operate on multiple hardware platforms or multiple operating 

systems or to interface with multiple software products (Jones, 

1997). 

Viral Protection Testing Stage. Major commercial software 

developers typically conduct viral protection testing to ensure that 

master copies of software packages do not contain viruses (Jones, 

1997). 

 A.3 USER-INVOLVED TESTING STAGES 

Usability Testing. Also known as the human factors testing, this 

stage is conducted to identify operations that will be difficult or 

inconvenient for users. Usability testing is generally performed 

before beta testing. It involves observing actual clients who use the 

software product under controlled or instrumented conditions. 

Usability testing is common for large commercial software 

developers (Jones, 1997). 

Field or Beta Testing. This stage is an external test involving 

customers. Beta testing usually occurs after system testing. External 

beta testing and internal usability testing may occur concurrently. 

Beta testing may involve special agreements with clients to avoid 

the risk of lawsuits if the software product has serious problems 

(Jones, 1997). The next two testing activities are associated with, or 

have similar goals as, field testing. 

Lab or Alpha Testing. These activities are typically used when 

special laboratories are involved to house complex new 

hardware/software products that prospective customers will test. 

Customers test these products under controlled conditions prior to 

having the software system installed at their own premises. 

Software developers who build complex software systems primarily 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

A-4 

use lab testing. In these cases typical beta testing is infeasible 

because of hardware or software constraints. 

Acceptance Testing. This process is used to determine whether a 

product satisfies predefined acceptance criteria. It is a combination 

of other types of tests to demonstrate that the product meets user 

requirements. Customer acceptance testing is commonly performed 

for contract software and for large systems such as PDM software 

systems, but it is rarely used in high-volume commercial “shrink 

wrapped” software products. Sometimes, alpha and beta testing are 

considered a part of acceptance testing (Jones, 1997; Kit, 1995). 

 A.4 TEST DESIGN AND DEVELOPMENT TOOLS 

Data Dictionary Tools. These tools are documentation tools for 

recording data elements and the attributes of the data elements. 

Under some implementations, they can produce test data to validate 

the system’s data edits. 

Executable Specification Tools. These tools provide a high-level 

interpretation of the system specifications to create response test 

data. Interpretation of expected software products requires system 

specifications to be written in a special high-level language so that 

those specifications can be compiled into a testable program. 

Exhaustive Path-Based Tools. The purpose of these tools is to 

attempt to create a test transaction for every possible condition and 

every path in the program. 

Volume Testing Tools. Volume testing tools identify system 

restrictions (e.g., internal table size) and then create a large volume 

of transactions designed to exceed those limits. Thus, volume 

generators facilitate the creation of specific types of test data to test 

predetermined system limits to verify how the system functions 

when those limits are reached or exceeded (Perry, 1995). 

Requirements-Based Test Design. These tools facilitate a highly 

disciplined approach based on cause–effect graph theory to design 

test cases that will help ensure that the implemented system meets 

the formally specified requirements. 
Appendix A — Glossary of Testing Stages and Tools 

A-5 

 A.5 TEST EXECUTION AND EVALUATION TOOLS 

Capture/Playback Tools. These tools capture user operations 

including keystrokes, mouse activity, and display output. These 

captured tests, including the output that has been validated by the 

tester, form a baseline for future testing of product changes. The 

tool can then automatically play back the previously captured tests 

whenever needed and validate the results by comparing them to the 

previously saved baseline. This frees the tester from having to 

manually re-run tests over and over again when fixes, 

enhancements, or other changes are made to the product (Kit, 

1995). 

Test Harnesses and Drivers Tools. Used for performance testing, 

these tools invoke a program under test, provide test inputs, control 

and monitor execution, and report test results. 

Evaluation tools, also referred to as analysis tools, focus on 

confirming, examining, and checking results to verify whether a 

condition has or has not occurred. These include the following 

tools. 

Memory Testing Tools. These provide the ability to check for 

memory problems, such as overwriting and/or overreading array 

bounds, memory allocated but not freed, and reading and using 

uninitialized memory. Errors can be identified before they become 

evident in production and can cause serious problems. Detailed 

diagnostic messages are provided to allow errors to be tracked and 

eliminated. Memory testing tools are also known as boundscheckers, memory testers, run-time error detectors, or leak 

detectors. 

Instrumentation Tools. These measure the functioning of a system 

structure by using counters and other monitoring instruments. 

Snapshot Monitoring Tools. These show the content of computer 

storage at predetermined points during processing. These tools print 

the status of computer memory at predetermined points during 

processing when specific instructions are executed, or when data 

with specific attributes are processed. 

System Log Reporting Tools. These tools provide an audit trail of 

monitored events occurring in the environmental area controlled by 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

A-6 

system software. The information can be used for analysis purposes 

to determine how well the system performed. 

Coverage Analysis Tools. These tools use mathematical 

relationships to demonstrate what percentage of the software 

product the testing process has covered. The resulting qualitative 

metric is used for predicting the effectiveness of the test process. 

This tool informs testers about which parts of the product have been 

tested and which parts have not. 

Mapping Tools. They analyze which parts of a computer program 

are exercised during the test and the frequency of execution of each 

statement or routine in a program. Mapping tools can be used to 

detect system flaws, determine how much of a program is executed 

during testing, and identify areas where more efficient code may 

reduce execution time. 

Simulation tools are also used to test execution. Simulation tools 

take the place of software or hardware that interacts with the 

software to be tested. Sometimes they are the only practical method 

available for certain tests, like when software interfaces with 

uncontrollable or unavailable hardware devices. These include the 

following tools. 

Disaster Testing Tools. These tools emulate operational and/or 

system failures to determine if the software product can survive 

or be correctly recovered after the failure. 

Modeling Tools. Modeling tools simulate the functioning of the 

software system and/or its environment to determine how 

efficiently the proposed system solution will achieve the system 

objectives. 

Symbolic Execution Tools. These tools are used to identify 

processing paths by testing the programs with symbolic rather 

than actual test data. The symbolic execution results in an 

expression that can be used to evaluate the completeness of the 

programming logic. It is a technique that does not require test 

data. 

System Exercisers. These tools stress or load subsystem 

components or physical devices by focusing on consuming 

critical system resources such as peripherals, memory, and CPU. 
Appendix A — Glossary of Testing Stages and Tools 

A-7 

For example, multiuser resource exercisers simulate full or 

maximum workload for several users (Wilson, 1995). 

 A.6 ACCOMPANYING AND SUPPORT TOOLS 

Code Comprehension Tools. These tools help us understand 

unfamiliar code. They improve understanding of dependencies, 

trace program logic, view graphical representations of the program, 

and identify areas that should receive special attention, such as 

areas to inspect. 

Flowchart Tools. Flowchart tools are used to graphically represent 

the system and/or program flow to evaluate the completeness of the 

requirements, design, or program specifications. 

Syntax and Semantic Analysis Tools. These tools perform extensive 

error checking to find errors that a compiler would miss, and they 

are sometimes used to flag potential defects before or during formal 

testing. 

Problem Management Tools. Problem management tools are 

sometimes called defect tracking tools, bug management tools, and 

incident control systems and are used to record, track, and generally 

assist with the management of defects and enhancements 

throughout the life cycle of software products. These include 

system control audit databases, scoring databases, and configuration 

management tools. 

B-1 

 Appendix B: 

 CAD/CAM/CAE/PDM 

 Use and 

 Development in the 

 Transportation 

 Sector 

The appendix provides background on the users of 

CAD/CAM/CAE/PDM software in the transportation sector and the 

vendors that supply the software systems. 

 B.1 TRANSPORTATION EQUIPMENT 

MANUFACTURERS (SECTOR 336) 

Establishments in this sector of the economy manufacture motor 

vehicles, ships, aircraft, railroad cars and locomotives, and other 

transportation equipment. An estimated 13,206 establishments in 

the U.S. produce transportation equipment. Their products include 

the following: 

Z motor vehicles (sector 3361) (e.g., complete automobiles 

and light duty motor vehicles [i.e., body and chassis or 

unibody], chassis); 

Z motor vehicle body and trailer manufacturing (sector 3362) 

(e.g., motor vehicle bodies and cabs; truck, automobile, and 

utility trailers, truck trailer chassis, detachable trailer bodies 

and chassis); 

Z motor vehicle parts (sector 3363) (e.g., new and rebuilt 

motor vehicle gasoline engines, engine parts; vehicular 

lighting equipment; motor vehicle electrical and electronic 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-2 

equipment; motor vehicle steering mechanisms and 

suspension components; motor vehicle brake systems and 

related components; motor vehicle transmission and power 

train parts; motor vehicle seating, seats, seat frames, seat 

belts, and interior trimmings; motor vehicle fenders, tops, 

body parts, exterior trim and molding; other motor vehicle 

parts and accessories); 

Z aerospace products and parts (sector 3364) (e.g., aerospace 

engines, propulsion units, auxiliary equipment and parts, 

prototypes of aerospace parts, converted aircraft, restored 

aircraft or propulsion systems); 

Z railroad rolling stock (sector 3365) (e.g., new and rebuilt 

locomotives, locomotive frames and parts; railroad, street 

and rapid transit cars and car equipment; rail layers, ballast 

distributors, rail tamping equipment, and other railway track 

maintenance equipment); 

Z ship and boat building (sector 3366) (e.g., new and rebuilt 

barges, cargo ships, drilling and production platforms, 

passenger ships, submarines, dinghies [except inflatable 

rubber], motorboats, rowboats, sailboats, yachts); and 

Z other transportation equipment (sector 3369) (e.g., 

motorcycles, bicycles, metal tricycles, complete military 

armored vehicles, tanks, self-propelled weapons, vehicles 

pulled by draft animals, and other transportation equipment 

not classified in sectors 3361-3366). 

In such a broad sector, many factors affect industry trends and the 

need for product innovation. This section highlights trends in two 

sectors of the transportation equipment industry: motor vehicles 

and aerospace. 

In the motor vehicle industry, more open trading policies and 

economies of scale make it efficient to use the same underpinnings, 

engines, and transmissions on different vehicle models produced in 

various parts of the world. In addition, the globalization of the 

industry means that the U.S. is competing with more recently 

industrialized nations that may have newer equipment and face a 

lower-paid labor force and less government regulation. The U.S. 

motor vehicle industry needs improved design technology to 

facilitate better communication between the parts producers and 

assemblers located in different parts of the world, to speed up the 

design process and to increase overall productivity (U.S. 

Department of Commerce, 1998). 

Growth of the U.S. aerospace industry is currently affected by 

constrained defense spending, foreign competition, investment in 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-3 

research and development, increased productivity, and 

technological innovation. For the civil aircraft industry, the 

importance of exports requires the expansion of foreign markets for 

future growth. At the same time, competition from foreign suppliers 

will challenge the U.S. aerospace industry’s global market share. 

Foreign research and development spending on aerospace 

technology is often supported by government policies. However, 

the recent GATT Aircraft Agreement should limit government 

intervention in the civil aircraft industry, placing the U.S. on more 

even footing with newer, foreign aircraft industries (U.S. Department 

of Commerce, 1998). 

Manufacturers of transportation equipment spent more than $718 

million on software and data processing services in 1997 (U.S. 

Census Bureau, 1999av through 1999bv) (24 six-digit sectors 

reporting out of 30). Computer-aided design (CAD) and mechanical 

computer-aided engineering (CAE) software is vital to this industry 

as manufacturers are attempting to meet demand for state-of-the-art 

design in record time. Auto manufacturers, for example, desire to 

shorten the product design process to as little as 24 months (U.S. 

Department of Commerce, 1998). CAD/CAE software allows quick 

design, quick design adjustments, simulation without prototype 

production, and easy transmission of product design information to 

every member of the product development team. Manufacturers of 

the Boeing 777 used CATIA in their design process and found the 

inherent software capabilities to be very important in letting the 

design and build teams see how all components and systems of the 

aircraft fit and work together before manufacturing began (U.S. 

Department of Commerce, 1998). 

Figures B-1a, b, and c provide examples of the ability of CAD and 

CAE software to enhance the design process for automobiles. The 

figures are CAD visualizations of “the Rocket,” designed by George 

Balaschak for a customer to display at the Geneva Auto Show. 

Balaschak’s three-dimensional Pro/ENGINEER software from 

Parametric Technology Corporation created the transparent and 

cutaway models in Figure B-1a as well the model for the body 

molds as shown in Figures B-1b and c. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-4 

Source: Pro/Engineer. 1999. <http://www.ptc.com/proe/overview/index.html>. 

Source: Pro/Engineer. 1999. <http://www.ptc.com/proe/overview/index.html>. 

Source: Pro/Engineer. 1999. <http://www.ptc.com/proe/overview/index.html>. 

Figure B-1a. Transparent 

and Cutaway Views of 

the Solid Model 

Pro/ENGINEER provides threedimensional visualization of 

“the Rocket” design. 

Figure B-1b. The Shaded 

Model of the Mold Used 

to Fabricate the Engine 

Hood Panel 

Pro/ENGINEER aided Mr. 

Balaschak in designing body 

molds. 

Figure B-1c. The Main 

Body Mold Was Machined 

in Four Sections and 

Then Assembled 

This view shows three of the 

mold sections. 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-5 

 B.2 CAD/CAM/CAE AND PDM SOFTWARE 

PRODUCTS AND THEIR CHARACTERISTICS 

Software provides the instructions that lead computer hardware to 

perform desired processes. It is the interface between computer 

users and computer processors and peripheral equipment. Software 

has a higher degree of specificity than the hardware on which it is 

run. That is, while software is written to perform a specific task or 

closely related set of tasks, the computer may be able to perform a 

wide variety of tasks depending on the software employed. 

There are two broad forms of software: systems software and 

applications software. Systems software controls, manages, and 

monitors computer resources and organizes data. Operating 

systems, compilers and interpreters, communications software, and 

database management systems are all types of systems software. 

Applications software instructs computers in performing more 

specific tasks such as word processing, graphic design, and 

accounting functions (Freeman and Luc Soete, 1999). 

CAD/CAM/CAE and PDM software are types of packaged 

applications software used to perform complex design and 

engineering functions. CAD/CAM/CAE software is a point tool in 

the product development cycle. PDM is a life-cycle software tool 

that manages the flow of information and data from one point tool 

to another point tool. 

 B.2.1 CAD/CAM/CAE Software Products 

CAD, CAM, and CAE refer to functions that a computer and 

peripheral equipment may perform for a user with the aid of 

application software. 

CAD software functions enable users to design products and 

structures with the aid of computer hardware and peripherals more 

efficiently than with traditional drafting technologies. The user 

creates a computer image of a two-dimensional or threedimensional design using a light pen, mouse, or tablet connected to 

a workstation or personal computer. The design can be easily 

modified. It can be viewed on a high-quality graphics monitor from 

any angle and at various levels of detail, allowing the user to readily 

explore its physical features. Designers can use CAD software to 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-6 

integrate drawings in such a way that adjusting one component 

alters every attached component as necessary. 

CAM software functions allow a manufacturer to automate 

production processes. CAM software includes programs that create 

instructions for manufacturing equipment that produces the design. 

In addition, the software provides instructions to other computers 

performing real-time control of processes, in using robots to 

assemble products, and in providing materials requirements 

associated with a product design (P.C. Webopaedia, 1996). 

CAE software functions allow users to conduct engineering analyses 

of designs produced using CAD applications to determine whether a 

product will function as desired. The engineering analysis may 

involve simulating the eventual operating conditions and 

performance of a designed product or structure. Or users can 

analyze the relationships between design components. 

Until the mid-1980s, CAD/CAM/CAE software was available only 

on computers constructed especially to perform the complex and 

specific design, engineering, and manufacturing functions a firm 

might need (P.C. Webopaedia, 1996). Now, the software is also 

sold for use on personal computers and more general-purpose 

workstations. 

A small number of software packages dominate the market for 

CAD/CAM/CAE software. Each of the leading software packages 

stores product designs in a unique file format. These software 

packages can be called software design platforms. Some software 

design platforms include translation programs that convert a file into 

a new format to be used with a different software package. 

However, all translations are somewhat imperfect. As a result, 

smaller software developers who wish to meet the unique demands 

for product “add-ons” or “plug-ins” usually license design formats 

from leading software design platform developers to ensure 

compatibility. 

The presence of a few dominant software applications could be 

explained by one of two phenomena: “lock-in,” as a result of 

switching costs, and quality domination, as a result of “instant 

scalability.” Lock-in occurs when software users continue to use an 

inferior product because of the high, up-front cost of switching to a 

superior one. Switching costs arise when learning is involved, as 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-7 

there is with all experience goods, and when that learning is not 

costlessly transferable to the alternative product. These costs may 

also exist because of network externalities. This phenomenon arises 

when incumbent users of a product receive welfare increases when 

additional consumers purchase the commodity. For example, as 

more firms use a particular piece of software, the firm that 

developed this software has an incentive to improve this product. 

These improvements accrue to the newly adopting firms as well as 

the incumbent users. Buyer switching costs can be an important 

source of competitive advantage enjoyed by “early movers”—firms 

that are first to market with a new product or design. “Lock-in” may 

be present in the CAD/CAM/CAE industry for several reasons: 

Z A firm using the same CAD/CAM/CAE design platform on 

multiple machines will find it costly to add a new type of 

software with a file format incompatible with the old 

software and its file formats. 

Z A firm that has used one software package consistently for 

years will lose the benefits of training and experience 

specific to that software package. 

Z By changing the CAD/CAM/CAE design platform, firms may 

lose complete or partial access to their historical data files. 

Z Already established CAD/CAM/CAE design platforms are 

likely to be complemented by a larger array of add-on 

software packages than are newly available software design 

platform. 

In contrast with the lock-in explanation for the limited number of 

CAD/CAM/CAE software products and few new market entrants, it is 

possible that markets are, in fact, dominated by the highest-quality 

software applications. Quality domination is an especially pertinent 

theory in examining software market domination because software 

production benefits from instant scalability—extra copies of 

software applications can be copied quickly and cheaply with the 

aid of relatively inexpensive CD-ROM duplicators (Liebowitz and 

Margolis, 1999). Because of the ease of reproducing software 

products, a high-quality product can quickly take over the market 

since its production can easily be scaled up to meet demand. 

Liebowitz and Margolis find that case studies and empirical research 

support the explanation of quality domination rather than lock-in in 

the market for software applications. 

Table B-1 identifies the dominant CAD/CAM/CAE software design 

platforms and describes how they are used in industry. The table 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-8 

Table B-1. Dominant CAD/CAM/CAE Software Products 

Several companies produce CAD/CAM/CAE software design platforms. 

Product Name Product Description Sources 

Bravo Mechanical design software with solid, surface, 

wireframe, piping, HVAC, sheet metal, 2D and 3D 

modeling capabilities. Features top-down design 

and numerical control capabilities for 

manufacturing 

http://www.ug.eds.com/products/ 

bravo/introduction.html 

CADKEY 3D, 2D, solids and surface modeling. Designs 

created with other platforms imported as 

“geometry” so that they can be manipulated as if 

created in CADKEY. 

http://www.cadkey.com/products/ 

index.html – CADKEY 98 

brochure in Adobe Acrobat format 

CATIAa Includes solid, hybrid and sheet metal part design 

capabilities. Allows creation and modification of 

mechanical and freeform surfaces. Integrates 

electrical product design information with 

mechanical design. Allows simulation. 

http://www.catia.ibm.com/catv5/ 

newv5r3.html 

Formality Allows integrated circuit designers to compare a 

design at any stage of the design process with the 

original design to check for functional equivalence. 

http://www.sec.gov/Archives/ 

edgar/data/883241/0000891618-

98-005466.txt 

HLDA Plus Allows integrated circuit designers to translate a 

graphic design into a textual hardware design 

language. Then, the software allows for simulation 

and verification of the design. 

http://www.sec.gov/Archives/ 

edgar/data/925072/0001047469-

99-009976.txt 

Helixa Mid-range surface and solid modeling package 

using a kernel modeler and constraint manager. 

Includes a suite of geometric editing tools for 

creating and modifying models, investigating 

design alternatives, determining interferences and 

clearances and calculating mass properties. 

http://www.microcadam.com/ 

product/pages/hds.html 

I-DEAS Mechanical design software specifically for users 

needing solid modeling technology. 

http://www.sec.gov/Archives/ 

edgar/data/820235/0000906318-

99-000032.txt 

IntelliCAD 2D design and drafting software that is highly (but 

not perfectly) compatible with the AutoCAD file 

format. Works with add-ons designed for 

AutoCAD. 

http://www.visio.com/company/ 

indepth.html 

IronCAD Provides mechanical engineers with solid modeling 

capabilities and easy manipulation of 3D objects. 

Facilitates design modification at all stages of the 

design process. 

http://www.ironcad.com/ 

Mechanical Desktop Provides solid modeling, surface modeling, 2D 

design/drafting and bidirectional associative 

drafting capabilities. Translates Desktop files for 

exchange with other design systems and produces 

a bill of materials. 

http://www.sec.gov/Archives/ 

edgar/data/769397/0000929624-

99-000172.txt 

(continued) 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-9 

Table B-1. Dominant CAD/CAM/CAE Software Products (continued) 

Product Name Product Description Sources 

Microstation 

Modelera

Facilitates solid, surface, and detailed modeling 

using a Windows interface. Includes a 3D parts 

library and translators to enable designers to 

exchange data with users of different design 

systems. 

http://www.phillynews.com/ 

inquirer/99/Oct/11/business/ 

BENT11.html 

Parasolid A solid modeling technology designed to be 

portable and used with multiple design systems. 

http://www.ugsolutions.com/ 

products/parasolid/ 

Pro/ENGINEER Facilitates design of detailed solid and sheet metal 

components. Aids in building assemblies. 

Produces fully documented production drawings 

and photorealistic images of designed product. 

http://www.ptc.com/proe/ 

overview/index.html 

Seamless® CoVerification 

Environment (CVE) 

Detects errors in hardware/software interfaces in 

embedded systems before prototype fabrication. 

http://www.mentorg.com/ 

press_release/jan00/ 

seamless_pr.html 

Solid Edge Mechanical design and drafting software with 2D 

and 3D capabilities. Uses unique STREAM 

technology to improve speed, effectiveness, and 

usability of the software. 

http://www.solid-edge.com/ 

prodinfo/v7/ 

SolidDesigner Facilitates dynamic modeling (computer reshaping 

of design components when one reference 

component is changed). Allows freeform and solid 

modeling. Provides accessories to aid team design. 

http://www.hp.com/pressrel/ 

dec95/05dec95a.html 

SolidWorksa 3D product design software that functions on a 

Windows platform. Features wide range of 

interoperability with other mechanical design 

formats. 

http://www.solidworks.com/html/ 

Company/cprofile.cfm 

SpeedSim Integrated circuit simulation software. Uses cyclebased technology to reduce the time requirements 

for simulation. 

http://www.sec.gov/Archives/ 

edgar/data/914252/0001012870-

99-001140.txt 

Think3 A mid-range product providing solids modeling 

and advanced surfacing capabilities. Facilitates the 

conversion of 2D designs into 3D design using 

wireframe modeling. For Windows®95 or NT®. 

http://www.think3.com/content/ 

docs.content.specs.html 

Unigraphics 

(UG/Solid Modeling, 

UG/CamBase, etc.) 

Full range of design capabilities, including freeform 

modeling. Available modules provide advance 

graphics display, a part library system, a mold 

wizard, assistance in building numerical control 

machines, and more. 

http://ugsolutions.com/products/ 

unigraphics/cad 

Vectorworks 

(formerly MiniCAD) 

2D and 3D design capabilities. Includes a 

database spreadsheet function, report generation, 

and customizable programmability. 

http://www.sec.gov/Archives/ 

edgar/data/819005/0000819005-

99-000003.txt 

aProduct developed by a foreign software developer. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-10 

also describes a few of the dominant electronic design automation 

software packages used for electronic design, verification, and 

simulation. 

 B.2.2 PDM Software Products 

Traditional approaches to engineering are linear. Each project has a 

set of specific tasks, performed by different groups, in a specific 

order that must be accomplished before the project can be 

completed. The product development cycle is envisioned as a 

series of independent sequential steps that starts at the generation of 

the product design and proceeds in an orderly manner to the 

finished product. Information is passed from one stage to the next 

with minimal feedback. This model is referred to as serial 

engineering. However, this model is not completely accurate. In 

reality, changes and updates are made to each part of the 

development cycle that affect the other phases. If the product 

development process is linear, then the changes would only affect 

downstream phases. But modern production processes are not 

linear; changes are made to product designs after they have passed 

through each stage (Gascoigne, 1995). 

Serial engineering is poorly equipped to handle this dynamic 

process because, as a project advances, engineering changes pose 

greater and greater expenses in the form of time delays and cost 

increases. Design changes force the whole project back to the 

planning phase for modification. Each of the design steps must then 

be repeated, resulting in additional effort and increased time to 

market. 

The modern approach, concurrent engineering, addresses this 

problem. Instead of a serial development process, engineers from 

all stages of the development and production processes can work 

on the project at the same time. Changes at any stage in the 

production process are addressed immediately and are incorporated 

into the production process. Feedback loops occur as soon as the 

change is made, and all phases in the product development cycle 

adjust. This approach decreases the time to market of new 

products, reduces development time and costs, and improves 

product quality (Gascoigne, 1995). 

While product development speed can increase and costs decrease 

with concurrent engineering, a problem develops. In serial 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-11 

engineering, each unit works on its part of the project in isolation. 

Once the unit is finished, it is passed on to the next unit. The 

passage of information is orderly. In concurrent engineering, 

multiple units are working on the project at the same time, and it is 

difficult to pass information from one group to the next in an orderly 

manner. Monitoring who made changes, incorporating the changes 

into the product, and updating the changes are paramount activities 

in exploiting the potential of concurrent engineering. PDM supports 

these activities. It can be divided in two components: data 

management and process management. 

Data Management 

As engineering work has become reliant on CAD/CAM/CAE, greater 

volumes of data are being produced. As more data are generated, 

searching data to find specific pieces of information becomes 

increasingly costly. Independent of changes dictated by the shift to 

concurrent engineering, the shear increase in the volume of data 

that is generated by shifting to computer-aided production 

techniques necessitated a change in the way data are handled and 

manipulated. Data management in PDM is concerned with 

monitoring and controlling the information generated in 

CAD/CAM/CAE. It is the aspect of the production process that is 

concerned with how individual users manipulate the data on which 

they are currently working. Data management is static in that it 

monitors the data at a particular point in time, but it does not 

monitor how it is being changed or altered through time. 

PDM can manage all of the product-related information generated 

throughout the product life-cycle. PDM creates a master document 

that can be logged out and held in a secure location. Other 

engineers working on the project can access a duplicate copy that 

they can use in their work. Whenever changes are made to the 

master copy, all users are notified and the copy that they are using 

is updated to reflect any changes. PDM tools focus on automating 

existing processes and managing electronic documentation, files, 

and images. PDM is used almost exclusively in CAD/CAM/CAE 

systems. 

Data management in PDM monitors both the attributes of the files 

as they change through time as well as the documentary information 

associated with any changes. Monitoring is widely defined and 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-12 

includes classification of components in the design, classification of 

the documents that have been produced, the structure of the 

product being produced, and a system for querying the data.

Process Management 

Process management systems encompass three functions: 

Z managing data when someone is working on it (work 

management), 

Z managing the flow of data between two people (workflow 

management), and 

Z tracking the changes to the data (work history management) 

(PDMIC, 2000). 

Process management is the dynamic aspect of PDM—it is 

concerned with the movement and transformations of data as it 

moves from one user to another. 

Engineers and developers are constantly changing and updating the 

product throughout the production process. Work management 

within PDM monitors and tracks the changes made to the data. It 

organizes the information and implications for other parts of the 

production process that are created by changes that one engineer 

makes to the product in different areas. Work management tracks 

every footstep, and the implications from those footsteps, that the 

engineer makes in the production process. 

Workflow management focuses on the movement of information 

across units within an organization. How information is passed 

back and forth between the units is the realm of workflow 

management. Workflow management bundles the project in logical 

units, often called packets, of information that allow each unit to 

work on the appropriate sections. When changes are made to each 

packet, information is then sent to all of the other units that need to 

know about the change. Workflow management tracks the changes 

that are made that determine what group or units need to see the 

data after a change has been made. 

Work history management tracks all of the changes that have been 

made by individual units or departments and how those changes 

have affected other units or departments. It captures and records 

the entire history of the project and all of the team members of the 

project. Work history management can then be used for auditing 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-13 

the production process as well as evaluating specific units with the 

production process. 

Benefits from PDM 

The most frequently cited benefit of PDM is reduction in time to 

market for new products. The time reduction from PDM occurs in 

several ways. First, the time to perform the overall task is decreased 

because data are made available as soon as they are needed. 

Second, because of concurrent engineering, bottlenecks do not 

develop in the production process because no queue exists in the 

project development process. Third, the feedback from changes is 

almost immediate, and all units know they are working on the latest 

version of the product this decreases the amount of time spent on 

corrections and reworking. Improved feedback loops have an 

additional advantage: by ensuring that all employees are working 

on the most recent version of the project and making changes 

available immediately, the risk of failure is also reduced. However, 

care still must be exercised. Just because the data are the most 

recent version does not mean the data are correct. 

In addition, PDM has the potential to generate other benefits. 

Because PDM reduces the amount of time spent searching for 

documents, checking the freshness of each document, and 

reworking existing products, engineers are able to spend more time 

on designing products and developing new and innovative ideas. 

Historically, over 25 percent of a designer’s effort is consumed by 

reworking or searching for documentation. PDM has the potential 

to substantially reduce this percentage (PDMIC, 2000). 

Another benefit from PDM is its ability to leverage knowledge from 

other products. Many problems already have a solution; it is a 

question of finding rather than rediscovering it. In the traditional 

approach to product development, it was often easier to reinvent an 

existing process or idea rather than track down an existing solution. 

Because PDM organizes existing knowledge and allows for easy 

searches of that knowledge, existing solutions will be easier to find; a 

shift from customized production to component production occurs. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-14 

 B.3 THE DEVELOPMENT AND DEVELOPERS OF 

CAD/CAM/CAE AND PDM SOFTWARE 

Two major groups of firms are involved in the development of 

CAD/CAM/CAE and PDM products, the developers of the software 

product and the testers of the software product. 

 B.3.1 Software Publishers (Sector 5112) 

Software publishers produce and distribute software of all types. 

Our focus is on the subset of the industry that produces the 

CAD/CAM/CAE and PDM software products described in 

Section 6.2. 

CAD/CAM/CAE Firms 

CAD/CAM/CAE software developers include many establishments; 

however, about 20 firms dominate the market. These well-known 

software developers include those that produce the design platforms 

for CAD/CAE software and the most respected EDA software 

developers. Testing services may be provided by the developer or 

contracted for from specialized suppliers in the computer systems 

design and related services sector. 

Table B-2 lists the U.S. developers of the most widely used 

CAD/CAE design platforms as well as the prominent EDA software 

developers. In some cases, the current owner of the proprietary 

rights to a software package is not the original owner. However, 

because the owners of the proprietary rights develop upgrades and 

new releases of the original software package, they are designated 

as developers in Table B-2. Developers who concentrate only on 

AEC or GIS software are not listed because they are outside the 

scope of this study. In most cases, data in the table come directly 

from annual reports filed with the Securities Exchange Commission. 

Where this is not the case, the data source is noted. The table 

includes revenues, costs, and employment, with specific 

information on R&D expenses. 

As noted previously, development is large cost factor in the 

production of software. Table B-2 shows that 7 to 35 percent of the 

total costs of CAD/CAM/CAE software developers were for R&D. 

Information on R&D spending for other industries in recent years 

shows such spending to be proportionately higher in the software 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-15 

Table B-2. Developers of CAD/CAM/CAE Software, 1997 

CAD/CAM/CAE software developers spend a larger percentage of their total revenues on R&D than do other U.S. 

industries. 

Costs ($thousands) 

Employment 

(thousands) 

Company 

Total Revenues 

($thousands)a Totalb R&Dc

R&D Costs 

as a Share of 

Total Costs Total R&D 

R&D 

Employment 

as a Share 

of Total 

Employment 

i2 Technologies (formerly 

Aspect Development) 

$1,126,000 5,653 1,875 33% 

Autodesk $632,358 $541,964 $122,432 23% 2,470 

Avant! Corporation $227,141 $179,441 $56,785 32% 822 404 49% 

Cadkey Corporationd NA 

Bentley Systems, Inc.e $175,000 960 

Cadence Design Systems, 

Inc. 

$1,216,070 $829,800 $179,400 22% 4,200 1,300 31% 

Hewlett-Packard (owner of 

CoCreate) 

$47,061,000 $43,220,000 $3,355,000 8% 

IKOS Systems Inc. $40,893 $64,250 $14,400 22% 256 100 39% 

Intergraph $1,032,790 $1,131,000 $83,786 7% 6,700 

International Business 

Machines—Software 

Segmentf

$11,841,715 $7,365,275 $731,670 10% 

International 

Microcomputer Software 

$62,472 $55,315 $8,600 16% 338 115 34% 

MacNeal Schwendlerg $125,397 $135,438 $13,666 10% 745 

Mentor Graphicsh $490,393 $332,712 $117,853 35% 2,600 

OrCAD $47,652 $45,446 $11,508 25% 261 101 39% 

Parametric Technologiesi $1,018,000 $732,960 $91,620 13% 4,911 958 20% 

Quickturn $104,109 $147,939 $23,425 16% 383 129 34% 

Structural Dynamics 

Research Corporation 

$403,025 $357,360 $64,182 18% 2,366 644 27% 

Summit Design Inc.! $31,439 $36,687 $7,749 21% 178 106 60% 

Synopsys, Inc. $717,940 $598,823 $154,400 26% 2,592 

Unigraphicsj

 $403,571 $406,116 $103,017 25% 2,200 

Visiok $100,775 $75,684 $16,124 21% 355 140 39% 

Wind River Systems Inc. $129,400 $92,311 $17,638 19% 598 181 30% 

aIncludes data from subsidiaries and revenues from hardware sales, where applicable. 

bIncludes costs of revenue and operating expenses; taxes and interest income excluded; acquired technology and 

merger expenses not included unless considered as part of research and development expenses in the annual report. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-16 

Table B-2. Developers of CAD/CAM/CAE Software (continued) 

cR&D expenditures may or may not include capitalization, depending on how the figure was presented on the balance 

sheet of the annual report. 

dRevenue and cost information not available. Cadkey is a private corporation. 

eSource: Bentley Systems Incorporated. Corporate Backgrounder obtained November 1999. 

<http://www.bentley.com/bentley/backgrnd.htm>. 

f

Source: International Business Machines. 1998. Annual Report. 

<http://www.ibm.com/annualreportt/1998/discussion/ibm98armd04.html>. 

gSource: The MacNeal Schwendler Corporation. 1998. Annual Report. 

<http://www.mscsoftware.com/ir/annual.html>. 

hSource: Mentor Graphics Corporation. 1998. Annual Report. 

<http://www.mentorg.com/investor_relations/MentorAnnual.pdf>. 

i

Source: Parametric Technology Corporation. 1998. Annual Report. 

<http://www.ptc.com/company/pmtc/1998/index.htm>. As obtained October 1999. 

j

Source: Unigraphics Solutions Incorporated website. <http://www.ug.eds.com>. 

kSource: 10K report and Visio Corporation. 1998. Annual Report. <http://www.visio.com/files/ar98/Visar98.pdf>. 

Source: National Science Foundation. 1999. Research and Development in Industry: 1997. 

industry than in other sectors of the economy. For example, R&D 

spending was only 2.9 percent of the net sales of all industries in 

1997 (National Science Foundation [NSF], 1999). The service 

industry, of which the software industry is a part, spent 8.6 percent 

of its net sales on R&D in 1997, still well below the average R&D 

expenditures of CAD/CAM/CAE industry leaders listed in Table B-2 

(NSF, 1999). In fact, the computer and data processing services 

industry, a more specific industry group including software 

developers, spent a larger proportion of its net sales on R&D 

(13.3 percent) than did any other industry group surveyed by the 

NSF in 1995. The above data actually underestimate the differences 

in R&D spending between the CAD/CAM/CAE industry and other 

industries, because the NSF data are based on net revenues (gross 

revenues minus operating expenses, cost of revenue and taxes), 

which are smaller than gross revenues. If the NSF percentages were 

based on total revenues, they would be even smaller. 

Appendix A provides additional information for the software 

developers included in Table B-2 as well as several hundred others. 

The appendix includes a partial list of developers of less well-known 

design CAD/CAE platforms and accessory software products as well as 

some EDA software developers that produce a smaller range of 

products than the often-cited developers listed in Table B-2. The 

software developers in the appendix constitute the population of 

software developers to be surveyed as part of this project. 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-17 

PDM Firms 

PDM systems emerged in the early 1980s when software and 

engineering companies originally developed in-house PDM 

projects. These firms realized that paper-based systems to monitor 

the flow of data were becoming increasingly unwieldy and 

uncontrollable. During the late 1980s, some of these companies 

started to market their internally developed PDM systems to other 

organizations. The initial products were large databases that 

engineers could search to find documents. Because most of the 

software firms that developed the original PDM products were in 

the CAD/CAM/CAE business, they focused their efforts on 

developing PDM systems for their existing customers. 

The early PDM systems’ main focus was on monitoring and 

controlling engineering data after the point of initial development to 

the end of the manufacturing process. Although PDM first focused 

on managing the manufacturing process, during the early 1990s it 

was also used to monitor activities farther upstream in the product 

cycle. During the product inception stage, PDM is now used to 

track the data generated by engineers. In the later half of the 1990s, 

business operations became more interrelated, and PDM systems 

are now used to manage CAD/CAM/CAE systems as well as other 

engineering and business programs. The recent innovations have 

transformed PDM from a database application to an entire workflow 

management system. 

Numerous firms sell or provide PDM services. Some encompass the 

entire PDM product cycle by developing, selling, installing, and 

supporting a specific product. Other firms only engage in specific 

parts of the production process. Table B-3 lists the categories of 

firms engaged in PDM and describes their activities. 

Over 50 domestic and 25 international firms produce PDM 

products. Table B-4a provides the relative market shares for the 

eight largest PDM software and services vendors. Table B-4b 

provides sales and employment information on the domestic PDM 

product vendors. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-18 

Table B-3. Categories of Firms Engaged in PDM 

Firm Type Description of Activities 

PDM Product Vendors Encompasses the whole organization by providing complete 

document management from planning to manufacturing 

Document and Image Management 

Product Vendors 

View, mark-up, plot, print, and convert document formats 

PDM Support Product Vendors Implementation, installation, training, modification, and 

conversion services and system consulting 

Value-Added Resellers Sale and installation of existing PDM products 

System Integrators Provides technical assistance, consulting, training and 

software design, integration, and development 

Consultants Design and develop customized applications to support 

customer-specific requirements 

Source: Product Data Management Information Center. <http://www.pdmic.com>. As obtained on March 13, 2000. 

Table B-4a. Market Shares for the Eight Largest PDM Software and Services Vendors 

Company Market Share (%) 

i2 Technologies (formerly Aspect Development) 8 

Documentation 7 

Engineering Animation Inc. 6 

IBM/Enovia 5 

MatrixOne 5 

Parametric Technology Corporation 4 

Structural Dynamics Research Corporation/Metaphase 3 

UGS, Inc. 3 

Source: CIMdata. 2000. <http://www.cimdata.com/PR000307B.htm>. 

Table B-4b. Developers of PDM Software, 2000 

Company Sales Employment 

Accel Technologies, Inc. 11 60 

Agile Software Corp. 16.8 156 

Applicon 135.5 200 

Autodesk Inc. 740.2 2,716 

Auto-trol Centura 2000 8.7 177 

BaanCompany 736 5,101 

(continued) 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-19 

Table B-4b. Developers of PDM Software, 2000 (continued) 

Company Sales Employment 

CACI-Federal, Inc. 441.7 4,228 

Think3, Inc. 32.03 250 

Ceimis Enterprises, Inc. 2.5 25 

CMstat Corporation 2.5 25 

CoCreate (subsidiary of Agilent Technologies 8 70 

CONCENTRA 100 450 

Concurrent Systems, Inc. NA NA 

Configuration Data Services 2.5 15 

ConsenSys Software Corporation 7.5 50 

Custom Programming Unlimited 2 30 

DataWorks Corporation 25.9 300 

Eignor & Partner, Inc. 19 95 

Engineering Animation Inc. 70.7 957 

Enovia Corp. 12 90 

Formation Systems Inc. 10.9 85 

FORMTEK, Inc. A Lockheed Martin Co. 22 150 

Gerber Information Systems NA NA 

i2 Technologies (formerly Aspect Development) 1,126.3 6,000 

IBM 8,093 316,303 

IDFM, Inc. 6 43 

Ingenuus Corporation 7.5 95 

Innovative Data Concepts, Inc. NA NA 

InSight NA NA 

Integrated Support Systems, Inc. 9.2 35 

Integrated Systems Technologies, Inc NA NA 

IntegWare NA NA 

Intergraph Corporation 690.5 4,600 

Intergraph Electronics Corporation NA NA 

Interleaf, Inc. 45.2 338 

Kruise Inc. NA NA 

Matrix One NA NA 

MERANT - PVCS 400 2,000 

Mesa Systems Guild, Inc. 2 35 

Metaphase Technology 403 2,500 

Modultek Inc. NA NA 

Mystic Management Systems, Inc. 2 9 

NEC Systems, Inc. NA NA 

NetIDEAS, Inc. NA NA 

Network Imaging Systems Corp NA NA 

NovaSoft Systems, Inc. NA NA 

Open Text Corp. 112.9 408 

Oracle Corporation 8827 43800 

(continued) 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-20 

Table B-4b. Developers of PDM Software, 2000 (continued) 

Company Sales Employment 

Parametric Technology Corporation 1057.6 4998 

Parametric Technology Corporation 928.4 4725 

Prefered Technology Corp. NA NA 

PROCAD, Inc. NA NA 

SDRC 2 9 

Sherpa Corporation NA NA 

Structural Dynamics Research Corporation/Metaphase 340.8 1637 

The Business Process Performance Co. NA NA 

TMSSequoia 5 46 

Unigraphics Solutions 400 2200 

Waware Systems NA NA 

Workgroup Technology, Inc. 8.6 110 

Metaphase Technology 403 2500 

Modultek Inc. NA NA 

Mystic Management Systems, Inc. 2 9 

NEC Systems, Inc. NA NA 

NetIDEAS, Inc. NA NA 

Network Imaging Systems Corp NA NA 

NovaSoft Systems, Inc. NA NA 

Open Text Corp. 112.9 408 

Oracle Corporation 8827 43800 

Parametric Technology Corporation 1057.6 4998 

Prefered Technology Corp. NA NA 

PROCAD, Inc. NA NA 

SDRC 2 9 

Sherpa Corporation NA NA 

Structural Dynamics Research Corporation/Metaphase 340.8 1637 

The Business Process Performance Co. NA NA 

TMSSequoia 5 46 

Unigraphics Solutions 400 2200 

Waware Systems NA NA 

Workgroup Technology, Inc. 8.6 110 

Prefered Technology Corp. NA NA 

PROCAD, Inc. NA NA 

SDRC 2 9 

Sherpa Corporation NA NA 

Structural Dynamics Research Corporation/Metaphase 340.8 1637 

The Business Process Performance Co. NA NA 

Source: Standard and Poor’s Net Advantage ; Reference USA; Hoovers Online, http://www.hoovers.com 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-21 

 B.3.2 Computer Systems Design and Related Services 

(Sector 5415) 

Establishments in this sector are affiliated with the CAD/CAM/CAE 

and PDM industry in two important ways: as suppliers of testing 

services to software developers and users and as service providers 

aiding CAD/CAM/CAE and PDM software in computer systems 

integration, software installation, and custom programming. 

Table B-5 presents current information on the number of 

establishments providing computer system design and related 

services. CAD/CAM/CAE and PDM software developers and service 

providers are a subset of the population listed in Table B-5. 

Table B-5. Industry Profile for Computer Systems Design and Related Services, 1997 

NAICS 

Code Description 

Number of 

Establishments 

Value of 

Shipments or 

Receipts 

(thousands) 

Number of 

Employees 

5415 Computer systems design and related 

services 

72,278 108,967,614 764,659 

541511 Custom computer programming services 31,624 38,300,515 318,198 

541512 Computer systems design services 30,804 51,212,916 337,526 

5415121 Computer systems integrators 10,571 35,270,055 207,741 

5415122 Computer systems consultants (except 

systems integrators) 

20,233 15,942,861 129,785 

541513 Computer facilities management services 1,445 15,114,194 71,821 

541519 Other computer related services 8,405 4,339,989 37,114 

Source: U.S. Census Bureau. December 1999bx. “1997 Economic Census, Professional, Scientific, and Technical 

Services.” Geographic Area Series. 

 B.4 PRODUCTION AND CONSUMPTION OF 

CAD/CAM/CAE AND PDM SOFTWARE 

PRODUCTS 

The world market for CAD/CAM/CAE software is about $8.0 billion 

annually. U.S. manufacturers purchased approximately $2.5 billion 

of CAD/CAM/CAE software in 1997. U.S. software developers sold 

twice that amount throughout the world in 1997. 

 B.4.1 Production 

The U.S. supplies the majority of the CAD/CAM/CAE software sold 

on the world market, although U.S. suppliers do compete with 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-22 

developers in Japan, Asian-Pacific countries, and Europe. In 1997, 

U.S. software developers sold about $5.4 billion worth of the almost 

$8.0 billion worth of CAD/CAM/CAE software sold in the world. 

Figure B-2 shows the relative world market shares of other world 

regions. Japan and Asian-Pacific countries supply 20 percent of the 

world’s CAD/CAM/CAE software. Europe supplies 10 percent (U.S. 

Department of Commerce, 1998). 

United States

68%

Japan/Asia

20%

Europe

10%

Rest of World

2%

Source: U.S. Department of Commerce. 1998. U.S. Industry & Trade Outlook 

‘98. New York: McGraw-Hill. 

 B.4.2 Consumption 

Although the U.S. supplies 68 percent of the world’s 

CAD/CAM/CAE software, world demand for the software is more 

evenly distributed. Because of this, more than 36 percent of the 

1997 revenues of U.S. CAD/CAM/CAE suppliers were derived from 

overseas sales. Figure B-3 shows the relative consumption of the 

software throughout several regions of the world. U.S. 

manufacturers accounted for 32 percent ($2.5 billion) of the world 

demand for the software. European manufacturers purchased nearly 

the same amount of software in 1997, accounting for another 

31 percent of the world demand. Japanese manufacturers 

accounted for nearly $2.0 billion (25 percent) of the demand (U.S. 

Department of Commerce, 1998). 

Figure B-2. The 

Producers of 

CAD/CAM/CAE Software, 

1997 

The U.S. produces the majority 

of CAD/CAM/CAE software on 

the world market. 
Appendix B — CAD/CAM/CAE/PDM Use and Development in the Transportation Sector 

B-23 

United States

32%

North America

3%

Europe

31%

Asia/Pacific

5%

Japan

25%

Rest of World

5%

Total Sales = $8.0 billion

Source: U.S. Department of Commerce, 1998, U.S. Industry & Trade Outlook 

‘98. New York: McGraw-Hill. 

Compared to CAD/CAM/CAE, a larger share of PDM system 

consumption is in North America. Figure B-4 shows the relative 

amount of consumption of PDM by geographic region in 1999. 

Based on estimated total sales of $1.76 billion, this implies that 

North America purchased over $800 million of PDM products, 

Europe purchased over $600 million worth of PDM products, and 

the Asia-Pacific region purchased under $250 million worth 

(CIMdata, 2000). 

Figure B-3. The 

Consumption of 

CAD/CAM/CAE Software, 

1997 

U.S. manufacturers purchase 

half as much CAD/CAM/CAE 

software as is sold by U.S. 

software developers. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

B-24 

United States

47%

Asia Pacific

14%

Europe

38%

Rest of World

1%

Total Sales = $1.76 billion

Source: CIMdata. 2000. <http://www.cimdata.com/PR000307B.htm> 

Figure B-4. Regional 

Distribution of PDM 

Revenues, 1999 
Appendix C: 

CAD/CAM/CAE/PDM 

Developers and Users 

Survey Instruments 
C-1 

Survey of CAD/CAM/CAE and PMD Software Developers 

Being conducted by 

Research Triangle Institute 

On behalf of 

National Institute of Standards and Technology 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-2 

OMB NO: 0693-0031 Expires 10/31/2002 

This survey is authorized under Executive Order 12862, “Setting Customer Service Standards.” 

Your response is voluntary and all data collected will be considered confidential. Public 

reportings for this collection of information is estimated to average 25 minutes per response, 

including the time of reviewing instructions, searching existing data sources, gathering and 

maintaining the data needed, and completing and reviewing the collection of information. Send 

comments regarding this estimate or any other aspects of this collection of information, including 

suggestions for reducing the length of this questionnaire, to the National Institute of Standards and 

Technology, 100 Bureau Drive, Stop 3220, Gaithersburg, MD, 20899-3220 and the Office of 

Management and Budget Information and Regulatory Affairs, Office of Management and Budget, 

Washington, DC 20503. 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-3 

Introduction 

As part of a research study for the National Institute of Standards and Technology (NIST), 

Research Triangle Institute (RTI) is conducting a survey of CAD/CAM/CAE and PDM software 

developers. The purpose of this survey is to learn about the incidence and cost of software bugs 

and errors to software developers and users. 

The National Institute of Standards and Technology (NIST) is a non-regulatory federal agency 

within the Commerce Department’s Technology Administration. NIST’s mission is to promote 

economic growth by working with industry to develop and apply technology, measurements, and 

standards. NIST carries out its mission through four interwoven programs: NIST Laboratories, 

Baldridge National Quality Program, Manufacturing Extension Partnership, and Advanced 

Technology Program. See http://www.nist.gov for more information about NIST’s work. 

Our study would greatly benefit from your insights and experience. The findings from the study 

will be used to assist NIST to identify and prioritize technical infrastructure needs in the area of 

software testing. In addition, your company could benefit from identifying and quantifying basic 

software testing inadequacies. All participants will receive a copy of the final report. 

Your participation is voluntary, and your responses will be kept strictly confidential. Please note 

that questions regarding the number and type of software bugs will only be used to estimate the 

cost of software errors for the entire industry and will not be available to the public or shared with 

other survey participants. Only aggregate results will be included in the final report. 

The survey will take about 25 minutes to complete. Please answer all questions as they pertain to 

your firm. Please answer each question by checking the appropriate answer(s) or providing your 

answer in the designated space. 

If you have any questions as you complete the survey, please contact Michelle Bullock at 

(919) 485-5599 or bullock@rti.org. 

Thank you in advance for your participation. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-4 

1. Background Information 

1.1 Please type your name, company name, and e-mail address on the lines below. 

Name: _______________________________________________ 

Company: ____________________________________________ 

E-mail: _______________________________________________ 

1.2 What types of software products are developed at your company? (Check all that apply.) 

1. CAD 

2. CAE 

3. CAM 

4. PDM 

5. Other (Specify): ________________________________________ 

1.3 What share of products can be classified as CAD/CAM/CAE or PDM software? 

___________________ %

1.4 Please choose the North American Industry Classification System (NAICS) code(s) under 

which your company is classified. 

1. 541511—Custom computer software analysis and design services 

2. 511210—Packaged computer software publisher 

3. 541519—Computer software installation services 

4. Other (Specify): ________________________________________ 

1.5 What was the approximate total number of employees employed by your company in 

2000? (Report a range of employees if necessary.)

_____________________ 

1.6 What was the approximate value of total revenues (sales) reported by your company in 

2000? (Report a range of sales if necessary.)

_____________________ 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-5 

2. Expenditures on Software 

Testing 

For the purpose of this survey, software testing is defined as: 

The process of exercising a product to identify differences between expected and actual results 

and performance. Typically testing is bottom-up: unit test, integrate test and finally system test. 

NOTE: This does not include software development. 

2.1 What were the total number of full-time equivalent (FTE) employees for your company in 

2000 who were involved in software testing and error correction? If you can’t answer this 

question for your entire company directly, take the total number of full-time equivalent 

(FTE) employees for your group/organization who were involved in software testing and 

error correction. Then multiply that number by the number of groups/organizations in 

your company that are involved in software testing and error correction. Please 

breakdown the total number according to the employee category. 

Employee Category 

Number of FTE Employees Involved in 

Software Testing and Error Correction 

Software Engineers/Programmers 

Software Engineers/Testers/QA Engineers 

Other: (Specify) ______________________ 

2.2 Did your company purchase testing software in 2000? 

_____ Yes 

_____ No (Go to Question 2.4) 

2.3 Please complete the following table based on testing software purchases that were made 

in 2000. 

Software Name 

Annual Expenditures for Test 

Software1 Type of Testing Conducted2

 

Notes: 

1. If Test Software was developed In-house, then estimate yearly internal expenditures, budget, or 

number of FTE employees engaged in development and maintenance of the test software. 

2. Choose all of the following types of testing that apply: a) Conformance to Specifications (also 

called Functional Verification Testing), b) Interoperability Testing, or c) Performance Testing (also 

called System Testing). 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-6 

2.4 Did your company purchase hardware to support software testing in 2000? 

_____ Yes 

_____ No (Go to Question 2.6) 

2.5 Please complete the following table based on testing hardware purchases that were made 

in 2000. 

Hardware Name 

Cost of 

Hardware 

Was the 

Hardware 

Leased? 

Expected Useful 

Life of the 

Hardware 

Type of Testing 

Conducted3

 

Notes: 

3. Choose all of the following types of testing that apply: a) Conformance to Specifications, b) 

Interoperability Testing, or c) Performance Testing. 

2.6 Did your company contract for external testing services in 2000? 

_____ Yes 

_____ No (Go to Section 3) 

2.7 How much did your company pay for external testing services in 2000 (expressed in 

dollars or as the number of Full Time Equivalent employees)? If you can’t answer this 

question for your entire company directly, take what your group/organization paid for 

external testing services in 2000. Then multiply that number by the number of 

groups/organizations in your company that contracted for external testing services in 

2000. 

$ ____________________ 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-7 

3. Incidence of Software Bugs and 

Errors 

In this section of the survey, we segment the software development process into five stages and 

investigate 

• where errors are introduced and 

• where bugs are typically detected. 

Software bugs and errors can be generally divided into three broad categories; design, 

implementation and delivery errors. Design bugs are flaws in the underlying design of the 

software architecture typically resulting in redesign. Implementation and delivery bugs are errors 

in the way the programmer tries to achieve the design during coding. For the purpose of this 

survey, we are limiting our definition of a software bug to implementation and delivery coding 

errors. 

The five stages of the development process are 

• requirements gathering and analysis/architectural design, 

• coding/unit testing, 

• integration and component/RAISE (Reliability, Availability, Install Serviceability, and Ease 

of Use) system testing, 

• early customer feedback/beta test programs, and 

• post-product release (found by customer after purchase or acceptance). 

For the following questions, please consider a representative new CAD, CAM, CAE, or PDM 

development project or new version. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-8 

3.1 Bugs are found throughout the software development process. Bugs are detected 

internally through formal testing and externally by users during beta testing and business 

operations. In the table below, please identify the stages in which bugs are typically 

found. Please list either the number of bugs typically detected (per development project 

or lines of code) or the distribution (percentage) of bugs detected across the five stages. 

Stages of Development 

Number of Bugs Detected 

at Each Stage or 

Distribution of Bugs 

Detected Across Stages 

Requirements gathering and 

analysis/ architectural design 

 

_____% 

Coding/unit testing _____% 

Integration and 

component/RAISE system testing 

 

_____% 

Early customer feedback/beta 

test programs 

 

_____% 

Post-product release _____% 

 per project 

 per _____ lines of code

 Total = 100%

3.2 Errors can be introduced into software at various stages in the development process. For 

bugs found during the coding/unit testing phase, in what stage was the error likely to be 

introduced (i.e., where was the source of the error)? Please indicate the probability of the 

error being introduced during the following stages. 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

100% Total 

3.3 For bugs found during the integration and component testing phase, in what stage was the 

error likely to be introduced? 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

_____% integration and component 

100% Total 

3.4 For bugs found during beta testing, in what stage was the error likely to be introduced? 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

_____% integration and component 

100% Total 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-9 

3.5 For bugs found by customers during the post-product release phase, in what stage was the 

error likely to be introduced? 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

_____% integration and component 

100% Total 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-10 

4. The Cost of Fixing Bugs 

In this section, we investigate the costs of locating the source of bugs and correcting the errors 

(referred to as fixing or repairing bugs). We are primarily interested in how these costs vary with 

respect to where the error was introduced and at what stage in the software development process 

the bug is detected. 

4.1 The severity and, hence, the cost of fixing a given bug may depend on what stage the bug 

was introduced into the software. In the table below, provide the average cost of fixing 

bugs (in terms of labor hours) that are introduced during the three main development 

stages presented in Section 3. For this cost table, assume that the bug is detected in the 

same stage that it was introduced. 

Stage the Bug was Introduced 

Average Number of Hours to Correct an 

Error Introduced and Found in this Stage 

Requirements gathering and analysis/ 

architectural design __________ hours 

Coding/unit testing __________ hours 

Integration and component/RAISE system testing __________ hours 

4.2 It is widely assumed that bugs caught later in the software development process are more 

expensive to repair. In the following table, please indicate how much more expensive it is 

to repair a bug created in the requirements gathering and analysis/architectural design

stage if it is not detected until later in the software development process (i.e., Not detected 

until coding, integration, beta testing, or post-product release). Provide your answer in 

terms of how many times more expensive it is to repair the bug in later stages compared to 

detecting and repairing it during the stage in which it was introduced. 

Stage Where Errors Introduced in 

Requirements Gathering and Analysis/ 

Architectural Design Stage are Detected 

How Many More Times as Costly is it to 

Repair a Bug if it is Detected After the 

Stage it is Introduced 

Requirements gathering and analysis/ 

architectural design Stage bug is introduced 

Coding/unit testing _____ times as costly to repair 

Integration and component/RAISE system testing _____ times as costly to repair 

Early customer feedback/beta test programs _____ times as costly to repair 

Post-product release _____ times as costly to repair 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-11 

4.3 Now consider bugs introduced during the coding stage. How much more costly is it to 

repair these bugs if they are detected in later stages? 

Stage Where Errors Introduced in 

Coding Stage are Detected 

How Many More Times as Costly is it to 

Repair a Bug if it is Detected After the 

Stage it is Introduced 

Coding/unit testing Stage bug is introduced 

Integration and component/RAISE system testing _____ times as costly to repair 

Early customer feedback/beta test programs _____ times as costly to repair 

Post-product release _____ times as costly to repair 

4.4 Finally, consider bugs introduced during the integration stage. How much more costly is 

it to repair these bugs if they are detected in later stages? 

Stage Where Error Introduced in 

Integration Stage are Detected 

How Many More Times as Costly is it to 

Repair a Bug if it is Detected After the 

Stage it is Introduced 

Integration and component/RAISE system testing Stage bug is introduced 

Early customer feedback/beta test programs _____ times as costly to repair 

Post-product release _____ times as costly to repair 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-12 

5. A World with Improved Testing 

Resources 

NIST is interested in estimating the costs of inadequate software testing tools and resources to U.S. 

companies. 

5.1 Please describe the shortcomings of the testing software (testware) and processes you 

currently use to detect and fix bugs in your software products. 

For example:

• Testware products are not as compatible with our software development environment 

as we had expected. 

• Testware products assume a software development process that is different than the 

one we use. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

5.2 What improvements would you like to see in software testing programs and procedures? 

For example:

• What are there capabilities you would like to see that are not available in current 

testware? 

• Could testware products function better if there were fewer user interface or 

interoperability problems? 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

For the questions below, please consider how the distribution of where bugs are detected and the 

cost of repairing bugs would change if the improved testing procedures and tools you described 

above were available. (Even if you were not able to describe all the software testing 

improvements you would like to see in Questions 7.1 through 7.4, feel free to broadly envision a 

world with an enhanced software testing infrastructure when answering the questions below.) 

Note: We are assuming that the number of bugs introduced during the software development 

process remains unchanged—only the developer’s ability to detect and fix bugs is changed in our 

hypothetical new and improved world. 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-13 

5.3 In a world with improved software testing tools, how would the distribution of where 

(when) bugs are detected change? 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

5.3a In the table below, we have repeated your response to Question 3.1. Please 

indicate how this distribution you provided would change in a world with 

improved software testing tools. 

Current World 

(your response to 3.1) 

World with improved 

Testing Tools 

Stages of Development 

Number of Bugs 

Detected Total 

Number of Bugs 

Detected Total 

Requirements gathering 

and analysis/ 

architectural design _____% _____% 

Coding/unit testing _____% _____% 

Integration and 

component/RAISE system 

testing _____% _____% 

Early customer 

feedback/beta test 

programs _____% _____% 

Post-product release _____% _____% 

 per project 

 per _____ 

lines of code

Total = 

100%

 per project 

 per _____ 

lines of code

Total = 

100%

5.4 How would the cost of repairing bugs change with improved software testing tools? 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-14 

5.4a In the table below, we have repeated your response to Question 4.1. Please 

indicate how the number of labor hours would change with improved tools for 

locating and repairing bugs. 

Stage the Bug was Introduced 

Current Labor 

Hours to Fix 

Average Bug 

(your response to 

Question 4.1)

World with 

Improved Testing 

Tools 

Requirements gathering and analysis/ 

architectural design _______ hours _______ hours 

Coding/unit testing _______ hours _______ hours 

Integration and component/RAISE system 

testing _______ hours _______ hours 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-15 

6. Time to Market 

6.1 For a representative new CAD/CAM/CAE or PDM product or new version developed by 

your company, what average production time to market? If you can’t answer this question 

for your entire company directly, use the average time to market for a representative new 

CAD/CAM/CAE or PDM product or new version developed by your group/organization. 

________________ years 

6.2 If no testing activities were needed (i.e., no error detection and repair), how long would it 

take for a typical project to progress from inception to completion? 

________________ years 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-16 

7. Customer Service Cost 

7.1 Does your company typically provide installation assistance for your CAD/CAM/CAE or 

PDM software products? 

_____ Yes 

_____ No (Go to Question 6.5) 

7.2 Please describe the type of installation assistance your company provides. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

7.3 What were your total installation costs (annual expenditures) in 2000? 

$ ____________ 

7.4 What percentage of your installation costs are due to bugs or errors found during 

installation? 

______________ percent 

7.5 Does your company provide long-term service contracts or other types of after-sales 

customer service? 

_____ Yes 

_____ No (Go to Section 7) 

7.6 Please describe the type of after-sales service your company provides. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

7.7 What were your total after-sales service costs (annual expenditures) in 2000? 

$ ____________ 

7.8 What percentage of your after-sales service costs are related to bugs or errors found by 

customers during business operations? 

______________ percent 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-17 

7.9 In a world with improved software testing tools, how much could your customer 

installation expenditures be reduced? 

______________ percent 

7.10 In a world with improved software testing tools, how much could your other after-sales 

customer service costs be reduced? 

______________ percent 

7.11 What percentage of this potential improvement do you expect can come from internal or 

vender-supplied software testing capability over the next five years? 

_____________ percent 

8. Comments 

8.1 Please provide any additional comments that would be helpful in evaluating how 

improved testing tools would impact your company’s software development costs and 

product quality. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

We thank you for your participation. 

Please indicate below if you would like to receive a copy of the final report. 

_____ Yes, please send a copy 

_____ No 

Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-19 

Survey of CAD/CAM/CAE and PDM Software Users 

Being conducted by 

Research Triangle Institute 

On behalf of 

National Institute of Standards and Technology 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-20 

OMB NO: 0693-0031 Expires 10/31/2002 

This survey is authorized under Executive Order 12862, “Setting Customer Service Standards.” 

Your response is voluntary and all data collected will be considered confidential. Public 

reportings for this collection of information is estimated to average 20 minutes per response, 

including the time of reviewing instructions, searching existing data sources, gathering and 

maintaining the data needed, and completing and reviewing the collection of information. Send 

comments regarding this estimate or any other aspects of this collection of information, including 

suggestions for reducing the length of this questionnaire, to the National Institute of Standards and 

Technology, 100 Bureau Drive, Stop 3220, Gaithersburg, MD, 20899-3220 and the Office of 

Management and Budget Information and Regulatory Affairs, Office of Management and Budget, 

Washington, DC 20503. 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-21 

Introduction 

As part of a research study for the National Institute of Standards and Technology (NIST), 

Research Triangle Institute (RTI) is conducting a survey of transportation manufacturing 

companies that use CAD/CAD/CAE and PDM software. The purpose of this survey is to learn 

about the incidence and cost of software bugs and errors to software users. 

The National Institute of Standards and Technology (NIST) is a non-regulatory federal agency 

within the Commerce Department’s Technology Administration. NIST’s mission is to promote 

economic growth by working with industry to develop and apply technology, measurements, and 

standards. NIST carries out its mission through four interwoven programs: NIST Laboratories, 

Baldridge National Quality Program, Manufacturing Extension Partnership, and Advanced 

Technology Program. See http://www.nist.gov for more information about NIST’s work. 

Our study would greatly benefit from your insights and experience. The findings from the study 

will be used to assist NIST to identify and prioritize technical infrastructure needs in the area of 

software testing. In addition, your company could benefit from identifying and quantifying how 

software bugs and errors affect companies in the financial service sector. All participants will 

receive a copy of the final report. 

Your participation is voluntary, and your responses will be kept strictly confidential. Please note 

that questions regarding the type and cost of software bugs will only be used to estimate the 

aggregate impacts for the entire industry, and individual responses will not be available to the 

public or shared with other survey participants. Only aggregate industry-level results will be 

included in the final report. 

Your establishment was randomly selected to participate in this survey. The survey will take 

about 30 minutes to complete. Please answer all questions as they pertain to your firm by 

checking the appropriate box(es) or providing text in the designated space. 

If you have any questions as you complete the survey, please contact Michelle Bullock at 

(919) 485-5599. 

Thank you in advance for your participation. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-22 

1. Background Information 

1.1 Please type your name, company name, and e-mail address on the lines below. 

Name: ___________________________________________________ 

Company: ________________________________________________ 

E-mail: ___________________________________________________ 

1.2 What types of products or subcomponents of products are produced by your company? 

(Circle all that apply.) 

1. Automotive 

2. Aerospace 

3. Shipping 

4. Rail 

5. Other (Specify): ________________________________________ 

1.3 Please fill in the North American Industry Classification System (NAICS) code(s) under 

which this establishment is classified. 

NAICS Code(s) 

1.4 What was the approximate total number of employees employed by your company in 

2000? (Report a range of employees if necessary.) 

_____________________ 

1.5 What was the approximate value of total revenues (sales) reported by your company in 

2000? 

_____________________ 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-23 

2. The Use of CAD/CAM/CAE and 

PDM Software 

2.1 In the table below, please list the CAD/CAM/CAE and PDM software your company 

currently maintains and indicate when it was installed and what you project to be its 

remaining life expectancy? 

Name of Software Product (all versions) Year Installed 

Number of Years Expected To 

Remain in Operation 

Example: CATIA Example: 1996 Example: 10 more years

 

2.2 What were the total number of (full-time equivalent [FTE]) employees in 2000 involved in 

operating and supporting the software listed in Table 2.1? 

Type of Activity 

Number of 

Employees 

CAD/CAM/CAE 

PDM 

 

Please Read Before Continuing!

In Sections 2 through 5, we ask about the incidence and cost of FEDI and clearinghouse software 

bugs and errors at this establishment. Personnel responsible for monitoring and maintaining FEDI 

and clearinghouse software at this establishment should be able to provide the best answers to 

these questions. If necessary, please share your password with colleagues at your company and 

ask them to complete the appropriate sections. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-24 

3. Incidence and Costs of Software 

Bugs and Errors 

This section focuses on the software bugs and errors your company encounters in the 

CAD/CAM/CAE and PDM systems you employ and how they affect your business operations. 

3.1 Does the information you are providing reflect all the CAD/CAM/CAE and PDM systems at 

your company? 

 _____ yes 

 _____ no: what percentage of your company’s systems are represented in your 

 responses? _____% 

 

3.2 What types of problems does your company encounter due to bugs and errors in 

CAD/CAM/CAE and PDM software (do not include short comings in basic product design 

and functionality)? 

_______________________________________________________________________________________ 

_______________________________________________________________________________________ 

_______________________________________________________________________________________ 

3.3 Software bugs can lead to either major problems (manual re-entry of data or design faults) 

or minor problems (requiring a slight correction to files being transferred). In 2000, how 

many major and minor software bugs or errors did your company encounter in your 

software? 

_______________ major 

_______________ minor 

 

3.4 For what percentage of those bugs or errors were the source of the problems found and 

fixed through code modification and patches? 

_______________ % found and fixed 

_______________ % not fixed 

100% total 

3.5 Was 2000 a typical year for software problems, or has your company been making an 

above average number of software upgrades, potentially leading to an uncharacteristically 

large number of software problems? 

_____ typical year 

_____ unusual year with _____% more software/system improvement projects than usual 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-25 

3.6 For the typical major error that you had in 2000, what was the impact on your company’s 

business operations? 

1. _____% lead to design faults that were detected downstream or after product release 

2. _____% lead to design time and cost increases 

_____ months delay 

_____ % cost increase 

3. Other impact: please explain _________________________________________________ 

3.7 Did your company experience any repair costs associated with the software failure, such 

as time to re-enter lost data or repair data archives? 

1. _____ No 

2. _____ Yes: ________ labor hours spent on repair 

 ________ value of lost information 

 ________ other repair or damage costs, 

 please explain ___________________________________________ 

3.8 Do you think your company experienced any long-run competitive effects from the 

software failure(s), such as lost reputation or lost market share? 

Yes/no: lost reputation 

Yes/no: lost market share 

Yes/no: Delayed product getting to market by _____ months, leading to lost sales of _____ $/month 

_____ other impacts 

3.9 For minor software bugs in your CAD/CAM/CAE or PDM software, did these result in 

increased operating costs or decreased efficiency? 

_____ No (Go to Section 4) 

_____ Yes: please explain __________________________________ 

3.9a Are these one-time expenditures due to developing patches and work arounds or 

are they ongoing problems affecting efficiency? 

 _____ one-time costs 

 _____ ongoing costs 

3.9b Approximately what are these annual expenditures? 

 $_____________ 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-26 

4. Software Life-Cycle Costs 

Associated with Bugs and Errors 

In this section, we investigate how software bugs and errors affect the life-cycle costs of 

purchasing and operating CAD/CAM/CAE and PDM software. 

The Purchase Decision 

4.1 How much time and resources are spent researching a new CAD/CAM/CAE and PDM 

software package before a purchase decision is made? 

_____ calendar time (months) 

_____ labor expenditures (number of or fraction of FTEs) 

4.2 Could the search time and resources have been reduced if you had better information 

about the quality of the software products you were comparing? 

_____ Yes: What would be the change in 

 _____ fewer months 

 _____ fewer number of FTEs 

4.3 Because of potential software bugs and errors, do you typically delay purchasing new 

versions of CAD/CAM/CAE and PDM software? 

_____Yes: What is the typical delay? _____ months 

_____ No 

Software Installation and Acceptance 

4.4 What was the average time it took for installation and acceptance testing for your 

CAD/CAM/CAE and PDM software? 

______________ months 

4.5 What parties were involved in the installation and 

performance testing of your CAD/CAM/CAE and PDM 

software, and what was the share of effort/expenditures? 

_____% software developers 

_____% your company 

_____% third-party integrator (consultant) 

100% 

Acceptance testing is the 

process of determining 

whether software 

determines predefined 

acceptance criteria. 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-27 

4.6 What were your company’s own expenditures on installing and performing acceptance 

testing of CAD/CAM/CAE and PDM software in 2000? 

______ total labor hours 

4.7 How much did your company spend on external consulting services for installation and 

acceptance testing services of CAD/CAM/CAE and PDM software in 2000? 

$ ____________________ 

4.8 If the software you purchased contained fewer bugs and errors, how much would your 

labor and external consulting expenditures for installation and acceptance testing have 

been reduced? 

______________ percent 

Maintenance Costs 

4.9 How much money did your company spend on maintenance 

contracts for CAD/CAM/CAE and PDM software in 2000? 

$____________________

4.10 In 2000, how much money did your company spend on 

CAD/CAM/CAE and PDM software upgrades and 

maintenance that were not covered by a maintenance 

contract? 

$____________________

4.11 What percentage of maintenance costs were associated with bugs and errors embedded in 

the software? 

______________ percent 

Redundant System Costs 

4.12 After installation and acceptance, did your company maintain redundant backup systems 

for some period of time in case the new software failed? 

_____Yes 

How long did you maintain the backup system? _____ months 

What was (is) the estimated cost of maintaining these systems? __________ $/month 

_____ No 

Maintenance contracts

include any agreements 

with outside agencies that 

those agencies will perform 

periodic checks of system 

integrity and/or provide 

free upgrades and/or 

correct errors in installed 

software. Contracts may 

include training and 

technical support.
The Economic Impacts of Inadequate Infrastructure for Software Testing 

C-28 

5. The Impact of Reducing the 

Number of Software Bugs and 

Errors 

In this section, we investigate how the costs associated with bugs and errors in CAD/CAM/CAE 

and PDM would change if the number of bugs and errors embedded in these software products 

were partially reduced. Our discussions with industry indicate that it is not feasible or 

economical for software developers to produce “bug-free” software. However, NIST is interested 

in knowing what the cost savings would be if your company encountered a 25, 50, 75, or 90 

percent reduction in software errors. 

We anticipate that the rate at which the cost of bugs decreases as the number of bugs decreases 

will not be the same for all of the cost categories that have been discussed previously in the 

survey. For example, some cost-bug relationships may be linear (i.e., a 50 percent reduction in 

bugs leads to a 50 percent reduction is costs), and some may be nonlinear (i.e., a 50 percent 

reduction in bugs may lead to less than a 50 percent reduction in costs because even a small 

number of bugs requires testing, backup, systems, etc.). 

5.1 In the table below, please estimate the percentage cost reduction associated with different 

percentage reductions in bugs for each of the major cost categories discussed earlier in the 

survey. Two examples are provided. In Example A, costs decline proportionally as the 

number of bugs are reduced. In Example B, costs do not decline proportionally, and a 

90 percent reduction in bugs does not eliminate over half of the costs because other 

normal costs may be associated with maintenance or installation. 

Cost Reductions as a Function of Bug Reductions 

Cost Categories 

Percentage Reduction in Bugs or Errors in 

FEDI and Clearinghouse Software 

 25% 50% 75% 90% 

 

Example A (linear) 25% 50% 75% 90% 

Example B (nonlinear) 10% 15% 40% 45% 

 

Major failure costs 

Minor failure costs 

Purchase decision costs 

Installation costs 

Maintenance costs 

Redundant system costs 
Appendix C — CAD/CAM/CAE/PDM Developers and Users Survey Instruments 

C-29 

6. Comments 

6.1 Please provide any additional comments that would help us evaluate the cost of 

CAD/CAM/CAE or PDM software bugs and errors to your company. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

We thank you for your participation. 

Please indicate below if you would like to receive a copy of the final report. 

_____ Yes, please send a copy 

_____ No 
D-1 

 Appendix D: 

 Financial Services 

 Software Use and 

 Development 

More and more communications are occurring electronically. 

Shipping orders, messages, and other notifications are now 

completed with minimal paper documentation. This is especially 

true of financial transactions. In 1999, over $19.5 trillion dollars 

worth of transactions occurred electronically, representing a 

282 percent increase since 1989 (NACHA, 2000). 

The generic term used to describe the transfer of information 

electronically is Electronic Data Interchange (EDI). EDI is the 

process of exchanging documents in a standardized format directly 

from a database in one agency to a database in a separate agency. 

EDI can potentially cover most exchanges that are made with paperbased communication, such as placing orders with suppliers and 

carrying out financial transactions. 

The internal structure of the message distinguishes EDI for other 

forms of electronic communication (such as e-mail). E-mail 

messages are written in a free format and are not intended to be 

processed in any systematic and repeated manner when they are 

received. The goal of EDI is to have the data within the message 

processed automatically (i.e., without user intervention) when the 

message is received. To accomplish this goal, EDI messages must 

have an internal structure and content that must be adhered to for 

the data within the message to be transferred from one party to 

another. Even though EDI has a well-defined basic structure 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-2 

established by standards such as ANSI X.12, it allows for significant 

flexibility in its application. For this reason, several industries have 

established their own EDI application protocols with additional 

specifications to support unique industry requirements. 

Financial Electronic Data Interchange (FEDI) is the process of 

electronically transferring data from one user to another. The 

process of transferring financial data electronically is more 

complicated than most other forms of electronic communication. 

FEDI transactions in the financial services sector not only must 

contain the information for the transaction that is being processed, 

but must also include the transfer of the financial resources. The 

reconciliation of accounts requires the use of a clearinghouse that 

adds a step to the FEDI process that does not exist in traditional EDI 

transactions. 

Computer software and hardware play two important roles in the 

transfer of information in the financial services sectors. First, FEDI 

and clearinghouse software are used to manage the information 

content once it has arrived at its appropriate location. Second, 

routers and switches (a combination of software and hardware) are 

used to manage the flow of information from one entity to the next 

via the Internet and company intranets. 

This Appendix is divided into three sections. The first section 

provides background on the role of software and hardware in 

financial transactions. The second section focuses on the FEDI and 

clearinghouse software used to conduct the financial transactions. 

The third section of this deliverable focuses on the routers and 

switches used in the transfer of that information. 

 D.1 OVERVIEW OF ELECTRONIC TRANSACTIONS 

IN THE FINANCIAL SERVICES SECTOR 

Financial transaction management is the overarching term used to 

describe the flow, monitoring, and control of data across and within 

banking institutions. It is defined as the firm’s ability to control and 

manage a range of transactions—from foreign exchange to securities 

deals—from their input through to their reconciliation and 

successful resolution. Financial transactions management can be 

subdivided into three general activities: financial transactions 
Appendix D — Financial Services Software Use and Development 

D-3 

reconciliation, financial transactions services, and financial 

transactions control. 

Z Financial Transaction Reconciliation—The financial 

transaction reconciliation software allows the automated 

reconciliation of payments, securities, and foreign 

transactions. A flexible matching algorithm within each 

reconciliation module allows users to set up matching 

criteria to optimally meet the needs of partner banks or 

brokers, which increases matching rates. 

Z Financial Transaction Services—Financial transaction 

services include on-line transactions, archiving and retrieval 

functionality, and other services to aid the end user. 

Z Financial Transaction Control—Financial transactions 

control is software used to develop profiles and govern 

access to all functions. Roles and users can be defined 

individually or in groups, and user IDs can be assigned to all 

actions, providing a full audit trail. Several institutions can 

work with the same system independently of each other, and 

firms also have the ability to outsource matching services, if 

required. 

Starting in the mid-1970s, the first major use of EDIs was for direct 

deposits of payrolls. Paper financial transactions were growing at a 

rate of more than one billion per year. Banks and other financial 

institutions realized that they needed to develop a more efficient 

way of managing the flow of information that was being created. 

Automated clearinghouses were developed in response to this 

realization. In 1975, the Social Security Administration established 

the option of direct deposit for recipients to boost the use of 

electronic transactions. With close to half of all recipients receiving 

benefits electronically in 1988, the Social Security Administration 

was far ahead of the private sector, where only 11 percent of all 

employees were paid electronically. 

 D.1.1 Characteristics of Electronic Transactions 

Electronic payments include all financial transactions that are made 

electronically without the use checks, sharedrafts, or other paper 

documents. Direct deposit of payroll checks, automated payments, 

PC banking, and debit card transactions are the most common forms 

of electronic payments. More recently, benefits payments, 

annuities, dividends, and Internet transactions are being conducted 

electronically. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-4 

While these transactions have been increasing dramatically in the 

past 10 years, the process of conducting the transaction has 

remained relatively constant. All financial transactions involve an 

originating entity, the group making the payment, and a receiving 

entity (the group receiving the payment). The originating entity 

starts the transaction by querying the recipient agency about 

purchase of a specific product or service. Once an agreed-upon 

price and quantity are reached, an order is processed and the 

product is delivered from the recipient to the originator. However, 

the originator and recipient only exchange information and the 

product; there is no direct financial exchange. Rather, the 

originator contacts the Originating Financial Depository Institution 

(OFDI). The OFDI works within the clearinghouse system to 

transfer funds from the originating entity’s account to an account in 

a Receiving Financial Depository Institution (RFDI) where the 

recipient has an account. The process is depicted in Figure D-1. 

Figure D-1. Electronic Transactions in the Financial Services Sector 

ODFI RDFI

Originator Recipient

INFORMATION/$

INFORMATION/PRODUCT

INFORMATION INFORMATION

For the information that is passed to be interpretable, all parties in 

the transaction must be able to understand and transfer that 

information. To understand the information that is being passed, a 

common set of standards is needed to define the information, 

software is needed to interpret and manage the information, and an 

infrastructure is needed to send and receive the information (Clarke, 

1998). 
Appendix D — Financial Services Software Use and Development 

D-5 

Z Standards define the structure of how the information is 

passed between entities. The standard needs to be an 

unambiguous method of presenting the informational 

content of the data that are being passed. The standards 

dictate what information is included in the message and the 

ordering of that information. Without agreed-upon 

standards, it is not possible to efficiently communicate the 

information. The standards can either be agreed upon 

between parties when the transactions or negotiation costs 

are low, or they can be set by a third party when those costs 

are high. 

 The standard consists of the syntax that will be used in the 

message, the message design rules, the ordering of 

directories within the message, and the message itself. 

Three different standards are commonly used with the EDI 

world. ANSI X.12 is the dominant standard in North 

America, Australia, and New Zealand, while parts of Europe 

currently use UNTDI. TRADACOMS is the most widely 

used standard and has the most international appeal.1 

However, some firms and industries have developed their 

own standards that may or may not be based on any of the 

three most commonly used standards. 

Z Data management and translation software is needed to 

manage the flow of information and to translate messages 

once they are put into an agreed-upon standard. The 

software manages five specific activities that occur within 

the FEDI process: 

1. extracting data from a specific computer application 

within the computer system, 

2. translating the data into a transmittable format, 

3. transmitting the data within a message to the receiving 

firm, 

4. interpreting the message and the data by the receiving 

entity, and 

5. loading information into a specific computer application 

within the receiving entity’s computer system. 

Z Communications infrastructure is the physical technology that 

will actually pass the information from one entity to another. 

Originally, the infrastructure was electronic data tapes or 

diskettes that were sent from one firm to another. This approach 

was replaced by the development of closed networks that 

participating firms could use to transmit information.2 More 

recently, the communications medium has become the Internet 

 

1ANSI is the American National Standards Institute, UNTDI is the United Nations 

Trade Date Interchange Standard, and EDIFACT is the EDI for Administration, 

Commerce, and Transportation. 

2These closed networks are often referred to as value-added networks (VAN). 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-6 

and the routers and switches that manage the flow of 

information in the infrastructure. 

 D.1.2 Volume of Electronic Transactions 

The amount of electronic transactions has increased dramatically in 

the last 10 years. Table D-1 shows the inflation-adjusted growth in 

the total dollar value of transactions that are conducted 

electronically through FEDI mechanisms. In 1999, 77 percent of 

social security recipients received electronic payments and 

96 percent of government employees were paid electronically. 

While government use has been growing slowly over the last 5 

years, with annual growth rates of 3 to 8 percent, commercial use of 

electronic transactions has increased dramatically during that same 

time with annual growth rates consistently over 15 percent 

(NACHA, 2000). In 1999, approximately half of all private sector 

employees were paid electronically (NACHA, 2000).3 

In the United States, the two entities that have emerged to perform 

the greatest amount of automated clearinghouse transactions are the 

Federal Reserve and the Network of Automated Clearing Houses 

(ACH). The ACH is a nationwide private network governed by a set 

of operating rules that controls the transfer of electronic funds 

between depository financial institutions. 

Not only is the dollar value of transactions increasing, so is the 

number of transactions. In 1989, 1.3 billion transactions were 

conducted electronically; by 1999, 6.2 billion electronic 

transactions passed through clearinghouses. Although the Federal 

Reserve dominates the number of transactions that take place, it has 

recently become less prominent. In 1994, the Federal Reserve 

accounted for 82 percent of the total number of transactions that 

were processed electronically; by 1998, that number fell to under 

70 percent (BIS, 2000). 

A continuing trend is that the average dollar value per transaction 

has been decreasing while the total value and volume of 

transactions have been increasing. This trend is due to the 

decreased cost of electronic transactions and the increased public 

awareness and acceptance of these systems. In 1989, the inflation- 

 

3Part of the explanation of increased use of EDI has been attributed to the 

aggressive marketing campaign by the Social Security Administration that has 

featured Gavin McLeod and Ricardo Monteban. 
Appendix D — Financial Services Software Use and Developmen

t

D-7 

Table D-1. Extent of Electronic Market 

 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 

Total ACH volume (millions) 1,549 1,964 2,206 2,559 2,933 3,407 3,929 4,549 5,344 6,122 6,882  % increase 16.4 26.8 14.8 16.0 14.6 16.2 15.3 15.8 17.5 14.6 12.4 Dollar value (trillions) $6.1 $6.9 $7.80 $8.8 $10.1 $11.1 $12.1 $14.0 $18.1 $19.1 $20.3 Commercial (millions) 1,030 1,193 1,375 1,603 1,879 2,211 2,566 3,010 3,523 3,858 4,360  % increase 22.0 15.8 15.3 16.6 17.2 17.7 16.1 17.3 17.0 9.5 13.0 Government (millions) 519 521 531 554 574 601 625 678 764 832 848  % increase 6.6 0.4 1.9 4.3 3.6 4.8 3.9 8.5 12.7 8.9 1.9 On-us (millions) 250 300 402 480 595 738 861 1057 1,432 1,675  % increase 20.0 34.0 19.4 24.0 24.0 16.7 22.8 35.5 16.9 Total commercial (millions) 1,030 1,443 1,675 2,005 2,359 2,806 3,304 3,871 4,580 5,290 6,034  % increase 22.0 40.1 16.1 19.7 17.6 19.0 17.7 17.2 18.3 15.5 14.1 Number of companies using 

the ACH network (thousands) 

100 130 150 300 400 500 600 725 2,000 2,500 2,500 

% government employees 

using direct deposit 

75 78 79 82 83 84 91 92 95 97 97 

% social security recipients 

using direct deposit 

50 51 53 54 58 59 63 69 75 77 78 

Source: NACHA: The Electronic Payment Association. 2001. <http://www.nacha.org/news/Stats/ACH_Statistics_Fact_Sheet_2000.htm>. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-8 

adjusted average dollar per transaction was just over $5,000. In 

1999, the average fell to just over $3,000. This drop shows that an 

increasingly large number of lower value transactions are being 

conducted electronically, indicating that FEDI is becoming more 

involved in the routine and daily operations of financial institutions. 

 D.1.3 Firms in the Financial Services Sector 

The Census Bureau aggregates firms engaged in financial 

transactions into four broad categories by NAICS code.4 Table D-2 

provides establishment, revenue, payroll, and employment 

information for each category. 

Table D-2. Characteristics of Firms in the Financial Services Sector, 1997

 Establishments 

Revenue 

(millions) 

Payroll 

(millions) Employees 

521 Monetary Authorities 42 24,581 903 21,674 

522 Credit Intermediation and Related 

Activities 

166,882 808,810 98,723 2,774,910 

523 Securities, Commodity Contracts, and 

Other Financial Investments and Related 

Activities 

54,491 274,986 71,281 706,053 

524 Insurance Carriers and Related 

Activities 

172,299 1,072,784 92,230 2,327,306 

Source: 1997 Economic Census, Finance and Insurance Subject Series. 

Firms within the Credit Intermediation and Related Activities sector 

(522) are the most dependent on software and hardware to support 

financial transactions. Sector 522 comprises firms engaged in 

financial transactions processing, reserve activities, and 

clearinghouse activities. Firms conducting clearinghouse activities 

(subsector 52232) are primarily engaged in financial transaction 

processing, reserve activities, and liquidity services or other 

financial instrument clearinghouse services. Firms in this sector are 

engaged in both automated and manual clearinghouse activities. In 

1997, the clearinghouse subsector included over 1,200 firms with 

over 60,000 employees. 

 

4The appendix provides descriptions for each of the NAICS codes in sector 52. 
Appendix D — Financial Services Software Use and Development 

D-9 

The finance and insurance sector of the economy (sectors 523 and 

524) comprises firms whose dominant line of business is either 

financial transactions or facilitating those transactions. Transactions 

are broadly defined to include the creation, liquidation, or change 

of ownership of a financial asset. Within this broad definition, firms 

can be subclassified based on their activities. Three types of broad 

activities are used for this subclassification (Census, 2000): 

1. Raising funds by taking deposits or issuing securities and in turn 

creating liabilities. Firms then use these funds to acquire assets, 

make loans, or purchase securities. 

2. Pooling risk by underwriting insurance and annuities. Firms 

collect fees, insurance premiums, or annuities from engaging in 

these contracts. 

3. Providing specialized services facilitating or supporting financial 

intermediation, insurance, and employee benefit programs. 

 D.2 SOFTWARE USED BY FINANCIAL SERVICES 

PROVIDERS 

There are two main types of software used to facilitate the exchange 

of information in the financial services sector: FEDI software and 

clearinghouse software. FEDI software manages the flow of 

information across firms while clearinghouse software manages the 

flow of funds between financial institutions and the clearinghouse. 

Figure D-2 builds on Figure D-1 and adds the types of software that 

are used within each activity in the financial services sector. 

 D.2.1 FEDI Software 

Of the three key elements involved in FEDI transactions, the one 

that users have the most choice over is the software that they will 

use. Standards are often set by third parties or have developed 

through time, and the communications infrastructure is commonly 

dictated by the level of physical technology capital in the region. In 

contrast, there is a wide variety of commercial software available to 

support FEDI transactions. Software used in sending FEDI 

transactions must be able to extract the needed information from the 

appropriate computer application and be able to translate that 

information into a transferable document. Software used by the 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-10 

Figure D-2. Software Used in Financial Transactions 

Automated

Clearing House ODFI (ACH) Software RDFI

Originator Recipient

FEDI

Software

FEDI Software/

ACH Software

FEDI Software/

ACH Software

receiving party must be able to interpret the information that is 

shipped and be able to place it into the correct application in an inhouse format. 

FEDI Software Attributes 

There are few hard and specific rules to be followed when 

developing FEDI software. However, like most software products, 

some specific attributes are more important than others when 

evaluating FEDI software products. 

Z Ease of Upgrade. As firms increase the amount of 

transactions that are being conducted electronically, there 

will be an increased demand for system capacity. 

Additionally, technological improvements in the future will 

decrease costs of conducting each transaction. Because of 

the dynamic changes that are occurring, the ease of upgrade 

of the existing package is an important attribute when 

evaluating FEDI packages. 

Z Interoperability in Connectivity. Most FEDI software 

packages are able to connect and transmit information to all 

of the other major FEDI software packages. However, for 

some software packages, interoperability between systems is 

still an issue. 

Z Interoperability in Data Standards. No common standard 

exists for all EDI users. Consequently, even when 

information is successfully passed from one user to another, 

it still might be meaningless due to the lack of formatting 

and organization within the message. Although some 

software may be able to decipher information sent based on 
Appendix D — Financial Services Software Use and Development 

D-11 

multiple standards, there is not one software product that 

can decipher and organize all potential standards. 

Z Interoperability with Existing Systems. For the EDI system to 

increase productivity at the financial institution, it needs to 

be able to interoperate with the firm’s current software 

system. The cost of integration can be significant for two 

reasons. Financial institutions often have programs and 

databases that are separated by geography and activity. 

Developing EDI systems that can work with all of the legacy 

systems is often complex and time consuming. Secondly, 

while there is a movement towards real-time systems, a 

significant number of legacy systems operate under a batch 

system where information may only be updated once a day. 

EDI systems operate in real time and must constantly be 

updated. Changing the legacy system from a batch to a 

continuous process can also be exceptionally costly 

(Economist, 2000). Some firms that have adopted EDI 

technologies have had such severe interoperability problems 

that they would receive orders and information 

electronically and then print out hard copies to re-enter 

them in the firms’ computer system (EDI Aware, 1994). 

Development of FEDI Software 

FEDI cannot be undertaken without software. Firms have three 

potential ways to acquire FEDI software applications: in-house 

development, custom packages, or commercial products. 

Z In-House Development. Firms may develop an EDI system 

in-house. This approach is potentially effective if the 

number of trading partners is limited and if the standards for 

communicating across firms are well defined. 

 However, this approach often increases the cost of the 

software product, as well as development and 

implementation time. In-house development requires 

additional testing, design, and development that commercial 

and custom products may have already overcome. In 

addition, upgrades to the product will require additional 

development and testing time that would not be experienced 

with commercial products. 

Z Custom Packages. For very specific or tailored applications, 

a firm may wish to contract out the development of their EDI 

system. Custom-built solutions are often developed faster 

and more efficiently than in-house development, but may be 

more costly. 

Z Commercial Packages. As EDI activities have become 

increasingly common, commercial firms are starting to 

develop software products that they can market to firms as 

an alternative to custom or in-house solutions. Software 

packages range from simple translators that are used on a PC 

to sophisticated mainframe packages. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-12 

 Commercial products have several advantages over in-house 

or custom packages for three reasons. Most commercial 

products will meet any of the commonly agreed-upon 

standards. Additionally, commercial packages are often 

cheaper than in-house or custom development. Updating 

and maintenance are often easier, cheaper, and more 

effective when conducted by third-party experts. 

 D.2.2 Clearinghouse Software 

Applications software used in banks and financial institutions to 

support the flow of dollars in the financial transaction management 

activities described in Section 1 can be grouped into two software 

categories: core accounting software and clearinghouse software. 

Core accounting software is intrafirm software used to manage the 

flow of information within the financial institution. It is used to 

support all financial organization processing and customer 

information requirements, including commercial banking and 

relationship management, on-line teller and platform, customer 

information, universal loans, deposits, safe deposit, general ledger, 

item processing, and information storage and retrieval. Core 

accounting software has been used in the financial services industry 

for decades. Most systems have been in use for several years and 

were developed from scratch and currently run on mainframe 

systems. The systems have been in existence for so long that most 

software bugs that were in the original programming have been 

edited or repaired. In addition, new systems are rarely developed; 

rather, existing systems are adapted through time. 

Banks use clearinghouse software to manage the flow of financial 

information. As more transactions are conducted electronically, an 

increasing need for interoperability among financial institutions is 

required. Clearinghouse software is the interfirm software used to 

manage the flow of information across firms. Electronic payments 

take the form of payrolls, travel and expense reimbursements, 

annuities and pensions, dividends, government payments such as 

Social Security and Veterans benefits, bill payments, retail 

purchases, Internet purchases, corporate payments and treasury 

management, and the provision of foods stamps and other 

government cash assistance. Table D-3 gives the amount of 

clearinghouse transaction for the top 50 banks in 1999 by dollar 

volume. 
Appendix D — Financial Services Software Use and Development 

D-13 

Table D-3. Volume of Clearinghouse Transactions by Bank, 1999 

Rank Company Debits Credits Total 

Annual 

Change 

1 Chase Manhattan Corp. 307,993,871 202,367,963 510,361,834 11% 

2 Bank One Corp. 258,612,535 151,374,665 409,987,200 28% 

3 Wells Fargo & Co. 153,870,924 250,619,653 404,490,577 11% 

4 BankAmerica Corp. 105,709,913 222,951,819 328,661,732 10% 

5 First Union Corp. 79,504,081 162,264,680 241,768,761 –5% 

6 FleetBoston Financial Corp. 31,492,190 109,938,731 141,430,921 64% 

7 Wachovia Corp. 70,731,560 62,135,960 132,867,520 133% 

8 Northern Trust Corp. 53,258,648 75,173,037 128,431,685 14% 

9 KeyCorp 47,453,555 75,116,095 122,569,650 –13% 

10 Citicorp 40,839,018 47,171,293 88,010,311 15% 

11 PNC Financial Services 23,613,380 62,610,435 86,223,815 17% 

12 Mellon Financial Corp. 26,421,087 47,392,024 73,813,111 9% 

13 Amsouth Bancorp 55,069,158 14,203,805 69,272,963 215% 

14 U.S. Bancorp 18,389,109 42,699,282 61,088,391 –2% 

15 SunTrust Banks Inc. 18,932,200 39,674,600 58,606,800 1% 

16 Regions Financial Corp 19,620,663 35,737,701 55,358,364 287% 

17 National City Corp. 10,545,386 32,616,933 43,162,319 17% 

18 EFS National Bank 33,389,224 33% 

19 First National of Nebraska 18,804,181 13,067,312 31,871,493 20% 

20 ABN Amro North America 16,295,500 15,529,989 31,825,489 37% 

21 Harris Bankcorp 12,654,795 17,351,548 30,006,343 –9% 

22 State Street Corp. 14,973,357 14,508,704 29,482,061 8% 

23 UMB Financial Corp. 16,921,438 11,640,290 28,561,728 9% 

24 Firstar Corp 16,242,848 12,143,478 28,386,326 14% 

25 Allfirst Financial Inc. 12,177,000 14,147,000 26,374,000 36% 

26 Comerica Inc. 5,221,626 15,357,505 20,579,131 33% 

27 Michigan National Corp. 8,183,000 11,400,918 19,583,918 21% 

28 Marshall & Ilsley Corp. 9,072,805 9,628,490 18,701,295 34% 

29 Bank of New York Co. 4,619,452 13,712,305 18,331,757 6% 

30 First Tennessee National Corp. 3,621,517 14,552,812 18,174,329 –5% 

31 Unionbancal Corp. 5,346,824 10,726,357 16,073,181 30% 

32 Fifth Third Bancorp 4,595,033 11,443,865 16,038,898 16% 

(continued) 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-14 

Table D-3. Volume of Clearinghouse Transactions by Bank, 1999 (continued) 

Rank Company Debits Credits Total 

Annual 

Change 

33 Huntington Bancshares 5,168,812 10,440,950 15,609,762 1% 

34 Pacific Century Financial Corp. 5,867,095 7,620,753 13,487,848 46% 

35 First Security Corp. 5,493,213 7,237,901 12,731,114 7% 

36 Mercantile Bancorp. 6,235,884 6,399,155 12,635,039 38% 

37 Imperial Bancorp 1,401,653 11,086,989 12,488,642 — 

38 Compass Bancshares 7,534,402 4,802,032 12,336,434 26% 

39 HSBC USA Inc. 1,851,585 8,494,842 10,346,427 8% 

40 Summit Bancorp 2,236,379 8,073,368 10,309,747 3% 

41 First Premier Bank 8,876,037 1,347,176 10,223,213 27% 

42 BOK Financial Corp. 7,248,818 2,873,858 10,122,676 20% 

43 USAA Federal Savings Bank 819,011 8,767,469 9,586,470 –25% 

44 Commerce Bancshares 3,213,378 6,170,289 9,383,667 17% 

45 Universal Savings Bank 6,463,723 1,604,910 8,068,633 26% 

46 SouthTrust Corp. 2,095,521 5,179,092 7,274,613 15% 

47 CentraBanc Corp. 210,838 6,878,485 7,089,323 — 

48 Arvest Inc. 56,213 6,721,337 6,777,550 — 

49 Old Kent Financial Corp. 1,747,499 4,909,567 6,657,066 41% 

50 First Hawaiian Bank 4,232,484 2,344,783 6,577,267 28% 

Source: AmericanBanker.com. <http://www.americanbanker.com/PSUser/ABO_Display.htm?type=RankingBanks& 

master=1999/Holding/ACHYE1999.html>. 

The use of clearinghouse software has increased dramatically during 

the last several years. Roughly 70 percent of all transactions now 

take place electronically (NACHA, 2000). This increased reliance 

on electronic data transactions has increased the importance of 

software used in these transactions. However, the software often 

used in these transactions occasionally fails. For example, in April 

1999, CheckFree, which produces a clearinghouse software 

application, announced that a bug existed in their product that 

could have effected up to 350 banks throughout the country 

(Sullivan, 1999). 
Appendix D — Financial Services Software Use and Development 

D-15 

 D.2.3 The Developers of FEDI and Clearinghouse Software 

When a firm is deciding on what FEDI or clearinghouse software to 

implement, it can either develop its own software, have the software 

custom built, or purchase a commercial application. While some 

FEDI and clearinghouse software applications are commercially 

available, they often have to be adapted and altered to fit in with 

the firm’s existing legacy system. 

Tables D-4 and D-5 list the firms that have developed commercial 

clearinghouse software and FEDI products, respectively, and 

provides a description of the products. 

 D.2.4 Impacts of Software Failures 

The economic cost associated with inadequate FEDI and 

clearinghouse software can be substantial (System Transformation, 

2000). In some cases, software failures prevent transactions from 

occurring; in other cases, expensive work-arounds for failures need 

to be implemented. Examples of the problems and associated costs 

resulting from FEDI and clearinghouse software failures include the 

following: 

Z Data interchange interruptions or errors. If software fails, 

originating and receiving financial institutions may not 

receive some or all of the information involved in the 

transaction. Solutions to this problem include switching to 

paper or alternate ways to conduct a transaction, manually 

correcting the data once they have been received, or 

working with the data creators to correct the problem. In 

the extreme, the entire transacting process may need to be 

frozen until the problem is remedied. 

Z Credit card processing failure in the banking system. If FEDI 

software fails and the volume of credit card transactions is 

low, the transactions can be approved manually, although at 

a much higher cost. However, if transactions volume is 

high, merchants may have to purchase a new system or 

discontinue use of credit card transactions until the problem 

is remedied.

Z Trading system failure. If major financial trading systems fail 

and no suitable remedies exist, the impacts of the failure 

could be significant. If a second trading system cannot be 

quickly modified to accept these transactions, trading would 

have to be halted until the problem is resolved.
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-16 Table D-4. Clearinghouse Software Developers 

Company Name Software Product Description Cost Contact Information 

AMDEV, INC. AUTODRAFT CP Windows-based software program designed to 

structure data into the standard ACH format. 

$299 

AUTODRAFT

SB 

Windows-based software program designed to structure data into the standard ACH format. 

$99 

 NOVA Windows-based software program designed to 

allow financial institutions and third-party processors an effective method to manage origination and receipt of ACH files. 

$6,500 – $15,000 P.O. Box 1147 

Cookeville, TN 38503 Phone: 800/628-7638 Fax: 931/520-0542 

 NOVA 2000 ACH processing for high-volume processors, is 

a high-level implementation of an SQL-based version of the NOVA application. NOVA 2000 provides more features, flexibility, and performance than that of current mainframe processing applications. 

$250,000 

 VIP Windows-based software program designed to 

allow financial institutions and third-party processors an effective method for managing ACH files created by their clients. The system is used to streamline ACH processing, address timing issues, manage risk, and reduce cost. 

$3,500 

Barrington Corporation 

ACH for the PC PC-based software system that lets customers 

create their own ACH transactions from their PCs. Customers are able to create direct deposit of payroll, collect receivables, make local, state and federal tax payments, and consolidate and disburse funds with minimal intervention from the financial institution. 

$1,600 and $2,000 607 NP Avenue, Third Floor 

Fargo, ND 58102 Phone: 800/779-0183 Fax: 701/241-9930 

(continued) 
Appendix D — Financial Services Software Use and Development

D-17 

Table D-4. Clearinghouse Software Developers (continued) 

Company Name Software Product Description Cost Contact Information 

Bottomline Technologies 

BankQuest NetTransact PayBase 

Provides web-enabled billing, payment, and electronic banking solutions for the businessto-business market. 

 1-800-472-1321 

www.bottomline.com btinfo@bottomline.com 

 Paybase Allows organizations to autonomously create 

and send any type of electronic payment. It can be used to extract data from any corporate financial application, including Lotus and Excel spreadsheets, accounting packages, and check-issuing software, and produce any type of payment. 

$19,995 – $69,995 155 Fleet Street 

Portsmouth, NH 03801-4050 Phone: 603/436-0700 Fax: 603/436-0300 

Brinkman Technologies, Inc. 

CAP 2000 ACH Origination 

Processes incoming FedLine ACH and EDI transactions, ACH transactions originated by third-party payroll service providers, and ACH and EDI transactions originated by a bank’s commercial accounts. 

$5,000 – $17,000 1445 MacArthur Drive, 

Suite 122 Carrollton, TX 75007 Phone: 972/242-8090 Fax: 972/242-8676 

 Powerline ACH 

(RETAIL) 

Gives any size corporation or organization the ability to originate electronic transactions—ACH transactions. 

$100 – $2,000 

Entegrity Solutions 

AssureMail Entegrity AssureWeb Entegrity-Enabled Entegrity SafePages Entegrity SDP Entegrity Secrets 

Entegrity Solutions secures standard, custom, and legacy applications for the business-tobusiness (B2B) e-commerce needs of the Global 1000. 

 2077 Gateway Place, Suite 200 

San Jose, CA 95110 Phone: 408-487-8600 X104 www.entegrity.com info@entegrity.com 

Equifax EBanking Solutions Customer-Link ACH Origination Helps financial institutions and corporations eliminate the processing and reconciliation of 

checks. 

$5,000 – $15,000 103 Commerce Street, Suite 

120 Lake Mary, FL 32746 Phone: 888/453-5323 Fax: 407/829-4452 

(continued) 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-18 Table D-4. Clearinghouse Software Developers (continued) 

Company Name Software Product Description Cost Contact Information 

Fundtech Corporation 

Fundtech ACH(TM) 

Generate ACH payments or collections quickly and easily in a highly secure environment. 

$5,000 – $75,000 30 Montgomery Street, Suite 

501 Jersey City, NJ 07302 Phone: 201/946-1100 Fax: 201/946-1313 

Global Payment Systems 

Bank-On-It Transact 

A Windows-based ACH origination system for financial institutions and their corporate customers. Customers create ACH transactions and electronically send the files in a secure NACHA format to the bank. 

Starting at $3,800 1109 Main Street 

Boise, ID 83702 Phone: 800/727-5009 Fax: 208/342-0964 

HICOR Provides solutions for high-speed translation 

and transmission of ACH, CHIPS, financial EDI and S.W.I.F.T. formats. Customer base includes leading financial institutions and national clearinghouses. 

 17197 N. Laurel Park Dr., 

Suite 201 Livonia, MI 48152 Phone: 734-462-2244 www.hicor.net paul.wrenn@hicor.net 

Mellon Global Cash Management 

 Offerings include Internet payment and bill 

presentment. 

 Three Mellon Bank Center 

Pittsburgh, PA 15259-0001 412-236-8723 412-236-0504 www.mellon.com/inst/gcm/ gcm_direct_pgh@mellon.com 

National City Corporation 

 National City offers domestic and international 

standards translation of secured EDI/ACH/Wire/Check payments outsourcing, remittance delivery, receipt reporting, AR/AP process integration, and program planning and setup assistance. 

 Corporate Banking Business 

Solutions 1900 East 9th Street Cleveland, OH 44114 216-222-3633 mark_d_schulte@nationalcity.com 

(continued) 
Appendix D — Financial Services Software Use and Development

D-19 

Table D-4. Clearinghouse Software Developers (continued) 

Company Name Software Product Description Cost Contact Information 

Payment Technologies Inc 

TransLink TransLink is a low-cost, PC software package 

that enables ACH receivers to provide financial EDI Translation and Remittance Delivery Services. 

 5000 Ritter Road, Suite 103 

Mechanicsburg, PA 17055 717-506-2200 gigiw@paytec.com 

Politzer & Haney Auto Cash 

Transfer 

Originates and processes ACH transfers including direct deposit of payroll, direct debits, cash concentration, book transfers, reversals, returns, adjustments and corporate payments including vendor payments and state and federal tax payments. 

$2,500 – $4,000 320 Nevada Street 

Newton, MA 02460 Phone: 617/796-7700 Fax: 617/243-0033 

 Web Payments Fund transfer initiation system that allows 

corporate customers to originate ACH transactions using a standard browser (Netscape or Explorer) on their PCs. 

$10,000+ 

Prestige Systems, Inc. 

Automated Transaction System II 

Used for receipt, management, and origination of ACH transactions. 

$3,500 – $10,000 P.O. Box 3633 

Oakbrook, IL 60522 Phone: 630/850-7940 Fax: 630/850-7644 

Southern Business Technologies, Inc. 

BANC-COM /ACH 

PC-based, stand alone, entry-level ACH origination system designed for the small business customer of a financial institution. 

$395 3300 Highlands Pkwy, 

Suite 255 Smyrna, GA 30082 Phone: 770/436-2332 Fax: 770/434-9844 

Sterling Commerce 

 Sterling Commerce, an SBC Communications 

Company, provides solutions to in the accounts receivables, accounts payables, and other business processes. 

 4600 Lakehurst Court 

Dublin, OH 43016 614-793-7000 www.sterlingcommerce.com ed_armstrong@stercomm.com 

(continued) 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-20 Table D-4. Clearinghouse Software Developers (continued) 

Company Name Software Product Description Cost Contact Information 

Suchak Data Systems, Inc. (SDS) 

ACH Origination System 

Allows a financial institution to be a “lead bank” and originate ACH transactions to the ACH or Federal Reserve. 

$1,995 license fee. Customer module: $495. Annual software maintenance: $395 

2085 Baseline Road Grand Island, NY 14072 Phone: 716/773-1483 Fax: 716/773-7692 

UMB Bank, N.A. ACH/EDI origination and receipt. Specializing 

in industry types with specific needs such as mutual fund, insurance, utility and health care, EBP-BSP, and Internet Payments Processing. 

 928 Grand, 3rd Floor 

Kansas City, MO 64141-6226 816-860-7045 www.umb.com patricia.engelage@umb.com 
Appendix D — Financial Services Software Use and Development

D-21 

Table D-5. FEDI Software Developers 

Company Name Software Product Description Cost Contact Information 

Fundtech Corporation 

RECON$TAR Reconciliation product that facilitates account 

reconciliation, transaction matching, and MIS reporting. 

$40,000 30 Montgomery Street, 

Suite 501 Jersey City, NJ 07302 Phone: 201/946-1100 Fax: 201/946-1313 

St. Paul Software SPEDI*TRAN High performance EDI translation software 

designed for integration into a client-server EDI environment and provides extremely flexible mapping capabilities. 

$3,000 – $50,000 1450 Energy Park Drive 

St. Paul, MN 55108 Phone: 800/998-4334 Fax: 651/603-4403 

The Weiland Financial Group, Inc. 

Bank Administrator 

Designed to track signatories, banks, contacts, division, and account relationships. 

$6,000 – $8,000. 

 The 822 Express Solves the problem of transmitting account 

analysis statements regardless of their format, via the ANSI X12 822 electronic transmission standard. 

$8,000 – $50,000 900 North Shore Drive, 

Suite 185 Lake Bluff, IL 60044 Phone: 847/735-0577 Fax: 847/735-0419 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-22 

 D.3 SOFTWARE EMBEDDED IN HARDWARE USED 

TO SUPPORT FINANCIAL TRANSACTIONS 

In addition to software used to support FEDI and clearinghouse 

transactions, software is also embedded in hardware that is used to 

facilitate the physical transfer of electronic information. The 

process of passing information from one user to another is called 

routing. The two key pieces of technology involved in routing are 

routers and switches, both of which are combination of hardware 

and software that manage the flow of information. However, the 

software used to manage the flow of information is often inoperable 

across firms, routers, and area networks. Different products use 

different languages and different algorithms when making decisions 

about the passage of information. These differing decision-making 

processes create an interoperability problem. 

The following section describes how information is passed through 

an internetwork to get from one user to another. This section 

describes how software is used in the routing of information and 

provides an overview of the markets for routers and switches. It 

concludes with on overview of current problems and inadequacies 

in the production and use of the software used in routers and 

switches. 

 D.3.1 Internetwork Systems 

Regardless of the sector of the economy or the type of software 

being used, the passage of information from users within or across 

firms is a complicated process. When information is passed from 

one user to another, it is separated into smaller pieces and then 

shipped through nternetworks, such as local area networks (LAN) 

and wide area networks (WAN), via a series of communication 

protocols. 

Internetworks manage the transportation of information within and 

across firms. An internetwork is a collection of individual networks 

connected by networking devices that allow individual networks to 

act as if they are parts of a larger network. Internetworks were first 

developed by IBM and Digital and were time-sharing networks that 

attached terminals to mainframes to increase the amount of users of 

a mainframe. With the development of PCs, LANs were introduced 

that allowed users within a specific, relatively small, geographic 
Appendix D — Financial Services Software Use and Development 

D-23 

region to share access to resources and files. WANs soon followed 

that connected multiple LANs across normal telephone lines and 

eliminated the geographic proximity that they required. Modern 

internetworking links high-speed LANs to support the transmission 

of voice, high-bandwidth applications, and videoconferencing. 

LANs offer three key advantages over individually linked PCs. First, 

LANs let users interact more efficiently; users no longer have to ship 

disks back and forth to communicate electronically. Second, LANs 

eliminate the duplication of resources because each individual no 

longer needs his/her own software and hardware. Third, LANs 

decrease the difficulty of managing a network by creating a 

centralized management structure that eases maintenance, troubleshooting, and other management responsibilities of information 

administrators. 

The most widely agreed-upon system for transporting data is the 

Open Systems Interconnection reference model established by the 

International Standards Organization. Within this model, 

information from a software application is passed through a series of 

seven phases that specify each activity that occurs within the 

network. The seven phases can be separated into two overarching 

activities: application activities and data transportation activities. 

The application process interacts with the communications 

components of the software application being used. The data 

transportation process determines how the data will be transmitted 

across LANs. 

 D.3.2 Routing 

Routing is the term used to describe the process of passing 

electronic information from one entity to another. It refers to the 

passing of transportable units, called packets, through intranets and 

the Internet so they can be reassembled at their destination.5 Once 

the data have been divided into packets, two basic routing activities 

occur. First, the optimal path that the data will travel is determined; 

second, the information has to be passed from the starting point to 

the destination. The routing and switching algorithms are used in 

these two activities. 

 

5The applications software described in Section 2 labels and divides information 

into packets and reassembles these packets. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-24 

Correspondingly, the two main physical components that are 

involved in transmitting electronic information are routers and 

switches. A router is a device that uses multiple metrics to 

determine the optimal path along which network data should travel. 

Switches are used to facilitate the process of transporting the 

information packets along their optimal path through intranets and 

the Internet. The switching process is relatively straightforward 

compared to the determination of the optimal path. Table D-6 

provides a technical description of routers, switches, and several 

related components used in the passage of information. 

Table D-6. Network Devices 

Technology Description 

Router Network layer device that uses one or more metrics to determine the optimal path along 

which network traffic should be forwarded. Routers forward packets from one network 

to another based on network layer information. 

LAN switch High-speed switch that forwards packets between data-link segments. Most LAN 

switches forward traffic based on MAC addresses. This variety of LAN switch is 

sometimes called a frame switch. LAN switches are often categorized according to the 

method they use to forward traffic: cut-through packet switching or store-and-forward 

packet switching. Multilayer switches are an intelligent subset of LAN switches. 

Multilayer 

switch 

Switch that filters and forwards packets based on MAC addresses and network 

addresses—a subset of LAN switch. 

Access server Communications processor that connects asynchronous devices to a LAN or WAN 

through network and terminal emulation software. Performs both synchronous and 

asynchronous routing of supported protocols. Sometimes called a network access server. 

Routing Algorithms for Path Determination 

Several metrics are widely used to determine the optimal path that a 

packet of information will take when it is transported from a sender 

to a user. The most common metric is path length. Various routing 

algorithms and tables are used to determine the shortest path length 

or to optimize on another metric. Routing algorithms fill out and 

update routing tables with different pieces of information, 

depending on the metric being used to determine the path that the 

packet of information will follow. 

A variety of messages are passed between routers to update the 

progress of the information being passed through an intranet or the 
Appendix D — Financial Services Software Use and Development 

D-25 

Internet. The purpose of these messages is to constantly update the 

topology that information will pass through so the routers can 

continuously update and redetermine the optimal path that a piece 

of information will travel. For example, each router sends out 

routing updates that consist of part, or all, of its routing table as it 

gains information about the specific part of the network that it is 

traveling through (e.g., is there a significant amount of traffic here, is 

there a problem with one of the routers or switches). When routers 

are deciding how to send that piece or other pieces of information 

along, it incorporates the data that it receives from routers 

throughout the network to recalculate the optimal path for all 

information in and entering the network. 

There are several different types of routing algorithms. The 

differences in the types of algorithms have the potential to decrease 

the interoperability of network systems. Several characteristics 

cause routing protocols to differ: 

Z algorithm designs may have different objectives, 

Z different types of routing algorithms have different impacts 

on network resources, or 

Z routing algorithms use different metrics to determine optimal 

paths. 

Routing algorithms often have different objectives, including 

optimality, simplicity, robustness, rapid convergence, and flexibility. 

Different designers have different goals depending on which 

attribute they wish to maximize. Because all potential users of the 

routers in a network do not maximize the same attributes, potential 

suboptimalities emerge across users. 

Routing algorithms also differ in how they interact with the network. 

Algorithms may be classified as either static or dynamic, single or 

multi-path, and flat or hierarchical. Flat versus hierarchical 

algorithms present an example of how differences in algorithms can 

create interoperability problems. Flat algorithms operate in a twodimensional plane, while hierarchical algorithms operate in a threedimensional plane and can use a routing backbone to send the 

packets of information from one major hub to another and then 

send the packet to its destination address over local lines. These 

and other differences in algorithms delay the amount of time that it 

takes for information to be passed from a host PC to a destination 

PC. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-26 

The third way that routing algorithms differ is through the use of 

different routing metrics. Routing tables are designed to contain 

information that is used by the software within the router to select 

the best route that each packet of information will take to reach its 

destination. However, different routes use different types of 

software to build their routing tables. Examples of metrics that can 

be used in constructing the routing tables are path length, reliability, 

delay, bandwidth, load, and communication costs. Additionally, 

some routing tables combine several different metrics to form hybrid 

metrics. 

Even if each individual algorithm functions effectively and correctly, 

it might not interoperate with other routers. For example, if routing 

tables are not updated and routing algorithms are not compatible, 

users may not be able to pass packets of information between each 

other in a timely fashion (if they receive the information at all). 

Each of the four major producers of routers test the software that is 

used within each router, but testing of the interoperability across 

routers is not occurring to the same degree. Improved testing of the 

software that is used in routers to transmit information could be an 

effective mechanism to increase interoperability between routers. 

Switching Algorithms 

Relative to routing, switching algorithms are simple and fairly 

consistent across routing protocols. Switching consists of a host 

receiving a piece of information that tells it to send a packet to 

another host. Each step in the transfer of the packet is called a 

“hop.” When the information is first separated into packets it is 

assigned a Transmission Control Protocol (TCP). The TCP is 

designed to verify the delivery of information from one location to 

another. In other words, it checks to see if the information that was 

sent was received. Each packet is also assigned an Internet Protocol 

(IP). The IP is responsible for moving the information from one 

node to another and controls the hops that the packet takes. 

When a router examines the packet’s IP address, it determines if it 

knows how to forward that information to the intended location. If 

the router knows, then it forwards the information, if it does not 

know, then it drops the packet of information. The next hop that 

the information takes is either to its final destination or to another 
Appendix D — Financial Services Software Use and Development 

D-27 

router or switch that passes the information along. If the packet is 

sent to another router, then the process repeats. 

The International Standards Organization has developed a 

hierarchical terminology to describe this process. End systems (ES) 

are network devices that cannot pass information along. 

Intermediate systems (IS) are network devices with this ability. IS 

can be subdivided into intradomain IS and interdomain IS, which 

transmit packets of information within and across domains, 

respectively. 

 D.3.3 Market for Routers and Switches 

The market supply for routers and switches is relatively 

concentrated with four companies accounting for the majority of 

U.S. productions. In contrast, almost every major domestic 

company on the demand side is a consumer of routers and 

switches. There are two major types of consumers of this 

technology. Companies that provide web hosting, Internet access, 

and other electronic services to be used over the Internet are major 

consumers of the product. These companies must have routers and 

switches in place for their business to operate. The second group of 

consumers consists of companies that use routers and switches to 

pass information over LANs or WANs. These companies use the 

technology to pass information throughout their business, but it is 

not the business’s core technology. Rather, it is a technology that 

increases the efficiency with which they conduct their business. 

Market Size 

The market for routers grew steadily throughout the end of the 

1990s, increasing from less than 500,000 units shipped in 1997 to 

close to 800,000 (est.) shipped in the United States in 2000. The 

market for switches has expanded even faster, growing from roughly 

7.5 million units shipped in 1997 to close to 25 million in 2000 

(IDC, 2000). Total worldwide revenues from the sale of routers are 

estimated at over $6 billion, while sales from switches are estimated 

to pass $8 billion in 2000. 

Table D-7 shows the total revenue from the sale of routers, by type, 

since the fourth quarter of 1997. Legacy routers only transmit data; 

access routers are able to transmit both voice and data. Remote 

access routers are used by employees outside of the firm to transmit 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-28 

Table D-7. Total Sales of Routers (millions $) 

Date Legacy Routers Access Routers Remote Access Routers 

4Q97 1,383 574 NA 

1Q98 1,297 541 NA 

2Q98 1,372 571 NA 

3Q98 1,478 616 NA 

4Q98 1,071 475 556 

1Q99 1,112 575 867 

2Q99 1,077 566 1,113 

3Q99 1,044 585 1,397 

NA – not available. 

Source: The Dell’Oro Group. 2001. <www.delloro.com>. 

both voice and data, and are faster and more efficient than 

traditional modems. While sales of legacy and access routers have 

been flat or slightly declining over the last 2 years, sales of remote 

access routers have been increasing substantially. In the last quarter 

of 1997, the technology did not even exist; by the third quarter of 

1999, it was the greatest in terms of sales. 

Major Producers of Routers and Switches 

Four major companies produce the routers that are used to transfer 

information: Cisco, Nortel, Lucent, and 3Com. Each major 

company uses its proprietary software to write switching and routing 

algorithms for use in its routers. Table D-8 presents a list of 

companies and the proprietary software that they use. 

Table D-8. Router Market Shares of Major Firms 

Company 

Number of 

Router Types 

Total Sales 

(millions in 3rd quarter, 1999) Market Share Software Product 

Cisco 16 $1,360 72% IOS, ConFig Maker 

Nortel 8 $51 3% Preside 

Lucent $278 15% Hybrid Access 

3Com 5 $196 10% Enterprise OS Software 

Source: The Dell’Oro Group. 2001. <www.delloro.com>. 
Appendix D — Financial Services Software Use and Development 

D-29 

The measure of the number of router types that each company has is 

a broad measure of product categories. Numerous potential 

configurations and upgrades are available to the end user within 

each broad router type, effectively increasing the number of 

available products. We use total sales in the third quarter of 1999 

to get a common metric for the relative share of the market for 

routers and switches held by each firm. 

Consumers of Routers and Switches 

Not surprisingly, the major market for routers and switches is North 

America. Data for Nortel, Lucent, and 3Com are not available for 

regional sales of routers and switches, but they are available for 

Cisco. Because of Cisco’s market dominance, its sales are likely to 

be representative of global sales. Table D-9 presents Cisco’s 

regional sales of routers, switches, and other support hardware and 

services, as well as the regional percentage of total sales. 

Table D-9. Cisco’s Regional Sales 

 1999 2000 2001 

Regions Sales Percent Sales Percent Sales Percent 

Americas $8,088 64% $12,924 65% $15,130 68% 

Europe $3,216 26% $4,770 24% $6,288 28% 

Asia Pacific $825 7% $1,705 9% $2,384 11% 

Japan Pacific $459 4% $566 3% $1,540 7% 

Source: Cisco Systems. Cisco 2001 Annual Report. 2001. 

<www.cisco.com/warp/public/749/ar2001/online/financial_review/mda.html>. 

 D.3.4 Current Market Inefficiencies 

Developing the software and hardware needed to run an effective 

internetwork is a difficult task. The rapid growth in the sales of 

switches and routers and the significant technological 

improvements that have occurred in the second half of the 1990s 

has created routers and switches that may not interoperate. 

Insufficient testing of the software algorithms that are used in the 

operation of the routers and switches is contributing to the lack of 

interoperability. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

D-30 

Failures in the software used to run internetworks, which can be 

attributed to inadequate testing, can cause serious information 

delivery problems. Attributes of the software used to run 

internetworks that are of concern to developers are connectivity, 

reliability, network management, and flexibility. Connectivity is a 

challenge because various sites use different types of technology 

that may operate at different speeds. Reliability is a concern 

because individual users need information from other users in a 

timely manner. Network management ensures that centralized 

support is available to all users. Flexibility deals with the ability to 

adapt, add on to, and improve the network. 

Failure on any of these measures leads to several potential impacts, 

including the following: 

1. Decreased speed of information delivery. Poor 

communication and lack of interoperability across routers 

increase the delay time between a message being sent and 

received. This problem is worsened due to the different 

types of information that a router can transmit (e.g., voice 

versus data). Additionally, if the algorithms within each 

router are not well tested, they may not choose the optimal 

route to transmit data from one location to another, thereby 

increasing delay time. 

2. Failure of information delivery. Poor communication may 

not only be delaying the delivery of some packets of 

information; it may also be preventing some packets from 

ever being delivered. A lack of adequate testing of the 

software across routers may be contributing to this failure. 

 Convergence is the process of agreement among all routers 

on the optimal route for a packet of information. When a 

router becomes unavailable, routers distribute routing 

update messages that permeate networks, stimulating 

recalculation of optimal routes and eventually causing all 

routers to agree on these routes. If convergence occurs 

slowly, routing loops (a packet of information being 

continuously cycled from one router to another without 

being delivered) or network outages may occur. 

3. Inefficient router algorithms. Routing algorithms should be 

designed to be as simple as possible. The less software 

utilization within the routing algorithm, the more efficient 

the algorithm. A lack of adequate testing may be increasing 

the amount of software utilization within each router, in turn 

decreasing its efficiency. This problem is of special concern 

when the computer that is either sending or receiving the 

information has limited resources. 

4. Lack of robust routers. Failure to adequately test routers or 

the interoperability of routers may lead to fragile and weak 
Appendix D — Financial Services Software Use and Development 

D-31 

routers. Routing algorithms need to be robust and perform 

correctly during software or hardware failures, high loads, or 

other uncommon circumstances. Routers are located at 

major network junction points, so they can cause 

considerable problems when they fail. 

5. Reduced security of Internet and intranet traffic. Lack of 

adequate testing of routing algorithms may also be 

contributing to security violations. 

6. Inability to run specific programs. Because of the slowness 

of the software within the router, certain programs simply 

cannot be run over or across networks. 
Appendix E: 

Financial Services 

Survey Instruments 
E-1 

Survey of Routers and Switches Software Developers 

Being conducted by 

Research Triangle Institute 

On behalf of 

National Institute of Standards and Technology 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-2 

OMB NO: 0693-0031 Expires 10/31/2002 

This survey is authorized under Executive Order 12862, “Setting Customer Service Standards.” 

Your response is voluntary and all data collected will be considered confidential. Public 

reportings for this collection of information is estimated to average 25 minutes per response, 

including the time of reviewing instructions, searching existing data sources, gathering and 

maintaining the data needed, and completing and reviewing the collection of information. Send 

comments regarding this estimate or any other aspects of this collection of information, including 

suggestions for reducing the length of this questionnaire, to the National Institute of Standards and 

Technology, 100 Bureau Drive, Stop 3220, Gaithersburg, MD, 20899-3220 and the Office of 

Management and Budget Information and Regulatory Affairs, Office of Management and Budget, 

Washington, DC 20503. 
Appendix E — Financial Services Survey Instruments 

E-3 

Introduction 

As part of a research study for the National Institute of Standards and Technology (NIST), 

Research Triangle Institute (RTI) is conducting a survey of Routers and Switches software 

developers. The purpose of this survey is to learn about the incidence and cost of software bugs 

and errors to software users. 

The National Institute of Standards and Technology (NIST) is a non-regulatory federal agency 

within the Commerce Department’s Technology Administration. NIST’s mission is to promote 

economic growth by working with industry to develop and apply technology, measurements, and 

standards. NIST carries out its mission through four interwoven programs: NIST Laboratories, 

Baldridge National Quality Program, Manufacturing Extension Partnership, and Advanced 

Technology Program. See http://www.nist.gov for more information about NIST’s work. 

Our study would greatly benefit from your insights and experience. The findings from the study 

will be used to prioritize NIST’s research and development efforts addressing software 

improvements for manufacturing and financial industries. In addition, your company could 

benefit from identifying and quantifying basic software testing inadequacies. All participants will 

receive a copy of the final report. 

Your participation is voluntary, and your responses will be kept strictly confidential. Please note 

that questions regarding the number and type of software bugs will only be used to estimate the 

cost of software errors for the entire industry and will not be available to the public or shared with 

other survey participants. Only aggregate results will be included in the final report. 

Your establishment was randomly selected to participate in this survey. Please answer all 

questions as they pertain to your firm. 

The survey will take about 25 minutes to complete. Please answer each question by checking the 

appropriate answer(s) or providing your answer in the designated space. 

If you have any questions as you complete the survey, please contact Michelle Bullock at 

(919) 485-5599 or bullock@rti.org. 

Thank you in advance for your participation. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-4 

1. Background Information 

1.1 Please type your name, company name, and e-mail address on the lines below. 

Name: _______________________________________________ 

Company: ____________________________________________ 

E-mail: _______________________________________________ 

1.2 What types of software products are developed at your company? (Check all that apply.) 

1. Routers 

2. Switches 

3. Other (Specify): ________________________________________ 

1.3 What share of products can be classified as router or switch software? 

___________________ %

1.4 Please choose the North American Industry Classification System (NAICS) code(s) under 

which your company is classified. 

1. 334210-Telephone apparatus equipment manufacture, including data communications 

equipment (e.g., bridges, gateways, routers) manufacturing 

2. 511210-Packaged computer software computer publishers 

3. 541511-Custom computer software analysis and design services 

4. Other (Specify): ________________________________________ 

1.5 What was the approximate total number of employees employed by your company in 

2000? (Report a range of employees if necessary.) 

_____________________ 

1.6 What was the approximate value of total revenues (sales) reported by your company in 

2000? (Report a range of sales if necessary.)

_____________________ 
Appendix E — Financial Services Survey Instruments 

E-5 

2. Expenditures on Software 

Testing 

2.1 What were the total number of full-time equivalent (FTE) employees for your company in 

2000 who were involved in software testing and error correction? If you can’t answer this 

question for your entire company directly, take the total number of full-time equivalent 

(FTE) employees for your group/organization who were involved in software testing and 

error correction. Then multiply that number by the number of groups/organizations in 

your company that are involved in software testing and error correction. Please 

breakdown the total number according to the employee category. 

Employee Category 

Number of FTE Employees Involved in 

Software Testing and Error Correction 

Software Engineers/Programmers 

Software Engineers/Testers/QA Engineers 

Other: (Specify) ______________________ 

Total 

2.2 Did your company purchase testing software in 2000? 

_____ Yes 

_____ No (Go to Question 2.4) 

2.3 Please complete the following table based on testing software purchases that were made 

in 2000. 

Software Name 

Annual Expenditures for Test 

Software1 Type of Testing Conducted2

 

Notes: 

1. If Test Software was developed In-house, then estimate yearly internal expenditures, budget, or 

number of FTE employees engaged in development and maintenance of the test software. 

2. Choose all of the following types of testing that apply: a) Conformance to Specifications (also 

called Functional Verification Testing), b) Interoperability Testing, or c) Performance Testing (also 

called System Testing). 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-6 

2.4 Did your company purchase hardware to support software testing in 2000? 

_____ Yes 

_____ No (Go to Question 2.6) 

2.5 Please complete the following table based on testing hardware purchases that were made 

in 2000. 

Hardware Name 

Cost of 

Hardware 

Was the 

Hardware 

Leased? 

Expected Useful 

Life of the 

Hardware 

Type of Testing 

Conducted3

 

Notes: 

3. Choose all of the following types of testing that apply: a) Conformance to Specifications, b) 

Interoperability Testing, or c) Performance Testing. 

2.6 Did your company contract for external testing services in 2000? 

_____ Yes 

_____ No (Go to Section 4) 

2.7 How much did your company pay for external testing services in 2000 (expressed in 

dollars or as the number of Full Time Equivalent Employees)? If you can’t answer this 

question for your entire company directly, take what your group/organization paid for 

external testing services in 2000. Then multiply that number by the number of 

groups/organizations in your company that contracted for external testing services in 

2000. 

$ ____________________ 
Appendix E — Financial Services Survey Instruments 

E-7 

3. Incidence of Software Bugs and 

Errors 

In this section of the survey, we segment the software development process into five stages and 

investigate 

• where bugs are typically detected and 

• where errors are introduced. 

Software bugs and errors can be generally divided into three broad categories; design, 

implementation and delivery errors. Design bugs are flaws in the underlying design of the 

software architecture typically resulting in redesign. Implementation and delivery bugs are errors 

in the way the programmer tries to achieve the design during coding. For the purpose of this 

survey, we are limiting our definition of a software bug to implementation and delivery coding 

errors. 

The five stages are 

• requirements gathering and analysis/architectural design, 

• coding/unit testing, 

• integration and component/RAISE (Reliability, Availability, Install Serviceability, and Ease 

of Use) system testing, 

• early customer feedback/beta test programs, and 

• post-product release (found by customer after purchase or acceptance). 

For the following questions, please consider a representative new router or switch software 

development project or a new version. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-8 

3.1 Bugs are found throughout the software development process. Bugs are detected 

internally through formal testing and externally by users during beta testing and business 

operations. In the table below, please identify the stages in which bugs are typically 

found. Please list either the number of bugs typically detected (per development project 

or lines of code) or the distribution (percentage) of bugs detected across the five stages. 

Stages of Development 

Number of Bugs Detected 

at Each Stage or 

Distribution of Bugs 

Detected Across Stages 

Requirements gathering and 

analysis/ architectural design 

 

_____% 

Coding/unit testing _____% 

Integration and component/ 

RAISE system testing 

 

_____% 

Early customer 

feedback/beta test programs 

 

_____% 

Post-product release _____% 

 per project 

 per _____ lines of code

 Total = 100%

3.2 Bugs can be introduced into software at various stages in the development process. For 

bugs found during the coding/unit testing phase, in what stage was the bug likely to be 

introduced? Please indicate the probability of the error being introduced during the 

following stages. 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

100% Total 

3.3 For bugs found during the integration and component testing phase, in what stage was the 

bug likely to be introduced? 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

_____% integration and component 

100% Total 

3.4 For bugs found during beta testing, in what stage was the bug likely to be introduced? 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

_____% integration and component 

100% Total 
Appendix E — Financial Services Survey Instruments 

E-9 

3.5 For bugs found by customers during the post-product release phase, in what stage was the 

bug likely to be introduced? 

_____% requirements gathering and analysis/architectural design 

_____% coding/unit testing 

_____% integration and component 

100% Total 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-10 

4. The Cost of Fixing Bugs 

In this section, we investigate the costs of locating the source of bugs and of correcting bugs 

(referred to as fixing or repairing bugs). We are primarily interested in how these costs vary with 

respect to where the bug was introduced and at what stage in the software development process 

the bug is detected. 

4.1 The severity and, hence, the cost of fixing a given bug may depend on what stage the bug 

was introduced into the software. In the table below, provide the average cost of fixing 

bugs (in terms of labor hours) that are introduced during the three main development 

stages presented in Section 3. For this cost table, assume that the bug is detected in the 

same stage that it was introduced. 

Stage the Bug was Introduced 

Average Number of Hours to Correct an Error 

Introduced and Found in this Stage 

Requirements gathering and analysis/ 

architectural design __________ hours 

Coding/unit testing __________ hours 

Integration and component/RAISE system testing __________ hours 

4.2 It is widely assumed that bugs caught later in the software development process are more 

expensive to repair. In the following table, please indicate how much more expensive it is 

to repair a bug created in the requirements gathering and analysis/architectural design

stage if it is not detected until later in the software development process (i.e., Not detected 

until coding, integration, beta testing, or post-product release). Provide your answer in 

terms of how many times more expensive it is to repair the bug in later stages compared to 

detecting and repairing it during the stage in which it was introduced. 

Stage Where Errors Introduced in 

Requirements Gathering and Analysis/ 

Architectural Design Stage are Detected 

How Many More Times as Costly is it to 

Repair a Bug if it is Detected After the 

Stage it is Introduced 

Requirements gathering and analysis/ 

architectural design Stage bug is introduced 

Coding/unit testing _____ times as costly to repair 

Integration and component/RAISE system testing _____ times as costly to repair 

Early customer feedback/beta test programs _____ times as costly to repair 

Post-product release _____ times as costly to repair 
Appendix E — Financial Services Survey Instruments 

E-11 

4.3 Now consider bugs introduced during the coding stage. How much more costly is it to 

repair these bugs if they are detected in later stages? 

Stage Where Errors Introduced in 

Coding Stage are Detected 

How Many More Times as Costly is it to 

Repair a Bug if it is Detected After the 

Stage it is Introduced 

Coding/unit testing Stage bug is introduced 

Integration and component/RAISE system testing _____ times as costly to repair 

Early customer feedback/beta test programs _____ times as costly to repair 

Post-product release _____ times as costly to repair 

4.4 Finally, consider bugs introduced during the integration stage. How much more costly is 

it to repair these bugs if they are detected in later stages? 

Stage Where Error Introduced in 

Integration Stage are Detected 

How Many More Times as Costly is it to 

Repair a Bug if it is Detected After the 

Stage it is Introduced 

Integration and component/RAISE system testing Stage bug is introduced 

Early customer feedback/beta test programs _____ times as costly to repair 

Post-product release _____ times as costly to repair 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-12 

5. A World with Improved Testing 

Resources 

NIST is interested in estimating the costs of inadequate software testing tools and resources to U.S. 

companies. 

5.1 Please describe the shortcomings of the testing software (testware) and processes you 

currently use to detect and fix bugs in your software products. 

For example: 

• Testware products are not as compatible with our software development environment 

as we had expected. 

• Testware products assume a software development process that is different than the 

one we use. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

5.2 What improvements would you like to see in software testing programs and procedures?

For example: 

• What are there capabilities you would like to see that are not available in current 

testware? 

• Could testware products function better if there were fewer user interface or 

interoperability problems? 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

For the questions below, please consider how the distribution of where bugs are detected and the 

cost of repairing bugs would change if the improved testing procedures and tools you described 

above were available. (Even if you were not able to describe all the software testing 

improvements you would like to see in Questions 7.1 and 7.2, feel free to broadly envision a 

world with an enhanced software testing infrastructure when answering the questions below.) 

Note: We are assuming that the number of bugs introduced during the software development 

process remains unchanged—only the developer’s ability to detect and fix bugs is changed in our 

hypothetical new and improved world. 
Appendix E — Financial Services Survey Instruments 

E-13 

5.3 In a world with improved software testing tools, how would the distribution of where 

(when) bugs are detected change? 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

5.3a In the table below, we have repeated your response to Question 3.1. Please 

indicate how this distribution you provided would change in a world with 

improved software testing tools. 

Current World 

(your response to 3.1) 

World with improved Testing 

Tools 

Stages of Development 

Number of 

Bugs Detected 

at Each Stage 

Or 

Distribution of 

Bugs Detected 

Across Stages 

Number of 

Bugs Detected 

at Each Stage 

Or 

Distribution of 

Bugs Detected 

Across Stages 

Requirements gathering 

and analysis/ 

architectural design _____% _____% 

Coding/unit testing _____% _____% 

Integration and 

component/RAISE system 

testing _____% _____% 

Early customer 

feedback/beta test 

programs _____% _____% 

Post-product release _____% _____% 

 per project 

 per _____ 

lines of code

Total = 100% per project 

 per _____ 

lines of code

Total = 100%

5.4 How would the cost of repairing bugs change with improved software testing tools? 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-14 

5.4a In the table below, we have repeated your response to Question 4.1. Please 

indicate how the number of labor hours would change with improved tools for 

locating and repairing bugs. 

Stage the Bug was Introduced 

Current Labor 

Hours to Fix 

Average Bug 

(your response to 

Question 4.1)

World with 

Improved Testing 

Tools 

Requirements gathering and analysis/ 

architectural design _______ hours _______ hours 

Coding/unit testing _______ hours _______ hours 

Integration and component/RAISE system 

testing _______ hours _______ hours 
Appendix E — Financial Services Survey Instruments 

E-15 

6. Time to Market 

6.1 For a representative new router or switch software product or new version developed by 

your company, what is the average time to market? If you can’t answer this question for 

your entire company directly, use the average time to market for a representative new 

router or switch software product or new version developed by your group/organization. 

________________ years 

6.2 If no testing activities were needed (i.e., no error detection and repair), how much would 

this shorten your time to market? 

________________ years 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-16 

7. Customer Service Cost 

7.1 Does your company typically provide installation assistance for your router and switch 

software products? 

_____ Yes 

_____ No (Go to Question 6.5) 

7.2 Please describe the type of installation assistance your company provides. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

7.3 What were your total installation costs (annual expenditures or Full Time Equivalent 

Employees) in 2000? 

$ ____________ 

7.4 What percentage of your installation costs are due to bugs or errors found during 

installation? 

______________ percent 

7.5 Does your company provide long-term service contracts or other types of after-sales 

customer service? 

_____ Yes 

_____ No (Go to Section 7) 

7.6 Please describe the type of after-sales service your company provides. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

7.7 What were your total after-sales service costs (annual expenditures) in 2000? 

$ ____________ 

7.8 What percentage of your after-sales service costs are related to bugs found by customers 

during business operations versus those costs related to user error or other causes not 

related to defective software? 

______________ percent 
Appendix E — Financial Services Survey Instruments 

E-17 

7.9 In a world with improved software testing tools, how much could your customer 

installation expenditures be reduced? 

______________ percent 

7.10 In a world with improved software testing tools, how much could your other after-sales 

customer service costs be reduced? 

______________ percent 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-18 

8. Comments 

8.1 Please provide any additional comments that would be helpful in evaluating how 

improved testing tools would impact your company’s software development costs and 

product quality. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

We thank you for your participation. 

Please indicate below if you would like to receive a copy of the final report. 

_____ Yes, please send a copy 

_____ No 
Appendix E — Financial Services Survey Instruments 

E-19 

Survey of FEDI and Clearinghouse Software Users 

Being conducted by 

Research Triangle Institute 

On behalf of 

National Institute of Standards and Technology 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-20 

OMB NO: 0693-0031 Expires 10/31/2002 

This survey is authorized under Executive Order 12862, “Setting Customer Service Standards.” 

Your response is voluntary and all data collected will be considered confidential. Public 

reportings for this collection of information is estimated to average 20 minutes per response, 

including the time of reviewing instructions, searching existing data sources, gathering and 

maintaining the data needed, and completing and reviewing the collection of information. Send 

comments regarding this estimate or any other aspects of this collection of information, including 

suggestions for reducing the length of this questionnaire, to the National Institute of Standards and 

Technology, 100 Bureau Drive, Stop 3220, Gaithersburg, MD, 20899-3220 and the Office of 

Management and Budget Information and Regulatory Affairs, Office of Management and Budget, 

Washington, DC 20503. 
Appendix E — Financial Services Survey Instruments 

E-21 

Introduction 

As part of a research study for the National Institute of Standards and Technology (NIST), 

Research Triangle Institute (RTI) is conducting a survey of financial service companies that use 

financial electronic data interchange (FEDI) and clearinghouse software. The purpose of this 

survey is to learn about the incidence and cost of software bugs and errors to software users. 

Our study would greatly benefit from your insights and experience. The findings from the study 

will be used to assist NIST to identify and prioritize technical infrastructure needs in the area of 

software testing. In addition, your company could benefit from identifying and quantifying how 

software bugs and errors affect companies in the financial service sector. All participants will 

receive a copy of the final report. 

Your participation is voluntary, and your responses will be kept strictly confidential. Please note 

that questions regarding the type and cost of software bugs will only be used to estimate the 

aggregate impacts for the entire industry, and individual responses will not be available to the 

public or shared with other survey participants. Only aggregate industry-level results will be 

included in the final report. 

Your establishment was randomly selected to participate in this survey. The survey will take 

about 20 minutes to complete. Please answer all questions as they pertain to your firm by 

checking the appropriate box(es) or providing text in the designated space. 

If you have any questions as you complete the survey, please contact Michelle Bullock at 

(919) 485-5599. 

Thank you in advance for your participation. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-22 

1. Background Information 

1.1 Please type your name, company name, and e-mail address on the lines below. 

Name: ___________________________________________________ 

Company: ________________________________________________ 

E-mail: ___________________________________________________ 

1.2 What types of products or services are provided at this establishment? (Circle all that 

apply.) 

1. Credit intermediation 

2. Securities, commodity contracts, and other financial services 

3. Insurance 

4. Other (Specify):___________________________________ 

1.3 Please fill in the North American Industry Classification System (NAICS) code(s) under 

which this establishment is classified. 

NAICS Code(s) 

1.4 What was the approximate total number of employees employed by your company in 

2000? (Report a range of employees if necessary.) 

_____________________ 

1.5 What was the approximate value of total revenues (sales) reported by your company in 

2000? 

_____________________ 
Appendix E — Financial Services Survey Instruments 

E-23 

2. The Use of FEDI and 

Clearinghouse Software 

Systems 

2.1 In the table below, please list the FEDI and clearinghouse software your company 

currently maintains and indicate when it was installed and what you project to be its 

remaining life expectancy? 

Name of Software Product (all versions) Year Installed 

Number of Years Expected To 

Remain in Operation 

Example: RECON$TAR Example: 1999 Example: 10 more years

 

2.2 What were the total number of (full-time equivalent [FTE]) employees in 2000 involved in 

operating and supporting the software listed in Table 2.1? 

Type of Activity 

Number of 

Employees 

FEDI transactions 

Clearinghouse transactions 

 

Please Read Before Continuing!

In Sections 2 through 5, we ask about the incidence and cost of FEDI and clearinghouse software 

bugs and errors at this establishment. Personnel responsible for monitoring and maintaining FEDI 

and clearinghouse software at this establishment should be able to provide the best answers to 

these questions. If necessary, please share your password with colleagues at your company and 

ask them to complete the appropriate sections. 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-24 

3. Incidence and Costs of Software 

Bugs and Errors 

This section focuses on the software bugs and errors your company encounters in the FEDI and 

clearinghouse systems you employ and how they affect your business operations. 

3.1 Does the information you are providing reflect all the FEDI and clearinghouse transactions 

at your company? 

 _____ yes 

 _____ no: what percentage of your companies transactions are represented in your 

 responses? _____% 

 

3.2 Software bugs can either be major (systems shut down) or minor (a slight inconvenience to 

your work). In 2000, how many major and minor software bugs or errors did your 

company encounter in your FEDI or clearinghouse software? 

_______________ major 

_______________ minor 

 

3.3 What percentage of those bugs or errors were attributable to problems with routers and 

switches vs. problems with your transaction software? 

_______________ % routers and switches 

_______________ % transaction software 

100% total 

3.4 Was 2000 a typical year for software problems, or has your company been making an 

above average number of software upgrades, potentially leading to an uncharacteristically 

large number of software problems? 

_____ typical year 

_____ unusual year with _____% more software/system improvement projects than usual 

3.5 For the typical major error that you had in 2000, what was the impact on your company’s 

business operations? 

1. Shut down all transactions for _________ hours, resulting in __________ transactions not 

completed and $__________ lost sales. 

2. Temporarily delayed transactions 

3. Other impact: please explain _________________________________________________ 
Appendix E — Financial Services Survey Instruments 

E-25 

3.6 Did your company experience any repair costs associated with the software failure, such 

as time to re-enter lost data or repair data archives? 

1. _____ No 

2. _____ Yes: ________ labor hours spent on repair 

 ________ value of lost information 

 ________ other repair or damage costs, 

 please explain ___________________________________________ 

3.7 Do you think your company experienced any long-run competitive effects from the 

software failure(s), such as lost reputation or lost market share? 

Yes/no: lost reputation 

Yes/no: lost market share 

Yes/no: Delayed product or service introduction by _____ month leading to lost sales of ___ $/month 

_____ other impacts 

3.8 For minor software bugs in your FEDI or clearinghouse software, did these result in 

increased operating costs or decreased efficiency? 

_____ No (Go to Section 4) 

_____ Yes: please explain __________________________________ 

3.8a Are these one-time expenditures due to developing patches and work arounds or 

are they ongoing problems affecting efficiency? 

 _____ one-time costs 

 _____ ongoing costs 

3.8b Approximately what are these annual expenditures? 

 $_____________ 
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-26 

4. Software Life-Cycle Costs 

Associated with Bugs and Errors 

In this section, we investigate how software bugs and errors affect the life-cycle costs of 

purchasing and operating FEDI and clearinghouse software. 

The Purchase Decision 

4.1 How much time and resources are spent researching a new FEDI or clearinghouse 

software package before a purchase decision is made? 

_____ calendar time (months) 

_____ labor expenditures (number of or fraction of FTEs) 

4.2 Could the search time and resources have been reduced if you had better information 

about the quality of the software products you were comparing? 

_____ Yes: What would be the change in 

 _____ fewer months 

 _____ fewer number of FTEs 

4.3 Because of potential software bugs and errors, do you typically delay purchasing new 

versions of FEDI or clearinghouse software? 

_____Yes: What is the typical delay? _____ months 

_____ No 

Software Installation and Acceptance 

4.4 What was the average time it took for installation and acceptance testing for your FEDI 

and clearinghouse software? 

______________ months 

4.5 What parties were involved in the installation and 

performance testing of your FEDI and clearinghouse software, 

and what was the share of effort/expenditures? 

_____% software developers 

_____% your company 

_____% third-party integrator (consultant) 

100% 

Acceptance testing is the 

process of determining 

whether software 

determines predefined 

acceptance criteria. 
Appendix E — Financial Services Survey Instruments 

E-27 

4.6 What were your company’s own expenditures on installing and performing acceptance 

testing of FEDI and clearinghouse software in 2000? 

______ total labor hours 

4.7 How much did your company spend on external consulting services for installation and 

acceptance testing services of FEDI and clearinghouse software in 2000? 

$ ____________________ 

4.8 If the software you purchased contained fewer bugs and errors, how much would your 

labor and external consulting expenditures for installation and acceptance testing have 

been reduced? 

______________ percent 

Maintenance Costs 

4.9 How much money did your company spend on maintenance 

contracts for FEDI and clearinghouse software in 2000? 

$____________________

4.10 In 2000, how much money did your company spend on FEDI 

and clearinghouse software upgrades and maintenance that 

were not covered by a maintenance contract? 

$____________________

4.11 What percentage of maintenance costs were associated with bugs and errors embedded in 

the software? 

______________ percent 

Redundant System Costs 

4.12 After installation and acceptance, did your company maintain redundant backup systems 

for some period of time in case the new system failed? 

_____Yes 

How long did you maintain the backup system? _____ months 

What was (is) the estimated cost of maintaining these systems? __________ $/month 

_____ No 

Maintenance contracts

include any agreements 

with outside agencies that 

those agencies will perform 

periodic checks of system 

integrity and/or provide free 

upgrades and/or correct 

errors in installed software. 

Contracts may include 

training and technical 

support.
The Economic Impacts of Inadequate Infrastructure for Software Testing 

E-28 

5. The Impact of Reducing the 

Number of Software Bugs and 

Errors 

In this section, we investigate how the costs associated with bugs and errors in FEDI and 

clearinghouse software would change if the number of bugs and errors embedded in these 

software products were partially reduced. Our discussions with industry indicate that it is not 

feasible or economical for software developers to produce “bug-free” software. However, NIST is 

interested in knowing what the cost savings would be if your company encountered a 25, 50, 75, 

or 90 percent reduction in software errors. 

We anticipate that the rate at which the cost of bugs decreases as the number of bugs decreases 

will not be the same for all of the cost categories that have been discussed previously in the 

survey. For example, some cost-bug relationships may be linear (i.e., a 50 percent reduction in 

bugs leads to a 50 percent reduction is costs), and some may be nonlinear (i.e., a 50 percent 

reduction in bugs may lead to less than a 50 percent reduction in costs because even a small 

number of bugs requires testing, backup, systems, etc.). 

5.1 In the table below, please estimate the percentage cost reduction associated with different 

percentage reductions in bugs for each of the major cost categories discussed earlier in the 

survey. Two examples are provided. In Example A, costs decline proportionally as the 

number of bugs are reduced. In Example B, costs do not decline proportionally, and a 90 

percent reduction in bugs does not eliminate over half of the costs because other normal 

costs may be associated with maintenance or installation. 

Cost Reductions as a Function of Bug Reductions 

Cost Categories 

Percentage Reduction in Bugs or Errors in 

FEDI and Clearinghouse Software 

 25% 50% 75% 90% 

 

Example A (linear) 25% 50% 75% 90% 

Example B (nonlinear) 10% 15% 40% 45% 

 

Major failure costs 

Minor failure costs 

Purchase decision costs 

Installation costs 

Maintenance costs 

Redundant system costs 
Appendix E — Financial Services Survey Instruments 

E-29 

6. Comments 

6.1 Please provide any additional comments that would help us evaluate the cost of FEDI or 

clearinghouse software bugs and errors to your company. 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

_____________________________________________________________________________________ 

We thank you for your participation. 

Please indicate below if you would like to receive a copy of the final report. 

_____ Yes, please send a copy 

_____ No