August 2004 

Measuring Benefits from the National 

Technology Transfer and Advancement Act 

Final Report 

Prepared for 

JoAnne Overman 

Kevin McIntyre 

Mary Saunders 

Standards Services Division 

National Institute of Standards and Technology 

Gaithersburg, MD 20899-2100 

Prepared by 

William J. White 

RTI International 

Health, Social, and Economics Research 

Research Triangle Park, NC 27709 

RTI Project Number 08628 

RTI Project Number 

08628 

Measuring Benefits from the National 

Technology Transfer and Advancement Act 

Final Report 

August 2004

Prepared for 

JoAnne Overman 

Kevin McIntyre 

Mary Saunders 

Standards Services Division 

National Institute of Standards and Technology 

Gaithersburg, MD 20899-2100 

Prepared by 

William J. White 

RTI International* 

Health, Social, and Economics Research 

Research Triangle Park, NC 27709 

 

*RTI International is a trade name of Research Triangle Institute.

Contents 

 

 Executive Summary ES-1

1 Introduction 1-1

1.1 Background Investigation.............................................................1-2

1.2 Analysis Framework and Metrics .................................................1-2

1.3 Data Collection and Survey Instruments......................................1-3

1.4 Pretest Results and Recommendations.......................................1-3

1.5 Organization of the Report...........................................................1-4

 2 Background Investigation 2-1

2.1 Requirements of NTTAA..............................................................2-1

2.2 Taxonomies of Standards Used within Government 

Organizations ...............................................................................2-2

2.2.1 Procurement Standards ..................................................2-4

2.2.2 Regulatory Standards .....................................................2-4

2.2.3 Conformity Assessment ..................................................2-5

2.3 Methods for Measuring Economic Benefits from the Use of 

VCS..............................................................................................2-6

2.3.1 Case Studies: Before-and-After or Counterfactual ........2-7

2.3.2 Surveys ...........................................................................2-7

2.3.3 Qualitative Analysis: Success Stories............................2-8

2.3.4 Program Evaluation: NPV or B/C of 

Standardization Process .................................................2-8

2.4 Literature Review: Economic Benefits of Using VCS..................2-9

2.4.1 Department of Defense and Its Service Branches..........2-9

2.4.2 Consumer Product Safety Commission........................2-12

2.4.3 Department of the Interior .............................................2-13

2.4.4 Occupational Safety and Health Administration............2-13

iii 
Measuring Benefits from the National Technology Transfer and Advancement Act 

2.4.5 Food and Drug Administration ......................................2-14

2.4.6 Department of Energy...................................................2-14

2.4.7 Environmental Protection Agency.................................2-15

3 Analysis Framework and Metrics 3-1

3.1 Taxonomy of Benefits Evaluation Methods..................................3-1

3.2 Brief Review of Relevant Economic Theory.................................3-4

3.3 Process for Conducting Counterfactual Analysis.........................3-7

3.3.1 Defining Scope of Analysis .............................................3-7

3.3.2 Identifying Cost Impact Areas .........................................3-9

3.3.3 Technical and Economic Metrics and Discussion 

Guides...........................................................................3-10

3.3.4 Developing Aggregation and Benefits 

Quantification ................................................................3-10

4 Data Collection and Survey Instruments 4-1

4.1 Consumer Product Safety Commission.......................................4-1

4.2 Facilities Construction in the Department of Defense..................4-2

4.3 Laboratory Accreditation under National Environmental 

Laboratory Accreditation Program ...............................................4-2

5 Pretest Results and Recommendations 5-1

5.1 Results of Agency Pretests..........................................................5-2

5.2 Implications for Broad-Scale Implementation of RTI's 

Methodology.................................................................................5-3

5.3 Recommendations about Next Steps ..........................................5-4

 References R-1

Appendix – Interview Guides for Case Studies A-1

iv
Contents 

 

Figures 

 

Figure 3-1. Mansfield’s Approach for Evaluating Benefits of Technological 

Change...................................................................................................3-5

Figure 3-2. Simplified Equilibrium Diagram of Government Agency 

“Production”............................................................................................3-6

v 
Measuring Benefits from the National Technology Transfer and Advancement Act 

Tables 

 

Table 4-1. Consumer Product Safety Commission Study.......................................4-3

Table 4-2. Military Construction Study ....................................................................4-4

 Table 4-3. Laboratory Accreditation Study..............................................................4-6

vi 
Executive Summary 

In fulfilling their missions, government agencies and organizations 

reference a variety of technical standards in rule making, in 

procurement, and in other activities. Since its enactment in 1996, 

the National Technology Transfer and Advancement Act (NTTAA) 

has required that all federal agencies and departments use 

standards that have been developed or adopted by Voluntary 

Consensus Standards (VCS) bodies, except where such use is 

inconsistent with existing laws or impractical. The requirements of 

NTTAA were implemented by the Office of Management and 

Budget (OMB) through revisions to their Circular A-119, entitled 

"Federal Participation in the Development and Use of Voluntary 

Consensus Standards and in Conformity Assessment Activities." 

NTTAA and the OMB circular assign significant implementation 

and coordination responsibilities to the National Institute of 

Standards and Technology (NIST). Each year NIST is required to 

report on progress made by federal agencies toward complying 

with NTTAA, as well as their participation in VCS organizations. 

NIST organized, and still leads, the Interagency Committee on 

Standards Policy (ICSP), through which it coordinates many of its 

leadership activities in this area. 

In the eight years since passage of NTTAA, NIST has reported 

significant progress toward meeting its objectives, but results have 

been variable across government organizations. Although several 

agencies have enthusiastically supported the use of VCS and 

reported steady annual improvement to NIST, others have shown 

little or no standards-related activity or have not sent in reports. A 

number of agencies have exempted significant portions of their 

standards activity due to statutory conflicts. 

ES-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

NIST’s Standards Services Division (SSD) has worked to improve 

the overall level of compliance with NTTAA and to help agencies 

realize the benefits of the use of VCS processes. Recently, SSD 

contracted with RTI International (RTI) to begin to document the 

economic benefits that agencies achieve by moving from agencyspecific standards to third-party VCSs. This report summarizes 

our efforts in this respect, and offers recommendations for future 

initiatives by SSD that may help build interest and commitment 

from agencies whose support has been lacking. 

RTI conducted background research to better understand NTTAA, 

its requirements and implications, and to speak with standards 

executives who have been supporting its implementation within 

the agencies. We have also devoted resources to understanding 

the complex conformity assessment process, including the 

accreditation of testing and calibration laboratories and the 

recognition of those accreditations by regulators, procurement 

agents, and other users of test/calibration data. These latter 

activities are consuming a large amount of effort from federal 

agencies, affected businesses, and NIST. 

We then applied an RTI methodology for benefit-cost analysis, 

treating the use of VCS as a beneficial technological change. We 

developed a counterfactual model with which to compare 

hypotheses about potential benefits and costs. We identified 

impact areas in which economic outcomes would differ between 

actual and counterfactual cases, i.e., the with-VCS and withoutVCS scenarios. A set of technical and economic metrics was 

developed for each of the impact areas, with a standard set of

measurement tools to be applied, including benefit-cost ratio, 

internal rate of return, and net present value. 

The scope of the work assignment included development and 

testing of data collection tools, but not a large-scale data collection 

effort. Due to the small number of compliant agencies involved 

and the complexity of the metrics being collected, we chose to

develop a set of interview guides rather than a formal survey. An 

interview guide is a set of questions that supports a structured

conversation between the researcher and the agency respondent. 

The interactive format allows for discussion of more complex 

topics than can typically be addressed in a survey. 

ES-2 
Executive Summary 

According to the requirements of the project, we then conducted a 

pretest of the data collection instrument to assess its usefulness 

for broad-scale implementation across compliant agencies. In all, 

we constructed three interview guides to pretest: one for agencies 

that primarily reference standards in their regulatory activities, one 

for agencies that use standards in procurement, and one for 

agency efforts in laboratory accreditation and recognition. 

Although we planned to test all three instruments, we were only 

able to get agency support to test the one designed for use by 

regulatory agencies. 

The results of the pretest were very informative, although not in 

the way we initially intended. Findings suggested that there needs 

to be additional groundwork before a successful economic 

analysis can be done across the universe of government 

agencies. The completed pretest, along with the informal 

conversations held with knowledgeable representatives at several 

agencies, has convinced us that the data needed to inform any 

comprehensive benefits analysis is not routinely collected or 

analyzed by the agencies themselves. It is our judgment that 

analysis using currently available data would be incomplete and 

perhaps misleading. 

Fortunately, SSD has the means to begin collecting the type of 

data that would be useful in performing economic assessments in 

the future. The questions posed in the annual report could be 

modified or augmented to yield much of the information from 

which quantitative performance measures can be derived. RTI 

recommends that SSD may wish to work through existing NTTAA 

coordinating groups to make suitable modifications to the annual 

report. 

The following specific changes may be considered. The report 

already poses questions on outputs of standards management 

processes, i.e., the number of new standards created and existing 

standards supported, both for VCS and agency- or governmentspecific standards. If conformity assessment continues to be an 

important area of agency activity, a suitable output measure would 

be helpful, such as the number of Memoranda of Understanding 

(MOUs) or other mutual recognition agreements being supported 

or newly negotiated. 

ES-3 
Measuring Benefits from the National Technology Transfer and Advancement Act 

On the input side, the data now being reported are less useful for 

quantitative economic analysis. Instead of gathering information 

on activity or interest, such as the number of agency employees 

participating in VCS meetings or conferences, it would be more 

helpful to request the level of effort devoted to creation and 

maintenance of VCSs and/or government-specific standards. 

Metrics could include the numbed of full-time-equivalent (FTE) 

employees devoted to negotiating and writing new standards and 

supporting existing ones.

If data of this type could be put in place over the next couple of 

years, it may be possible to perform the type of quantitative 

assessment envisioned by RTI within perhaps 3-5 years. It is also 

possible that some agencies we were unable to contact already 

have such metrics in place. If so, additional pretests performed by 

SSD might shed more light on the question of the economic 

impact of NTTAA on government agencies. 

ES-4 
 1 Introduction 

In February 1996, Congress passed the National Technology 

Transfer and Advancement Act (NTTAA) and the president signed 

the Act into law the following month. Recognizing the importance 

of technological innovation in creating economic growth, NTTAA 

made a number of statutory changes intended to spur the 

development and diffusion of innovations within the United States. 

One important provision of NTTAA requires that all federal 

agencies and departments use technical standards that have 

been developed or adopted by Voluntary Consensus Standards 

(VCS) bodies, except where such use is inconsistent with existing 

laws or impractical. NTTAA also directed the National Institute of 

Standards and Technology (NIST) to coordinate the conformity 

assessment activities of government and private standards 

organizations, to eliminate complexity and duplication of effort. 

NTTAA requirements were implemented by the Office of 

Management and Budget (OMB) through revisions to their Circular 

A-119, entitled "Federal Participation in the Development and Use 

of Voluntary Consensus Standards and in Conformity Assessment 

Activities." The OMB circular assigned NIST the roles of collecting 

information on activities of departments and agencies on an 

annual basis, publishing the compiled data in an annual report, 

and leading a coordinating committee that monitors compliance 

with the provisions NTTAA. 

NIST's Standards Services Division (SSD) currently manages the 

activities enumerated in NTTAA and the OMB circular, which have 

1-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

remained largely unchanged since initially promulgated. In order to 

assess the impacts of their efforts, as well as to encourage agency 

involvement in VCS and conformity assessment activities, SSD has 

sought research on related economic benefits achieved by 

compliant agencies. SSD recently contracted with RTI 

International (RTI) to conduct background research and develop a 

methodology for measuring the benefits accruing to agencies from 

their use of VCS. This report summarizes our efforts and results in 

pursuing these objectives. 

 1.1 BACKGROUND INVESTIGATION 

To develop an analytical framework for estimating economic 

benefits, RTI needed to assemble a great deal of background

information about NTTAA and OMB's directive for compliance with 

its provisions, roles of key stakeholders in developing VCSs and 

government-specific standards, and standards in general. To

support this need, we engaged in a significant amount of library 

and online research, conducted a detailed literature review on 

both academic and business literature, and contacted a number of 

key players at NIST, other government agencies, and at 

Standards Development Organizations (SDOs). In addition, we 

had informal discussions with in-house RTI resources, each of 

whom has significant experience with one or more key agencies. 

This type of intensive research is an important piece of every 

economic analysis we perform, and the complexity of standards in 

general and conformity assessment in particular made this 

research especially critical for this study. Still, our primary skill and 

expertise is in economics, and our understanding of the technical 

details of the field we are studying pales in comparison to that of 

our clients and those with whom we consult. As with most of our 

projects, this step concluded with the preparation of a background 

memorandum, which we forwarded to NIST. We summarize the 

contents of this memorandum in Section 2 of this report. 

1.2 ANALYSIS FRAMEWORK AND METRICS 

The Technology Economics and Policy (TEP) program at RTI has 

developed a methodology for use in performing benefit-cost 

analysis across a wide variety of government and private 

innovation processes. When proposing to evaluate the benefits of 

1-2 
Section 1 — Introduction

NTTAA, we treat the use of VCS processes as a form of 

technological change, i.e., an innovation to an existing process. In 

order to quantify potential benefits, we first identified areas in 

which the proposed change has an impact. We next developed a 

set of technical and economic metrics that will help us evaluate 

economic costs and benefits in each of the impact areas. 

Finally, a data collection process was proposed to inform the 

metrics and allow calculation of net economic benefits. 

Depending on the scope of the analysis, we typically estimate 

measures of private benefit (the profit impact on private firms), 

public benefits (which accrue to government agencies or 

consumers in the public at large), or the total social benefit (which 

combines both public and private economic benefits). In the case 

of VCS and NTTAA, we were asked to focus specifically on the 

potential benefits that would accrue to compliant federal agencies, 

although there are costs and benefits that are felt by participating 

private firms as well. 

 1.3 DATA COLLECTION AND SURVEY INSTRUMENTS 

The scope of the current project included development of tools for 

data collection and pretesting those tools. A case study approach 

was recommended, and our preliminary analysis convinced us 

that case studies were the best way to proceed. In order to 

provide some structure to the study, however, we often prepare 

interview guides, which contain a list of questions we expect will 

yield the quantitative and qualitative information needed to inform 

the economic metrics and to allow estimation of benefits. 

Because our research involves a limited number of agencies, we 

suspected that selection of target organizations would not be a 

challenge, and the respondents for the annual NTTAA report 

made an ideal contact list for placing the interviews. As the body 

of the report details, we were able to establish contacts in each of 

the areas for which we developed an interview guide, although we 

had more difficulty in placing the pretests required by the project. 

 1.4 PRETEST RESULTS AND RECOMMENDATIONS 

The pretests themselves were envisioned as face-to-face or 

telephone interviews with knowledgeable individuals at each of the 

targeted agencies. The primary audience would be standards

1-3 
Measuring Benefits from the National Technology Transfer and Advancement Act 

executives who have a broad focus on voluntary and agency- or 

government-specific standards. As with any data collection 

activity, follow-up activities were planned to clarify initial 

responses and to obtain additional qualitative and quantitative 

data. 

We anticipated successful pretests in the regulatory- and 

procurement-focused agencies, which we had identified early on 

as key segments of the agency population. After receiving 

feedback from SSD, we designed and requested a pretest of an 

instrument focused on laboratory accreditation, an area of special 

interest to NIST and one in which there is a great deal of need for 

voluntary consensus approaches. With the results of the pretests 

in hand, recommendations on how to proceed would follow. 

1.5 ORGANIZATION OF THE REPORT

The remainder of this report provides details of the sections

summarized above, including the results of interactions with 

agency personnel during the study. We have also included an

appendix containing the interview guides that were sent to the

agency representatives from whom we requested assistance. 

1-4 
 Background 2 Investigation 

A major objective of government departments and agencies 

should be to deliver valuable services to the public as efficiently as 

possible. The pursuit of this goal has been central to many efforts 

to improve the performance of government departments and 

agencies during the past quarter century, including those initiated 

by various administrations and several mandated by Congress. 

Academic papers published as early as the 1970s show that, in 

areas where private-sector businesses, educational institutions, 

and other nonprofit organizations have significant expertise, there 

is a great deal to be gained if all parties work cooperatively to 

optimize resource use and avoid duplication of effort (Whitaker, 

1972; Miller, 1985). Congress applied this reasoning with respect 

to the use of VCS in its passage of NTTAA in 1996. 

2.1 REQUIREMENTS OF NTTAA 

Under the provisions of NTTAA, all federal agencies are required 

to use VCS in preference to agency- or government-specific 

standards for procurement and rule-making activities, except 

where such use is impossible or inappropriate. OMB issued a 

revised version of OMB Circular A-119, entitled “Federal 

Participation in the Development and Use of Voluntary Consensus 

Standards and in Conformity Assessment Activities,” to implement 

the provisions of NTTAA, and to define reporting and oversight 

responsibilities. The OMB circular assigned NIST the following

2-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

roles: collecting information on the activities of departments and 

agencies on an annual basis, publishing the complied data in an 

annual report, and leading a coordinating committee to monitor 

compliance with the provisions of NTTAA. In addition, NIST’s 

strengths in information management and communication made it 

a natural clearinghouse for VCS-related publications and 

materials. 

Although it was not clearly specified in either the initial act or the 

OMB circular that implemented it, testimony in a 2000 

congressional hearing strongly suggested that quantitative benefitcost analyses and better communication of success stories would 

support OMB and NIST’s efforts with NTTAA (U.S. House, 2000). 

It seems evident that the members of Congress who worked to 

pass the bill were philosophically inclined to believe that the use of 

VCS would generate benefits; however, some evidence would 

also be required to convince others that the bill’s objectives were 

being met. As early as 2000, SSD was committed to begin the 

process of measuring economic benefits from the use of VCS. 

In addition to NTTAA, several other public laws or agency 

initiatives require or encourage the use of voluntary standards. 

Statutes governing activities of the Consumer Product Safety 

Commission (CPSC) have directed it to use VCS whenever 

possible. Since 1981, the commission has been required to make 

a determination that no effective voluntary standard is in place 

before issuing a mandatory safety standard. The Health 

Insurance Portability and Accountability Act (HIPAA) of 1996, the 

Telecommunications Act from the same year, and the FDA 

Modernization Act of 1997 all contain language that encouraged 

or required the use of VCS. The massive Department of Defense 

(DoD) standardization initiative, also called the Milspec Reform, 

had as one of its most important elements the replacement of 

DoD-specific procurement standards with voluntary standards that 

are supported by independent SDOs. 

2.2 TAXONOMIES OF STANDARDS USED WITHIN 

GOVERNMENT ORGANIZATIONS 

Several different types of standards are used by governmental

organizations, and these standards function in a variety of ways to 

help the agencies perform their missions. The International 

2-2 
Section 2 — Background Investigation 

Organization for Standardization (IOS) has identified eight general 

types of standards in common use: basic, terminology, testing, 

product, process, service, interface, and data requirements 

(Breitenberg, 1987). A more useful taxonomy for this project

divides government standards by function into procurement and 

regulatory standards. Procurement standards are used in the 

purchase of goods and services, and typically describe 

specifications, dimensions, performance characteristics, or quality 

attributes of the items being purchased. Regulatory standards are 

those specified within the regulations that businesses and other 

affected entities must follow. 

Critical to the use and acceptance of any standard is the process 

of conformity assessment, the means by which firms can show

that their products, processes, or services adhere to the standard 

in question. For process- or service-oriented standards, an audit 

or inspection of the firm’s operations may be required to 

demonstrate conformity. For product standards, the assessment 

may be accomplished by physical measurement, performance 

testing, chemical or biological analysis, or a variety of other means 

and often conducted in a company-owned or independent thirdparty laboratory. These laboratories, in turn, must be evaluated to 

make sure that their results will be accepted; this important aspect 

of conformity assessment is called “certification or accreditation.” 

Procurement and regulatory standards are both likely to include 

and depend on laboratory testing and accreditation for their 

effectiveness. 

The use of any type of standard provides several kinds of benefits 

to the parties involved, some of which are easily measurable and 

others that are more difficult to quantify. Actions that need to be 

performed many times can be done consistently and repeatedly, 

with low transaction costs and minimal risk. Quality assurance is 

made simpler, less costly, and more effective by adherence to

standards and established procedures. The efficiency and low 

cost of communicating information should support higher quality 

decision making, foster increased competition, and thus lower 

overall costs. In the next two subsections, we discuss the use and 

benefits of each of the standards classes described above. 

2-3 
Measuring Benefits from the National Technology Transfer and Advancement Act 

 2.2.1 Procurement Standards 

Government organizations, like firms in the private sector, 

purchase a large number of goods and services in order to fulfill 

their objectives, including such disparate items as vehicles, office 

supplies, laboratory equipment, construction and maintenance 

services, and computer equipment. In addition, departments and 

agencies that produce final goods and services, such as DoD, the 

National Aeronautics and Space Administration (NASA), the

Department of Energy’s National Laboratories, and the 

Department of Veteran’s Affairs, purchase a wide array of raw 

materials, capital equipment, and labor services to support their 

production activity. The use of suitable standards allows agencies 

to purchase exactly what they need, as often as they need it, 

without investing an inordinate amount of time in engineering,

determining specifications, and negotiations. These agencies 

reduce the risks that the products or services will fail to function as 

intended (quite important for components of jet fighters or rockets) 

and ensure that the vendor will deliver the expected quality and 

performance. 

Potential benefits from the use of VCS in procurement are fairly 

straightforward. Because private firms also buy many of the

products and services purchased by governments, there is an

incentive for a single standard that can be used in both arenas. It 

makes little economic sense to have a government- or agencyspecific standard that is different from one developed for use in 

the private sector by an industry trade organization or SDOs. In 

fact, if government agencies make up a small fraction of demand 

for the product or service, there may be a significant cost penalty 

paid by any agency that insists its own standards be met. Even if 

the government is the only buyer for a product, as is often the 

case for DoD’s military equipment, the inclusion of vendors in the 

VCS process may lead to a higher quality standard and lower 

product costs. 

 2.2.2 Regulatory Standards 

Just as large procurement-intensive government departments 

require and benefit from standards in their purchasing activities, 

regulatory agencies specify many standards in the rules that they 

promulgate. These organizations include the Environmental 

Protection Agency (EPA), Food and Drug Administration (FDA), 

2-4 
Section 2 — Background Investigation 

Occupational Safety and Health Administration (OSHA), National 

Highway Traffic Safety Administration (NHTSA), CPSC, and

several others. The rule-making efforts of these organizations 

typically require specific actions by affected firms to comply with 

the provisions of the regulation in question. In the case of 

environmental rules, firms may be required by EPA to install 

equipment that is designed in accordance with a set of materials 

standards, to measure effluents from their facilities using several 

process standards, and to check instrument calibrations using 

specified laboratory standards. CPSC adopts existing voluntary 

consensus safety and performance standards (or develops new 

mandatory standards if no suitable VCSs are in place) for each of 

the hundreds of products that the commission is responsible for 

regulating. 

The benefits of the standards themselves include many of the 

cost, quality, and efficiency considerations noted previously for 

procurement standards. In addition, using an appropriate set of 

standards allows a regulatory agency to enact and enforce a rule 

that applies equally to all of the firms under that agency’s 

jurisdiction, ensuring fairness and consistency across affected

industries. The use of VCS should bring economic benefits to the 

rule-making agencies, although more indirectly than in the case of 

procurement. The VCS process allows the government agency to 

share standards development and maintenance costs with SDOs 

or trade associations, thereby lowering the overall costs to the 

regulated firms as well as to the government. By making it easier 

and cheaper for firms to comply with the rule, VCS should also 

allow agencies to better achieve their regulatory intent. Finally, 

the inclusion of the affected firms in the rule-making process may 

increase those firms’ incentives to cooperate with the agency, 

both during and after the regulations are promulgated, thereby 

potentially lowering enforcement costs. 

2.2.3 Conformity Assessment 

A typical government agency purchase contract may specify that 

all required physical measurements and performance tests on the 

products being purchased, as well as on raw and intermediate 

materials, be conducted at and certified by an accredited testing 

firm. Environmental regulations almost always require firms that 

emit pollutants to air, water, or other media to use chemical 

2-5 
Measuring Benefits from the National Technology Transfer and Advancement Act 

analyses performed by a laboratory that has been accredited for 

the specific tests involved. Often, these agencies maintain one or 

more agency-specific accreditation processes, which they insist 

that suppliers or affected firms use. For example, testing drinking 

water for bacteria or mercury might involve separate and 

incompatible accreditation processes for each of the states in the 

United States, as well as for the federal EPA. Laboratories that 

perform tests on several environmental media such as air, water, 

and solid waste, are currently subject to separate accreditation

from three different offices within EPA. 

Clearly, there are potential benefits from coordinating certification 

and accreditation processes between agency departments, 

among agencies within the federal government, across federal 

and state governments, and internationally as well. A number of 

initiatives have begun to stimulate the required sharing of 

information and resources. NIST leads the National Voluntary 

Laboratory Accreditation Program (NVLAP), which provides thirdparty accreditation in 25 different program areas. NIST supports 

the National Cooperation for Laboratory Accreditation (NACLA), 

whose primary mission is to evaluate and recognize laboratory 

accreditation bodies in the U.S. EPA heads up the National 

Environmental Laboratory Accreditation Program (NELAP), which 

is attempting to coordinate mutual recognition of laboratory 

accreditation across states in the U.S., beginning with drinking 

water and wastewater analysis. 

2.3 METHODS FOR MEASURING ECONOMIC BENEFITS

FROM THE USE OF VCS 

The literature on program evaluation and RTI’s experience in 

microeconomic assessment suggest several ways in which the 

economic benefits generated by the use of the VCS process could 

be measured. The relatively small number of government 

agencies involved suggests that case studies could be profitably 

employed for a few high-profile or very active organizations. 

Placement of a survey would provide a more systematic 

assessment, although at a greater cost and with a significant

increase in effort. Once a measurement methodology to evaluate 

individual initiatives was proven, it could be included as an 

optional (or even mandatory) part of the agencies’ reports to SSD. 

2-6 
Section 2 — Background Investigation 

Of course, success stories and other types of qualitative reporting 

provide useful illustrations of economic impacts, even if detailed 

quantitative analysis is not forthcoming. Finally, major agency 

initiatives, like the DoD’s standardization initiative, might warrant a 

full-blown program evaluation, with estimation of net present value 

and benefit to cost ratios for the organization’s investment in VCS. 

Each of these potential approaches is discussed further in this 

section of the report. 

2.3.1 Case Studies: Before-and-After or Counterfactual 

A case study is an ideal way to determine the impact of specific 

change or systems improvement for a government department or 

agency. The amount of time devoted to locating key resources, 

establishing communications, and collecting data offers the 

opportunity to obtain a richness of information not easily 

obtainable by other means. The qualitative and quantitative 

information obtained can help inform subsequent data collection 

activities in which time and complexity are more limited. 

Two types of measurement methods are often used to quantify 

costs and benefits in case studies: before-and-after and 

counterfactual. In the former method, revenues (benefits) and 

capital and operating costs are estimated for the periods prior and 

subsequent to implementation of the change being studied; the 

difference between the net costs represents the benefit of the 

change. In a counterfactual analysis, a hypothetical scenario is 

evaluated, and the scenario’s net costs are compared to those 

costs that are actually incurred in order to determine the impact of 

the systems improvement. As many of the costs in developing 

VCSs or agency-specific standards are one-time in nature, we 

expect that a counterfactual analysis might prove more productive 

in the present project. 

 2.3.2 Surveys 

Although case studies can provide detailed analyses of specific 

events or innovations, they tend to be idiosyncratic in nature, and 

extrapolation of results from one case is unlikely to prove an 

accurate basis for estimating impacts across disparate 

organizations or systems improvements. A more systematic tool 

is needed both to provide this cross-cutting analysis and to 

aggregate individual results up to the level of an entire agency or 

2-7 
Measuring Benefits from the National Technology Transfer and Advancement Act 

department. If NIST is interested in evaluating the economic 

impact of NTTAA on the entire population of federal agencies, a 

structured survey is a more appropriate tool. Using information 

gained from representative case studies, a survey instrument can 

be designed and placed that will request numerical or categorical 

responses to a limited number of questions. RTI has had great 

success conducting small surveys with between 10 and 150 

respondents for a number of NIST program evaluations and 

microeconomic assessments, and the methodologies used can be 

easily adapted to quantify the benefits from the use of VCS. 

2.3.3 Qualitative Analysis: Success Stories 

As the literature review in the next section details, the simplest 

method for determining the benefits derived from an innovation is 

the creation of a qualitative “success story.” In this approach, 

costs and benefits are described verbally rather than numerically, 

and directions of changes are indicated: quality was improved, 

labor input was reduced, prices fell, etc. Success stories 

frequently compare results of two or more initiatives and attempt 

to place these initiatives in rank order by some subjective criteria. 

Although these simple analyses are valuable to building 

understanding and creating energy to support change, these 

analyses cannot provide much in the way of estimating benefits or 

helping guide resource allocation decisions. 

2.3.4 Program Evaluation: NPV or B/C of Standardization Process 

If one improvement initiative is complex enough or of significant 

size, it may be worthwhile to conduct a stand-alone assessment of 

that effort using program evaluation techniques. This is, in effect, 

an extended case study, with the outcome being a determination 

as to whether the resources invested in the effort were properly 

allocated. In this quantitative analysis, detailed costs and 

revenues are collected for the entire duration of the project, and 

measures of economic return, such as net present value (NPV) 

and/or benefit-to-cost ratio (B/C), are calculated. Of the efforts we 

have reviewed as a part of this VCS evaluation task, only the DoD 

standardization initiative (Milspec Reform) would seem worthy of a 

separate program evaluation; however, such an effort would be 

well beyond the scope of this project. 

2-8 
Section 2 — Background Investigation 

 2.4 LITERATURE REVIEW: ECONOMIC BENEFITS OF

USING VCS 

Even though we expended a great deal of effort in searching for 

journal articles, papers, press releases, and Web pages 

discussing economic benefits from the use of VCS, we found very 

few documented savings estimates and a relatively small number 

of success stories. Even the SDOs and industry trade 

associations, who spend a high percentage of their effort (and

their member firms’ financial contributions) developing voluntary 

standards, seem to have almost no examples of quantified 

benefits. Our experience matches a comment made about 2 

years ago by a member of the National Society of Professional 

Engineers: “Despite the achievements noted earlier, success 

stories have still been the exception rather than the rule” 

(Ruggieri, 2001). 

Nonetheless, we were able to find a fair number of papers that 

claimed economic benefits, and a few that provided numerical 

estimates of savings from the use of VCS or from the substitution 

of an agency- or government-specific standard with a voluntary 

one. The results of this literature search are organized by 

department or agency, beginning with DoD, the most active in 

standards reform. 

2.4.1 Department of Defense and Its Service Branches 

DoD has been called the most diversified and largest developer 

and user of standards in the United States, and possibly the world. 

DoD primarily uses standards for procurement purposes, requiring 

equipment and parts suppliers to provide products that conform to 

detailed product specifications. 

Beginning after World War II and responding to inadequate or

nonexistent private-sector standards, DoD developed its own set 

of agency-specific standards, known as Milspecs. For products 

that were common to military and civilian uses, Milspecs in most 

cases did not conform to the industry standards and thereby 

forced suppliers to develop separate methods and machinery to 

accommodate both markets. A study of military contractors by the 

management consulting company Cooper & Lybrand determined 

that the use of a group of 120 Milspecs increased the price DoD 

2-9 
Measuring Benefits from the National Technology Transfer and Advancement Act 

pays for industry goods and services by about 18 percent 

(Coopers & Lybrand/TASC, 1994). 

DoD has been working to reduce these excess costs by 

supporting the use of private-sector standards since 1962; and in 

fact, the original 1982 version of OMB Circular A-119 was based 

on DoD’s established policy. These ongoing efforts were 

accelerated as a result of the Milspec Reform Initiative, which was 

ordered by Secretary of Defense William Perry in 1994. This 

reform’s immediate purpose was to review the current militaryunique specifications and determine if any of those specifications 

could be replaced by reasonable private-sector standards. For 

the future, Milspec emphasized a greater use of and reliance on 

performance and commercial specifications and standards by the 

DoD (American National Standards Institute [ANSI], 2003a). 

Since the beginning of Milspec Reform, DoD has reviewed over 

29,000 agency-specific standards and has cancelled or replaced 

9,600 of them. These profound efforts continue today under the 

direction of the Defense Standardization Program Office. 

One DoD reform success story is the C-17 Program. The C-17 

Program develops cargo aircraft. These aircraft are used to 

perform tactical airlift and airdrop missions. In the 1990s, program 

costs were out of control, and aircraft were being delivered late 

and with poor quality. The program decided to eliminate the 

traditional Milspecs and replace them with commercial systems. 

In addition, the military developed a quality control program based 

on Boeing’s successful D1-9000 system. The C-17 Program

reports that initially the production line slowed down even more, 

causing even greater delays. Slowly, the delivery schedule 

started improving, and by 1997 the program was awarded the 

California Golden State Quality Award for being the best of the 

state’s large manufacturing enterprises. The program now reports 

that it is delivering about 6 months early on contracts with one of 

the lowest “costs of quality” of any aircraft currently being 

manufactured. “Production span time as well as fly-away cost of 

the aircraft has been cut in half” (Department of Defense [DoD], 

2002a). 

As part of the DoD Reform efforts, the Navy led an effort to 

replace 33 MIL & FED Specs for insulation with ASTM 

International standards. ASTM International, formerly the 

2-10 
Section 2 — Background Investigation 

American Society for Testing and Materials, is a not-for-profit SDO 

that specializes in the development and publication of voluntary 

consensus standards for materials, products, systems, and 

services. According to a DoD publication, the use of ASTM 

standards has resulted in $89 million total life-cycle savings 

(ANSI, 2002). 

One specific example of the Navy’s successful work with ASTM 

standards is its application of mechanically attached pipe-fitting 

(MAF) testing. MAF was a new pipe-fitting technology that 

promised substantial improvements over existing pipe-joining 

technologies, such as welding and brazing. Most MAFs offered 

easy fabrication, high reliability, and lower installation costs. To 

the Navy, these assets would mean cost of maintenance savings 

while maintaining fleet material readiness. The problem was that 

the Navy could not implement the MAFs without proof that the 

MAFs would indeed provide better integrity than brazing or 

welding. No testing methods were in place, and the Navy had to 

use various testing procedures provided by each of the individual 

manufacturers. It was difficult to compare the results and to

ensure that the Navy would get the best MAF at the best price 

(ASTM International, 2001). 

Instead of developing its own Milspec for MAF, the Navy asked 

ASTM to work with them and the manufacturers to develop a 

common standard. During the development of this standard, 

manufacturers funded and conducted about $1 million worth of

MAF testing to prove concepts and validate tests. At minimum, 

the Navy saved this $1 million, because it would have had to 

conduct similar MAF testing if they had developed an in-house 

standard. The Navy further estimates that by involving ASTM, the 

MAF standard was develop 3 years ahead of the Navy’s timetable 

for standard development. This progress in turn meant that the 

Navy could take advantage of MAF savings (which are about $136 

per installed MAF) 3 years ahead of schedule. Because the Navy 

installs about 42,500 MAFs each year, this improvement resulted 

in a total 10-year savings of about $58 million (DoD, 2002b). 

Another example of an area of concern was sealant and 

elastomers. The Milspecs written for these products were 

outdated and were causing high prices, waste of material, and 

delays in getting the products. DoD worked with the Society of 

2-11 
Measuring Benefits from the National Technology Transfer and Advancement Act 

Automotive Engineers (SAE) to develop the portion of the National 

Defense Contractor Accreditation Program (NADCAP) responsible 

for sealants and elastomers. NADCAP is a system used to 

evaluate quality systems for suppliers and distributors. By using 

NADCAP to evaluate sealants and elastomers, the number of 

product returns decreased by 90 percent. The quality of materials 

supplied to the field was improved, and waste associated with

shelf life has been eliminated. The General Service 

Administration (GSA) was able to discontinue lab testing for a 

savings of $200,000 to $300,000, while the industry saved $2.2 

million. Future government savings are expected to be around 

$760,000 yearly (DoD, 2002c). 

2.4.2 Consumer Product Safety Commission 

CPSC is charged with protecting the public from unreasonable 

risks of death and injury associated with consumer products. In 

order to fulfill that objective, CPSC researches, develops, and 

enforces product safety standards. The Consumer Product Safety 

Act of 1996 requires CPSC to rely on voluntary standards when 

such standards are likely to result in the adequate reduction of the 

risk of injury and when it is likely that there will be substantial 

compliance. If there are no suitable standards for the product in 

question, or if an existing standard is not being followed by 

industry, CPSC is expected to develop an agency-specific 

standard. As a result, the commission spends about 70 percent of 

its regulatory effort participating in voluntary standards-setting 

efforts and about 30 percent developing government-specific 

standards. 

As it considers the development of a new standard, CPSC 

estimates the benefits that will be generated, both in terms of 

avoided medical costs and the value of lives that will be saved. 

Our research did not uncover any instances in which the 

commission estimated savings from using a VCS process rather 

than developing its own mandatory standard. Recent cooperative 

regulatory efforts for which it might be possible to quantify benefits 

include: 

• work with ASTM on safety standards to protect children in 

cribs and on playgrounds; 

2-12 
Section 2 — Background Investigation 

• work with the Water Heater Joint Research and Development 

Consortium and the Gas Appliance Manufacturers Association 

to reduce the risk of injuries and deaths from gas-fired water 

heaters; and

• joint efforts with the National Fire Protection Association 

(NFPA) and Congressional Fire Services Institute to evaluate 

the effectiveness of smoke detectors during fire incidents and 

to develop related product safety standards (Consumer 

Product Safety Commission [CPSC], 1997). 

2.4.3 Department of the Interior 

The Department of the Interior (DOI) is the nation’s principal 

conservation agency. This agency has been tasked with the 

responsibility of fostering sound use of land and water resources 

and conserving and protecting fish and wildlife. In DOI’s 

regulatory capacity, the agency sometimes looks to industry 

associations and standards for guidance. For instance, the U.S. 

Mineral Management Service (MMS), a bureau of DOI, worked

with the American Petroleum Institute (API), the primary trade 

association of the oil and gas industry, to develop Safety and 

Environmental Management Programs (SEMP) for offshore 

operations. According to API, the SEMP standards saved the 

petroleum industry $200 million in compliance costs (American 

Petroleum Institute [API], 2002). 

2.4.4 Occupational Safety and Health Administration 

OSHA, an agency within the Department of Labor (DOL), was 

created to ensure safe working conditions for employees in the 

United States. OSHA works closely with accredited SDOs, 

including the American National Standards Institute (ANSI). ANSI 

is a private, nonprofit organization that administers and 

coordinates the U.S. voluntary standardization and conformity 

assessment system. ANSI does not itself develop American 

National Standards (ANSs); rather ANSI facilitates development 

by establishing consensus among qualified groups. For instance, 

ANSI adopted protective electrical equipment standards 

developed by ASTM. OSHA, in turn, cited six of these standards 

in its protective electrical equipment regulation. In all, OSHA 

regulations reference over 200 of the existing 800 ANSs for safety 

and health (ANSI, 2003b). Although we found a variety of 

2-13 
Measuring Benefits from the National Technology Transfer and Advancement Act 

documents describing OSHA’s extensive use of and support for 

ANSs, none of these documents estimated quantitative savings or 

described qualitative benefits from the use of VCS. 

2.4.5 Food and Drug Administration 

FDA is responsible for ensuring the safety of processed and 

formulated foods, prescription and over-the-counter drugs, 

cosmetics, and medical devices manufactured for sale in the 

United States. In addition, FDA evaluates new drugs, food 

additives, and medical devices that firms would like to market, and 

FDA approval is necessary before any of these new products can 

be marketed. The regulations by this agency, which is a part of 

the Department of Health and Human Services (HHS), typically 

place the burden on firms to establish procedures to ensure their 

products’ safety and wholesomeness. As a result, FDA 

regulations require the use of fewer standards than most rulemaking agencies. In recent years, the agency has increased its 

use of VCSs to replace the limited number of government-specific 

standards it has cited. For instance, FDA is working with the 

Human Milk Banking Association of North America (HMBANA) to 

adapt the HMBANA donor human milk guidelines into federal 

regulation (Food and Drug Administration [FDA], 2001). 

2.4.6 Department of Energy

DOE is one of the largest users of procurement standards in the 

federal government for its various agencies and through its 

contractor-managed National Laboratories, including Oak Ridge, 

Argonne, Sandia, Los Alamos, and others. DOE’s role in 

regulating nuclear power operations in the United States gives the 

department a prominent role in regulatory standards as well. DOE 

uses a VCS as a basis for some of its programs but also relies 

extensively on government-specific standards, either its own or 

those developed by other governmental agencies such as DoD. 

For example, DOE maintains a Laboratory Accreditation Program 

(DOELAP) that is separate from the NVLAP operated by NIST. 

Within the DOELAP, however, some VCSs are used, such as the 

quality of radiation dosimetry measurement performance, which is 

based on quality system criteria set forth by the International 

Organization for Standardization (ISO) and the International 

Electrotechnical Commission (Department of Energy [DOE], 

1998). 

2-14 
Section 2 — Background Investigation 

It is not always the case that a governmental organization agency, 

such as DOE, will use an existing VCS or work with SDOs to 

develop a new one in order to meet its procurement or rulemaking needs. If the agency concludes that no adequate industry 

standard exists, the agency may petition an SDO to declare an 

existing government-developed standard as a VCS and ask the 

private organization to support the VCS in the future. DOE has 

used this process extensively in its efforts and has described the 

process in its Technical Standards Program Guide (DOE, 1999). 

In this case, the economic benefits are likely to be small, as no 

efficiencies are generated in standard development, and the lack 

of industry involvement makes cost reductions in procurement 

unlikely. The agency will reduce its costs of maintaining and

improving the standard if the SDO agrees to take over these 

responsibilities. 

2.4.7 Environmental Protection Agency

EPA was founded in the 1960s to protect human health and to

safeguard the natural environment—air, water, and land—on 

which life depends. EPA’s primary activities involving standards 

are in rule-making. In order to achieve its mission, EPA is one of 

the most active regulation-producing agencies in the U.S. 

government. As required by the underlying legislation authorizing 

its activities, along with its longstanding practice, the rules 

promulgated by EPA are very prescriptive in describing the 

actions that firms must take and how they must comply with the 

regulations. As such, there are a large number of standards 

referenced in EPA regulations, including specifications of 

materials and equipment to be used, laboratory tests that must be 

run, maintenance and calibration activities that need to be 

performed, information that must be recorded and subsequently 

reported to EPA. 

This wealth of details means that EPA has a greater opportunity to 

benefit from the use of VCS than most or all other regulatory 

agencies. In addition, the large number of conformity assessment 

requirements written into EPA rules has created a significant 

potential for economic benefits in laboratory accreditation 

activities. Not surprisingly, EPA has supported the development 

of VCS. EPA not only offers grants for VCS research to SDOs, 

EPA also conducts training sessions to familiarize rule-makers 

2-15 
Measuring Benefits from the National Technology Transfer and Advancement Act 

with the VCS information provided by the NSSN, an online 

national resource for global standards (U.S. General Accounting 

Office, 2000). NSSN is a cooperative partnership between ANSI, 

private domestic standards organizations, government agencies, 

and international standards organizations (NSSN, 2003). 

To our knowledge, EPA has not estimated or publicized the 

economic benefits realized as a result of their VCS activities. 

Nonetheless, the agency is aware both of the potential benefits 

from regulatory flexibility in general as well as of the relation 

between reductions in industry compliance cost and improved

environmental results.

2-16 
Analysis Framework 3 and Metrics 

The heart of any economic benefit-cost analysis is the analytical 

framework that was developed to estimate benefits and costs. 

Key elements of this effort are defining the scope of the analysis, 

applying an analytical methodology, articulating key benefit and 

cost impact areas, developing technical and economic metrics to 

measure these impacts, and proposing data collection strategies 

for informing the metrics and creating results measures. 

In this study of the economic impact of NTTAA, it was important to 

clearly define the analytical methodology, as a number of different 

approaches were feasible. In this section, we first restate the 

taxonomy of evaluation methods from the previous section, with 

special emphasis on how these methods could be applied in this 

case. Next, a brief review of relevant economic theory illustrates 

our decision to treat the use of VCS as a beneficial technological 

change. Finally, we describe impact areas, metrics, and 

measures, and then discuss appropriate data collection strategies. 

3.1 TAXONOMY OF BENEFITS EVALUATION METHODS 

In formulating a case study design for analyzing benefits from

agencies’ use of VCS, an initial choice needed to be made about 

the balance between descriptive and quantitative analysis. There 

are a number of potential approaches that could be used, each of 

which having advantages, resource requirements, and limitations. 

Although most of these methods are used in various applications 

at RTI, for studies in technology economics and policy, we tend to 

focus on the more quantitative approaches. A taxonomy of the 

most used methods appears below. 

3-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

• Success stories are almost always descriptive rather than 

analytical. These stories can provide a richness of detail 

and narrative description that helps build an understanding 

of goals and accomplishments and facilitates

communication across and between organizations. 

However, their selective nature and lack of quantitative 

data make success stories unsuitable for program 

evaluation or economic assessment. 

• Qualitative case studies are an excellent tool for 

communicating comprehensive results of a key initiative, 

especially if quantification is difficult or impossible. 

Interviews with key decision-makers and stakeholders can 

allow a more intimate look than is afforded by the more 

detached success story. Although some metrics are 

commonly presented, there are seldom sufficient data to 

support quantitative analysis. 

• Corporate-style project analyses are most often 

conducted prior to authorizing, funding, and executing new 

product ventures, capital projects, and major systems 

improvements. In a typical application, a discounted cashflow analysis is performed to yield projected rates of return, 

which must exceed corporate hurdle rates for the project to 

be accepted. Although this process is rigorously 

quantitative, it is usually selective—not all of the revenues 

and costs affected by the project are included in the cashflow analysis. In addition, ex-post assessments are rarely 

conducted or publicized, even if the project is successful. 

• Before-and-after case studies are one of the two types of 

comprehensive, quantitative analyses commonly employed 

in assessing major systems changes or improvement 

initiatives. These studies can also be used successfully in 

program evaluation. In these studies, a baseline is 

established prior to the planned change, with a full slate of 

metrics defined and measured. Once a steady state has 

been reestablished after the change, these same metrics 

are collected and compared to the baseline. Economic 

benefits are determined from aggregates, such as profit or 

total cost. This approach works well if the system change 

is large enough relative to day-to-day or secular changes 

3-2 
Section 3 — Analysis Framework and Metrics 

in prices, costs, revenues, etc. If results are noisy, or if 

other significant events disturb the aggregates being 

measured, before-and-after analysis can produce

misleading results. 

• Counterfactual analysis involves the comparison of 

actual results with a hypothetical situation that would be 

expected to occur in absence of the intervention being 

studied. Just as rigorous as before-and-after analysis, 

counterfactuals also can take into account changes in 

secular variables that might confound other estimates. 

Although the creation of the counterfactual involves some 

judgment and subjectivity, it is almost always possible to 

develop reasonable hypotheses by conducting in-depth 

interviews with key participants and stakeholders. 

Although it is not often discussed, almost all historical and 

economic analyses involve some degree of counterfactual 

reasoning (Fogel, 1979). 

For the VCS task, therefore, we built our case study methodology 

using counterfactual analysis. With this approach, the actual 

costs of supporting the VCS being studied are compared to those 

that would have existed had the agency used government-specific 

standards instead. Our initial concept involved evaluating an 

entire organization rather than a specific initiative. The 

organization could be an entire agency; or, more likely, a 

department or office within an agency. 

DSPO has produced a series of seven quantitative case studies 

that implicitly apply a counterfactual approach. It should be 

pointed out that, by and large, the DSPO analyses focus on 

estimating benefits from standardization in general, comparing 

actual results to a situation in which no standards are available at 

all. Our task, however, is to compare the benefits of VCS with 

government-specific standards, a significantly different empirical 

proposition. Nonetheless, we have benefited from the example 

provided by the three DSPO case studies that document the use 

of consensus standards: Mechanically-Attached Pipe Fittings, 

Navy Self-Contained Breathing Apparatus, and Conversion of MilStd-100 to a Non-Government Standard. A fourth initiative, the 

NAVSTAR Global Positioning System, was started as a 

government-specific initiative, but this initiative has since been 

3-3 
Measuring Benefits from the National Technology Transfer and Advancement Act 

broadened to include civil applications and is now largely under 

the control of a consensus process as well. 

3.2 BRIEF REVIEW OF RELEVANT ECONOMIC THEORY 

Standards and conformity assessment tools are indirect inputs to 

production processes, and the impacts of these tools can be

analyzed like other economic inputs, including labor and capital. 

This capability remains true regardless of whether the focus of

analysis is a private firm (such as a regulated manufacturing 

facility, government supplier, or testing laboratory) or the 

government agency itself. EPA, for example, produces products 

(regulations, analytical methods, accreditations), as does DoD 

(weapons systems, military facilities, national security). An 

improvement in the performance or decrease in the cost of using 

standards lowers agency costs and allows that agency to fulfill its 

mission more effectively. Replacement of government-specific 

standards with VCS can be thought of as a beneficial 

technological change, and economic principles from that field of 

study can help frame the current methodology. 

In a classic set of industrial studies more than 25 years ago, 

Edwin Mansfield explored the measurement of the benefits of

technical change (Mansfield et al., 1977). Using partial 

equilibrium analysis, Mansfield demonstrated the two impacts of 

improved processes and products: an increase in consumer and 

producer surplus as fewer resources are required to produce the 

existing level of output, and an expansion of output as supplier 

firms respond to the gap that opens between their marginal 

revenue and marginal cost. The increase in output provides an 

additional social benefit as purchasers are able to expand their 

consumption. 

When cost or price reduction is the primary result of the 

improvement, a straightforward algebraic approach can be used. 

Figure 3-1 illustrates Mansfield's approach. An innovation 

affecting an input or production process lowers the firm's marginal 

cost of production from MC0 to MC'. The firm can therefore cut its 

price accordingly, and with downward-sloping demand, will 

increase output and sales from Q0 to Q'. The increase in social 

welfare includes the impact of the lower firm costs (the shaded 

rectangular area) and the increase in demand due to a lower price 

3-4 
Section 3 — Analysis Framework and Metrics 

Figure 3-1. Mansfield’s 

Approach for Evaluating 

Benefits of Technological 

Change 

Welfare Increase to Be Measured

Demand

MC0

MC'

Quantity

Price/

Cost

P0

Q0

P'

Q'

Welfare Increase to Be

Ignored

(the triangle). In much empirical work, economists measure the 

cost-reduction benefits and ignore the smaller demand impacts. 

In the case of a government agency, we are not dealing with a 

profit-maximizing entity, so the supply behavior may be quite

different. We may choose to assume that the level of output is set 

exogenously: by federal laws (as detailed in the Code of Federal 

Regulations, for instance), by a defined mission and strategy, and 

by congressional authorizations and appropriations. This 

assumption is equivalent to assuming that demand for agency 

“output” is totally inelastic. Figure 3-2 illustrates the resulting

model, with a slightly more realistic upward-sloping supply curve 

(increasing marginal costs as activity is raised) replacing 

Mansfield's assumption of constant marginal cost. The equilibrium 

with the government-specific standards shows a unit cost on the 

supply curve at P0 and output fixed at Q0. Substitution of voluntary 

consensus standards or conformity assessment tools allows a 

3-5 
Measuring Benefits from the National Technology Transfer and Advancement Act 

Figure 3-2. Simplified 

Equilibrium Diagram of 

Government Agency

“Production”

Demand

Price/

Cost

reduction in costs to the supply curve S', with unit costs falling to 

P’ in equilibrium. 

The social benefit from the innovation is represented by the 

shaded area between the supply curves, with quantity remaining 

constant. This relationship can be shown algebraically to be equal 

to (P0 – P') Q0. To operationalize a model of this type, therefore, 

we need only to estimate the change in cost per unit of output 

(whether defined as the number of standards supported or 

perhaps the cost per regulation), and multiply that per-unit savings 

by the appropriate affected population. This dynamic is discussed 

in greater depth in the next section.

One the critical aspects of moving from government-specific 

standards and government-mandated conformity assessments to 

voluntary consensus processes is the maintenance or 

improvement of quality in the resulting systems. Standards, after 

all, are intended to support the core function of a government

agency or private firm; standards are not to exist for their own 

sake. If the movement from one type of standard to another 

caused a decrease in the quality of products or services, any 

savings realized by the consensus process would be illusory. In 

fact, Steve Lowell, Deputy Director, Defense Standardization 

3-6 
Section 3 — Analysis Framework and Metrics 

Program Office suggested examples of substantial improvement 

in quality and/or performance following some moves to 

nongovernmental standards, while other initiatives resulted in

quality degradation that had to be corrected by additional effort 

and expense. One example of this quality improvement is the 

move to commercial off-the-shelf semiconductors and other 

integrated circuits. These off-the-shelf items have a markedly 

shorter expected life than the military-only devices, which had 

become increasingly expensive as the military's share of the 

computer-acquisition market declined. 

For the purposes of this assessment, we will make an assumption 

that the agency or department will act to protect the quality of its 

products, services, and internal processes during any transitions 

to voluntary consensus standards. NTTAA and the OMB circular 

do not force agencies to adopt VCSs at any cost, but rather allow 

them to maintain their agency-specific standards if required. As a 

result, we will assume that the quality of the regulation, procured 

part, laboratory analysis, or accreditation is the same in the actual 

and hypothetical cases. 

3.3 PROCESS FOR CONDUCTING COUNTERFACTUAL

ANALYSIS 

RTI has established a well-defined process for conducting 

economic benefits analysis using the counterfactual approach, a 

process that consists of five major steps. In this section, we 

discuss each of these steps, including critical issues and actual 

outcomes. Application of this process to the task of estimating 

benefits from implementation of NTTAA is deferred to the next 

section, which contains descriptions of three separate case 

studies we constructed to inform the metrics. 

3.3.1 Defining Scope of Analysis 

Identifying the appropriate boundaries and the depth of 

penetration for the counterfactual strongly impacts the likelihood of 

success in the subsequent analysis. In the present case, we

needed to decide whether to attempt to measure costs and 

benefits across an entire agency or department, to select broad 

activity areas within an agency, or to focus on the implementation 

of a single standard or set of standards. In an effort 

encompassing an entire department, we would measure the total 

3-7 
Measuring Benefits from the National Technology Transfer and Advancement Act 

number of standards of each type (government-specific or VCS) 

and estimate average support costs for those standards. It is 

necessary to avoid penetrating too much into the details of each 

standard and focus on aggregates and averages to create our 

benefits estimates. Although this practice gives summary 

measures that appear to be comprehensive, the great degree of 

variability in resource intensity, applicability, and overall costs 

across standards may lead to a lack of precision in the reported 

figures. 

Focusing on the level of an individual standard, in contrast, allows 

a great deal of accuracy in specifying actual and counterfactual 

costs, at the expense of extrapolation to the entire agency's 

activities or to other governmental entities. The DSPO case 

studies, for example, are all at the standard level—they are 

looking at a small set or even a single standard among the 

thousands of Milspecs and Non-Governmental Standards DoD 

supports. For the purpose of this evaluation, it seems that the 

standard level is too narrow a focus; however, the data needed to 

assess an entire agency could be quite daunting. 

A couple of other possibilities exist that we believe would strike a 

balance between precision and coverage. As is explained in more 

detail in Section 4, CPSC adopts VCS and issues governmentspecific standards in meeting its mission, offering an unusual sideby-side comparison of the costs for supporting both systems. We 

could create and complete a case study to develop an estimate of 

the average annual cost of developing and supporting a 

government-specific standard and a second measure of the 

average annual cost of assisting in development and ongoing

support of a VCS. If the experience of CPSC was thought to be 

similar enough to that of other agencies, NIST could use these 

estimates to convert the activity data that was already reported 

annually into a rough approximation of the newly created benefits. 

One disadvantage of this method, aside from the potential of poor 

precision, is that each agency would be unable to tie the projected 

savings to specific initiatives, which may make it more difficult to 

develop confidence in the reported results. 

A second potential approach was to measure benefits across 

agencies for a few widely applicable types of standards. As an 

example, Greg Saunders, Director of DSPO, observed during our 

3-8 
Section 3 — Analysis Framework and Metrics 

recent meeting that some of the clearest successes in the use of 

VCS across all of DoD have been in military construction. A 

detailed case study of the benefits achieved could find wide 

reapplication across other agencies, as many of these agencies 

are involved in constructing facilities—most notably the General 

Services Administration (GSA), but also the DOI, Bureau of Land 

Management (BLM), DOE, and others. 

Another common application area is in the inspection and 

accreditation of laboratories. This is a major issue, not only for the 

federal government, but also for states. Often, a laboratory that 

seeks to do business with multiple state and federal agencies will 

need to undergo scores of accreditation evaluations and 

inspections, as well as participate in routine round-robin or blind 

standard evaluations. It seems very likely that estimates of the 

benefits from voluntary consensus processes in this area could 

have reapplication possibilities across many agencies. 

In order to maximize our chances of collecting useful metrics and 

being able to derive net benefits data during the pretests, we

elected to pursue both approaches in subsequent phases of the 

study. The plan was to evaluate agency-wide results on the 

regulatory side, while focusing more narrowly on specific 

segments of procurement-based agencies. We also decided to 

pursue quantification of the interesting accreditation issues being 

faced by most, if not all key agencies. 

3.3.2 Identifying Cost Impact Areas 

Once the scope of analysis had been clearly defined, the next task 

was to identify those costly activities that will differ between the 

actual and hypothetical scenarios. In some of these areas, the 

VCS process should be expected to lead to smaller investments in 

costly resources than for government-specific standards. On the 

other hand, we would expect a greater level of costly activities 

under VCS. One simple example of such activities is labor effort 

and travel to support the participation in the VCS process itself. A 

second example is purchasing written copies of the standards 

themselves, a nontrivial expense for often cash-starved agency 

departments. 

3-9 
Measuring Benefits from the National Technology Transfer and Advancement Act 

3.3.3 Technical and Economic Metrics and Discussion Guides 

For each impact area identified in the previous step, a specific 

technical metric was developed that provided raw material for 

costing out the savings figures, whether positive or negative. 

Following this step, all of the technical metrics were associated 

with an appropriate economic metric, which will be an expenditure 

expressed in dollars. Once the impact areas and metrics were 

defined, case study discussion guides or survey instruments were 

developed to elicit responses from knowledgeable contacts. 

Although full-scale deployment of the pretested instruments was 

beyond the scope of this study, the case study discussion guides 

were designed to support subsequent placement with all 

compliant agencies. These guides can support a structured 

conversation between the RTI interviewer and the agency 

representative, and the quantitative responses can be aggregated 

for statistical and mathematical analysis. During the in-depth data 

collection process, agency respondents will occasionally need to 

consult with knowledgeable persons from government or other 

stakeholder organizations. For this reason, the discussion guides 

would be sent out in advance, along with instructions for the data 

collection process. 

3.3.4 Developing Aggregation and Benefits Quantification 

For those case studies that look at agency-wide activities, the 

basis for aggregation is simple: a per-standard benefit is 

multiplied by the number of standards used by that agency to 

arrive at a total estimate. In the narrower selective approach, 

information is needed, as a part of the case study, on the number 

of affected projects or the fraction of the total agency the data will 

represent. For instance, with a finding that the use of VCS in 

building construction offers savings of 2 percent of the total 

construction costs, we would need data on the total construction 

expenditures to estimate the benefits. Once we eliminate the 

possibility of benefits totaling $1000 per accreditation, the logical 

multiplier is the total number of inspection visits supported by 

laboratories.

The data collected, suitably aggregated to represent the entire 

affected population, would then be used to calculate a number of 

measures of economic benefit, including benefit-to-cost ratio, net 

3-10 
Section 3 — Analysis Framework and Metrics 

present value, and internal rate of return. The following is a 

description of each of these benefits calculations: 

Benefit-to-Cost Ratio (B/C)—Annual time series of benefits and 

costs, derived from the aggregated stakeholder data, are 

assembled. Letting Bt be the net benefits accrued in year t and Ct

be the total costs incurred in that year, then the benefit-cost ratio 

for the program is given by 

 (B/C) = 

∑

i=0

n B(t+i)

(1+r)i

∑

i=0

n C(t+i)

(1+r)i

 , (3.1) 

where t is the first year in which benefits or costs occur, n is the 

number of years the benefits and/or costs occur, and r is the 

social rate of discount. In most of our studies, r is set at 7 percent, 

the OMB-specified level. Because benefits and program costs 

may occur at different periods, both are expressed in presentvalue terms before the ratio is calculated. 

Net Present Value (NPV)—The NPV of an initiative to implement 

VCS or conformity assessment processes can be calculated as 

 NPV = ∑

i=0

n

⎣

⎢

⎡

⎦

⎥

⎤ B(t+i)

(1+r)i

 – Ct+i

(1+r)i , (3.2) 

where the terms have the same meanings as identified for the B/C 

determination. Any project that yields a positive NPV is 

considered to have been economically successful. It should be

noted that the 7 percent real discount rate required by OMB is a 

rather high hurdle for project analysis, ensuring that projects that 

show a positive NPV are quite socially advantageous. 

Internal Rate of Return (IRR)—The IRR is the value of r that sets 

the NPV equal to 0 in Eq. (3.2). The IRR’s value can be 

compared to conventional real rates of return for comparable or 

alternate investments. Risk-free capital investments, such as

government bonds, can be expected to yield rates of return under 

5 percent in real terms, while equities seldom return more than 10 

percent over an extended period. It should be noted that, in cases 

for which costs exceed benefits, an IRR cannot be calculated. 

3-11 

Data Collection and 4 Survey Instruments 

As a result of continued background research and in-depth 

discussions with several government agency personnel, we have 

developed three detailed case study scenarios that we would like 

to execute in Task B of this project. Because the scenarios differ 

significantly in scope, target agencies, and required metrics, we 

have chosen to describe these scenarios more fully in this 

methodology document. The first study is most consistent with 

the work plan as initially proposed; this study attempts to measure 

for the CPSC benefits captured by its VCS efforts at the agency 

level. The second study intends to investigate standards-related 

activities in facility construction, an area of activity common to a 

number of federal agencies and one in which the DSPO believes 

there have been significant unrecognized benefits. Finally, we 

would like to assess benefits in the conformity assessment area, 

and the large amount of effort being expended in laboratory 

accreditation makes this area a natural focus for additional 

attention. 

4.1 CONSUMER PRODUCT SAFETY COMMISSION 

CPSC is directed by law to seek industry-supported voluntary 

standards for the products they regulate and to issue governmentspecific mandatory standards if no private-sector standards exist 

that would adequately protect the public from harm. As a result, 

CPSC is actively engaged in both processes of interest to our 

counterfactual proposition and should be in an ideal position to 

help us evaluate relative costs and benefits of each. Although 

CPSC's primary mission is to protect human health and save lives, 

4-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

the reality of the budgeting process forces them to control 

expenditures closely. A study that describes and quantitatively 

estimates economic benefits from CPSC’s use of voluntary 

industry standards may help them build support for their strategic 

plan and future development needs. Table 4-1 lists a number of 

impact areas and technical and economic metrics of CPSC. 

4.2 FACILITIES CONSTRUCTION IN THE DEPARTMENT 

OF DEFENSE 

A large number of government agencies construct facilities to

support their core missions and objectives, to provide office and 

laboratory areas, and even to house their employees. The most 

important of these is DoD, which designs, constructs, and 

operates permanent and temporary buildings and structures on 

military bases and in office and research complexes. According to 

their 2003 appropriation, DoD, its service branches, and reserves 

were authorized to spend $5.6 billion in military construction

during the fiscal year, along with $1.6 billion in construction of 

family housing. 

Despite this immense level of spending and activity, military 

construction is a small fraction of total construction activity in the 

United States, which amounted to $846 billion in 2002 (U.S. 

Census Bureau, 2003). As a result, there are large potential 

benefits from DoD specifying voluntary consensus standards

already in place in the private sector, rather than a set of Milspecs 

that they would have to initiate, maintain, and enforce. Table 4-2 

lists a number of impact areas and technical and economic 

metrics that we would expect to be useful in quantifying benefits 

from use of VCS in DoD or other agency construction activities. 

 4.3 LABORATORY ACCREDITATION UNDER NATIONAL 

ENVIRONMENTAL LABORATORY ACCREDITATION 

PROGRAM 

EPA regulations require the use of a variety of standards and 

conformity assessment methods, many of which are agencyspecific, while others are the product of voluntary consensus

efforts. As a part of these activities, EPA specifies thousands of 

analytical tests that must be performed by private-sector firms, 

state and local governments, and EPA itself. Most, if not all, of 

4-2 
Section 4 — Data Collection and Survey Instruments 

Table 4-1. Consumer Product Safety Commission Study

Activity Area Technical Metric Economic Metric 

Areas affecting agency costs:

Efforts of in-house experts to develop new 

standards

Labor hours Labor costs 

Number of training courses 

supported 

Training and development of in-house Training costs 

experts 

Supervisory effort Labor costs 

Hardware and software Info systems cost 

Conferences attended Travel costs 

Maintenance/improvement of existing 

standards

Effort hours for maintenance Labor costs 

Memberships in SDOs Membership fees

Meetings attended Travel costs 

Support for voluntary consensus standards 

efforts

Purchase of hard-copy 

standards

Unit cost x number of 

standards

Areas affecting regulated firms: 

Contractor efforts to interpret and comply

with standards

Labor hours Labor costs 

Number of training courses 

supported 

Training and development of standards Training costs 

experts at firms 

Supervisory effort Labor costs 

Change control and dissemination for Hardware and software Info systems cost

existing standards 

Effort hours for maintenance Labor costs 

Memberships in SDOs Membership fees 

Meetings attended Travel costs 

Support for voluntary consensus standards 

efforts

Purchase of hard-copy 

standards

Unit costs x number of

standards

Audits and inspections to ensure firm Inspection labor hours Labor costs or audit fees 

complies with standards 

Legal consultation hours Legal fees

Conformity assessment and demonstration Laboratory tests Testing fees

CofCs produced Labor hours 

Inspection visits Overtime pay

4-3 
Measuring Benefits from the National Technology Transfer and Advancement Act 

Table 4-2. Military Construction Study

Activity Area Technical Metric Economic Metric 

Areas affecting agency costs:

Efforts of in-house experts to develop new 

standards

Labor hours Labor costs 

Number of training courses 

supported 

Training and development of in-house Training costs 

experts 

Supervisory effort Labor costs 

Hardware and software Info systems cost 

Conferences attended Travel costs 

Maintenance/improvement of existing 

standards

Effort hours for maintenance Labor costs 

Memberships in SDOs Membership fees

Meetings attended Travel costs 

Support for voluntary consensus standards 

efforts

Purchase of hard-copy 

standards

Unit costs x number of

standards

Contract development and negotiation Purchasing agent hours Labor costs 

Legal consultation hours Legal fees

Evaluation of contractor conformance to Hours reviewing documents Labor costs 

standards/specs 

Inspection visits Travel costs 

Areas affecting price of purchased goods:

Contractor efforts to interpret and comply

with standards

Labor hours Labor costs 

Number of training courses 

supported 

Training and development of Milspec experts Training costs 

at firms

Supervisory effort Labor costs 

Change control and dissemination for Hardware and software Info systems cost

existing standards 

Effort hours for maintenance Labor costs 

Support for voluntary consensus standards Memberships in SDOs Membership fees 

efforts

Meetings attended Travel costs 

Contract development and negotiation Sales/Marketing hours Labor costs 

Legal consultation hours Legal fees

Laboratory tests Testing fees

CofCs produced Labor hours 

Conformity assessment and demonstration 

Inspection visits Labor hours 

4-4 
Section 4 — Data Collection and Survey Instruments 

these tests are required by law or EPA regulations to be 

performed at an accredited analytical laboratory, whether 

operated by the regulated firm, a government organization, or a 

third-party service provider. 

In the absence of coordinating activities, such as those facilitated 

by NELAP or SDO-led voluntary consensus processes, it is 

possible that a single laboratory intending to provide services 

across different media (water, air, and solid waste, for example) 

may require inspection and accreditation from several branches of 

EPA, as well as the corresponding governmental bodies in each 

state for which that laboratory does business. This process 

contains excessive costs in terms of laboratory effort and 

materials, agency effort and travel expense, and communications 

and coordination costs.

In a hypothetical world in which a single accreditation could serve 

to meet the requirements of several EPA agencies and the 

associated state bodies, a great deal of economic benefit could be 

derived. Agency costs could potentially be reduced, and, just as 

importantly, costs for the affected firms and the third-party 

providers would fall. This scenario would allow regulatory 

agencies, such as EPA, to achieve more of their core mission 

without adding to the burden of the regulated firms and 

municipalities. The differences in costs between these two 

scenarios are an accurate measure of the economic benefits 

generated by NELAP. 

Table 4-3 illustrates activity areas and technical and economic 

metrics that may be useful in estimating benefits from NELAP and 

similar accreditation-recognition programs. Currently, NELAP has 

11 states enrolled in their program for drinking-water and 

wastewater analysis, and additional state organizations are in the 

process of joining or have agreed to accept the accreditation from 

one of the enrolled states. Future plans for NELAP call for an

expansion into air monitoring and other media regulated by EPA. 

In addition to estimating benefits to EPA and the member states 

for the existing program, the case study could project future 

benefits from this plan across EPA and even to other federal 

agencies. 

4-5 
Measuring Benefits from the National Technology Transfer and Advancement Act 

Table 4-3. Laboratory Accreditation Study

Activity Area Technical Metric Economic Metric 

Areas affecting U.S. EPA and state EPA costs:

Manage accreditation and performance 

testing (PT) process 

Manager effort hours Labor costs 

Prepare and distribute PT samples Laboratory effort Labor hours 

Shipping charges Shipment costs 

Conduct initial and follow-up

assessments/inspections 

Inspector effort Labor and travel costs 

Number of training courses 

supported 

Training and development of in-house Training costs 

assessors 

Supervisory effort Labor costs 

Memberships in SDOs Membership fees

Meetings hosted Meeting costs 

Meetings attended Travel costs 

Support for NELAP or voluntary 

consensus standards efforts

Purchase of hard-copy standards Unit costs x number of 

standards

Areas affecting testing laboratories:

Proficiency tests run Cost/test x number of PTs

x states 

Accreditation fees Cost per fee x states 

Accreditation efforts for each state and 

federal agency 

Effort to support audits Labor hours x states 

Number of training courses 

supported 

Training and development of standards Training costs 

experts at firms 

Supervisory effort Labor costs 

Support for NELAP or voluntary Committee participation Labor and travel costs 

consensus standards efforts Purchase of hard-copy standards Unit cost x number of 

standards

One note should be added to the discussion at this point. NELAP 

is not, strictly speaking, a voluntary consensus organization in that 

voting rights have always resided in government representatives, 

both at the state and federal levels. Still, private testing 

laboratories and regulated firms have actively participated in the 

working groups and annual conferences of NELAP, and most of 

these laboratories and firms felt the process was as collaborative 

as an SDO-led VCS process. 

4-6 
Pretest Results and 5 Recommendations 

The benefit-cost methodology outlined in earlier sections of this 

report, which underlies the analytical approach and data collection 

instruments developed during the project, has been used 

successfully in a number of economic studies conducted by RTI's 

TEP program. In the case of estimating agency benefits from

utilization of VCS, however, it is not clear at this point that our 

approach will produce the comprehensive, robust results desired 

by SSD. Based on the feedback received from our contacts in 

federal agencies, as well as from the completed pretest, it appears 

that agencies and departments may have a great deal of difficulty 

assembling and reporting the information required to fully inform 

our methodology. 

In this final section of the report, we describe briefly the pretest 

results, discuss the implications for broad-scale implementation of 

the methodology, and make a few recommendations about next 

steps. Although we recognize that this result is not what we 

anticipated or hoped for at the outset of this project, we hope that 

NIST understands our findings and the rationale for RTI's caution. 

In the recommendations section, we offer several options that we 

now believe may have a higher chance of achieving SSD's goals 

of building energy and commitment to the VCS process. 

5-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

5.1 RESULTS OF AGENCY PRETESTS 

The interview guides discussed in the last section were shared 

with our contacts at the three agencies we believed would offer a 

thorough pretest of the questions and data needs required to

inform our economic analysis. Copies of each of the interview 

guides we developed appear in the Appendix to this report. 

To evaluate the benefits from the use of VCS referenced in 

regulation, we contacted Colin Church, Voluntary Standards and 

International Activities Coordinator at CPSC. For the use of 

procurement-related VCS, we requested help from Greg Saunders 

of DoD, who attempted to find a suitable respondent in the military 

construction purchasing area. To help illuminate costs and 

benefits from voluntary laboratory accreditation, we sent the 

appropriate interview guide to Lara Autry of EPA, who functions as 

the director of NELAP. 

The discussion with Mr. Church at CPSC provided a great deal of 

qualitative support for the overall benefits of the use of VCS by 

that agency. As mentioned previously, CPSC is unique among 

federal organizations in that the use of VCS is required by its 

authorizing legislation, unless the commission makes a judgment 

that no nongovernmental organization could produce a suitable 

safety standard for the product under review. In addition, CPSC’s 

status as a small, independent agency, with about 500 employees 

and a $50 million annual budget, encourages CPSC to rely on

third-party standards-developing organizations whenever possible. 

As a result, CPSC has reported that in recent years it has 

promulgated about seven times as many voluntary standards as 

mandatory ones. Only about 5 percent (or around 25) of its 

employees are involved in supporting voluntary efforts, which are 

led by industry groups and third-party SDOs. One possible 

conclusion from these two pieces of data is that CPSC's efforts in 

support of voluntary safety standards are highly efficient, although 

no information was available on the actual effort expended in 

voluntary and mandatory rule support.

The lack of quantitative information on distribution of effort and 

costs is very troublesome, as it relates to CPSC's ability to 

respond to the questions contained in the interview guide. Mr. 

5-2 
Section 5 — Pretest Results and Recommendations 

Church believes that the detailed data on labor effort, training

costs, travel expenses, and other items required to inform our 

metrics is not currently available within CPSC, and that collecting 

this information would represent a significant burden on a small 

organization that is focused on saving lives and reducing injuries. 

 5.2 IMPLICATIONS FOR BROAD-SCALE 

IMPLEMENTATION OF RTI'S METHODOLOGY 

Taken in a broader context, the pretest results suggest that other 

government agencies might also have a great deal of difficulty 

responding to the detailed questions contained in the interview 

guide. An organization like CPSC, solidly focused on using 

voluntary consensus processes, might reasonably be expected to 

track data on expenditures in each of the two regimes that the 

organization must operate. An organization such as EPA, whose 

controlling regulations obligate it to use its own laboratory 

methods and many other standards, is even less likely to have 

access to the data we require for the quantitative analysis. At a 

minimum, this dynamic suggests that the pretest should be 

repeated prior to broad-scale adoption; in the extreme, this 

dynamic casts doubt on the feasibility of data collection. 

The results of the pretest have additional implications, both in

terms of RTI's experience in completing the study and in 

formulating recommendations to pass along to SSD. From 

conversations with several stakeholders in the existing VCS and 

NTTAA compliance process, it appears that federal agencies and 

their standards representatives have had a great deal of difficulty 

obtaining the information needed to meet their annual reporting 

requirements to NIST and, in turn, to OMB. The reasons for this 

difficulty are not relevant to our study, but the implied lack of time 

for VCS issues is critically important if SSD is considering adding 

an additional reporting burden by means of implementing our 

proposed methodology. 

This dilemma is especially acute in view of the lack of solid 

benefit-cost analysis now available. Aside from the excellent DoD 

case studies mentioned earlier in this report, there are few 

published reports, either inside or outside government. However, 

DoD's tremendous success with Milspec reform, and its 

willingness to invest resources in documenting some of DoD’s key 

5-3 
Measuring Benefits from the National Technology Transfer and Advancement Act 

successes, suggests a final implication of our study. We would 

look to the agencies themselves to make a decision on whether to 

invest the (perhaps considerable) resources required to perform a 

solid benefit-cost study on a selected group of voluntary 

standards, or perhaps on the agency as a whole. Despite a lack 

of confirmation from the pretest, we believe that the methodology 

and metrics developed in this project are appropriate and 

sufficient to perform the type of analysis required. Agency 

personnel would also have an easier time establishing access to 

information owned by other key stakeholders, such as SDO and 

industry representatives.

 5.3 RECOMMENDATIONS ABOUT NEXT STEPS 

The project assignment as we received it from NIST involved 

background research, development of an analytical framework, 

and creation and pretesting of one or more case studies. It was 

clear from the outset, however, that SSD intended to conduct the 

case studies, or otherwise collect data from compliant agencies, 

as soon as the methodology was tested and proven feasible. 

Based on our experience in the last couple of stages of this study, 

RTI would urge SSD to consider carefully before making a 

decision to proceed with data collection. In this final section of the 

study, we make some alternative recommendations for next steps. 

As detailed earlier in this section, we now believe that agencies 

may have a great deal of difficulty providing the necessary 

information to inform our metrics and net-benefit measures. Given 

the potential difficulty of obtaining accurate and useful data, 

agencies would face two difficult choices: 

• to invest a substantial amount of effort and cost in 

searching out the necessary data; or 

• to provide guesses as to the magnitude of effort and costs 

requested in the interview guides. 

In either case, SSD is unlikely to achieve the objectives it sought 

when conceiving this study. 

In lieu of this option, RTI offers the following suggestions for next 

steps along the path to estimating the benefits of NTTAA: 

5-4 
Section 5 — Pretest Results and Recommendations 

• Enlist the help of willing federal agencies or departments to 

gather the detailed information necessary to test the 

methodology and to estimate benefits, from one or more 

projects or for the entire agency. This approach, which 

RTI attempted but did not complete, might be easier to 

execute through the interagency working group than by an 

outsider. A successful trial, with a demonstration of 

significant agency benefits, might go a long way toward 

building energy for NTTAA reporting and building support 

from currently uninvolved agencies.

• Work with an SDO or an umbrella organization, such as 

ASTM or ANSI, to rigorously evaluate one or more 

standards they developed to replace government-specific 

standards, using a comprehensive benefit-cost approach 

like the one specified in this report. Cooperation from the 

agency would be critical, but the SDO should be in a 

position to obtain that cooperation. One of the curious 

findings on this project was the lack of involvement by 

SDOs in performing benefit-cost analysis for use as a 

marketing tool in their day-to-day efforts. The few studies 

we saw were not thorough and often misidentified costs 

and benefits in the analysis. 

• Begin to request data from agencies that would support 

this type of economic analysis in the future. As a part of 

the annual report for OMB, SSD could ask agencies to 

share some data on key elements from the interview 

guides, such as effort hours or FTEs of personnel involved 

in VCS support, training costs, travel expenditures for 

various standards-related activities, etc. Because the 

number of new and existing standards supported by the 

agency is already requested, measures of costs and 

benefits, or at least relative efficiency, could be calculated. 

5-5 

 

 References 

American National Standards Institute (ANSI). 2002. “NonGovernment Standards Policies, Procedures, & Goals in 

the Defense Department.” As accessed on November 8, 

2002 at http://www.ansi.org/rooms/room_52/public/ 

dodpresent/sld011.htm. 

American National Standards Institute (ANSI). 2003a. “A Strong 

Private-Public Partnership.” As accessed on April 30, 

2003 at http://www.ansi.org/ 

government_affairs/partnership.aspx?menuid=6.

American National Standards Institute (ANSI). 2003b. 

“Significant Federal Laws and Policies.” As accessed on 

April 30, 2003 at http://www.ansi.org/ 

government_affairs/laws_policies/laws.aspx?menuid=6. 

American Petroleum Institute (API). 2002. “API’s Standardization 

Program – From Dallas to Denpasar.” As accessed on 

December 2, 2002 at http://www.sesstandards.org/library/01proceedings/MillerD.pdf. 

ASTM International. 2001. “Mechanically Attached Pipe Fittings.” 

Standardization News. As accessed on November 8, 2002 

at http://www.ast.org/SNEWS/NOVEMBER_2001/ 

case_nov01.html. 

Breitenberg, M. A. 1987. The ABCs of Standards-Related 

Activities in the United States. NBSIR 87-3576. 

Gaithersburg, MD: National Bureau of Standards. 

Consumer Product Safety Commission (CPSC). 1997. “Better 

Data Needed to Help Identify and Analyze Potential 

Hazards.” GAO/HEHS-97-147. Washington, DC: United 

States General Accounting Office. 

Coopers & Lybrand/TASC. 1994. The DoD Regulatory Cost

Premium: A Quantitative Assessment. 

R-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

Department of Defense (DoD). 2002a. “C-17 Advanced Quality 

System.” ASC Engineering Fact Sheet. As obtained on 

November 8, 2002 at http://public.ascen.wpafb.af.mil/ 

success/C-17_Advanced_Quality_3_07_.pdf. 

Department of Defense (DoD). 2002b. “Defense Standardization 

Program Case Study: Mechanically Attached Pipe 

Fittings.” As accessed on November 8, 2002 at 

http://www.dsp.dla.mil/documents/cases/pipefitting.pdf. 

Department of Defense (DoD). 2002c. “Year 2001 Defense 

Standardization Program Award Nomination.” As 

accessed on November 8, 2002 at 

http://engineering.wpafb.af.mil/engstds/DSP_Award_Fletch

er.pdf. 

Department of Energy (DOE). 1998. “The Department of Energy 

Laboratory Accreditation Program Administration.” DOESTD-1111-98. As accessed on November 8, 2002 at 

http://tis.eh.doe.gov/techstds/standard/std1111/

std1111.pdf.

Department of Energy (DOE). 1999. “Technical Standards 

Program Guide.” Doe G 252.1-1. As accessed on 

November 8, 2002 at http://tis.eh.doe.gov/techstds/g2521-

1.pdf. 

Food and Drug Administration (FDA). 2001. As accessed on

November 8, 2002 at http://www.fda.gov/ohrms/dockets/ 

dockets/00n0001/ts00019.pdf. 

Miller, J. C. III. 1985. “The FTC and Voluntary Standards: 

Maximizing the Net Benefits of Self-Regulation.” Cato 

Journal 4(3):897-903. 

NSSN. 2003. “About NSSN.” As accessed on April 30, 2003 at 

http://www.nssn.org/about.html. 

Ruggieri, J. 2001. “Consensus Standards Development Invites 

Engineering Perspectives.” Engineering Times Letters. As 

accessed on December 2, 2002 at 

http://computerdesigntools.com/Publications/ 

NSPE_PEG_JR_JAN_01.htm. 

U.S. General Accounting Office (GAO). 1993. “Automated 

Medical Records: Leadership Needed to Expedite 

Standards Development.” GAO/IMTEC-93-17. 

Washington, DC: GAO.

R-2 
References 

U.S. General Accounting Office (GAO). 2000. Voluntary 

Consensus Standards: Agencies’ Compliance with the 

National Technology Transfer and Advancement Act. 

GAO/T-RCED-00-122. Washington, DC: GAO.

U.S. House of Representatives. 2000. Committee on Science, 

Subcommittee on Technology. “Hearing on Standards 

Conformity and the Federal Government: A Review of 

Section 12 of Public Law 104-113,” March 15, 2000. 

Washington, DC: GPO.

Whitaker, B. 1972. “The Role of Voluntary Standards in Assuring 

Product Safety and Consumer Choice.” Food Drug 

Cosmetic Law Journal. 1985 April:187-194. 

Fogel, R. W. 1979. "Notes on the Social Saving Controversy." 

The Journal of Economic History 39(1):1-54. 

Mansfield, E., J. Rapoport, A. Romeo, S. Wagner, and G. 

Beardsley. 1977. "Social and Private Rates of Return 

from Industrial Innovations." Quarterly Journal of

Economics 91:221-240. 

U.S. Census Bureau. 2003. “Annual Value of Construction Put in 

Place in the United States.” As obtained at 

<http://www.census.gov/const/C30/c30tab1.rpt>. 

Washington, DC: U.S. Census Bureau. 

R-3 

Appendix – Interview 

Guides for Case Studies 

A.1 INTERVIEW GUIDE FOR GOVERNMENT AGENCIES 

ON THE USE OF VOLUNTARY CONSENSUS 

STANDARDS IN PROCUREMENT ACTIVITIES 

This interview guide has been designed for use by agencies that 

reference standards in procurement. This instrument focuses on the 

choice agencies face to develop and maintain their own standards or to 

rely on standards development organizations (SDOs) to coordinate these 

activities. Our intent is to gather relevant information about costs and

benefits from the use of Voluntary Consensus Standards (VCSs) in 

procurement, in regulation, and in laboratory accreditation, as part of a 

study undertaken for NIST’s Standards Services Division. 

Please answer the following questions about your operations 

over the past several years, including as much numerical data as

possible. These responses will be used to construct an aggregate 

measure of the net benefits generated from governmental agency use of 

voluntary consensus standards. Individual responses will be kept 

confidential by NIST, except that total realized benefits for each agency 

may be included in future annual reports about the implementation of the 

NTTAA. 

[This Interview Guide has been tailored for use in agencies that 

are engaged in facilities construction, especially those connected to the 

Department of Defense. Our understanding is that their conversion to 

private-sector standards is fairly recent, and there would be 

knowledgeable experts in these agencies who could respond to 

questions about costs under both procurement regimes.] 

I Questions about standards-related DoD expenditures during 

era in which agency-specific standards (primarily Milspecs) 

were in use 

1. Please estimate the annual expenditures on construction 

activities undertaken by your agency or division during this 

A-1 
Measuring Benefits from the National Technology Transfer and Advancement Act 

period. (Select any recent year for which records can be easily 

gathered.) 

2. Did your agency/division have a separate standards organization

during this period? If so, approximately how many people (in 

FTEs) worked in this standards group? Do you have an estimate

as to their annual budget?

3. Please estimate the annual labor effort (in FTEs) expended by 

personnel in developing new standards. Can you partition this 

number by skill level, i.e. how many engineers, technicians, 

administrative support staff? 

4. How much training and development effort was required for 

these people, including both initial training and continuing 

education? Can you quantify this in terms of out-of-pocket 

dollars (materials and travel), effort hours, and supervisory 

expenditures? 

5. Please estimate the annual labor effort expended in 

maintenance and improvement of existing standards. Were the 

people involved in these activities similar in skill level to those 

developing new standards? 

6. What other annual expenditures were involved in maintaining 

existing standards? (Information systems hardware and 

software purchase, service and support; travel cost for 

conferences or seminars; others) 

7. What standards-related expenses were incurred as a part of 

contract development and negotiation? Did purchasing agents 

and legal support staff need relevant training or technical support 

for their roles? Can you estimate annual effort required from

these highly-compensated individuals? 

II Questions about contractor expenses during period in 

which agency-specific standards were in use (which affect

prices of construction goods & services) 

1. What would you estimate to be each contractor’s effort required 

to interpret and comply with Milspec standards? (It may be 

easiest to estimate all of the quantities in this section as a 

percentage of contract value.) 

2. How would these contractors train and develop their Milspec 

experts? Can you estimate the annual effort for these activities, 

including supervisory costs?

3. Contractors need to maintain databases and support change 

control for existing standards. Can you estimate the information 

systems costs, and supervisory effort required for each 

contractor? 

A-2 
Appendix 

4. Would contractors experience contract development and 

negotiation costs of similar magnitude to those faced by DoD? If 

not, can you estimate these costs? 

III Questions about conformity assessment during period in 

which agency specific standards were in use 

1. How did you assess conformance to the standards specified in 

your procurement systems? (Supplier declarations, supplier test 

results, third-party analysis, in-house testing, or other) 

2. Can you estimate the effort expended by DoD in reviewing 

documents (including Certificates of Conformance) for conformity 

to standards?

3. If measurements, chemical analyses, or other tests were 

required, how were the capabilities of the organizations

performing the tests assessed? Was formal accreditation 

required? Did DoD act as an Accrediting Body (AB)?

4. Can you estimate the costs to contractors or third-parties, 

including laboratories, for demonstrating conformity? (This may 

include testing fees, hosting inspections, document preparation, 

etc.) 

IV Questions about current standards-related DoD 

expenditures, with non-governmental standards having 

replaced many of the Milspecs 

1. What fraction of the total construction-related standards have 

been converted from Milspecs to non-governmental VCSs? 

Approximately how many Milspecs are still being used by your 

division? Are any new Milspecs being developed? 

2. Does your agency/division have a separate standards

organization in today’s regime? If so, approximately how many 

people (in FTEs) work in this standards group? Do you have an 

estimate as to their annual budget? 

3. Please estimate the annual labor effort (in FTEs) expended by 

personnel in developing new standards. Can you partition this 

number by skill level, i.e. how many engineers, technicians, 

administrative support staff? 

4. How much training and development effort is required for these 

people, including both initial training and continuing education? 

Can you quantify this in terms of out-of-pocket dollars (materials

and travel), effort hours, and supervisory expenditures? 

5. Please estimate the annual labor effort expended in 

maintenance and improvement of existing standards. Are the 

people involved in these activities similar in skill level to those 

developing new standards? 

A-3 
Measuring Benefits from the National Technology Transfer and Advancement Act 

6. What other annual expenditures are involved in maintaining 

existing standards? (Information systems hardware and 

software purchase, service and support; travel cost for 

conferences or seminars; others) 

7. What standards-related expenses are incurred as a part of 

contract development and negotiation? Do purchasing agents 

and legal support staff need relevant training or technical support 

for their roles? Can you estimate annual effort required from

these highly-compensated individuals? 

8. Please estimate the annual support costs for the nongovernmental standards now being used by your division. 

Include annual fees and travel expenses for participation in 

SDOs, purchase costs for hard-copy standards, or other 

expenses.

A-4 
Appendix 

A.2 INTERVIEW GUIDE FOR GOVERNMENT AGENCIES 

ON THE USE OF VOLUNTARY CONSENSUS 

STANDARDS IN REGULATORY ACTIVITIES 

This interview guide has been designed for use by agencies that 

reference standards in their regulations. This instrument focuses on the 

choice agencies face to develop and maintain their own standards or to 

rely on standards development organizations (SDOs) to coordinate these 

activities. Our intent is to gather relevant information about costs and

benefits from the use of Voluntary Consensus Standards (VCSs) in 

procurement, in regulation, and in laboratory accreditation, as part of a 

study undertaken for NIST’s Standards Services Division. 

Please answer the following questions about your operations 

over the past several years, including as much numerical data as

possible. These responses will be used to construct an aggregate 

measure of the net benefits generated from governmental agency use of 

voluntary consensus standards. Individual responses will be kept 

confidential by NIST, except that total realized benefits for each agency 

may be included in future annual reports about the implementation of the 

NTTAA. 

[This Interview Guide has been tailored for use in agencies that 

create and implement their own standards and also make reference to 

voluntary consensus standards developed by third-parties. The target 

agency for this data collection effort is the Consumer Product Safety 

Commission (CPSC). The background below is specific to the CPSC] 

Background: We understand that the Consumer Product Safety 

Commission is obligated by law to make use of industry-developed or 

other voluntary consensus standards for product safety, if in the expert 

judgment of the CPSC, an existing standard will adequately protect the 

public. If there is no such standard in place, the CPSC must develop an 

agency-specific standard. This offers RTI (and NIST) an ideal 

opportunity to make a side-by-side comparison of the costs of these two 

modes of operation. We hope that you can help us quantify some of the 

benefits of the NTTAA’s requirement for VCS. 

I Questions about standards-related expenditures within the 

CPSC in cases where the Commission develops new

standards to insure product safety

1. Do you have data on the average number of new standards you 

initiate each year? By comparison, how many third-party 

standards (VCSs) do you accept annually? 

2. Please estimate the annual labor effort (in FTEs) expended by 

CPSC personnel in developing new standards. Can you partition

this number by skill level, i.e. how many engineers, technicians, 

administrative support staff? 

A-5 
Measuring Benefits from the National Technology Transfer and Advancement Act 

3. How much training and development effort is required for these 

people (in support of new standards development), including 

both initial training and continuing education? Can you quantify 

this in terms of out-of-pocket dollars (materials and travel), effort 

hours, and supervisory expenditures? 

4. Please estimate the annual labor effort (also in FTEs) expended 

in maintenance and improvement of existing CPSC standards. 

Are the people involved in these activities similar in skill level to 

those developing new standards? 

5. What other annual expenditures are involved in maintaining 

existing CPSC standards? (Include information systems 

hardware and software purchase, service and support; travel 

cost for conferences or seminars; other expenditures.) 

6. How do you assess conformance to the standards you issue? 

(manufacturer declarations, product test results, third-party 

analysis, in-house testing, or other) 

7. Can you estimate the annual effort expended by CPSC 

personnel in reviewing documents for conformity to standards? 

8. If physical measurements, chemical analyses, and/or other tests 

are required, how are the capabilities of the organizations 

performing the tests assessed? Does CPSC require formal 

accreditation? Does the Commission serve as an Accrediting 

Body (AB), or do you support third-party accreditation?

II Questions about product manufacturer expenses in the 

case that CPSC develops agency-specific standards. 

5. What would you estimate to be each manufacturer’s labor effort 

required to interpret and comply with a CPSC standard? Will this 

level of effort vary with the size of the company? ...with the 

annual unit volume of production of affected products? ...with the 

number of unique products affected? 

6. How do these manufacturers train and develop their standards 

experts? Can you estimate the annual costs for these activities, 

including supervisory costs?

7. Manufacturers need to maintain databases and support change 

control to support existing standards. Do you have an estimate 

of their information systems costs, and the supervisory effort 

required for standards maintenance and support? 

8. Does CPSC or its designee audit manufacturers for compliance 

to standards? If so, are the producers subject to multiple audits

on an annual basis? Can you estimate the effort hours involved 

in supporting an audit? 

A-6 
Appendix 

III Questions about standards-related expenditures within the 

CPSC in cases where the Commission adopts industry

standards or other VCSs

1. Please estimate the annual labor effort (in FTEs) expended by 

CPSC personnel in supporting development of new VCSs. Can 

you partition this number by skill level, i.e. how many engineers, 

technicians, administrative support staff?

2. How much training and development effort is required for these 

people (in support of new standards development), including 

both initial training and continuing education? Can you quantify 

this in terms of out-of-pocket dollars (materials and travel), effort 

hours, and supervisory expenditures? 

3. Please estimate the annual labor effort (also in FTEs) expended 

in maintenance and improvement of existing VCSs. Are the 

people involved in these activities similar in skill level to those 

developing new standards? 

4. What other annual expenditures are involved in maintaining 

existing CPSC standards? (Include information systems 

hardware and software purchase, service and support; travel 

cost for conferences or seminars; other expenditures.) 

5. Please estimate the annual support costs for the third-party VC 

standards now being supported by the CPSC. Include annual 

fees and travel expenses for participation in SDOs, purchase 

costs for hard-copy standards, or other expenses. 

6. Who accepts the responsibility for assessing conformity with a 

VCS, the SDO issuing a standard or the CPSC? If the 

Commission retains ownership of this responsibility, how do you 

assess conformance to the VC standards? (audits, manufacturer 

declarations, product test results, third-party analysis, in-house 

testing, or other) 

7. Can you estimate the annual effort expended by CPSC 

personnel (if any) in reviewing documents for conformity to 

standards? ...in conducting audits? ...in operating or supporting 

accreditation systems? 

IV Questions about product manufacturer expenses in the 

case that CPSC adopts a VC standard. 

1. Would you expect manufacturers to experience a different level 

of labor, training, or information systems costs in complying with 

a VCS versus CPSC standard? If so, what would cause the 

costs to differ?

2. Can you estimate the annual costs to a manufacturer for the 

following expenses specific to voluntary consensus standards: 

annual fees and travel expenses for participation in SDOs; 

purchase costs for hard-copy standards; and other costs? 

A-7 
Measuring Benefits from the National Technology Transfer and Advancement Act 

A.3 INTERVIEW GUIDE FOR GOVERNMENT AGENCIES 

ON THE USE OF THIRD-PARTY LABORATORY 

ACCREDITATION 

This interview guide has been designed for use by agencies that 

require laboratory accreditation in their regulatory or procurement 

activities. Accreditation is a critical element in the conformity 

assessment process, allowing affected firms and government agencies 

to have confidence that the required analyses, measurements, and/or 

calibrations are being conducted by competent laboratory organizations. 

This instrument focuses on the choice agencies face to develop 

laboratory accreditation processes and serve as an accrediting body 

(AB) or to rely on third-party organizations to coordinate the necessary 

accreditation activities. Our intent is to gather relevant information about 

costs and benefits from the use of Voluntary Consensus Standards 

(VCSs) in procurement, in regulation, and in laboratory accreditation, as 

part of a study undertaken for NIST’s Standards Services Division.

Please answer the following questions about your operations 

over the past several years, including as much numerical data as

possible. These responses will be used to construct an aggregate 

measure of the net benefits generated from governmental agency use of 

voluntary consensus standards. Individual responses will be kept 

confidential by NIST, except that total realized benefits for each agency 

may be included in future annual reports about the implementation of the 

NTTAA. 

[The target organization for this interview guide is EPA’s National 

Environmental Laboratory Accreditation Program (NELAP), which has 

recently undergone a transformation from a voluntary consensus SDO to 

one completely within the purview of the EPA. This experience should 

be useful in helping highlight the differences between the two operating 

regimes.] 

I Questions about agency costs incurred in support of a role as

an accrediting body (AB) and accreditation services provider: 

1. Do you manage laboratory accreditation activities within your 

organization, or employ an outside organization under contract to

manage these processes?

2. Approximately how many laboratories are currently accredited by 

your agency? What is the population of other laboratories

seeking accreditation but not yet accredited by your agency? 

3. Which outside organizations recognize your agency 

accreditations (if any) and what form does this recognition take? 

4. Please estimate the following internal agency costs or contractor

expenses for operating your accreditation programs: 

• Administrative costs - management and staff salaries,

capital expenditures (including computer hardware & 

software), database management 

A-8 
Appendix 

• Travel and salary costs for site inspection visits

• Operating expenses for proficiency testing (PT) 

programs

• Revenue from accreditation fees, PT program charges 

(subject to government regulation on these sources of

revenue) 

5. For those positions that involve agency personnel, can you 

estimate the annual labor effort (in FTEs, if possible) expended 

in performing the activities enumerated above? How is this effort 

partitioned across skill levels – i.e., how many professionals, 

technicians, administrative support staff?

6. How much training and development effort is required for these 

people (for support of new accreditations and management of 

the existing infrastructure)? Can you quantify this in terms of 

out-of-pocket dollars, effort hours, and supervisory 

expenditures? 

7. Can you describe any activities your agency is involved in to 

extend recognition of this accreditation across: additional 

government agencies, geographies, or to new types of analytical 

tests? Please estimate annual travel and salary costs for 

negotiating and maintaining MOUs for mutual recognition 

agreements (MRAs), including third-party, interagency, 

international, or state-by-state agreements. 

8. What other annual expenditures are incurred by your agency in 

acting as an accrediting body and/or managing accreditation 

processes?

II Questions about costs borne by laboratories and/or thirdparty organizations who participate in agency-led

accreditation processes

1. From discussions you have had with participating laboratories, 

what is the cost to each laboratory of analyzing PT samples, 

providing required documentation, conducting mandatory 

training, and hosting inspections for accreditation? Do any of 

these costs apply only to initial accreditation or are they incurred 

annually? 

2. Are there fees charged to laboratories by your agency or 

authorized contractors for participation in the accreditation 

process? Are these one-time charges or annual fees? Do the 

laboratories have to pay third-party organizations for required 

documents or services performed? What is the annual cost to 

each laboratory for these charges and fees? 

3. If the laboratories are subject to multiple inspections (by different 

agencies or jurisdictions), how many separate inspections and/or 

performance tests must be supported for the laboratories to 

function? Are there geographic areas or agencies not served by 

laboratories due to this lack of mutual recognition? 

A-9 
Measuring Benefits from the National Technology Transfer and Advancement Act 

4. In the cases where mutual recognition is not in place, what types 

(and magnitudes) of costs are incurred by ABs in supporting 

these separate processes? Are their costs similar in magnitude

to those incurred by the agency? [As an example, if each state 

requires an accreditation separate from that of the USEPA, do 

the state EPAs expend similar amounts of effort to that of the 

federal agency?] 

III Questions about agency costs incurred in support of 

accreditation if a third-party organization acts as an AB and 

provides accreditation services: 

This situation may be more difficult to estimate as we have few 

good examples of 'outside management of accreditation' within EPA. As 

a result, I expect that responses to this section will be somewhat 

speculative – more of a description of an ideal state, rather than a 

realized alternative. 

1. Can you estimate administrative costs in the case that a nongovernmental third-party managed all accreditation activities 

(even if the agency retains final authority over decisions on 

whether or not to accredit a specific laboratory)? Include all of 

the cost areas described in question I-4 above. 

2. How much would you estimate the agency would be required to 

pay for hard-copy documents, reports, and access to databases? 

Do you have any relevant examples of situations in which these 

costs are being incurred today? 

3. It is likely that travel and meeting costs may increase

substantially in the case that a voluntary consensus organization

takes a leadership role in accrediting. What would you estimate 

as the annual cost, in terms of out-of-pocket expense and staff 

time (annual hours or FTEs) to support this type of a structure? 

4. Can you quantify the costs of any additional oversight or risk 

mitigation effort that the agency would require in evaluating the 

effectiveness of the organization managing the accreditation and 

recognition tasks? 

IV Questions about costs borne by laboratories and/or thirdparty organizations who participate in third-party managed 

accreditation and recognition processes 

1. We would take an initial position that the costs for managing

inspections, databases, and PT sample programs would remain 

constant in this new scenario, i.e., that the federal government is 

no more or no less efficient in providing these services than an 

outside organization. Are there any aspects in which these costs 

would likely differ? Are there economies of scale or scope that 

one organization or the other would be able to bring to this 

management task? 

A-10
Appendix 

2. An annual accreditation fee would undoubtedly be charged to the 

laboratories by the AB, as one of its major sources of revenue. 

The magnitude of this charge does not affect the overall 

economics under consideration, as it is merely a transfer from

one private-sector firm to another. Still, it might be interesting to 

estimate the likely amount of this charge. Do you have any 

thoughts on this? 

3. Participating laboratories would also have to purchase

documents from the AB and any organizations coordinating 

mutual recognition. It seems reasonable to estimate the same 

cost per document to these firms as that incurred by the agency. 

How many documents would be needed for an average 

laboratory? 

4. Finally, a training and development infrastructure would be 

needed for bringing new laboratories, technicians, agency 

personnel, and outside organizations up to speed on details of 

the programs involved. Does the agency have any information 

on the costs of this type of a program in the private (or 

government) sector? 

A-11