INFORMATION 

MANAGEMENT

National Technical 

Information Service’s 

Dissemination of 

Technical Reports 

Needs Congressional 

Attention

Report to Congressional Committees

November 2012

GAO-13-99

United States Government Accountability Office

GAO
United States Government Accountability Office

Highlights of GAO-13-99, a report to 

congressional committees

November 2012

INFORMATION MANAGEMENT 

National Technical Information Service’s 

Dissemination of Technical Reports Needs 

Congressional Attention

Why GAO Did This Study

NTIS was established by statute in 

1950 to collect scientific and technical 

research reports, maintain a 

bibliographic record and repository of 

these reports, and disseminate them to 

the public. NTIS charges fees for its 

products and services and is required 

by law to be financially self-sustaining 

to the fullest extent possible.

GAO was mandated by Congress to 

update its 2001 report on aspects of 

NTIS’s operations and the reports in its 

collection. Specifically, GAO’s 

objectives were to determine (1) how 

NTIS is currently organized and 

operates, including its functions, 

current staffing level, reported cost of 

operations, and revenue sources; (2) 

the age of and demand trends for 

reports added to NTIS’s repository; 

and (3) the extent to which these 

reports are readily available from other 

public sources. To do this, GAO 

reviewed agency documentation, 

analyzed a sample of reports added to 

NTIS’s collection from fiscal years 

1990 through 2011 (reports from the 

period since GAO’s last study and 

other older reports), and interviewed 

relevant agency officials.

What GAO Recommends

GAO is suggesting that Congress 

reassess the appropriateness and 

viability of the fee-based model under 

which NTIS currently operates for 

disseminating technical information to 

determine whether the use of this 

model should be continued. In 

comments on a draft of this report, the 

Department of Commerce stated that 

NTIS believes GAO’s conclusions do 

not fully reflect the value that the 

agency provides. However, GAO 

maintains that its conclusions and 

suggestion to Congress are warranted. 

What GAO Found

As a component of the Department of Commerce, the National Technical 

Information Service (NTIS) is organized into five primary offices that offer the 

public and federal agencies a variety of products and services. As of late October 

2012, NTIS was supported by 181 staff, all except 6 of which held full-time 

positions. NTIS reports its progress toward agency goals to the Deputy Secretary 

of Commerce, and the Director of NTIS reports to the Director of Commerce’s 

National Institute of Standards and Technology. In addition, NTIS receives 

oversight of its functions and strategic direction from an advisory board with 

members appointed by the Secretary of Commerce. NTIS’s product and service 

offerings include, among other things, subscription access to reports contained in 

its repository in both print and electronic formats, distribution of print-based 

informational materials to federal agencies’ constituents, and digitization and 

scanning services. 

NTIS revenues are generated exclusively from direct sales or subscriptions for its 

products and services. NTIS reported that net revenues from all its functions 

(products and services) totaled about $1.5 million in fiscal year 2011. However, 

over most of the last 11 years, its costs have exceeded revenues by an average 

of about $1.3 million for its products. While NTIS has not recovered all of its costs 

for products through subscriptions and other fees, it has been able to remain 

financially self-sustaining because of revenues generated from its services such 

as distribution and order fulfillment, web hosting, and e-training. The NTIS 

strategic plan states that the electronic dissemination of government technical 

information by other federal agencies has contributed to reduced demand for 

NTIS’s products. As a result, the agency is taking steps to reduce its net costs, 

such as improving business processes and increasing the breadth and depth of 

its collection.

NTIS’s repository has been growing with mostly older reports, but the demand for 

more recent reports is greater. Specifically, NTIS added approximately 841,500 

reports to its repository during fiscal years 1990 through 2011, and approximately 

62 percent of these had publication dates of 2000 or earlier. However, the 

agency was more likely to distribute (by direct sale or through a subscription) 

reports published more recently. For example, GAO estimated that 100 percent 

of the reports published from 2009 through 2011 had been distributed at least 

once, while only about 21 percent of reports published more than 20 years ago

had been. 

Of the reports added to NTIS’s repository during fiscal years 1990 through 2011, 

GAO estimates that approximately 74 percent were readily available from other 

public sources. These reports were often available either from the issuing 

organization’s website, the federal Internet portal (http://www.USA.gov), or from 

another source located through a web search. Reports published from 1990 to 

2011 were more likely to be readily available elsewhere than those published in 

1989 or earlier. Further, GAO estimated that 95 percent of the reports available 

from sources other than NTIS were available free of charge. NTIS’s declining 

revenue associated with its basic statutory function and the charging for 

information that is often freely available elsewhere suggests that the fee-based 

model under which NTIS currently operates for disseminating technical 

information may no longer be viable and appropriate.

View GAO-13-99. For more information, 

contact Valerie C. Melvin at (202) 512-6304 or 

melvinv@gao.gov
Page i GAO-13-99 Commerce's National Technical Information Service

Letter 1 

Background 4

NTIS Carries out a Variety of Functions Related to Its Mission, but 

Currently Is Financially Sustained by Its Service Offerings 7

Additions to NTIS’s Repository Have Mostly Included Older 

Reports, but Demand for More Recent Reports Is Greater 21

Most Reports Added Since 1990 Are Freely Available from Other 

Websites 24

Conclusions 29

Matter for Congressional Consideration 29

Agency Comments and Our Evaluation 29

Appendix I Objectives, Scope, and Methodology 32

Appendix II Comments from the Department of Commerce 36

Appendix III GAO Contact and Staff Acknowledgments 41

Tables

Table 1: NTIS Staff as of October 31, 2012 10

Table 2: NTIS Office Responsibilities and Functions 10

Table 3: Reported Net Earned Revenue (or Net Cost) for NTIS 

Products and Services, Fiscal Years 2001-2011 18

Figures

Figure 1: NTIS Organization Chart 8

Figure 2: Fiscal Years 2001-2011 Net Earned Revenues and Net 

Costs for NTIS Products and Services 19

Figure 3: Year of Publication for Documents Added to NTIS 

Repository, 1990 through 2011 22

Figure 4: Estimated Percentage of Reports Distributed and Not 

Distributed by Publication Date during Fiscal Years 2000 

through 2011 23

Figure 5: Estimated Availability of Reports by Year of Publication 25

Contents 
Page ii GAO-13-99 Commerce's National Technical Information Service

Figure 6: Estimated Availability of Reports Distributed and Not 

Distributed During Fiscal Years 2000 through 2011 26

Figure 7: Of Reports Found Elsewhere, Estimated Percent 

Available for Free and Not Free, by Year of Publication 28

Abbreviations

Commerce Department of Commerce

NASA National Aeronautics and Space Administration

NIST National Institute of Standards and Technology

NTIS National Technical Information Service

SSA Social Security Administration

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-13-99 Commerce's National Technical Information Service

United States Government Accountability Office

Washington, DC 20548

November 19, 2012

The Honorable Barbara Mikulski

Chairwoman

The Honorable Kay Bailey Hutchison

Ranking Member

Subcommittee on Commerce, Justice,

 Science, and Related Agencies

Committee on Appropriations

United States Senate

The Honorable Frank Wolf

Chairman

The Honorable Chaka Fattah

Ranking Member

Subcommittee on Commerce, Justice,

 Science, and Related Agencies

Committee on Appropriations

House of Representatives

The Department of Commerce’s National Technical Information Service 

(NTIS) serves as a permanent repository and disseminator of scientific, 

technical, engineering, and business-related information.1 NTIS acquires 

such information—largely in the form of research reports—primarily from 

federal agencies and their contractors and grantees, as well as from other 

domestic and foreign sources. NTIS estimates that it maintains in its 

central repository more than 2.5 million records2

 

1

In the remainder of this report, we use the term “technical information” to refer collectively 

to these types of information.

covering 378 technical 

and business-related subject areas. In addition, NTIS performs various 

fee-based information services for other federal agencies. The legislation 

2

These 2.5 million records contain, among other things, the metadata that identify all of 

the reports held by NTIS, such as the report’s author, publication date, and the agency 

where the report originated. These records link to the actual reports, which NTIS 

maintains elsewhere in hardcopy or as electronic documents or microfiche. According to 

the agency, these records represent all reports that have been catalogued in the 

electronic repository since 1964. As part of our further discussion in this report, we use the 

term “reports” in referring to the repository holdings.

 
 

Page 2 GAO-13-99 Commerce's National Technical Information Service

that established NTIS3

In May 2001, we reported on NTIS’s operations,

requires all of the services and functions it 

provides to be financially self-sustaining, to the extent feasible.

4 noting among other 

things, the availability of many of the reports maintained in its repository 

from other sources, such as the originating agencies’ websites. House 

Report 112-169, accompanying the Consolidated and Further Continuing 

Appropriations Act, 2012,5

To determine how NTIS is organized and operates, we reviewed the 

agency’s strategic plan and documentation on its organizational and 

reporting structure, office staffing level and assigned responsibilities, and 

types of products and services offered. We also reviewed cost data 

contained in the agency’s financial reports.

directed GAO to update aspects of its prior 

study. Our specific objectives were to determine (1) how NTIS is currently 

organized and operates, including its various functions, current staffing 

level, reported costs of operations, and revenue sources; (2) the age of 

and demand trends for reports added to NTIS’s repository; and (3) the 

extent to which these reports are readily available from other public 

sources.

6 In addition, we analyzed 

relevant laws and regulations describing NTIS’s authority and 

responsibilities,7 and reviewed our previous reports that discussed its 

mission and operations.8

 

3

15 U.S.C. 1153.

We supplemented our analyses with interviews 

of the Director of NTIS and other relevant agency officials; we also 

interviewed officials of the Department of Commerce and its National 

Institute of Standards and Technology (NIST), which have specific 

reporting relationships with NTIS.

4

GAO, Information Management: Dissemination of Technical Reports, GAO-01-490 

(Washington, D.C.: May 18, 2001).

5

Pub. L. No. 112-55 (Nov. 18, 2011). 

6

The cost data reflected in our report were taken from documentation provided by NTIS. 

However, we did not test the reliability of NTIS’s cost accounting system, from which the 

agency derives its cost data. 

7

For example, 15 U.S.C.1151-57, 15 U.S.C. 3704b-1 & 2, and 15 C.F.R. Part 1180.

8

GAO, Information Policy: NTIS’ Financial Position Provides an Opportunity to Reassess 

Its Mission, GAO/GGD-00-147 (Washington, D.C.: June 30, 2000) and GAO-01-490. 
 

Page 3 GAO-13-99 Commerce's National Technical Information Service

To determine the age of and demand trends for reports added to NTIS’s 

repository, we first analyzed the distribution of the publication dates for all 

reports added to the repository since 1990. We chose to include reports 

added since 1990 to determine the current availability of older documents 

in light of NTIS’s role as a permanent repository, and to focus on the 

reports added since our previous review in 2001.9 We subsequently 

selected a stratified random sample from this body of reports10

To determine the extent to which reports NTIS collects are readily 

available from other public sources, we searched the Internet to 

determine if each of the reports included in our sample could be found 

elsewhere and at no cost. Using a tiered approach, we searched the 

following four sources in the order shown: (1) the issuing organization’s 

website; (2) the U.S. Government Printing Office’s Federal Digital System 

website—http://www.gpo.gov/fdsys; (3) the federal government Internet 

portal USA.gov—http://www.USA.gov; and (4) a web search conducted 

using the commercial search engine http://www.Google.com. Specifically, 

we determined whether each report was first available at no cost on the 

issuing organization’s website and, if so, concluded the Internet search at 

this point. However, if the report was not available, then the search 

continued to the second source, and so on, until either the report was 

found to be available at one of the remaining sources, or all sources were 

exhausted. We then used our results to estimate the percentage of the 

total population of NTIS reports added to the repository during fiscal years 

1990 through 2011 that was available from other public sources.

and then 

used cumulative data provided by NTIS on the sales of any of these 

reports to analyze the demand for reports in the repository.

All of the results derived from the sample analyses constituted estimates 

that are subject to sampling errors. These sampling errors measure the 

extent to which the sample size and structure are likely to differ from the 

population they represent. Because we followed a probability procedure 

based on random selections, our sample is only one of a large number of 

samples that we might have drawn. Since each sample could have 

provided different estimates, we express our confidence in the precision 

of our particular sample’s results as a 95 percent confidence interval. This 

 

9

We did not include reports added for fiscal year 2012 because our study only focused on 

those fiscal years that had been completed at the time that our study was initiated. 

10Our selected sample consisted of 384 reports.
 

Page 4 GAO-13-99 Commerce's National Technical Information Service

is the interval that would contain the actual population value for 95 

percent of the samples we could have drawn.

A more detailed discussion of our objectives, scope, and methodology is 

provided in appendix I.

We conducted this performance audit from February 2012 to November 

2012 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

NTIS’s basic statutory function is to collect research reports, maintain a 

bibliographic record and permanent repository of these reports, and 

disseminate them to the public. In addition, NTIS has developed a variety 

of information-related services. NTIS charges user fees for the sale of its 

products to the public and services to federal agencies.

Under statutory authority enacted in 1950,11 NTIS collects reports 

containing scientific, technical, and engineering information from both 

domestic and foreign sources in a repository and makes the information 

available to (1) business and industry, (2) state and local governments, 

(3) other federal agencies, and (4) the general public to increase U.S. 

competitiveness in the global economy. The statute does not define 

scientific, technical, and engineering information. However, the Secretary 

of Commerce has interpreted this to include “all types of information 

which have a more or less direct bearing on business and industry 

generally.” 12

 

1115 U.S.C. 1152.

The Secretary of Commerce described such information as 

including “economic information, market information, and related 

information so long as it is reasonably specific and bears some direct 

relationship to the organization and operation of industrial or business 

enterprise.” NTIS’s enabling legislation authorized it to charge fees for its 

12In 1954, the Comptroller General found this to be a reasonable interpretation of the law’s 

use of the word “technical.” 34 Comp. Gen. 58 (1954).

Background 
 

Page 5 GAO-13-99 Commerce's National Technical Information Service

products and established a policy to recover all costs, as feasible, through 

the fees.13

NTIS’s authority was revised in the National Technical Information Act of 

1988.14 This act gave the agency authority to also enter into contracts, 

cooperative agreements, joint ventures, and other transactions as 

necessary in the conduct of the business of NTIS, and declared the NTIS 

repository to be a permanent federal function that could not be transferred 

to the private sector without congressional approval. The act was 

subsequently amended by the American Technology Preeminence Act of 

1991,15

• required all costs associated with acquisition, processing, storage, 

bibliographic control, and archiving to be recovered primarily by fees;

which

• required agencies to transfer unclassified scientific, technical, and 

engineering information which results from federally funded research 

and development to NTIS; and

• provided that NTIS’s use of new methods or media for information 

dissemination should include producing and disseminating information 

in electronic format.

Further, the Commerce, Justice, and State, the Judiciary, and Related 

Agencies Appropriations Act of 199316

Our prior work found that, from fiscal years 1988 through 1994, NTIS 

showed excess revenue from the sale of its products and services in all 

years except one and generated cumulative net earned revenues of $5.8 

million. However, over the next 5-year period (from fiscal year 1995 

through fiscal year 1999), NTIS had a cumulative net cost

established a revolving fund for the 

payment of all expenses incurred by NTIS and gave it the authority to use 

that fund without further appropriations action by Congress.

17

 

1315 U.S.C. 1153.

of about $4.2 

14Pub. L. No. 100-519 (Oct. 24, 1988); 15 U.S.C. 3704b. 

15Pub. L. No. 102-245 (Feb. 14, 1992); 15 U.S.C. 3704b-1.

16Pub. L. No. 102-395 (Oct. 6, 1992).

17In the remainder of this report, we use the term “net costs” to refer to financial losses or 

gross costs that exceeded revenues. 
 

Page 6 GAO-13-99 Commerce's National Technical Information Service

million. Accordingly, in August 1999, the Secretary of Commerce 

proposed closing NTIS by September 30, 2000, because he believed that 

declining sales revenues would not continue to be sufficient to recover all 

of the agency’s operating costs. The Secretary attributed this decline 

partly to other agencies’ practice of making their research results 

available to the public for free through the Web. He also proposed 

transferring NTIS’s research report archives to the Library of Congress 

and requiring federal agencies to give the public free online access to 

new research reports.

In June 2000,18

We again reported on NTIS’s operations in May 2001.

we reported that NTIS could possibly remain solvent for a 

few years because Commerce and NTIS officials had acted in fiscal year 

1999 to reduce costs when it appeared there would not be sufficient funds 

to operate. However, we noted that the agency’s solvency beyond fiscal 

year 2003 was in doubt based on our forecast of its revenues and costs, 

which suggested that costs would exceed revenues beginning in fiscal 

year 2001 and continue. We pointed out that, if these net costs were 

realized, cumulative net earned revenues from previous years’ operations 

could be absorbed and the agency could run out of operating funds after 

fiscal year 2003. Based on these findings, we suggested that Congress 

consider whether NTIS (or another agency) was needed to serve as a 

permanent repository and disseminator for research reports; we also 

suggested that Congress could be aided in this assessment by reviewing 

NTIS annual financial and operational reports that Commerce was 

required to submit, but which it had not done since fiscal year 1995.

19

 

18

We noted that, 

according to its Deputy Director, the agency had significantly reduced its 

costs after our June 2000 report and had shown net earned revenues of 

$2.2 million for fiscal year 2000. Moreover, the Deputy Director had 

projected a net earned revenue of $1.5 million for fiscal year 2001. We 

also noted that the agency was providing a variety of other fee-based 

services for agencies and that, while demand for electronic products was 

on the rise, research reports and other scientific, technical, and 

engineering information maintained by NTIS were also becoming 

increasingly available on agency websites and through other public 

sources—often at no cost. We included as a matter for congressional 

GAO/GGD-00-147. 

19GAO-01-490. 
 

Page 7 GAO-13-99 Commerce's National Technical Information Service

consideration that Congress look at how this information was defined; 

whether there was a need for a central repository of this information; and 

if a central repository was maintained, whether all information should be 

retained permanently, and what business model should be used to 

manage it.

The Secretary of Commerce agreed with our assessment and raised as a 

primary question whether there was a need for a central repository in 

view of the increasing availability of newer publications from sources 

other than NTIS. The Secretary also noted that the need for a central 

repository depended on whether the information would be permanently 

maintained by agencies and whether the information would be easy to 

locate without the kind of bibliographic control that NTIS provides.

Subsequent to the issuance of our reports, in December 2003, Congress 

passed the 21st Century Nanotechnology Research and Development 

Act, which provided a coordinated federal approach to stimulating 

nanotechnology research and development. The act directed the 

Secretary of Commerce to establish a clearinghouse for information 

related to the commercialization of nanotechnology research using the 

resources of NTIS to the extent possible. As of September 2012, NTIS 

noted that it held over 700 publications in its nanotechnology collection.

NTIS currently operates as one of 12 independent bureaus within 

Commerce, with the mission to help promote the nation’s economic 

growth by providing access to information that stimulates innovation and 

discovery. Headquartered in Alexandria, Virginia (with warehouses in 

Alexandria, Virginia and in Brandywine, Maryland), the agency is 

organized into five primary offices. To carry out its statutory functions of 

collecting and maintaining a permanent repository and bibliographic 

record of research reports, and to disseminate them, the agency offers a 

variety of products, such as fee-based access to the reports in its 

repository. In addition, NTIS offers information-related services to federal 

agencies, such as distribution and order fulfillment, web hosting, and etraining, that are less directly related to its basic statutory function. While 

NTIS’s service offerings have resulted in increased revenues, allowing 

the agency to remain financially self-sustaining, it has experienced a net 

cost relative to its products, calling into question whether the agency’s 

function of acting as a self-financing repository of technical information is 

still viable.

NTIS Carries out a 

Variety of Functions 

Related to Its Mission, 

but Currently Is 

Financially Sustained 

by Its Service 

Offerings
 

Page 8 GAO-13-99 Commerce's National Technical Information Service

NTIS is led by a director, who is aided by two executives—a chief 

information officer and a chief financial officer. In addition, three 

operational offices have a variety of responsibilities for providing products 

and services that include collecting and disseminating technical reports, 

offering access to other information sources, and providing informationrelated services to federal agencies. Figure 1 displays the NTIS 

organization.

Figure 1: NTIS Organization Chart

NTIS operates as a unit within Commerce and receives oversight from 

the Deputy Secretary of Commerce, the Director of NIST,20

 

20NIST is a Commerce operating unit that, among other things, promotes U.S. innovation 

and industrial competitiveness by advancing measurements for science, standards, and 

technology.

and an 

advisory board. In this regard, the NTIS Director communicates progress 

toward agency goals to the Deputy Secretary of Commerce. For example, 

the Director participates in biweekly executive management meetings that 

are held with the Deputy Secretary. At these meetings, executive leads 

from each of Commerce’s components report on the status of 

performance and strategic goals within their offices, among other things.

NTIS Receives Guidance 

and Oversight from 

Commerce Department 

Entities and Carries out Its 

Functions through Five 

Primary Offices
 

Page 9 GAO-13-99 Commerce's National Technical Information Service

In addition, under a Commerce departmental order,21

Further, NTIS receives guidance on its operations from the NTIS Advisory 

Board, which was established by law

the NTIS Director 

also reports to the Director of NIST. As outlined in the Commerce 

organization chart and stated in the Commerce departmental organization 

order, the NTIS Director is to report and be responsible to the Director of 

NIST. In turn, the Director of NIST prepares the NTIS Director’s annual 

performance appraisal. Beyond this, the Director of NIST stated that the 

agency does not provide any other operational or financial oversight 

functions for NTIS. For example, NIST does not approve NTIS’s budget 

(although it does coordinate NTIS’s budget for final inclusion in the 

department’s overall budget, which is approved by Commerce).

22 in 1988 to review the general 

policies and operations of NTIS, including policies related to fees and 

charges for its products and services. The board,23

As of late October 2012, NTIS was supported by 181 staff, all except 6 of 

which held full-time positions. These included 103 NTIS employees and 

78 contractors. Table 1 shows the number of staff dedicated to the Office 

of the Director and each of the agency’s other primary offices as of 

October 2012.

comprised of a 

chairperson and four members appointed by the Secretary of Commerce, 

is required to meet at least every 6 months to discuss NTIS’s activities. 

The board’s last meeting took place in late October 2012, and, according

to the notice of open meetings from the Federal Register, the intended 

focus was on the agency’s strategic business plan. The board submits an 

annual report to the Secretary of Commerce, which includes strategic and 

tactical recommendations regarding NTIS’s future operations.

 

21Department of Commerce Departmental Organization Order 30-7A (Washington, D.C.: 

November 2008). 

2215 U.S.C. 3704b(c).

23NTIS Advisory Board members can include individuals who are both internal and 

external to the Department of Commerce. 
 

Page 10 GAO-13-99 Commerce's National Technical Information Service

Table 1: NTIS Staff as of October 31, 2012

NTIS office

Full-time 

employees

Part-time 

employees

Full-time 

contractors

Part-time 

contractors Total

Director 2 0 0 0 2

Chief Financial Officer 13 0 0 0 13

Chief Information 

Officer 26 0 26 5 57

Federal Services 15 0 3 0 18

Product Management 

and Acquisition 29 1 5 0 35

Production Services 17 0 39 0 56

Total staff 102 1 73 5 181

Source: NTIS data.

Among these offices, executive functions, such as directing NTIS 

activities and developing NTIS budgets and policies for the use of 

information technology, are carried out by the Office of the Director, along 

with the offices of the Chief Financial Officer and the Chief Information 

Officer. The three other operational offices have a variety of 

responsibilities for providing NTIS’s products and services. Table 2 

summarizes the specific responsibilities and functions of these offices.

Table 2: NTIS Office Responsibilities and Functions

Office Key responsibilities

Office of the Director Manages and is responsible for all activities of NTIS. In addition to the Director, officials in this office 

include a contracting officer, legal personnel, and program analysts. 

Office of the Chief Financial 

Officer

Develops NTIS budgets, financial plans, financial reports, and analyses.

Develops, maintains, and operates NTIS financial systems.

Provides financial advice to the Director of NTIS.

Provides management analysis, administrative management, procurement, facilities management, 

property management, and office support services to NTIS components and other agencies.

Provides financial management services to NTIS components, customers, and source clients.

Office of the Chief 

Information Officer

Develops and administers plans, policies, and standards for NTIS’s acquisition and use of information 

technology.

Represents NTIS on information technology matters, including computer security issues.

Plans, develops, operates, and services centralized and decentralized network computer-based 

capabilities and associated telecommunications networks.

Provides training, consultation, technical documentation, and operation of data processing and 

telecommunications.
 

Page 11 GAO-13-99 Commerce's National Technical Information Service

Office Key responsibilities

Office of Federal Services Develops strategies for information dissemination services and Internet-based service business 

opportunities for federal agencies.

Manages service projects using in-house capabilities or through joint venture partners.

Assists agencies in meeting their multimedia product requirements.

Coordinates marketing and exhibit activities for information dissemination and Internet-based services.

Office of Product 

Management and 

Acquisition 

Builds the NTIS bibliographic database by cataloging, indexing, and abstracting information products 

from either hard copy or electronic media.

Identifies and acquires scientific, technical, and business-related information from domestic and 

international sources.

Coordinates the production and sale of clearinghouse products.

Manages customer relationships involving order receipt/entry, inquiry and complaint resolution, and 

product identification functions.

Gathers data about customers’ or potential customers’ needs and expectations.

Promotes available federal government information, services, and programs to customers through 

demand and subscription sales.

Office of Production 

Services 

Provides services relative to order fulfillment, information product dissemination and service delivery, 

order processing, product manufacturing, warehousing, specialized storage, inventory management, and 

distribution.

Operates, maintains, and manages specialized equipment and computer-based systems for the imaging, 

publication, and reproduction of information products.

Provides project- and product-management support for individual products and services.

Source: GAO analysis of Commerce data.

As part of its basic statutory function to collect and disseminate technical 

reports, NTIS offers a variety of fee-based products. According to the 

agency, its customer base for these products include scientists, 

engineers, the business community, librarians, information specialists in 

government, academia, and the general public. The agency has 

organized its products into three lines of business—”Technical Reports,” 

“Clearinghouse,” and “Publishing”—as described below.

To carry out its statutory function of collecting and disseminating such 

information, NTIS maintains a searchable repository that contains 

bibliographic records for the over 2.5 million scientific, technical, 

engineering, and business research reports it has acquired from federal 

government agencies, state and local governments, international 

organizations, and private-sector organizations. NTIS’s bibliographic 

database provides for the classification and cataloging of the records in 

this database. For example, according to the NTIS Database Search 

Guide, NTIS classifies its records into 39 categories. These categories 

can be used for searching the contents of NTIS’s database. The database 

covers a host of scientific and technical subjects, such as biology, 

NTIS Offers Technical 

Reports and Other 

Products to a Variety of 

Customers 

Technical Reports 
 

Page 12 GAO-13-99 Commerce's National Technical Information Service

chemistry, physics, transportation, health care, and the environment. Of 

the 2.5 million reports, NTIS noted that approximately 700,000 reports 

have been digitized, with the remainder in physical form, such as paper or 

microfilm.

Access to the reports is provided both through the direct sale of individual 

reports and by subscription. Individual reports can be purchased via 

postal mail; phone; e-mail; and NTIS’s online ordering system, accessed 

through its website, http://www.ntis.gov. Subscription-based access to the 

reports is obtained through, among other things, NTIS’s National 

Technical Reports Library, which provides subscribers with the ability to 

search the bibliographic records repository and to access the 

approximately 700,000 digitized reports in portable document format 

(PDF).24

Prices for individual reports and subscriptions vary. For example, an 

electronic copy of a report from the Economics and Statistics 

Administration, Benefits of Manufacturing Jobs: Executive Summary, can 

be purchased for $15; a “customized CD” for this report can be purchased 

for $30. Further, an electronic copy of a report from the National 

Aeronautics and Space Administration (NASA) Marshall Space Flight 

Center, NASA Robotics for Space Exploration, can be purchased for $15; 

a “customized CD” for this report can be purchased for $30. With regard 

to subscriptions, access to the National Technical Reports Library is sold 

as an annual subscription to institutions based on the number of 

individuals accessing the library. For example, as of October 2010, an 

annual subscription providing access for up to 3,000 individuals costs 

$2,100, while an annual subscription providing access for 18,001 to 

28,000 individuals costs $11,200. In addition, subscriptions can be 

purchased for a specific number of issues of a particular document type. 

For example, six issues of the Livestock, Dairy, and Poultry Outlook can 

be purchased for $91.

In addition, NTIS’s Selected Research Service allows a 

subscriber to select from more than 378 subcategories and automatically 

receive reports tailored to that area of interest. According to NTIS, reports 

that have not been digitized can be provided in a digital format when a 

customer purchases a copy.

 

24The Merriam-Webster Dictionary defines PDF as “a computer file format for the 

transmission of a multimedia document that is not intended to be edited further and 

appears unaltered in most computer environments.” Merriam-Webster Online, accessed 

October 5, 2012, http://www.merriam-webster.com/dictionary/pdf. 
 

Page 13 GAO-13-99 Commerce's National Technical Information Service

In addition to the technical reports that it collects and disseminates, NTIS 

disseminates publications covering a wide array of topics on behalf of 

other federal agencies. According to NTIS, these agencies request that 

NTIS distribute the publications in print or electronically. The following are 

examples of the federal products distributed:

• Standard Occupational Classification Manual. A manual containing 

information on all occupations in the national economy classified 

according to the system used by federal statistical agencies for the 

purpose of collecting, calculating, analyzing, or disseminating 

occupational data.

• Food and Drug Administration Food Code Manual. A code and 

reference document that provides technical and legal information 

about the regulations of the retail and food service industry.

• North American Industry Classification System. A publication that 

details a system for the collection, analysis, and dissemination of 

industrial statistics used by government policy analysts, academics, 

researchers, the business community, and the public.

• Export Administration Regulations. A compilation of regulations issued 

by Commerce’s Bureau of Industry and Security relating to the control 

of certain exports, re-exports, and activities.

• National Correct Coding Policy Manual. A manual developed by the 

Centers for Medicare and Medicaid Services to control improper 

coding leading to inappropriate payment in Medicare Part B claims. 

The manual provides guidance for providers on the correct coding of 

claims being submitted for reimbursement.

Through a memorandum of understanding or interagency agreement, 

NTIS also provides access to information collected from federal agencies, 

which it refers to as its “Publishing” line of business. In some instances, 

NTIS repackages the information with additional features. According to 

NTIS, agencies initiate the request for these services. These offerings 

include the following:

• Drug Enforcement Administration database. The Drug Enforcement 

Administration database identifies persons and organizations 

authorized to handle certain controlled drug substances and 

chemicals under the Controlled Substance Act. NTIS is the authorized 

official distributor of the database. NTIS provides online subscription 

Clearinghouse

Publishing
 

Page 14 GAO-13-99 Commerce's National Technical Information Service

access to the database on a weekly, monthly, or quarterly basis. NTIS 

also provides this information via a searchable CD-ROM.

• Death Master File. This file, maintained by the Social Security 

Administration (SSA), contains approximately 85 million records of 

deaths that have been reported since 1936. The file is used by 

government; credit reporting organizations; and financial, 

investigative, medical research, and other industries to verify deaths. 

Through an agreement with SSA, NTIS is the only authorized official 

distributor of the Death Master File. In this regard, NTIS provides 

access to this information, including on DVD, and provides the means 

to search and download the Death Master File online.

• World News Connection. NTIS offers access to this online news 

service, which provides translated and English-language news and 

information from non-U.S. media sources. The information is obtained 

from full-text, and summaries of newspaper articles, conference 

proceedings, television and radio broadcasts, periodicals, and 

nonclassified technical reports. The material in World News 

Connection is provided to NTIS by the Open Source Center, a U.S. 

government agency that provides analysis of foreign open source 

intelligence.

In addition to its product offerings, NTIS offers a variety of fee-based 

services to federal agencies that are less directly related to its basic 

statutory function of collecting and disseminating scientific and technical 

information. These include services that leverage capabilities NTIS has 

developed in the course of carrying out its mission. Its five service 

offerings are distribution and order fulfillment, web-based services and 

federal cloud computing, brokerage services, e-training and knowledge 

management services, and digitization and scanning services. To provide 

its services, NTIS enters into memorandums of understanding or 

interagency agreements with agencies. Further, NTIS offers some of 

these services through public-private partnerships with private industry, 

which it refers to as “joint venture partners.” The five types of services are 

described below.

Through a memorandum of understanding or interagency agreement, 

NTIS distributes large amounts of information products for federal 

agencies. According to NTIS officials, these services differ from that 

provided by its Clearinghouse in that they are used for distributing large 

quantities of agencies’ products rather than selling individual copies of 

NTIS Provides Fee-Based 

Services to Federal 

Agencies

Distribution and Order 

Fulfillment Services 
 

Page 15 GAO-13-99 Commerce's National Technical Information Service

publications. NTIS identified five primary clients for these services: the 

Internal Revenue Service, the Department of Agriculture, the Department 

of Education, the Pension Benefit Guaranty Corporation, and the 

Administrative Office of the U.S. Courts. For example, NTIS currently has 

an agreement with the Department of Agriculture to distribute on its behalf 

health and nutrition educational materials in the form of brochures, 

posters, and similar nutritional products. It also has an agreement with the 

Department of Education to perform similar services.

In 1988, Congress required NTIS to implement new methods or media for 

disseminating technical information; a 1992 amendment25 specified that 

this should include producing and disseminating products in electronic 

formats. According to NTIS, this has been a primary basis for NTIS’s 

transformation from a static paper-based distribution operation to a 

modern, computer-based model and also a basis for NTIS to provide 

information-dissemination services to other agencies. For example, 

NTIS’s information systems infrastructure enables it to host federal 

agencies’ applications and websites. The agency currently has an 

agreement with the Federal Insurance and Mitigation Administration to 

provide hosting for two of its websites and associated systems and to 

host its web-based tool for reviewing underwriting and claims operations. 

NTIS has also expanded its infrastructure to provide cloud computing26

services and, according to the agency, is currently offering infrastructureas-a-service and software-as-a-service.27 For example, along with joint 

venture partner Carney, Inc., NTIS had an agreement with the National 

Archives and Records Administration and currently has an agreement 

with the Social Security Administration to configure and host the “Jive” 

platform.28

 

2515 U.S.C. 3704b(e)(5).

According to NTIS, the agency has eight primary clients for its 

26Cloud computing is an emerging form of computing that relies on Internet-based 

services and resources to provide computing services to customers.

27According to NIST, infrastructure-as-a-service is a model where the service provider 

delivers and manages the basic computing infrastructure of servers, software, storage, 

and network equipment on which a platform to develop and execute applications can be 

developed by the consumer. Software-as-a-service is a model where the service provider 

delivers one or more applications and the computational resources and underlying 

infrastructure to run them for use on demand as a turnkey service.

28“Jive” is a social networking product for enterprise implementation. See its website: 

http://www.jivesoftware.com. More specifically, it intends to help employees, customers, 

and partners to connect, communicate, and collaborate. 

Web-Based Services and 

Federal Cloud Computing
 

Page 16 GAO-13-99 Commerce's National Technical Information Service

web-based services and federal cloud computing offerings, including the 

Department of Homeland Security, the Internal Revenue Service, and 

other federal agency initiatives.29

NTIS provides billing and collection services on a reimbursable basis to 

other agencies that, like itself, charge for products and services but which 

lack the necessary financial infrastructure to do their own billing and 

collecting. The agency refers to these as its “Brokerage Services.” For 

example, NTIS had an agreement with the National Agricultural Library to 

develop, implement, and operate account maintenance, invoicing, and 

collection procedures for the fees charged by the National Agricultural 

Library to users of its photocopy and loan services. In addition, the 

agency had an agreement with the National Library of Medicine to 

perform invoicing, accounting, and collection services for its Interlibrary 

Loan services. NTIS officials stated, however, that the agency plans to 

stop marketing its brokerage services due to the decrease in demand for 

this service.

NTIS’s service offerings have also been expanded to e-training and 

knowledge management. Specifically, in conjunction with joint venture 

partners, the agency provides collaborative software solutions, learning 

management systems and support services, training evaluation software, 

and talent management applications. For example, NTIS entered into an 

agreement with Booz Allen Hamilton to provide, among other things, 

program management; secure Internet hosting; and operations, 

maintenance, and support services for the Defense Manpower Data 

Center’s enterprise training program. According to NTIS, as of May 2012, 

it had 28 primary government clients for this service offering, including the 

Departments of Justice, the Interior, and the Treasury, and the U.S. 

Patent and Trademark Office.

 

29The other six primary clients are the Defense Manpower Data Center, Department of 

Commerce, Wage Determination Online, U.S. Commission on Civil Rights, Joint Task 

Force National Capital Region, and National Telecommunications and Information 

Administration. 

Brokerage Services 

E-Training and Knowledge 

Management Services 
 

Page 17 GAO-13-99 Commerce's National Technical Information Service

As another service, NTIS digitizes various document types, such as 

microfilm or microfiche and paper forms, to assist agencies in complying 

with section 508 standards.30 Further, it offers storage and distribution for 

the documents that it digitizes. For example, NTIS has an agreement with 

SSA to provide alternative modes of receiving SSA notices and other 

communications. NTIS provides this service with the assistance of its joint 

venture partners, Vastec, Inc. and Braille Works, Inc. NTIS offers this 

information in data compact disc, large print, and audio compact disc. As 

of May 2012, the agency had six primary clients for its digitization and 

scanning services, including SSA and the Department of Justice.

As a fee-for-service entity, NTIS’s revenues are generated exclusively 

from its products and services, and all its revenues, expenses, and capital 

expenditures are deposited into and paid out of its revolving fund. Overall, 

NTIS had net earned revenues for 8 of the last 11 fiscal years. For

example, for fiscal year 2011, the agency reported that net earned 

revenues from all its functions (products and services) totaled about $1.5 

million. According to NTIS’s Financial Report for fiscal year 2011, the 

revolving fund ending unobligated balance31

However, over most of the last decade, the agency has incurred net costs 

for its products. Specifically, NTIS product expenditures exceeded 

revenues for 10 out of the past 11 fiscal years. The agency lost, on 

average, about $1.3 million over the last 11 years on its products.

was approximately $7.4 

million.

In contrast, NTIS’s overall financial performance has been supported by 

revenues from its service offerings. The agency’s service revenues 

increased, on average, about $1.8 million over the last 11 years. In 

particular, for fiscal year 2011, revenues were about $53.5 million, costs 

incurred were about $52 million, and the overall net earned revenue was 

approximately $1.5 million from its service offerings. NTIS officials 

attribute most of the net earned revenue to the agency’s agreements with 

 

30Section 508 of the Rehabilitation Act of l973 requires federal agencies to ensure that 

their development, procurement, maintenance, or use of electronic and IT equipment 

takes into account the need for people with disabilities to have access to and use of 

information that is comparable to that of individuals without disabilities. 

31The unobligated balance represents the funds in NTIS’s revolving fund that may be 

carried over into the next fiscal year.

Digitization and Scanning 

Services 

NTIS’s Reported Revenues 

and Costs over the Past 

Decade
 

Page 18 GAO-13-99 Commerce's National Technical Information Service

the Departments of Agriculture and Education and the Social Security 

Administration for various service offerings. Table 3 identifies the net 

earned revenue or net cost for NTIS products and services over the last 

11 fiscal years, as reported by NTIS. In those years in which NTIS had 

net costs, the agency was sustained by cumulative net earned revenues 

from previous years’ operations.

Table 3: Reported Net Earned Revenue (or Net Cost) for NTIS Products and 

Services, Fiscal Years 2001-2011 

Fiscal year Products Services

Net earned 

revenues (net cost) 

2001 $398,233 $1,772,011 $2,170,244

2002 (1,197,257) 2,276,535 1,079,278

2003 (1,409,026) 1,685,945 276,919

2004 (1,315,773) 1,824,157 508,384

2005 (1,973,646) 2,465,149 491,503

2006 (1,889,437) 2,179,183 289,746

2007 (1,683,794) 834,741 (849,053)

2008 (2,072,519) 768,982 (1,303,537)

2009 (2,217,263) 1,099,377 (1,117,886)

2010 (730,347) 3,041,803 2,311,456 

2011 (690,736) 2,182,535 1,491,799 

Total $(14,781,565) $20,130,418 $5,348,853 

Source: GAO analysis of NTIS-reported data.

Figure 2 illustrates the net earned revenues and net costs associated with 

NTIS’s products and services from fiscal year 2001 through fiscal year 

2011.
 

Page 19 GAO-13-99 Commerce's National Technical Information Service

Figure 2: Fiscal Years 2001-2011 Net Earned Revenues and Net Costs for NTIS 

Products and Services

The decline in revenue for its products continues to call into question 

whether NTIS’s basic statutory function of acting as a self-financing 

repository and disseminator of scientific and technical information is still 

viable. This is further highlighted by the fact that the services which are 

financially sustaining the agency are less directly related to this function.

Recognizing its financial stance, NTIS has conducted analyses, and 

identified in its 2011-2016 Strategic Plan actions to help address net 

costs from its products, including its technical reports. The plan 

emphasizes that the agency’s collection, culture, and information 

technology infrastructure are its main strengths; that continued use of less 

robust business systems and an aging work force are its primary 

weaknesses; that growth opportunities still exist in the various sectors 

served by NTIS, whether through products or services; and that NTIS is 

threatened by and will have to overcome a shrinking customer base for its 

products. The plan identifies three strategic initiatives to guide NTIS 

during this period:
 

Page 20 GAO-13-99 Commerce's National Technical Information Service

1. Increase revenue by enhancing the number of acquisitions, creating 

new products, reaching more customers, and adding value to what 

NTIS collects, and to reduce costs by reviewing and improving key 

business processes.

2. Improve NTIS’s utilization by other agencies by increasing the breadth 

and depth of its own collection and enhancing the suite of information 

management services that it can provide.

3. Achieve workforce excellence by focusing on identifying and acquiring 

the critical workforce skills required to accomplish the agency’s 

mission in a rapidly changing world.

Beyond these initiatives identified in the strategic plan, the Director of 

NTIS also provided information on several other initiatives under way to 

address the budget shortfalls from products. These initiatives include the 

following product and organizational improvements:

• Enhancing the accessibility of federal science content by shifting from 

a pricing model for stand-alone products (e.g., paper/print, microfiche, 

and compact disk media) to one that is subscription-based.

• Repositioning NTIS to support open government initiatives in 

science—meeting with agencies such as NIST, the National Oceanic 

and Atmospheric Administration, and the Government Printing Office 

to address how NTIS can reposition its programs to support current 

science information needs.

• Building collections of reports based on themes and categories that 

will be supported through subscriptions.

• Adjusting the NTIS business model to support the increased demand 

for subscriptions to customers.

• Reducing staff—for example, the agency has received authorization 

to provide early retirement for eligible employees and has stopped 

hiring additional staff. The agency anticipates employee attrition to 

further reduce current staffing levels.

Notwithstanding these efforts, NTIS could likely continue to face 

challenges in recouping the costs of its products given the increasing 

availability of technical information from other sources.
 

Page 21 GAO-13-99 Commerce's National Technical Information Service

From fiscal year 1990 through 2011,32

• 79,943 reports published in 1989 and prior years,

most of the additions to NTIS’s 

repository were older reports published in the year 2000 or before; 

however, the greater demand was for more recently published reports. In 

this regard, the agency added 841,502 reports to its repository from 1990 

through 2011. Of the reports added, approximately 62 percent, or 

524,256, had publication dates of 2000 or earlier, while approximately 38 

percent, or 317,246, of the reports were published from 2001 to 2011. 

Specifically, reports that were added to the repository during this period 

were as follows:

• 444,313 reports published from 1990 through 2000,

• 129,591 reports published from 2001 through 2004,

• 126,225 reports published from 2005 through 2008, and

• 61,430 reports published from 2009 through 2011.

Figure 3 shows the distribution of reports that were added by each 

publication date age group from 1990 through 2011.

 

32We did not include reports added for fiscal year 2012 because our study only focused on 

those fiscal years that had been completed at the time that our study was initiated. 

Additions to NTIS’s 

Repository Have 

Mostly Included Older 

Reports, but Demand 

for More Recent 

Reports Is Greater
 

Page 22 GAO-13-99 Commerce's National Technical Information Service

Figure 3: Year of Publication for Documents Added to NTIS Repository, 1990 

through 2011

With regard to demand for the reports, we estimate that, during fiscal 

years 2000 through 2011, NTIS distributed (sold) one or more copies of 

about 419,657, or almost 50 percent, of the 841,502 reports added to its 

repository during fiscal years 1990 through 2011.33 Of these 419,657 

reports, approximately 78 percent were distributed through a subscription 

only.34

 

33The 95 percent confidence interval for the estimated percent of reports distributed one 

or more times is (44.9, 54.8) percentage points. For purposes of this report, demand is 

defined as the distribution (sale) of a report by direct sale or subscription. 

NTIS officials attributed this to the fact that subscriptions are a 

cost-effective way for libraries to meet their collection development 

requirements within a specific or broad area of interest. The officials 

noted that direct sales are generally to customers interested in specific 

topics.

34The 95 percent confidence interval for the estimated percent of reports distributed 

through a subscription only is (69.7, 84.4) percentage points. 
 

Page 23 GAO-13-99 Commerce's National Technical Information Service

However, as shown in figure 4, the agency was more likely to distribute a 

higher percentage of more recently published reports than older ones. For 

example, we estimate that between 96 and 100 percent of the reports 

published from 2001 through 2011 had been distributed, while only about 

21 percent of reports published in 1989 or earlier were distributed during 

this time period. That is, the demand for older holdings in the NTIS 

repository is lower than for newer publications.

Figure 4: Estimated Percentage of Reports Distributed and Not Distributed by 

Publication Date during Fiscal Years 2000 through 2011

Note: This figure presents estimates based upon our random sample of 384 reports. The heights of 

the bars show the percentage point estimate for each category on the horizontal axis relative to the 

others. The line on each bar represents the 95 percent confidence interval associated with each 

estimate. Estimates are rounded to the nearest tenth of a percent, and may sum to greater than 100 

percent due to rounding. See app. I for an explanation of confidence intervals.
 

Page 24 GAO-13-99 Commerce's National Technical Information Service

Based on our sample, we estimate that most (about 74 percent) of the 

reports added to NTIS’s repository during fiscal years 1990 through 2011 

were readily available from other public websites, and nearly all of these 

(95 percent) could be obtained for free.35 Specifically, we estimate that 

approximately 621,917, or about 74 percent, of the 841,502 reports 

added to NTIS’s repository from fiscal years 1990 through 2011 are 

readily available from one of the other four publicly available sources we 

searched (i.e., the issuing organization’s website; the Government 

Printing Office’s Federal Digital System website; the U.S. government’s 

official web portal, USA.gov; or from another website located through a 

Google search).36

http://www.Google.com

The source that most often had the report we were 

searching for was another website located through 

. 

In addition, of the reports added since fiscal year 1990, those with more 

recent publication dates were more likely to be available from other public 

sources than older ones. For example, approximately 87 percent of the 

reports with publication dates from 2009 to 2011 were available 

elsewhere, while 55 percent of those published in 1989 or earlier were.37

 

35The 95 percent confidence interval for the estimated percentage of reports available 

elsewhere which could be obtained for free is (90.7, 97.5) percentage points. 

Figure 5 shows the estimated availability of reports added to NTIS’s 

repository since fiscal year 1990 by date of publication.

36The 95 percent confidence interval for the estimated percentage of reports available 

through one or more of the four publically available sources we searched is (67.9, 80.0)

percentage points. 

37For reports published between 2009 and 2011, we estimate that 86.5 percent are 

available elsewhere. The 95 percent confidence interval for this estimate is (74.2, 93.5) 

percentage points. For reports published in 1989 or earlier, we estimate that 55 percent 

are available elsewhere. The 95 percent confidence interval for this estimate is (34.1, 

74.3) percentage points. 

Most Reports Added 

Since 1990 Are Freely 

Available from Other 

Websites 
 

Page 25 GAO-13-99 Commerce's National Technical Information Service

Figure 5: Estimated Availability of Reports by Year of Publication

As shown in figure 6, of the reports that were found to be readily available 

from one of the other four sources that we searched, about 61 percent of 

those reports had been distributed (sold) by NTIS. Conversely, of the 

reports that were not found to be readily available from one of the other 

four sources, most, or about 82 percent, had not been distributed by 

NTIS.38

 

38The 95 percent confidence interval for the estimated percent available reports that were 

distributed is (54.0, 67.4) percentage points. The 95 percent confidence interval for the 

estimated percent of reports not available and not distributed is (73.2, 87.8) percentage 

points. 
 

Page 26 GAO-13-99 Commerce's National Technical Information Service

Figure 6: Estimated Availability of Reports Distributed and Not Distributed During 

Fiscal Years 2000 through 2011

Not only were most of the reports in our sample available from sources 

other than NTIS, but about 95 percent of the reports available elsewhere 

could be obtained free of charge from one of the four other sources we 

searched.39 The remaining 5 percent were available from the public 

sources for a fee.40

Hazardous Waste Characteristics Scoping Study, November 1996, 

Environmental Protection Agency, 278 pages. (At NTIS, print on demand 

costs $73, electronic $25.)

Moreover, the year of publication did not appear to 

have an effect on whether a report was available free of charge. For 

example, the following reports available for a fee from NTIS were 

available free of charge from the issuing organization’s website:

 

39The 95 percent confidence interval for this estimate is (90.7, 97.5) percentage points. 

40The 95 percent confidence interval for this estimate is (2.5, 9.3) percentage points. 
 

Page 27 GAO-13-99 Commerce's National Technical Information Service

“Homeland Security: Intelligence Indications and Warning,” December 

2002, Naval Postgraduate School, 5 pages. (At NTIS, print on demand 

costs $17, electronic $15.)

Export Controls: System for Controlling Exports of High Performance 

Computing Is Ineffective, 2000, GAO, 60 pages. (At NTIS, print on 

demand costs $48, electronic $15.)

FDA Enforcement Report: July 20, 2011, July 2011, Food and Drug 

Administration, 28 pages. (At NTIS, print on demand costs $33, electronic 

$15.)

Principal Rare Earth Elements Deposits of the United States: A Summary 

of Domestic Deposits and a Global Perspective, 2010, Geological Survey, 

104 pages. (At NTIS, print on demand costs $60, electronic $25.)

Of those reports we found available elsewhere, figure 7 shows the 

estimated percent that were available elsewhere for free by year that the 

document was published.
 

Page 28 GAO-13-99 Commerce's National Technical Information Service

Figure 7: Of Reports Found Elsewhere, Estimated Percent Available for Free and 

Not Free, by Year of Publication

Note: These estimates are for reports readily available from one of the other four publicly available 

sources that we searched.

The Director of NTIS acknowledged two factors that contributed to the 

free and public availability of reports from other sources:

• Federal agencies are providing information, including their federal 

scientific, technical, and engineering information products, on their 

websites in electronic format and on central federal information 

websites such as http://www.data.gov and http://www.science.gov. 

The information is available for immediate download at no cost.

• Federal agencies are participating in programs with Internet search 

engines that permit the public to locate their products for free or for 

less than they are when purchased from NTIS. In addition, 

commercial vendors are also able to obtain these information 

products from agency websites or through Internet search and are 

able to make these products available for free or at a price lower than 

that offered by NTIS.
 

Page 29 GAO-13-99 Commerce's National Technical Information Service

Further, NTIS acknowledged in its strategic plan that because the Internet 

continues to change the way people acquire and use information and 

permits federal agencies to make their information products available for 

free, NTIS is challenged to meet its statutory mandate as a self-financing 

repository and disseminator of technical information. Notwithstanding 

these acknowledgments, NTIS continues to charge for reports that are 

freely available from other public sources.

NTIS serves as a permanent repository and disseminator of technical 

information, and by statute, is required to be financially self-sustaining, to 

the fullest extent feasible, by charging fees for its products and services. 

While the agency had cumulative net earned revenues as of September 

30, 2011, its costs exceeded revenue by an average of about $1.3 million 

over the last 11 years from the sale of technical information. The agency’s 

net revenue now comes primarily from services that are less directly 

related to its basic statutory function. The decline in sales of technical 

information is due in part to the increasing availability of this information 

from other sources, including websites and Internet search tools, and 

often at no charge. Charging for information that is freely available 

elsewhere is a disservice to the public and may also be wasteful insofar 

as some of NTIS’s customers are other federal agencies. Taken together, 

these considerations suggest that the fee-based model under which NTIS 

currently operates for disseminating technical information may no longer 

be viable or appropriate.

In light of the agency’s declining revenue associated with its basic 

statutory function and the charging for information that is often freely 

available elsewhere, Congress should consider examining the 

appropriateness and viability of the fee-based model under which NTIS 

currently operates for disseminating technical information to determine 

whether the use of this model should be continued.

The Acting Secretary of Commerce provided written comments on a draft 

of this report, which are reprinted in appendix II. In its comments, 

Commerce expressed appreciation for our study and for our focus on the 

initiatives that NTIS has undertaken. However, the department said NTIS 

did not believe our conclusions (that the fee-based model under which it 

operates for disseminating technical information may no longer be viable 

or appropriate) fully reflect the additional value that NTIS provides with 

the work that it performs. Commerce stated that, through its federal 

Conclusions 

Matter for 

Congressional 

Consideration 

Agency Comments 

and Our Evaluation
 

Page 30 GAO-13-99 Commerce's National Technical Information Service

clearinghouse and repository, the agency provides federally funded 

reports that are not otherwise readily available, such as most of those 

issued prior to l989. Additionally, Commerce stated that NTIS recognizes 

that it cannot remain financially solvent solely through sales and 

subscriptions of technical reports with expectations that these products 

will be widely available for free. It added that NTIS is moving in the 

direction of “open and improved access” to information, but recognizes 

that it needs to maintain a sustainable financial model and continue 

providing enhanced value to the information generated by other federal 

agencies. In this regard, Commerce described features that it believes 

have added value to the technical reports that NTIS maintains, how these 

features improve access to the documents and related data, and specific 

information services that the agency provides to science information 

professionals. 

Our report highlighted various initiatives that NTIS has undertaken to 

provide older reports that might not otherwise be readily available, and to 

increase the value of its technical reports, information management 

services, and technology transfer capabilities. However, as discussed in 

the report, we found that the demand for older holdings in the agency’s 

repository is lower than for new publications. For example, we estimate 

that between 96 and 100 percent of the reports published from 2001 

through 2011 had been distributed, while only about 21 percent of reports 

published in l989 or earlier were distributed during this time period. 

Further, as the agency acknowledged, its financial health is based on 

both its information product and service missions. Also, as we state in 

our report, the agency’s net revenue now comes primarily from services 

that are less directly related to its basic statutory function, while sales of 

its technical information products have resulted in net losses. This decline 

in sales of NTIS’s technical reports is due in part to the increasing 

availability of this information from other sources, including websites and 

Internet search tools, and often at no charge. With these factors in mind, 

we stand by our conclusion that the fee-based model under which NTIS 

currently operates for disseminating technical information may no longer 

be viable or appropriate.

NTIS also provided technical comments on the report via e-mail, which 

we have incorporated as appropriate.

We are sending copies of this report to interested congressional 

committees. We are also sending copies to the Secretary of Commerce; 

the Director, NTIS; the Director, NIST; and other interested parties. 
 

Page 31 GAO-13-99 Commerce's National Technical Information Service

Copies of this report will also be available at no charge on GAO’s website 

at http://www.gao.gov.

Should you or your staffs have any questions on information discussed in 

this report, please contact me at (202) 512-6304 or melvinv@gao.gov. 

Contact points for our Offices of Congressional Relations and Public 

Affairs may be found on the last page of this report. GAO staff who made 

major contributions to this report are listed in appendix III.

Valerie C. Melvin

Director

Information Management

 and Technology Resources Issues
Appendix I: Objectives, Scope, and 

Methodology

Page 32 GAO-13-99 Commerce's National Technical Information Service

Our objectives were to determine (1) how the National Technical 

Information Service (NTIS) is currently organized and operates, including 

its various functions, current staffing level, reported cost of operations, 

and revenue sources; (2) the age of and demand trends for reports added 

to NTIS’s repository; and (3) the extent to which these reports are readily 

available from other public sources.

To determine how NTIS is organized and operates, we reviewed the 

agency’s strategic plan, and documentation on its organizational and 

reporting structure, office staffing level and assigned responsibilities, and 

types of products and services offered. We also reviewed cost data 

contained in the agency’s annual financial reports.1 In addition, we 

reviewed relevant laws and regulations on NTIS’s authority and 

responsibilities2 and our previous reports that discussed its mission and 

operations.3

For our second and third objectives, we obtained from NTIS the full list of 

document accession numbers for reports added to its repository since 

fiscal year 1990. We chose to include reports added since fiscal year 

1990 to determine the current availability of older documents in light of 

NTIS’s role as a permanent repository, and to focus on the reports added 

since our previous review in 2001.

We supplemented our analyses with interviews of the 

Director of NTIS and other relevant agency officials; we also interviewed 

officials of the Department of Commerce and its National Institute of 

Standards and Technology (NIST), which have specific reporting 

relationships with NTIS.

4

 

1

The cost data reflected in our report were taken from documentation provided by NTIS. 

However, we did not test the reliability of NTIS’s cost accounting system, from which the 

agency derives its cost data. 

We then stratified the list of document 

accession numbers into four strata by the year in which a document was 

added to the repository. We subsequently selected a stratified random 

sample with 96 document accession numbers per strata, for a total 

2

For example 15 U.S.C. 1151-57, 15 U.S.C. 3704b-1 & 2, and 15 C.F.R. Part 1180. 

3

GAO, Information Policy: NTIS’ Financial Position Provides an Opportunity to Reassess 

Its Mission, GAO/GGD-00-147 (Washington, D.C.: June 30, 2000) and Information

Management: Dissemination of Technical Reports, GAO-01-490 (Washington, D.C.: May 

18, 2001).

4

We did not include reports added for fiscal year 2012 because our study only focused on 

those fiscal years that had been completed at the time that our study was initiated. 

Appendix I: Objectives, Scope, and 

Methodology 
Appendix I: Objectives, Scope, and 

Methodology

Page 33 GAO-13-99 Commerce's National Technical Information Service

sample size of 384 reports. All of the estimates made with this sample are 

weighted to reflect the stratified design. NTIS provided us with the full 

bibliographic data for each document in our sample.

To determine the age of reports added to NTIS’s repository since fiscal 

year 1990, we used the year of publication for the reports in our sample to 

estimate the age range (prior to 1990; 1990-2000; 2001-2004; 2005-

2008; and 2009-2011) for all documents added from fiscal year 1990 

through fiscal year 2011. To determine the demand trends for reports 

added to NTIS’s repository during fiscal years 1990 through 2011, we 

requested the sales data from fiscal year 2000 through fiscal year 2011 

for the 384 reports in our stratified sample. NTIS provided the distribution 

data for both direct sales and subscriptions for all of these documents. 

We then used our sample and the sales data to estimate the extent and 

prevalence of the sales among all reports added to the NTIS repository. 

Specifically, we used these data to estimate the (1) total number of 

reports distributed by direct sale and subscription, (2) total number of 

reports distributed one or more times, (3) percentage of reports 

distributed relative to the year the reports were published, and (4) 

percentage of reports distributed relative to their availability elsewhere. 

The observations were statistically weighted in the estimation process to 

reflect the stratified sample design that we used.

To determine the extent to which reports in the repository are readily 

available from other public sources, we first developed a methodology for 

conducting systematic Internet searches to determine availability 

elsewhere. More specifically, as part of this methodology, we searched 

the Internet to determine if each of the reports included in our sample of 

384 reports could be found elsewhere and at no cost. Using a tiered 

approach, we searched the following four sources in the order shown: (1) 

the issuing organization’s website; (2) the U.S. Government Printing 

Office’s Federal Digital System website—http://www.gpo.gov/fdsys; (3) a 

web search conducted using the federal government Internet portal 

USA.gov—http://www.USA.gov; and (4) a web search conducted using 

the commercial search engine http://www.Google.com.5

 

5

These sources and the order in which they were searched were based on an effort to 

duplicate the methodology used in our previous 2001 study of NTIS. 

Specifically, with 

this methodology, we determined whether each report was first available 

at no cost on the issuing organization’s website and, if so, concluded the 
Appendix I: Objectives, Scope, and 

Methodology

Page 34 GAO-13-99 Commerce's National Technical Information Service

Internet search at this point. However, if the report was not available, then 

the search continued to the second source, and so on, until either the 

report was found to be available at one of the remaining sources, or all 

sources were exhausted.6

All of the results derived from the sample analyses constituted estimates 

that are subject to sampling errors. These sampling errors measure the 

extent to which samples of this size and structure are likely to differ from 

the population they represent. Because we followed a probability 

procedure based on random selections, our sample is only one of a large 

number of samples that we might have drawn. Since each sample could 

have provided different estimates, we express our confidence in the 

precision of our particular sample’s results as a 95 percent confidence 

interval. This is the interval that would contain the actual population value 

for 95 percent of the samples we could have drawn.

We then used our results to estimate the 

percentage of the total population of NTIS reports added to the repository 

during fiscal years 1990 through 2011 that was available from other public 

sources.

To determine the reliability of the data provided from NTIS’s repository of 

reports, we performed basic steps to ensure the data provided were valid, 

and reviewed relevant information describing the database supporting the 

repository. We tested for duplicate records, missing values, and out-ofrange values in the data received from NTIS. We did not assess the 

reliability of the system used to maintain these data or the processes 

used in extracting the data for our engagement purposes. To determine 

the reliability of the sales data provided by NTIS, we conducted interviews 

with agency officials to gain an understanding of the process by which 

accounts receivable records are added and managed within NTIS’s 

system of accounts receivable–”CIS.PUB.” Further, we asked cognizant 

agency officials specific questions to understand the controls in place for 

ensuring the integrity and reliability of the data contained in CIS.PUB. In 

addition, we met with NTIS officials to discuss data collected from NTIS 

and obtained their assertions regarding the data it provided. Based on the 

results of these efforts, we found the data sources to be sufficiently 

reliable, given the way they are reported herein.

 

6

In identifying the reports’ availability elsewhere, we did not assess if the source at which 

the report was found was the original issuer of the report or whether the report’s content 

was unaltered from its original issuance. 
Appendix I: Objectives, Scope, and 

Methodology

Page 35 GAO-13-99 Commerce's National Technical Information Service

We conducted this performance audit from February 2012 to November 

2012 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.
Appendix II: Comments from the Department 

of Commerce

Page 36 GAO-13-99 Commerce's National Technical Information Service

Appendix II: Comments from the Department 

of Commerce
Appendix II: Comments from the Department 

of Commerce

Page 37 GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department 

of Commerce

Page 38 GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department 

of Commerce

Page 39 GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department 

of Commerce

Page 40 GAO-13-99 Commerce's National Technical Information Service
Appendix III: GAO Contact and Staff 

Acknowledgments

Page 41 GAO-13-99 Commerce's National Technical Information Service

Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov

In addition to the contact named above, Cynthia Scott (Assistant 

Director), Carl Barden, Virginia Chanley, Elena Epps, Nancy Glover, Alina 

J. Johnson, Lee McCracken, Constantine Papanastasiou, David Plocher, 

Bradley Roach, and Tina Torabi made key contributions to this report.

Appendix III: GAO Contact and Staff 

Acknowledgments 

GAO Contact

Staff 

Acknowledgments 

(310986)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov. 

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations 

Public Affairs

Please Print on Recycled Paper.