Information Management: Dissemination of Technical Reports
Released: 2001-06-19T00:00:00

Letter

Background

Under statutory authority enacted in 1950, NTIS is directed to collect scientific, technical, and engineering information from both domestic and foreign sources and to make the information available to (1) business and industry, (2) state and local governments, (3) other federal agencies, and (4) the general public to increase U.S. competitiveness in the global economy. The statute does not define scientific, technical, and engineering information. However, a 1954 Comptroller General Decision found the Secretary of Commerce’s position reasonable that the statute included “all types of information which have a more or less direct bearing on business and industry generally.” The Secretary specifically described such information as including “economic information, market information, and related information so long as it is reasonably specific and bears some direct relationship to the organization and operation of industrial or business enterprise.” “The head of each Federal executive department or agency shall transfer in a timely manner to the National Technical Information Service unclassified scientific, technical, and engineering information which results from federally funded research and development activities for dissemination to the private sector, academia, State and local governments, and Federal agencies. Only information which would otherwise be available for public dissemination shall be transferred under this subsection. Such information shall include technical reports and information, computer software, application assessments generated pursuant to section 3710(c) of this title, and information regarding training technology and other federally owned or originated technologies.”

This statute is implemented by NTIS regulation 15 C.F.R. Part 1180 (2000). The regulation, in general, provides for transfer to NTIS of (1) at least one copy of every final research product resulting from an agency’s federally funded research and development activities and (2) a summary of an agency’s new and ongoing research that is likely to result in a final product.

NTIS’ basic statutory function is to collect research reports from agencies, maintain a bibliographic record and permanent repository of reports, and disseminate these reports to the public as required. NTIS charges user fees for the sale of its products and services to the public. NTIS’ enabling legislation requires that, to the fullest extent feasible, all of the services and functions it provides be self-sustaining. From fiscal years 1988 through 1994, NTIS showed a net profit in all years except 1 and generated a cumulative net profit of $5.8 million. However, over the next 5-year period of fiscal years 1995 through 1999, NTIS had a cumulative net loss of about $4.2 million.

In August 1999, the Secretary of Commerce proposed closing NTIS by September 30, 2000, because he believed that declining sales revenues soon would not be sufficient to recover all of NTIS’ operating costs. The Secretary partly attributed this decline to other agencies’ practice of making their research results available to the public for free through the Web. He also proposed transferring NTIS’ research report archives to the Library of Congress and requiring federal agencies to give the public free on-line access to new research reports.

In June 2000, we issued a report noting that NTIS may be able to remain self-sustaining for a few years, but that its solvency beyond fiscal year 2003 remained in doubt. Since the issuance of our June report, however, NTIS’ Deputy Director said NTIS has significantly reduced costs and shown a profit of $2.2 million for fiscal year 2000, and the Deputy Director projects a profit of $1.5 million for fiscal year 2001.

NTIS Provides a Variety of Services in Addition to Its Statutory Function of Collecting and Disseminating Research Reports

In addition to NTIS’ basic statutory function of collecting and disseminating technical reports, NTIS also provides various other fee- based services for agencies. According to NTIS, these include brokerage services such as billing and collecting for other agencies that charge for products and services; distribution services such as disseminating other agencies’ products or operating the IRS’ Web site; and Web services such as FedWorld (an on-line information network). According to NTIS financial data, revenue from all functions totaled $35 million in fiscal year 2000.

NTIS’ four major functions and the revenue they generated are described below with some examples: Clearinghouse repository function. This is NTIS’ basic statutory function of collecting and disseminating technical reports, and the function includes seven product lines. According to NTIS, these product lines generated a total of almost $19 million (55 percent of NTIS’ total revenue) in fiscal year 2000: Full-text products. This product line refers to NTIS’ basic document sales program. It includes reports that agencies and foreign sources provide either in hard copy or microfiche (retail); bulk sales to resellers (wholesale); and sales from special collections, such as military publications. It generated a reported revenue of about $6 million in fiscal year 2000.

Announcement products. This product line refers to information about new research reports available from NTIS. Included in this line are NTIS’ bibliographic database, which is disseminated to the public for a fee by vendors who share the resulting revenue with NTIS, and “NTIS Alerts,” a biweekly product announcing the availability of new products in specific subject categories. It generated a reported revenue of about $1.2 million in fiscal year 2000.

Computer products. This product line includes all sales of software, CD- ROMs, and diskettes—containing such items as export regulations, IRS forms, and IRS’ Small Business Resource Guides. It generated a reported $5 million in fiscal year 2000.

Paper subscription products. These are requests by users for a specific information product that is issued or updated on a regular, predictable basis and in hard copy. Examples of these products include Foreign Agricultural Service publications and EPA Quarterly Abstracts. The product line generated a reported $890,000 in fiscal year 2000. Electronic subscription products. These are products that are disseminated electronically on a predictable basis, such as World News Connection (an on-line foreign news service), Federal Research in Progress, and Davis-Bacon Act wage determinations. The product line generated a reported $3 million in fiscal year 2000.

Standing order products. These are similar to subscriptions, except that the product is issued or updated on an irregular basis, such as National Transportation Safety Board Accident Reports. The product line generated a reported revenue of almost $2 million in fiscal year 2000.

Audiovisual products. In addition to paper and electronic formats, NTIS sells a number of audiovisual products, such as videocassettes, that contain training, health and safety, environmental, and other forms of information. These products are collected by NTIS’ audiovisual center, which was transferred to NTIS from the National Archives and Records Administration. The product line generated a reported $590,000 in fiscal year 2000.

Brokerage services. NTIS provides billing and collection services on a reimbursable basis to other agencies that, like NTIS, charge for products and services, but lack the necessary financial infrastructure to do their own billing and collecting. The product line generated about $340,000 in fiscal year 2000, according to NTIS officials.

Distribution services. As an outgrowth of its own activities to disseminate research reports, NTIS will also produce and/or disseminate specific information products to an agency’s constituents under a reimbursable agreement with the agency, such as operation of the IRS’ Web site. These products may be distributed in hard copy, on videocassette, on CD-ROM, or via facsimile or the Web. The agency may supply a mailing list or direct its constituents to call NTIS. NTIS may provide related “help desk” services to assist those constituents in using the material. At times, NTIS will develop multiformat packages for an agency, such as videocassettes and printed instructional material for use in training courses. In addition, NTIS offers agencies comprehensive services to integrate hardware and software in the design of systems that allow agencies to conduct transactions over the Internet. According to NTIS, distribution services generated about $15 million in fiscal year 2000.

Web services. NTIS’ statutory charter calls for it to develop new methods or media for disseminating information; in 1992, the statute was amended specifically to include electronic media. In response, according to NTIS officials, NTIS developed the FedWorld platform as a means of handling high-volume traffic over the Internet in a secure environment—with sufficient capacity to allow NTIS to offer it for a fee to other agencies having a similar need. Web services generated a reported revenue of about $660,000 in fiscal year 2000.

According to NTIS, in addition to the four major functions, it provides other miscellaneous services, such as experimenting with new ways to help agencies reach their constituents. For several years, NTIS has offered CD-ROM mastering and premastering services. Recently, NTIS and the Department of the Navy developed a program that provides sophisticated satellite broadcast services to agencies that need to reach employees or constituents around the country or the globe.

Quantity, Age, and Demand Trends of Reports in NTIS’ Repository

We estimate that NTIS’ repository has about 2.46 million reports— 2.04 million recorded in a database and 420,000 older reports that were manually recorded on index cards. Of these estimated 420,000 older reports, we estimate that 200,000 were maintained and disseminated by other agencies on behalf of NTIS.

Our estimate of 2.46 million reports is somewhat lower than the nearly 3 million reports NTIS stated in its Web site and financial audit report for fiscal year 2000. NTIS overstated its holdings because it (1) double counted reports that were in both its database and on index cards and (2) overstated the number of reports maintained and disseminated by other agencies on behalf of NTIS (i.e., “source fill” reports).

As shown in figure 1, we estimate that about 75 percent of the reports in NTIS’ repository were over 12 years old (i.e., published in 1988 or earlier). The most recent reports—those published in 2000—accounted for less than 1 percent of all reports in NTIS’ repository. Specifically, there were 10,000 reports published in 2000, 97,000 reports published from 1997 to 1999, 516,000 reports published from 1989 to 1996, and an estimated 1,835,000 reports published in 1988 or earlier.

As shown in figure 2, in recent years, NTIS’ repository has been growing at a slower pace—from about 56,000 reports published in 1995 to just over 10,000 published in 2000 (as of mid-November 2000). However, the Assistant to the Director of NTIS said that since about 75 to 80 percent of the reports in NTIS’ collection for a given year are added in the years following the year of publication, the final number of reports for 2000 might increase significantly. Nevertheless, the Assistant to the Director of NTIS acknowledges that the total number of reports added to the repository in recent years has been declining.

Although total demand for NTIS reports has been increasing since fiscal year 1996, the public’s demand for NTIS paper reports has declined, while demand for electronic products increased, as shown in figure 3.

Demand for products was primarily from business and industry sources. As shown in table 1, nearly three-fourths of NTIS’ sales revenues during the last 3 fiscal years from the clearinghouse (except subscription sales) came from business and industry. The next largest customer segments were schools/universities, the federal government, and state and local or foreign governments.

Fewer Than One in Ten Reports in Repository Has Sold Since 1995

Since 1995, we estimate that NTIS has sold one or more copies of about 187,000 (8 percent) of its 2.46 million reports. As shown in figure 4, a higher percentage of reports published from 1989 to 1999 have sold than the newest reports or the oldest reports. Of the 1.8 million reports over 12 years old (pre-1989), we estimate that only about 1 percent has sold since 1995.

NTIS’ Best Sellers Comprise a Variety of Reports on Various Topics

The Assistant to the Director of NTIS described “best sellers” as products that have a broad audience appeal or that sometimes are targeted for a particular audience. These are products for which NTIS does special product development, marketing, and promotion activities to enhance distribution. NTIS officials sometimes work with agencies and its contractors to preserve or resurrect a popular publication that the agency can no longer afford to produce on its own. NTIS also helps an agency customize its product for particular audiences, such as the medical community.

According to NTIS data, revenue from the sale of only the best sellers in fiscal year 2000 totaled over $1 million—about 6 percent of the total $19 million in revenue from the sale of all paper and electronic products in the repository that year. “Any PIA (Primary Insurance Amount) - A Program to Estimate Social Security Monthly Benefit Payments”—Explains how much Social Security benefits will be at retirement. “Federal Tax Products from the IRS”—Includes forms, publications, and bulletins. “Statistical Abstract of the United States”—Reference book containing vital statistics on the United States. “US Industry and Trade Outlook”—Business forecasts for over 200 industries. “Armed Forces Recipe Cards”—1,300 recipes for groups of 100 that can be adjusted up or down.

According to the Assistant to the Director of NTIS, these best sellers are usually sold in one or more of the following formats: Published books. These books are either printed at GPO (although not necessarily distributed by GPO) and purchased by NTIS or developed by NTIS and a joint venture partner. According to NTIS records, these books generated almost $700,000 in sales in fiscal year 2000.

CD-ROM. In most cases, these NTIS products are developed from published books as an alternative format and are available only from NTIS. The one major exception is the IRS CD-ROM. According to NTIS records, these products generated over $200,000 in sales in fiscal year 2000.

Audiovisual. These products cover a variety of topics. For example, NTIS sells a 21-minute tape called “Ergonomic Programs That Work” for $55.00 from the Occupational Safety and Health Administration. According to NTIS records, these generated about $136,000 in sales in fiscal year 2000.

Nineteen Percent of the Reports in NTIS’ Repository Were Available From Other Public Sources

We estimate that about 470,000 of NTIS’ 2.46 million reports (19 percent) could be readily obtained through one or more of the four sources at the time that we searched. The remaining 2 million reports (81 percent) could be readily obtained only from NTIS.

As shown in figure 5, the same NTIS reports often were available at more than one of the four Web sites. For example, of the 10,000 reports published in 2000, we estimate that 56 percent were available from the issuing organization, 40 percent from sources identified through Google.com, 17 percent from sources identified through FirstGov.gov, and 6 percent from GPO. For reports in all age categories, the source with the greatest probability of having the NTIS report was the issuing organization, and the source with the least probability was GPO. For reports published from 1989 to 2000, Google.com was able to locate more than twice the percentage of reports than either FirstGov.gov or GPO’s Web site search engines could locate.

As shown in figure 6, reports published in 2000 were over five times more likely to be available elsewhere than reports published in 1988 or earlier, and the percentage not readily available from these sources increased with the age of the report. For example, we estimate that only 12 percent of the 1.8 million reports published in 1988 or earlier were readily available from the other sources.

Thirty-Seven Percent of NTIS’ Reports Available Elsewhere Were Free From the Internet

“Arizona - 1997 Census of Agriculture State and County Data. Volume 1. Geographic Area Series Part 3,” March 1999, National Agricultural Statistics Service, 328 pages. (At NTIS, a paper copy costs $25, microfiche $23.) “Packaging - and Transportation-Related Occurrence Reports - 1993 Annual Report,” June 1994, Oak Ridge National Laboratory, Department of Energy, 116 pages. (At NTIS, a paper copy costs $36.50, microfiche $21.) “Consumer Use of Information - Implications for Food Policy,” June 1999, Economic Research Service, 32 pages. (At NTIS, a paper copy costs $14, microfiche $12.) “Summary Travel Characteristics, District of Columbia. 1995 American Travel Survey,”

October 1997, Bureau of Transportation Statistics, 26 pages. (At NTIS, a paper copy costs $23, microfiche $12.)

As shown in figure 7, reports published from 1997 to 2000 were considerably more likely to be available free on the Internet than were reports published in other years.

Reports That NTIS Had Sold Since 1995 Were More Readily Available Elsewhere Than Those Not Sold

As shown in figure 8, we found reports that NTIS had sold since 1995 were more likely to be available elsewhere than reports that had not sold. Specifically, we estimate that 44 percent of the reports that sold in all age strata since 1995 were readily available elsewhere, compared with only 17 percent of the reports that did not sell since 1995.

Agencies Did Not Always Send Their Reports to NTIS, as Required by Law

Executive branch agencies are required by 15 U.S.C. 3704b-2 to transfer to NTIS unclassified scientific, technical, and engineering information resulting from federally funded research and development activities. NTIS’ implementing regulation 15 C.F.R. Part 1180 (2000) defines scientific, technical, and engineering information as (1) basic and applied research that results from the efforts of scientists and engineers in any medium (including new theory and information obtained from experimentation, observation, instrumentation, or computation in the form of text, numeric data, or images) and (2) information that bears on business and industry generally, such as economic information and market and related information, if the agency determines such information would be of value to consumers of the information described in the first part of this definition. The regulation requires agencies to send NTIS at least one copy of every final research product and further provides that agencies can enter into agreements with NTIS to transfer reports using alternative means. Such alternative means include providing the location for reports on the agency’s Web site so that NTIS can review the reports and select ones it believes meet the regulation’s requirements.

However, according to NTIS, agencies are not providing all of the required reports. To obtain more of those reports, in 2000, NTIS began harvestingelectronic versions of reports from agency Web sites. Although it would be difficult to identify the universe of federal reports that agencies should have provided to NTIS, harvesting provides some perspective on the extent to which agencies are complying with the regulation’s intent. For example, in 2000, an employee in NTIS’ Information Services Division harvested 1,127 reports from agency Web sites that NTIS believes fit the definition in their regulation and should have been provided. NTIS harvested 10 or more of the 1,127 reports from each of the following agencies’ Web sites: Federal Reserve System (169 reports) International Trade Commission (151) Office of National Drug Control Policy (108) Commerce’s International Trade Administration (95) Department of Transportation’s Federal Highway Administration (61) Commerce’s Bureau of the Census (52) Department of Justice (41) Federal Emergency Management Agency (24) Federal Communications Commission (20) Department of Energy’s Energy Information Administration (18) Agency for International Development (18) The White House (13) Department of the Interior’s Fish and Wildlife Service (11) Department of Agriculture’s Forest Service (10)

Commerce’s National Institute of Standards and Technology (10) Federal Trade Commission (10)

The seven agencies with the most harvested reports had one or more of three general reasons for not sending their reports to NTIS: they were not aware of the law (2 agencies); they would incur additional costs and duplication of effort to format and transfer the information to NTIS that is free on the agency’s Web site (4 agencies); and/or they thought that a few of the reports harvested by NTIS were not scientific, technical, or engineering reports and, thus, did not fall within the law’s scope (3 agencies).

For example, Justice officials did not think that their harvested reports School Shooter: A Threat Assessment Perspective and Reducing Illegal Firearms Trafficking: Promising Practices and Lessons Learned were covered because they were not scientific, technical, or engineering reports. These officials said that additional guidance was needed on the type of reports covered under the law. Similarly, Transportation did not think that its Inspector General’s strategic plan was covered under the law. It is unclear whether reports like these fall under the definition and should be submitted to NTIS. However, our cursory review of the titles of the 1,127 harvested reports found that most, although not all, appeared to contain scientific, technical, engineering, or business-related information.

NTIS has taken steps to improve compliance and thereby reduce the need for harvesting. The Assistant to the Director of NTIS said that its Office of Agency Liaison (1) helps agencies identify the kind of material they should be sending and (2) reminds agencies of their legal requirements, although NTIS’ efforts are limited by available resources. In addition, at one time, NTIS and Energy had an agreement—now expired—that NTIS would harvest those reports it required directly from Energy’s Web site, rather than Energy’s sending NTIS the reports. The Assistant to the Director of NTIS said that this arrangement saved NTIS the costs of scanning paper reports and simplified the agency’s responsibilities under the regulation. He also noted that Energy made harvesting easy by systematically collecting reports from the various Energy offices and components at one location on its Web site. This agreement was not renewed, however, when NTIS and Energy could not agree on the specific terms under which they would continue this arrangement.

NTIS, however, has not pursued other ways of improving compliance. Although NTIS found the agreement with Energy to be beneficial, NTIS has no additional agreements in place. NTIS does provide instructions to agencies on alternative means for transferring reports—such as identifying the location of reports on agency Web sites or submitting by other electronic formats—but only when requested. Further, the Assistant to the Director of NTIS stated that NTIS does not routinely notify agencies that it harvested their reports and/or discuss with them why these reports were harvested.

As a way of keeping the Congress informed on whether agencies are sending their technical reports to NTIS, the Secretary of Commerce is required, under subsection (b) of 15 U.S.C. 3704b-2 to report annually to the Congress “on the status of efforts under this section to ensure access to federal scientific and technical information by the public.” The Assistant to the Director of NTIS said that (1) they have not submitted any of these required reports because they understood the Congress and the executive branch were trying to eliminate surplus reporting requirements and (2) NTIS had erroneously assumed that their annual financial statement audit reports would include the necessary information to meet this requirement.

Improving Compliance Would Not Diminish the Need to Address Fundamental Policy Issues

Attempting to achieve greater compliance with the requirement for agencies to send research reports to NTIS is appropriate under the government’s current strategy for disseminating scientific, technical, and engineering information. However, the rising demand for electronic products, coupled with increasing availability of this information on the Web, raises more fundamental issues about how the information should be collected, stored, and disseminated—and, specifically, about the future of NTIS itself. These issues include whether the government’s current approach—a central repository to provide permanent public access on a self-sustaining basis—is still appropriate in the current environment.

In recognition of this and in response to a request from several Members of Congress, the National Commission on Libraries and Information Science (NCLIS)—an independent agency that provides advice to the President and the Congress on information issues—conducted a study of public information dissemination, which included an assessment of NTIS. NCLIS’ final report was issued in January 2001.

Specifically, the policy issues raised by NCLIS and our studies include the following: How should “scientific, technical, and engineering information” be defined in the current environment? NTIS has defined scientific, technical, and engineering information broadly to include information directly related to business and industry that results from the efforts of scientists and engineers. NTIS, as the agency responsible for administering the law, is given discretion to define such terms. However, this broad definition has led to the agency’s acquiring and disseminating publications that appear to bear little relationship to business and industry, such as some of the “best sellers.” This raises the question of whether policymakers believe that the mission of NTIS should be to disseminate this variety of documents or whether a narrower interpretation of scientific, technical, and engineering information would be more appropriate in the current environment.

NCLIS also addressed this issue. According to NCLIS, NTIS—in an effort to find revenue to support its operations—has expanded the scope of its collection well beyond its primary mission of collecting, organizing, and disseminating scientific and technical information. NCLIS concluded that NTIS’ scope should not be restricted to science and technology narrowly defined. However, they added that the scope should not include general public information that does not have a strong and direct relationship with business, industry, or technology.

Is there a need for a central repository to disseminate scientific, technical, and engineering information? The increasing availability of newer publications from sources other than NTIS, along with the relatively low demand for older reports, may raise the question of whether a central repository for scientific, technical, and engineering information is still needed. Answering this question requires knowing (1) the extent to which agencies could be relied upon to provide permanent public access to publications and (2) in the absence of a central repository, what type of bibliographic information and control over government publications are needed to ensure that reports can be identified and located indefinitely.

In its study, NCLIS stated that the notion of a central source for government technical information to the public should not be abandoned. NCLIS, however, recommended consolidating the missions, functions, and programs of NTIS—along with other government entities responsible for disseminating government information to the public—into new independent agencies in each branch of the government.

What obligations should a central repository have to retain all reports indefinitely? We estimate that only 1 percent of the NTIS reports over 12 years old has sold since 1995. This information suggests that it may be useful to consider whether all scientific, technical, and engineering information should be retained permanently. How long to retain such information, however, would have to be based on demand patterns for older documents and on the costs and benefits of retaining documents of various age strata. This decision should also be based on a policy determination about permanent access and the need to maintain a central repository, as opposed to reliance on other repositories such as libraries.

The NCLIS study also addressed the issue of permanent access to federal information. The study concluded that permanent public availability of and access to government information are critical parts of the overall strategy to meet the public information needs of the public and need to be strengthened. Further, the study concluded that NTIS is the only fail-safe source to make research results available when they are no longer available from the originating agency.

If NTIS continues to exist as a central repository, what should be its business model? Issues concerning the financial viability of NTIS and the availability of research reports on agency Web sites raise questions about the appropriateness of NTIS’ business model. NTIS currently is required to be self-sustaining while maintaining permanent public access to scientific, technical, and engineering information in their repository.

In its report, NCLIS recommended one such alternative business model. Specifically, NCLIS identified a number of NTIS functions that they believe generally benefit the public and recommended that these functions be funded with appropriations. These so-called “public good” functions included (1) collection or acquisition of reports, (2) cataloging and indexing, and (3) offering the NTIS database and full text of reports—to the extent that they are not available on agency Web sites—on NTIS servers for free public access.

Conclusions

In addition to its statutory function of being a permanent repository and disseminator of scientific, technical, and engineering information, NTIS provides a variety of other fee-based services for agencies. In NTIS’ permanent repository of about 2.5 million reports, the number being added has been declining in recent years, and about 75 percent of the reports are now over 12 years old. While demand for electronic NTIS products is rising, research reports and other scientific, technical, and engineering information maintained by NTIS are also increasingly available on agency Web sites and through other public sources—often at no cost.

Further, although NTIS has taken action to encourage agency compliance with the requirement to send it their research reports, agencies are often unaware of the law or confused about its application. NTIS has not been successful in pursuing agreements with agencies to facilitate the transfer of required documents. Finally, NTIS has not submitted its required annual report on the status of its efforts to ensure public access to scientific and technical information. As a result, NTIS has missed an opportunity to inform the Congress of the challenges that it faces.

Broader policy questions remain to be answered about how government information should be disseminated in this Internet era and about NTIS’ role, if any, in that effort.

Matters for Congressional Consideration

In light of developments in report dissemination through the Internet, the Congress may wish to reexamine how scientific, technical, and engineering information is disseminated to the public. In this reexamination, the Congress could specifically look at how this information is defined; whether there is a need for a central repository of this information, such as at NTIS; and, if a central repository is maintained, whether all information should be retained permanently, and what business model should be used to manage it.

Recommendations for Executive Action

In the interim, while the Congress is reexamining the broader policy issues of the dissemination of government research information, we recommend that the Secretary of Commerce instruct the Director, NTIS, to contact the major departments and agencies to outline the law’s requirements for sending reports to NTIS and offer assistance in helping these organizations comply; pursue agreements with agencies to facilitate harvesting reports from agency Web sites as an alternative method of achieving compliance; establish, in consultation with the President’s Council on Integrity and Efficiency, a method for monitoring future compliance, which could involve the participation of agency inspectors general; and submit annual reports to the Congress, as required, on the status of actions taken to ensure access by the public to federal scientific, technical, and engineering information, including any action taken to implement these recommendations.

Agency Comments

We provided a draft of this report to the Secretary of Commerce for review and comment. The Secretary’s comments are reprinted as appendix II.

In a letter dated April 26, 2001, the Secretary of Commerce agreed with our identification of the key policy issues—how to define scientific and technical information, whether there is a need for a central repository, how long reports should be maintained, and what is the proper business model for NTIS. The Secretary stated that, of these issues, the key question is whether there is a need for a central repository in view of the increasing availability of newer publications from sources other than NTIS. The Secretary also noted the need for a central repository depends on whether the information will be permanently maintained by agencies and whether it will be easy to locate absent the kind of bibliographic control NTIS provides.

The Secretary concurred with the thrust of our recommendations and said he would determine how best to implement them. He said he will ask the Acting Under Secretary for Technology to determine if there are any practical ways, consistent with the statutory requirement that NTIS be self-sustaining, to make NTIS’ information collection more widely available to the general public.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we will not distribute it until 30 days from the date of this report. At that time, we will send copies to the Honorable Donald L. Evans, the Secretary of Commerce; the Honorable Mitchell E. Daniels, Jr., Director, OMB; the Honorable Michael F. DiMario, the Public Printer; Mr. Ronald Lawson, Director, NTIS; and other interested congressional committees. Copies will also be available at our Web site at www.gao.gov.

Should you have any questions on matters contained in this report, please contact me at (202) 512-6240 or Alan Stapleton, Assistant Director, at (202) 512-3418. We can also be reached by e-mail at koontzl@gao.gov or stapletona@gao.gov, respectively. Other key contributors to this report included Wendy Ahmed, William Isrin, Michael Jarvis, Frank Maguire, Warren Smith, and Ray Wessmiller.

Appendix I: Scope and Methodology

To identify the functions that NTIS performs, we interviewed National Technical Information Service (NTIS) officials and reviewed various reports and studies, including NTIS’ handbook NTIS Services for Federal Agencies.

To determine the quantity of NTIS’ research holdings, we added the number of newer reports recorded in NTIS’ “STAR” database to the estimated number of older reports recorded on index cards in NTIS’ file drawers. To estimate the number of reports recorded on index cards, we multiplied the number of drawers times the average number of cards in selected drawers to arrive at our initial estimate of 1.5 million reports.

To determine if the data in STAR were valid and reliable, we randomly selected 59 accession numbers from the total recorded in STAR and attempted to locate these reports in NTIS to verify that the title and date of issuance were accurate and randomly selected 59 reports that were located near (above, below, to the right, or to the left) of the 59 reports to verify that they were recorded in STAR or in the index card drawers, depending on when the reports were issued.

NTIS officials said that old reports were recorded on index cards located in hundreds of drawers. To verify that NTIS had the reports recorded on these cards and that the titles and dates were accurate, we randomly selected 75 cards and asked NTIS to attempt to locate these reports. These same 75 cards were part of our random sample of 150 reports 12 years or older.

We found that NTIS’ computer generated data were sufficiently valid, complete, and reliable for purposes of our review, but the data in the index card drawers were not reliable. In particular, we found that the reports were available and the titles were accurately recorded in STAR for all 59 randomly selected accession numbers; the year of the report’s issuance was accurately recorded for 54 of the 59 accession numbers, but the 5 remaining dates did not match exactly; and nearly all of the reports—57 of 59—that we randomly selected from NTIS’ warehouse shelves were accurately recorded either in STAR or in the index card drawers.

However, 40 of the 75 reports randomly selected from the drawers were also found in STAR. This duplication was not expected because NTIS officials said they make their entries into the database directly from the document and no longer create an index card. We did not anticipate finding cards in the drawers that were evidently created after 1964, which is the year NTIS began the database that was later incorporated into STAR. However, an NTIS official in the Information Services Division said that during the period of 1964-80, NTIS created cards as well as an electronic record. They did this because at that time not all employees who needed to work with the database had the easy electronic access to it that is now routinely provided by personal computers. This double-counting of reports in the two systems means that our earlier estimate of 1.5 million reports recorded in the index card drawers was overstated by about 53 percent. Therefore, we reduced our estimate of the number of reports recorded on index cards accordingly, from 1.5 million to 700,000.

Of the remaining 35 unduplicated reports, NTIS was not able to locate 14 of them. Thus, we reduced by 40 percent the estimated number of reports that do not duplicate the STAR file and that NTIS can find and provide to customers—from 700,000 to 420,000 reports. This does not mean, however, that NTIS lost nearly 300,000 reports from its repository, since 10 of the 14 cards in the sample (representing an estimated 200,000 cards in the entire index file) were “source filled”—that is, NTIS retains only a bibliographic record (not the report) and asks the issuing organization to send the report directly to the customer, if it is available. The Information Services Division official said that these reports may no longer be available from the sources that originally issued them.

Adding the 420,000 reports from the index cards to the 2.04 million reports recorded in STAR, we estimate that NTIS has about 2.46 million reports that it can find in the repository or at other agencies. This total is understated to the extent that subscriptions are entered annually in STAR even though a subscription actually represents the dissemination of more than one document over the course of a year.

To determine the age of NTIS’ research holdings, we used the reports’ date of publication in the STAR database as of mid-November 2000 to sort the reports into four age strata: 2000; 1997 to 1999; 1989 to 1996; and 1988 or earlier. For the 1988 or earlier category, we also added the estimated number of reports recorded on index cards.

To determine the demand for NTIS’ research holdings, we analyzed various NTIS statistical and financial data on sales of reports, including the financial statement audit reports for fiscal year 1997 through the draft report for fiscal year 2000. Sales data came primarily from NTIS’ database called “CIS.PUB,” which has records of all sales of reports since May 1995.

To determine if the reports in NTIS’ repository were readily available elsewhere, we first randomly selected 600 report accession numbers from the repository—525 recorded in STAR and an additional 75 accession numbers from the index card drawers. These 600 reports consisted of 150 randomly selected from each of our 4 age strata. The 150 in the oldest category (1988 or earlier) consisted of 75 randomly selected from the approximately 1.4 million reports in that age category in STAR plus 75 randomly selected from those recorded in index card drawers.

Because we selected a sample of reports, all of the results from this sample are estimates that are subject to sampling errors. These sampling errors measure the extent to which samples of this size and structure are likely to differ from the population they represent. Each of the sample estimates is surrounded by a 95-percent confidence interval, indicating that we can be 95 percent confident that the interval contains the actual population value. Unless otherwise noted, the 95-percent confidence intervals for percentage estimates throughout this report do not exceed plus or minus 10 percentage points around the estimate.

Next, we searched the Internet for each of the 600 reports in our sample, using the following steps (in this order): (1) on the issuing organization’s Web site and, if unavailable there, through e-mail to its Office of Public Information or equivalent; (2) on the Government Printing Office’s Web site; (3) on the FirstGov.gov Web site; and (4) on Google.com’s Web site.Because organization officials may be more responsive to an e-mail request from us than they would be from the public, we did not use our computers to send the e-mail requesting reports. Thus, the agencies’ responses to our e-mails should be similar to what any citizen would expect who inquires about how to obtain a report. If an official at any of these sources told us the report was available for a fee, we assumed it was available and did not order the report because of the time and expense involved. If we found in a previous step (e.g., searching an organization’s Web site) that the report was available, we still attempted to find the report at all remaining sources (except the Office of Public Information).

Finally, we used our sample results to estimate what portion of the total population of NTIS reports was available through the four different sources discussed above in each age category. We performed a separate analysis of our sample results to show whether reports that had sold one or more copies since 1995 were more or less readily available through the same four sources, compared with reports that had not sold.

To determine whether federal agencies were sending their research reports to NTIS for sale to the public during calendar year 2000 as required by law, we asked NTIS officials to provide automated data from the STAR database that would allow us to identify all of the reports in their repository that they had harvested from the issuing agency’s Web site during that period. Then, to determine why agencies had not provided the reports, we asked officials at the seven agencies with the most harvested reports why they had not sent their reports to NTIS. These seven agencies represented more than half of all harvested reports. Although we did not review the content of all 1,127 harvested reports to determine if they should have been sent to NTIS, we reviewed the report titles to determine if they appeared to be scientific, technical, engineering, or business-related information. We did not evaluate whether each of the harvested reports was covered under the law.

We conducted our review from September 2000 to April 2001, in accordance with generally accepted government auditing standards.

Appendix II: Comments From the Department of Commerce