Information Management: National Technical Information Service's Dissemination of Technical Reports Needs Congressional Attention
Published: 2012-11-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: NTIS was established by statute in 1950 to collect scientific and technical research reports, maintain a bibliographic record and repository of these reports, and disseminate them to the public. NTIS charges fees for its products and services and is required by law to be financially self-sustaining to the fullest extent possible.
GAO was mandated by Congress to update its 2001 report on aspects of NTIS's operations and the reports in its collection. Specifically, GAO's objectives were to determine (1) how NTIS is currently organized and operates, including its functions, current staffing level, reported cost of operations, and revenue sources; (2) the age of and demand trends for reports added to NTIS's repository; and (3) the extent to which these reports are readily available from other public sources. To do this, GAO reviewed agency documentation, analyzed a sample of reports added to NTIS's collection from fiscal years 1990 through 2011 (reports from the period since GAO's last study and other older reports), and interviewed relevant agency officials.
section_title: What GAO Found
paragraphs: As a component of the Department of Commerce, the National Technical Information Service (NTIS) is organized into five primary offices that offer the public and federal agencies a variety of products and services. As of late October 2012, NTIS was supported by 181 staff, all except 6 of which held full-time positions. NTIS reports its progress toward agency goals to the Deputy Secretary of Commerce, and the Director of NTIS reports to the Director of Commerce's National Institute of Standards and Technology. In addition, NTIS receives oversight of its functions and strategic direction from an advisory board with members appointed by the Secretary of Commerce. NTIS's product and service offerings include, among other things, subscription access to reports contained in its repository in both print and electronic formats, distribution of print-based informational materials to federal agencies' constituents, and digitization and scanning services.
NTIS revenues are generated exclusively from direct sales or subscriptions for its products and services. NTIS reported that net revenues from all its functions (products and services) totaled about $1.5 million in fiscal year 2011. However, over most of the last 11 years, its costs have exceeded revenues by an average of about $1.3 million for its products. While NTIS has not recovered all of its costs for products through subscriptions and other fees, it has been able to remain financially self-sustaining because of revenues generated from its services such as distribution and order fulfillment, web hosting, and e-training. The NTIS strategic plan states that the electronic dissemination of government technical information by other federal agencies has contributed to reduced demand for NTIS's products. As a result, the agency is taking steps to reduce its net costs, such as improving business processes and increasing the breadth and depth of its collection.
NTIS's repository has been growing with mostly older reports, but the demand for more recent reports is greater. Specifically, NTIS added approximately 841,500 reports to its repository during fiscal years 1990 through 2011, and approximately 62 percent of these had publication dates of 2000 or earlier. However, the agency was more likely to distribute (by direct sale or through a subscription) reports published more recently. For example, GAO estimated that 100 percent of the reports published from 2009 through 2011 had been distributed at least once, while only about 21 percent of reports published more than 20 years ago had been.
Of the reports added to NTIS's repository during fiscal years 1990 through 2011, GAO estimates that approximately 74 percent were readily available from other public sources. These reports were often available either from the issuing organization's website, the federal Internet portal (http://www.USA.gov), or from another source located through a web search. Reports published from 1990 to 2011 were more likely to be readily available elsewhere than those published in 1989 or earlier. Further, GAO estimated that 95 percent of the reports available from sources other than NTIS were available free of charge. NTIS's declining revenue associated with its basic statutory function and the charging for information that is often freely available elsewhere suggests that the fee-based model under which NTIS currently operates for disseminating technical information may no longer be viable and appropriate.
section_title: What GAO Recommends
paragraphs: GAO is suggesting that Congress reassess the appropriateness and viability of the fee-based model under which NTIS currently operates for disseminating technical information to determine whether the use of this model should be continued. In comments on a draft of this report, the Department of Commerce stated that NTIS believes GAO's conclusions do not fully reflect the value that the agency provides. However, GAO maintains that its conclusions and suggestion to Congress are warranted.

section_title: Letter
paragraphs: The Department of Commerce’s National Technical Information Service  (NTIS) serves as a permanent repository and disseminator of scientific,  technical, engineering, and business-related information. NTIS acquires  such information—largely in the form of research reports—primarily from  federal agencies and their contractors and grantees, as well as from other  domestic and foreign sources. NTIS estimates that it maintains in its   covering 378 technical  central repository more than 2.5 million recordsand business-related subject areas. In addition, NTIS performs various  fee-based information services for other federal agencies. The legislation  that established NTISprovides to be financially self-sustaining, to the extent feasible.   requires all of the services and functions it  In May 2001, we reported on NTIS’s operations, things, the availability of many of the reports maintained in its repository  from other sources, such as the originating agencies’ websites. House  Report 112-169, accompanying the Consolidated and Further Continuing  Appropriations Act, 2012, directed GAO to update aspects of its prior  study. Our specific objectives were to determine (1) how NTIS is currently  organized and operates, including its various functions, current staffing  level, reported costs of operations, and revenue sources; (2) the age of  and demand trends for reports added to NTIS’s repository; and (3) the  extent to which these reports are readily available from other public  sources.
To determine how NTIS is organized and operates, we reviewed the  agency’s strategic plan and documentation on its organizational and  reporting structure, office staffing level and assigned responsibilities, and  types of products and services offered. We also reviewed cost data  contained in the agency’s financial reports. In addition, we analyzed  relevant laws and regulations describing NTIS’s authority and  responsibilities, and reviewed our previous reports that discussed its  mission and operations. We supplemented our analyses with interviews  of the Director of NTIS and other relevant agency officials; we also  interviewed officials of the Department of Commerce and its National  Institute of Standards and Technology (NIST), which have specific  reporting relationships with NTIS.
GAO, Information Management: Dissemination of Technical Reports, GAO-01-490  (Washington, D.C.: May 18, 2001).
To determine the age of and demand trends for reports added to NTIS’s  repository, we first analyzed the distribution of the publication dates for all  reports added to the repository since 1990. We chose to include reports  added since 1990 to determine the current availability of older documents  in light of NTIS’s role as a permanent repository, and to focus on the  reports added since our previous review in 2001. We subsequently  selected a stratified random sample from this body of reports and then  used cumulative data provided by NTIS on the sales of any of these  reports to analyze the demand for reports in the repository.
To determine the extent to which reports NTIS collects are readily  available from other public sources, we searched the Internet to  determine if each of the reports included in our sample could be found  elsewhere and at no cost. Using a tiered approach, we searched the  following four sources in the order shown: (1) the issuing organization’s  website; (2) the U.S. Government Printing Office’s Federal Digital System  website—http://www.gpo.gov/fdsys; (3) the federal government Internet  portal USA.gov—http://www.USA.gov; and (4) a web search conducted  using the commercial search engine http://www.Google.com. Specifically,  we determined whether each report was first available at no cost on the  issuing organization’s website and, if so, concluded the Internet search at  this point. However, if the report was not available, then the search  continued to the second source, and so on, until either the report was  found to be available at one of the remaining sources, or all sources were  exhausted. We then used our results to estimate the percentage of the  total population of NTIS reports added to the repository during fiscal years  1990 through 2011 that was available from other public sources.
All of the results derived from the sample analyses constituted estimates  that are subject to sampling errors. These sampling errors measure the  extent to which the sample size and structure are likely to differ from the  population they represent. Because we followed a probability procedure  based on random selections, our sample is only one of a large number of  samples that we might have drawn. Since each sample could have  provided different estimates, we express our confidence in the precision  of our particular sample’s results as a 95 percent confidence interval. This  is the interval that would contain the actual population value for 95  percent of the samples we could have drawn.
A more detailed discussion of our objectives, scope, and methodology is  provided in appendix I.
We conducted this performance audit from February 2012 to November  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: NTIS’s basic statutory function is to collect research reports, maintain a  bibliographic record and permanent repository of these reports, and  disseminate them to the public. In addition, NTIS has developed a variety  of information-related services. NTIS charges user fees for the sale of its  products to the public and services to federal agencies.
Under statutory authority enacted in 1950, NTIS collects reports  containing scientific, technical, and engineering information from both  domestic and foreign sources in a repository and makes the information  available to (1) business and industry, (2) state and local governments,  (3) other federal agencies, and (4) the general public to increase U.S.  competitiveness in the global economy. The statute does not define  scientific, technical, and engineering information. However, the Secretary  of Commerce has interpreted this to include “all types of information  which have a more or less direct bearing on business and industry  generally.” including “economic information, market information, and related  information so long as it is reasonably specific and bears some direct  relationship to the organization and operation of industrial or business  enterprise.” NTIS’s enabling legislation authorized it to charge fees for its   The Secretary of Commerce described such information as  products and established a policy to recover all costs, as feasible, through  the fees.
NTIS’s authority was revised in the National Technical Information Act of  1988. cooperative agreements, joint ventures, and other transactions as  necessary in the conduct of the business of NTIS, and declared the NTIS  repository to be a permanent federal function that could not be transferred  to the private sector without congressional approval. The act was  subsequently amended by the American Technology Preeminence Act of  1991, This act gave the agency authority to also enter into contracts,  required all costs associated with acquisition, processing, storage,  bibliographic control, and archiving to be recovered primarily by fees; required agencies to transfer unclassified scientific, technical, and  engineering information which results from federally funded research  and development to NTIS; and provided that NTIS’s use of new methods or media for information  dissemination should include producing and disseminating information  in electronic format.
Further, the Commerce, Justice, and State, the Judiciary, and Related  Agencies Appropriations Act of 1993 established a revolving fund for the  payment of all expenses incurred by NTIS and gave it the authority to use  that fund without further appropriations action by Congress.
Pub. L. No. 100-519 (Oct. 24, 1988); 15 U.S.C. 3704b. million. Accordingly, in August 1999, the Secretary of Commerce  proposed closing NTIS by September 30, 2000, because he believed that  declining sales revenues would not continue to be sufficient to recover all  of the agency’s operating costs. The Secretary attributed this decline  partly to other agencies’ practice of making their research results  available to the public for free through the Web. He also proposed  transferring NTIS’s research report archives to the Library of Congress  and requiring federal agencies to give the public free online access to  new research reports.
GAO-01-490. consideration that Congress look at how this information was defined;  whether there was a need for a central repository of this information; and  if a central repository was maintained, whether all information should be  retained permanently, and what business model should be used to  manage it.
The Secretary of Commerce agreed with our assessment and raised as a  primary question whether there was a need for a central repository in  view of the increasing availability of newer publications from sources  other than NTIS. The Secretary also noted that the need for a central  repository depended on whether the information would be permanently  maintained by agencies and whether the information would be easy to  locate without the kind of bibliographic control that NTIS provides.
Subsequent to the issuance of our reports, in December 2003, Congress  passed the 21st Century Nanotechnology Research and Development  Act, which provided a coordinated federal approach to stimulating  nanotechnology research and development. The act directed the  Secretary of Commerce to establish a clearinghouse for information  related to the commercialization of nanotechnology research using the  resources of NTIS to the extent possible. As of September 2012, NTIS  noted that it held over 700 publications in its nanotechnology collection.
section_title: NTIS Carries out a Variety of Functions Related to Its Mission, but Currently Is Financially Sustained by Its Service Offerings
paragraphs: NTIS currently operates as one of 12 independent bureaus within  Commerce, with the mission to help promote the nation’s economic  growth by providing access to information that stimulates innovation and  discovery. Headquartered in Alexandria, Virginia (with warehouses in  Alexandria, Virginia and in Brandywine, Maryland), the agency is  organized into five primary offices. To carry out its statutory functions of  collecting and maintaining a permanent repository and bibliographic  record of research reports, and to disseminate them, the agency offers a  variety of products, such as fee-based access to the reports in its  repository. In addition, NTIS offers information-related services to federal  agencies, such as distribution and order fulfillment, web hosting, and e- training, that are less directly related to its basic statutory function. While  NTIS’s service offerings have resulted in increased revenues, allowing  the agency to remain financially self-sustaining, it has experienced a net  cost relative to its products, calling into question whether the agency’s  function of acting as a self-financing repository of technical information is  still viable.
subsections: section_title: NTIS Receives Guidance and Oversight from Commerce Department Entities and Carries out Its Functions through Five Primary Offices
paragraphs: NTIS is led by a director, who is aided by two executives—a chief  information officer and a chief financial officer. In addition, three  operational offices have a variety of responsibilities for providing products  and services that include collecting and disseminating technical reports,  offering access to other information sources, and providing information- related services to federal agencies. Figure 1 displays the NTIS  organization.
NTIS operates as a unit within Commerce and receives oversight from  the Deputy Secretary of Commerce, the Director of NIST,advisory board. In this regard, the NTIS Director communicates progress  toward agency goals to the Deputy Secretary of Commerce. For example,  the Director participates in biweekly executive management meetings that  are held with the Deputy Secretary. At these meetings, executive leads  from each of Commerce’s components report on the status of  performance and strategic goals within their offices, among other things.   and an  In addition, under a Commerce departmental order,also reports to the Director of NIST. As outlined in the Commerce  organization chart and stated in the Commerce departmental organization  order, the NTIS Director is to report and be responsible to the Director of  NIST. In turn, the Director of NIST prepares the NTIS Director’s annual  performance appraisal. Beyond this, the Director of NIST stated that the  agency does not provide any other operational or financial oversight  functions for NTIS. For example, NIST does not approve NTIS’s budget  (although it does coordinate NTIS’s budget for final inclusion in the  department’s overall budget, which is approved by Commerce).
Further, NTIS receives guidance on its operations from the NTIS Advisory  Board, which was established by law policies and operations of NTIS, including policies related to fees and  charges for its products and services. The board, comprised of a  chairperson and four members appointed by the Secretary of Commerce,  is required to meet at least every 6 months to discuss NTIS’s activities.  The board’s last meeting took place in late October 2012, and, according  to the notice of open meetings from the Federal Register, the intended  focus was on the agency’s strategic business plan. The board submits an  annual report to the Secretary of Commerce, which includes strategic and  tactical recommendations regarding NTIS’s future operations.   in 1988 to review the general  As of late October 2012, NTIS was supported by 181 staff, all except 6 of  which held full-time positions. These included 103 NTIS employees and  78 contractors. Table 1 shows the number of staff dedicated to the Office  of the Director and each of the agency’s other primary offices as of  October 2012.
15 U.S.C. 3704b(c).
Among these offices, executive functions, such as directing NTIS  activities and developing NTIS budgets and policies for the use of  information technology, are carried out by the Office of the Director, along  with the offices of the Chief Financial Officer and the Chief Information  Officer. The three other operational offices have a variety of  responsibilities for providing NTIS’s products and services. Table 2  summarizes the specific responsibilities and functions of these offices.
As part of its basic statutory function to collect and disseminate technical  reports, NTIS offers a variety of fee-based products. According to the  agency, its customer base for these products include scientists,  engineers, the business community, librarians, information specialists in  government, academia, and the general public. The agency has  organized its products into three lines of business—”Technical Reports,”  “Clearinghouse,” and “Publishing”—as described below.
To carry out its statutory function of collecting and disseminating such  information, NTIS maintains a searchable repository that contains  bibliographic records for the over 2.5 million scientific, technical,  engineering, and business research reports it has acquired from federal  government agencies, state and local governments, international  organizations, and private-sector organizations. NTIS’s bibliographic  database provides for the classification and cataloging of the records in  this database. For example, according to the NTIS Database Search  Guide, NTIS classifies its records into 39 categories. These categories  can be used for searching the contents of NTIS’s database. The database  covers a host of scientific and technical subjects, such as biology,  chemistry, physics, transportation, health care, and the environment. Of  the 2.5 million reports, NTIS noted that approximately 700,000 reports  have been digitized, with the remainder in physical form, such as paper or  microfilm.
Access to the reports is provided both through the direct sale of individual  reports and by subscription. Individual reports can be purchased via  postal mail; phone; e-mail; and NTIS’s online ordering system, accessed  through its website, http://www.ntis.gov. Subscription-based access to the  reports is obtained through, among other things, NTIS’s National  Technical Reports Library, which provides subscribers with the ability to  search the bibliographic records repository and to access the  approximately 700,000 digitized reports in portable document format  (PDF). In addition, NTIS’s Selected Research Service allows a  subscriber to select from more than 378 subcategories and automatically  receive reports tailored to that area of interest. According to NTIS, reports  that have not been digitized can be provided in a digital format when a  customer purchases a copy.
Prices for individual reports and subscriptions vary. For example, an  electronic copy of a report from the Economics and Statistics  Administration, Benefits of Manufacturing Jobs: Executive Summary, can  be purchased for $15; a “customized CD” for this report can be purchased  for $30. Further, an electronic copy of a report from the National  Aeronautics and Space Administration (NASA) Marshall Space Flight  Center, NASA Robotics for Space Exploration, can be purchased for $15;  a “customized CD” for this report can be purchased for $30. With regard  to subscriptions, access to the National Technical Reports Library is sold  as an annual subscription to institutions based on the number of  individuals accessing the library. For example, as of October 2010, an  annual subscription providing access for up to 3,000 individuals costs  $2,100, while an annual subscription providing access for 18,001 to  28,000 individuals costs $11,200. In addition, subscriptions can be  purchased for a specific number of issues of a particular document type.  For example, six issues of the Livestock, Dairy, and Poultry Outlook can  be purchased for $91.
In addition to the technical reports that it collects and disseminates, NTIS  disseminates publications covering a wide array of topics on behalf of  other federal agencies. According to NTIS, these agencies request that  NTIS distribute the publications in print or electronically. The following are  examples of the federal products distributed:
Standard Occupational Classification Manual. A manual containing  information on all occupations in the national economy classified  according to the system used by federal statistical agencies for the  purpose of collecting, calculating, analyzing, or disseminating  occupational data.
Food and Drug Administration Food Code Manual. A code and  reference document that provides technical and legal information  about the regulations of the retail and food service industry.
North American Industry Classification System. A publication that  details a system for the collection, analysis, and dissemination of  industrial statistics used by government policy analysts, academics,  researchers, the business community, and the public.
Export Administration Regulations. A compilation of regulations issued  by Commerce’s Bureau of Industry and Security relating to the control  of certain exports, re-exports, and activities.
National Correct Coding Policy Manual. A manual developed by the  Centers for Medicare and Medicaid Services to control improper  coding leading to inappropriate payment in Medicare Part B claims.  The manual provides guidance for providers on the correct coding of  claims being submitted for reimbursement.
Through a memorandum of understanding or interagency agreement,  NTIS also provides access to information collected from federal agencies,  which it refers to as its “Publishing” line of business. In some instances,  NTIS repackages the information with additional features. According to  NTIS, agencies initiate the request for these services. These offerings  include the following:
Drug Enforcement Administration database. The Drug Enforcement  Administration database identifies persons and organizations  authorized to handle certain controlled drug substances and  chemicals under the Controlled Substance Act. NTIS is the authorized  official distributor of the database. NTIS provides online subscription  access to the database on a weekly, monthly, or quarterly basis. NTIS  also provides this information via a searchable CD-ROM.
Death Master File. This file, maintained by the Social Security  Administration (SSA), contains approximately 85 million records of  deaths that have been reported since 1936. The file is used by  government; credit reporting organizations; and financial,  investigative, medical research, and other industries to verify deaths.  Through an agreement with SSA, NTIS is the only authorized official  distributor of the Death Master File. In this regard, NTIS provides  access to this information, including on DVD, and provides the means  to search and download the Death Master File online.
World News Connection. NTIS offers access to this online news  service, which provides translated and English-language news and  information from non-U.S. media sources. The information is obtained  from full-text, and summaries of newspaper articles, conference  proceedings, television and radio broadcasts, periodicals, and  nonclassified technical reports. The material in World News  Connection is provided to NTIS by the Open Source Center, a U.S.  government agency that provides analysis of foreign open source  intelligence.
section_title: NTIS Provides Fee-Based Services to Federal Agencies
paragraphs: In addition to its product offerings, NTIS offers a variety of fee-based  services to federal agencies that are less directly related to its basic  statutory function of collecting and disseminating scientific and technical  information. These include services that leverage capabilities NTIS has  developed in the course of carrying out its mission. Its five service  offerings are distribution and order fulfillment, web-based services and  federal cloud computing, brokerage services, e-training and knowledge  management services, and digitization and scanning services. To provide  its services, NTIS enters into memorandums of understanding or  interagency agreements with agencies. Further, NTIS offers some of  these services through public-private partnerships with private industry,  which it refers to as “joint venture partners.” The five types of services are  described below.
Through a memorandum of understanding or interagency agreement,  NTIS distributes large amounts of information products for federal  agencies. According to NTIS officials, these services differ from that  provided by its Clearinghouse in that they are used for distributing large  quantities of agencies’ products rather than selling individual copies of  publications. NTIS identified five primary clients for these services: the  Internal Revenue Service, the Department of Agriculture, the Department  of Education, the Pension Benefit Guaranty Corporation, and the  Administrative Office of the U.S. Courts. For example, NTIS currently has  an agreement with the Department of Agriculture to distribute on its behalf  health and nutrition educational materials in the form of brochures,  posters, and similar nutritional products. It also has an agreement with the  Department of Education to perform similar services.
In 1988, Congress required NTIS to implement new methods or media for  disseminating technical information; a 1992 amendment specified that  this should include producing and disseminating products in electronic  formats. According to NTIS, this has been a primary basis for NTIS’s  transformation from a static paper-based distribution operation to a  modern, computer-based model and also a basis for NTIS to provide  information-dissemination services to other agencies. For example,  NTIS’s information systems infrastructure enables it to host federal  agencies’ applications and websites. The agency currently has an  agreement with the Federal Insurance and Mitigation Administration to  provide hosting for two of its websites and associated systems and to  host its web-based tool for reviewing underwriting and claims operations.  NTIS has also expanded its infrastructure to provide cloud computing  services and, according to the agency, is currently offering infrastructure- as-a-service and software-as-a-service. For example, along with joint  venture partner Carney, Inc., NTIS had an agreement with the National  Archives and Records Administration and currently has an agreement  with the Social Security Administration to configure and host the “Jive”  platform. According to NTIS, the agency has eight primary clients for its  web-based services and federal cloud computing offerings, including the  Department of Homeland Security, the Internal Revenue Service, and  other federal agency initiatives.
NTIS provides billing and collection services on a reimbursable basis to  other agencies that, like itself, charge for products and services but which  lack the necessary financial infrastructure to do their own billing and  collecting. The agency refers to these as its “Brokerage Services.” For  example, NTIS had an agreement with the National Agricultural Library to  develop, implement, and operate account maintenance, invoicing, and  collection procedures for the fees charged by the National Agricultural  Library to users of its photocopy and loan services. In addition, the  agency had an agreement with the National Library of Medicine to  perform invoicing, accounting, and collection services for its Interlibrary  Loan services. NTIS officials stated, however, that the agency plans to  stop marketing its brokerage services due to the decrease in demand for  this service.
NTIS’s service offerings have also been expanded to e-training and  knowledge management. Specifically, in conjunction with joint venture  partners, the agency provides collaborative software solutions, learning  management systems and support services, training evaluation software,  and talent management applications. For example, NTIS entered into an  agreement with Booz Allen Hamilton to provide, among other things,  program management; secure Internet hosting; and operations,  maintenance, and support services for the Defense Manpower Data  Center’s enterprise training program. According to NTIS, as of May 2012,  it had 28 primary government clients for this service offering, including the  Departments of Justice, the Interior, and the Treasury, and the U.S.  Patent and Trademark Office.
As another service, NTIS digitizes various document types, such as  microfilm or microfiche and paper forms, to assist agencies in complying  with section 508 standards. Further, it offers storage and distribution for  the documents that it digitizes. For example, NTIS has an agreement with  SSA to provide alternative modes of receiving SSA notices and other  communications. NTIS provides this service with the assistance of its joint  venture partners, Vastec, Inc. and Braille Works, Inc. NTIS offers this  information in data compact disc, large print, and audio compact disc. As  of May 2012, the agency had six primary clients for its digitization and  scanning services, including SSA and the Department of Justice.
section_title: NTIS’s Reported Revenues and Costs over the Past Decade
paragraphs: As a fee-for-service entity, NTIS’s revenues are generated exclusively  from its products and services, and all its revenues, expenses, and capital  expenditures are deposited into and paid out of its revolving fund. Overall,  NTIS had net earned revenues for 8 of the last 11 fiscal years. For  example, for fiscal year 2011, the agency reported that net earned  revenues from all its functions (products and services) totaled about $1.5  million. According to NTIS’s Financial Report for fiscal year 2011, the   was approximately $7.4  revolving fund ending unobligated balancemillion.
However, over most of the last decade, the agency has incurred net costs  for its products. Specifically, NTIS product expenditures exceeded  revenues for 10 out of the past 11 fiscal years. The agency lost, on  average, about $1.3 million over the last 11 years on its products.
In contrast, NTIS’s overall financial performance has been supported by  revenues from its service offerings. The agency’s service revenues  increased, on average, about $1.8 million over the last 11 years. In  particular, for fiscal year 2011, revenues were about $53.5 million, costs  incurred were about $52 million, and the overall net earned revenue was  approximately $1.5 million from its service offerings. NTIS officials  attribute most of the net earned revenue to the agency’s agreements with  the Departments of Agriculture and Education and the Social Security  Administration for various service offerings. Table 3 identifies the net  earned revenue or net cost for NTIS products and services over the last  11 fiscal years, as reported by NTIS. In those years in which NTIS had  net costs, the agency was sustained by cumulative net earned revenues  from previous years’ operations.
Figure 2 illustrates the net earned revenues and net costs associated with  NTIS’s products and services from fiscal year 2001 through fiscal year  2011.
The decline in revenue for its products continues to call into question  whether NTIS’s basic statutory function of acting as a self-financing  repository and disseminator of scientific and technical information is still  viable. This is further highlighted by the fact that the services which are  financially sustaining the agency are less directly related to this function.
Recognizing its financial stance, NTIS has conducted analyses, and  identified in its 2011-2016 Strategic Plan actions to help address net  costs from its products, including its technical reports. The plan  emphasizes that the agency’s collection, culture, and information  technology infrastructure are its main strengths; that continued use of less  robust business systems and an aging work force are its primary  weaknesses; that growth opportunities still exist in the various sectors  served by NTIS, whether through products or services; and that NTIS is  threatened by and will have to overcome a shrinking customer base for its  products. The plan identifies three strategic initiatives to guide NTIS  during this period:  1.  Increase revenue by enhancing the number of acquisitions, creating  new products, reaching more customers, and adding value to what  NTIS collects, and to reduce costs by reviewing and improving key  business processes. 2.  Improve NTIS’s utilization by other agencies by increasing the breadth  and depth of its own collection and enhancing the suite of information  management services that it can provide. 3.  Achieve workforce excellence by focusing on identifying and acquiring  the critical workforce skills required to accomplish the agency’s  mission in a rapidly changing world.
Beyond these initiatives identified in the strategic plan, the Director of  NTIS also provided information on several other initiatives under way to  address the budget shortfalls from products. These initiatives include the  following product and organizational improvements:
Enhancing the accessibility of federal science content by shifting from  a pricing model for stand-alone products (e.g., paper/print, microfiche,  and compact disk media) to one that is subscription-based.
Repositioning NTIS to support open government initiatives in  science—meeting with agencies such as NIST, the National Oceanic  and Atmospheric Administration, and the Government Printing Office  to address how NTIS can reposition its programs to support current  science information needs.
Building collections of reports based on themes and categories that  will be supported through subscriptions.
Adjusting the NTIS business model to support the increased demand  for subscriptions to customers.
Reducing staff—for example, the agency has received authorization  to provide early retirement for eligible employees and has stopped  hiring additional staff. The agency anticipates employee attrition to  further reduce current staffing levels.
Notwithstanding these efforts, NTIS could likely continue to face  challenges in recouping the costs of its products given the increasing  availability of technical information from other sources.
section_title: Additions to NTIS’s Repository Have Mostly Included Older Reports, but Demand for More Recent Reports Is Greater
paragraphs: From fiscal year 1990 through 2011,repository were older reports published in the year 2000 or before;  however, the greater demand was for more recently published reports. In  this regard, the agency added 841,502 reports to its repository from 1990  through 2011. Of the reports added, approximately 62 percent, or  524,256, had publication dates of 2000 or earlier, while approximately 38  percent, or 317,246, of the reports were published from 2001 to 2011.  Specifically, reports that were added to the repository during this period  were as follows:   most of the additions to NTIS’s
79,943 reports published in 1989 and prior years,
444,313 reports published from 1990 through 2000,
129,591 reports published from 2001 through 2004,
126,225 reports published from 2005 through 2008, and
61,430 reports published from 2009 through 2011.
Figure 3 shows the distribution of reports that were added by each  publication date age group from 1990 through 2011.
With regard to demand for the reports, we estimate that, during fiscal  years 2000 through 2011, NTIS distributed (sold) one or more copies of  about 419,657, or almost 50 percent, of the 841,502 reports added to its  repository during fiscal years 1990 through 2011. Of these 419,657  reports, approximately 78 percent were distributed through a subscription  only. NTIS officials attributed this to the fact that subscriptions are a  cost-effective way for libraries to meet their collection development  requirements within a specific or broad area of interest. The officials  noted that direct sales are generally to customers interested in specific  topics.
However, as shown in figure 4, the agency was more likely to distribute a  higher percentage of more recently published reports than older ones. For  example, we estimate that between 96 and 100 percent of the reports  published from 2001 through 2011 had been distributed, while only about  21 percent of reports published in 1989 or earlier were distributed during  this time period. That is, the demand for older holdings in the NTIS  repository is lower than for newer publications.
section_title: Most Reports Added Since 1990 Are Freely Available from Other Websites
paragraphs: Based on our sample, we estimate that most (about 74 percent) of the  reports added to NTIS’s repository during fiscal years 1990 through 2011  were readily available from other public websites, and nearly all of these  (95 percent) could be obtained for free. Specifically, we estimate that  approximately 621,917, or about 74 percent, of the 841,502 reports  added to NTIS’s repository from fiscal years 1990 through 2011 are  readily available from one of the other four publicly available sources we  searched (i.e., the issuing organization’s website; the Government  Printing Office’s Federal Digital System website; the U.S. government’s  official web portal, USA.gov; or from another website located through a  Google search).searching for was another website located through  http://www.Google.com  The source that most often had the report we were  .
In addition, of the reports added since fiscal year 1990, those with more  recent publication dates were more likely to be available from other public  sources than older ones. For example, approximately 87 percent of the  reports with publication dates from 2009 to 2011 were available  elsewhere, while 55 percent of those published in 1989 or earlier were.  Figure 5 shows the estimated availability of reports added to NTIS’s  repository since fiscal year 1990 by date of publication.
As shown in figure 6, of the reports that were found to be readily available  from one of the other four sources that we searched, about 61 percent of  those reports had been distributed (sold) by NTIS. Conversely, of the  reports that were not found to be readily available from one of the other  four sources, most, or about 82 percent, had not been distributed by  NTIS.
Not only were most of the reports in our sample available from sources  other than NTIS, but about 95 percent of the reports available elsewhere  could be obtained free of charge from one of the four other sources we  searched. The remaining 5 percent were available from the public  sources for a fee. Moreover, the year of publication did not appear to  have an effect on whether a report was available free of charge. For  example, the following reports available for a fee from NTIS were  available free of charge from the issuing organization’s website:  Hazardous Waste Characteristics Scoping Study, November 1996,  Environmental Protection Agency, 278 pages. (At NTIS, print on demand  costs $73, electronic $25.)  “Homeland Security: Intelligence Indications and Warning,” December  2002, Naval Postgraduate School, 5 pages. (At NTIS, print on demand  costs $17, electronic $15.)
Export Controls: System for Controlling Exports of High Performance  Computing Is Ineffective, 2000, GAO, 60 pages. (At NTIS, print on  demand costs $48, electronic $15.)
FDA Enforcement Report: July 20, 2011, July 2011, Food and Drug  Administration, 28 pages. (At NTIS, print on demand costs $33, electronic  $15.)
Principal Rare Earth Elements Deposits of the United States: A Summary  of Domestic Deposits and a Global Perspective, 2010, Geological Survey,  104 pages. (At NTIS, print on demand costs $60, electronic $25.)
Of those reports we found available elsewhere, figure 7 shows the  estimated percent that were available elsewhere for free by year that the  document was published.
The Director of NTIS acknowledged two factors that contributed to the  free and public availability of reports from other sources:
Federal agencies are providing information, including their federal  scientific, technical, and engineering information products, on their  websites in electronic format and on central federal information  websites such as http://www.data.gov and http://www.science.gov.  The information is available for immediate download at no cost.
Federal agencies are participating in programs with Internet search  engines that permit the public to locate their products for free or for  less than they are when purchased from NTIS. In addition,  commercial vendors are also able to obtain these information  products from agency websites or through Internet search and are  able to make these products available for free or at a price lower than  that offered by NTIS.
Further, NTIS acknowledged in its strategic plan that because the Internet  continues to change the way people acquire and use information and  permits federal agencies to make their information products available for  free, NTIS is challenged to meet its statutory mandate as a self-financing  repository and disseminator of technical information. Notwithstanding  these acknowledgments, NTIS continues to charge for reports that are  freely available from other public sources.
section_title: Conclusions
paragraphs: NTIS serves as a permanent repository and disseminator of technical  information, and by statute, is required to be financially self-sustaining, to  the fullest extent feasible, by charging fees for its products and services.  While the agency had cumulative net earned revenues as of September  30, 2011, its costs exceeded revenue by an average of about $1.3 million  over the last 11 years from the sale of technical information. The agency’s  net revenue now comes primarily from services that are less directly  related to its basic statutory function. The decline in sales of technical  information is due in part to the increasing availability of this information  from other sources, including websites and Internet search tools, and  often at no charge. Charging for information that is freely available  elsewhere is a disservice to the public and may also be wasteful insofar  as some of NTIS’s customers are other federal agencies. Taken together,  these considerations suggest that the fee-based model under which NTIS  currently operates for disseminating technical information may no longer  be viable or appropriate.
section_title: Matter for Congressional Consideration
paragraphs: In light of the agency’s declining revenue associated with its basic  statutory function and the charging for information that is often freely  available elsewhere, Congress should consider examining the  appropriateness and viability of the fee-based model under which NTIS  currently operates for disseminating technical information to determine  whether the use of this model should be continued.
section_title: Agency Comments and Our Evaluation
paragraphs: The Acting Secretary of Commerce provided written comments on a draft  of this report, which are reprinted in appendix II. In its comments,  Commerce expressed appreciation for our study and for our focus on the  initiatives that NTIS has undertaken. However, the department said NTIS  did not believe our conclusions (that the fee-based model under which it  operates for disseminating technical information may no longer be viable  or appropriate) fully reflect the additional value that NTIS provides with  the work that it performs. Commerce stated that, through its federal  clearinghouse and repository, the agency provides federally funded  reports that are not otherwise readily available, such as most of those  issued prior to l989. Additionally, Commerce stated that NTIS recognizes  that it cannot remain financially solvent solely through sales and  subscriptions of technical reports with expectations that these products  will be widely available for free. It added that NTIS is moving in the  direction of “open and improved access” to information, but recognizes  that it needs to maintain a sustainable financial model and continue  providing enhanced value to the information generated by other federal  agencies. In this regard, Commerce described features that it believes  have added value to the technical reports that NTIS maintains, how these  features improve access to the documents and related data, and specific  information services that the agency provides to science information  professionals.
Our report highlighted various initiatives that NTIS has undertaken to  provide older reports that might not otherwise be readily available, and to  increase the value of its technical reports, information management  services, and technology transfer capabilities. However, as discussed in  the report, we found that the demand for older holdings in the agency’s  repository is lower than for new publications. For example, we estimate  that between 96 and 100 percent of the reports published from 2001  through 2011 had been distributed, while only about 21 percent of reports  published in l989 or earlier were distributed during this time period.  Further, as the agency acknowledged, its financial health is based on  both its information product and service missions. Also, as we state in   our report, the agency’s net revenue now comes primarily from services  that are less directly related to its basic statutory function, while sales of  its technical information products have resulted in net losses. This decline  in sales of NTIS’s technical reports is due in part to the increasing  availability of this information from other sources, including websites and  Internet search tools, and often at no charge. With these factors in mind,  we stand by our conclusion that the fee-based model under which NTIS  currently operates for disseminating technical information may no longer  be viable or appropriate.
NTIS also provided technical comments on the report via e-mail, which  we have incorporated as appropriate.
We are sending copies of this report to interested congressional  committees. We are also sending copies to the Secretary of Commerce;  the Director, NTIS; the Director, NIST; and other interested parties.
Copies of this report will also be available at no charge on GAO’s website  at http://www.gao.gov.
Should you or your staffs have any questions on information discussed in  this report, please contact me at (202) 512-6304 or melvinv@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine (1) how the National Technical  Information Service (NTIS) is currently organized and operates, including  its various functions, current staffing level, reported cost of operations,  and revenue sources; (2) the age of and demand trends for reports added  to NTIS’s repository; and (3) the extent to which these reports are readily  available from other public sources.
To determine how NTIS is organized and operates, we reviewed the  agency’s strategic plan, and documentation on its organizational and  reporting structure, office staffing level and assigned responsibilities, and  types of products and services offered. We also reviewed cost data  contained in the agency’s annual financial reports. In addition, we  reviewed relevant laws and regulations on NTIS’s authority and  responsibilities and our previous reports that discussed its mission and  operations. We supplemented our analyses with interviews of the  Director of NTIS and other relevant agency officials; we also interviewed  officials of the Department of Commerce and its National Institute of  Standards and Technology (NIST), which have specific reporting  relationships with NTIS.
We did not include reports added for fiscal year 2012 because our study only focused on  those fiscal years that had been completed at the time that our study was initiated. sample size of 384 reports. All of the estimates made with this sample are  weighted to reflect the stratified design. NTIS provided us with the full  bibliographic data for each document in our sample.
To determine the age of reports added to NTIS’s repository since fiscal  year 1990, we used the year of publication for the reports in our sample to  estimate the age range (prior to 1990; 1990-2000; 2001-2004; 2005- 2008; and 2009-2011) for all documents added from fiscal year 1990  through fiscal year 2011. To determine the demand trends for reports  added to NTIS’s repository during fiscal years 1990 through 2011, we  requested the sales data from fiscal year 2000 through fiscal year 2011  for the 384 reports in our stratified sample. NTIS provided the distribution  data for both direct sales and subscriptions for all of these documents.  We then used our sample and the sales data to estimate the extent and  prevalence of the sales among all reports added to the NTIS repository.  Specifically, we used these data to estimate the (1) total number of  reports distributed by direct sale and subscription, (2) total number of  reports distributed one or more times, (3) percentage of reports  distributed relative to the year the reports were published, and (4)  percentage of reports distributed relative to their availability elsewhere.  The observations were statistically weighted in the estimation process to  reflect the stratified sample design that we used.
To determine the extent to which reports in the repository are readily  available from other public sources, we first developed a methodology for  conducting systematic Internet searches to determine availability  elsewhere. More specifically, as part of this methodology, we searched  the Internet to determine if each of the reports included in our sample of  384 reports could be found elsewhere and at no cost. Using a tiered  approach, we searched the following four sources in the order shown: (1)  the issuing organization’s website; (2) the U.S. Government Printing  Office’s Federal Digital System website—http://www.gpo.gov/fdsys; (3) a  web search conducted using the federal government Internet portal  USA.gov—http://www.USA.gov; and (4) a web search conducted using   Specifically, with  the commercial search engine http://www.Google.com.this methodology, we determined whether each report was first available  at no cost on the issuing organization’s website and, if so, concluded the  Internet search at this point. However, if the report was not available, then  the search continued to the second source, and so on, until either the  report was found to be available at one of the remaining sources, or all  sources were exhausted. We then used our results to estimate the  percentage of the total population of NTIS reports added to the repository  during fiscal years 1990 through 2011 that was available from other public  sources.
All of the results derived from the sample analyses constituted estimates  that are subject to sampling errors. These sampling errors measure the  extent to which samples of this size and structure are likely to differ from  the population they represent. Because we followed a probability  procedure based on random selections, our sample is only one of a large  number of samples that we might have drawn. Since each sample could  have provided different estimates, we express our confidence in the  precision of our particular sample’s results as a 95 percent confidence  interval. This is the interval that would contain the actual population value  for 95 percent of the samples we could have drawn.
To determine the reliability of the data provided from NTIS’s repository of  reports, we performed basic steps to ensure the data provided were valid,  and reviewed relevant information describing the database supporting the  repository. We tested for duplicate records, missing values, and out-of-  range values in the data received from NTIS. We did not assess the  reliability of the system used to maintain these data or the processes  used in extracting the data for our engagement purposes. To determine  the reliability of the sales data provided by NTIS, we conducted interviews  with agency officials to gain an understanding of the process by which  accounts receivable records are added and managed within NTIS’s  system of accounts receivable–”CIS.PUB.” Further, we asked cognizant  agency officials specific questions to understand the controls in place for  ensuring the integrity and reliability of the data contained in CIS.PUB. In  addition, we met with NTIS officials to discuss data collected from NTIS  and obtained their assertions regarding the data it provided. Based on the  results of these efforts, we found the data sources to be sufficiently  reliable, given the way they are reported herein.
We conducted this performance audit from February 2012 to November  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Cynthia Scott (Assistant  Director), Carl Barden, Virginia Chanley, Elena Epps, Nancy Glover, Alina  J. Johnson, Lee McCracken, Constantine Papanastasiou, David Plocher,  Bradley Roach, and Tina Torabi made key contributions to this report.