Nanotechnology: Improved Performance Information Needed for Environmental, Health, and Safety Research
Released: 2012-06-20T00:00:00

Letter

Background

Nanomaterials come in a variety of forms based both on their chemical composition and physical structure. For example, carbon-based nanomaterials can be produced in a number of physical structures such as sheets (graphene), tubes (carbon nanotubes), and particles (carbon black). Nanomaterials can enter the marketplace as materials themselves, as intermediates that either have nanoscale features or incorporate nanomaterials, and as final nano-enabled products.

The use of nanomaterials in commercial applications raises questions about the potential risks that might arise from exposures to nanomaterials and the differences in exposure during their manufacture, use, and disposal. For example, some, but not all, research studies have shown adverse respiratory or cellular effects in animals exposed to some types of carbon nanotubes. Observed effects include early onset and persistence of pulmonary fibrosis and interference with cell division. The risk posed by a material is a combination of the hazard or negative effect that material may have on an organism and the extent of the organism’s exposure to that material. Therefore, a highly poisonous material—that is, one with high hazard—may nonetheless pose little risk if susceptible groups have little or no contact with the material. For instance, many household chemicals are hazardous to human health but pose little risk when exposure is limited by safe handling. Conversely, a material with relatively mild health effects may pose a large risk if people or the environment are exposed to large amounts or over prolonged periods.

NRC, A Research Strategy for Environmental, Health, and Safety Aspects of Engineered Nanomaterials (2012).

Development has developed a list of 13 representative manufactured nanomaterials now or soon to be in commerce as priority testing targets.

Regulatory agencies, like FDA, CPSC, and EPA, may not have complete information on the uses or risks of some nanomaterials. In 2010, we reported that uncertainties persist about how to evaluate the safety of engineered nanomaterials in food and that nanomaterials may enter the food supply in certain products generally recognized as safe without We also reported in 2010 that EPA has taken a FDA’s knowledge. variety of actions to better understand and regulate the risks of nanomaterials but that the agency faces challenges that might impede its ability to regulate nanomaterials effectively.that the Toxic Substances Control Act gives EPA authority to issue rules requiring companies to submit certain information about chemicals. EPA recently amended a regulation to require companies to report certain information regarding production of chemicals above certain thresholds but those thresholds may not capture nanomaterials if they are produced in amounts below the thresholds. EPA currently has no plans to further reduce the thresholds. In addition, according to officials at CPSC, which is charged with protecting the public from unreasonable risks of injury or death from thousands of types of consumer products, CPSC does not have the statutory authority to require pre-market approval for products, including those incorporating nanomaterials.

For example, we reported The NNI was codified in law by the 21st Century Nanotechnology Research and Development Act in 2003. The act requires the President to implement a national nanotechnology program and charges the NSTC itself—or through a subgroup—with overseeing the planning, management, and coordination of the program. The NSTC carries out these tasks through the NSET Subcommittee, which includes a co-chair from OSTP as well as representatives from the member agencies of the NNI. The NSET oversees working groups, including the Nanotechnology Environmental and Health Implications (NEHI) working group which supports federal activities to protect public health and the environment. Figure 1 provides an overview of the organizational structure of the NNI.

The NNI does not fund research directly; rather, each of its member agencies determines its nanotechnology activities based on its individual mission and priorities. The NNI provides a framework for a comprehensive nanotechnology research and development program by establishing shared goals, priorities, and strategies among member agencies; and providing avenues for member agencies to leverage the resources of all participating agencies. The four goals of the NNI are to (1) advance a world-class nanotechnology research and development program; (2) foster the transfer of new technologies into products for commercial and public benefit; (3) develop and sustain educational resources, a skilled workforce, and the supporting infrastructure and tools to advance nanotechnology; and (4) support responsible development of nanotechnology.

Efforts of the NNI member agencies are reported through triennial strategic plans and annual budget supplements. The act directs the NSTC, itself or through an appropriate subgroup it designates or establishes, to develop and update every 3 years a strategic plan to guide the activities of the program. The NSTC published its most recent strategic plan in February 2011. In addition to the statutorily required strategic plan, in 2008, the NSTC published a Strategy for Nanotechnology-Related Environmental, Health, and Safety Research and updated this document in 2011. The EHS strategies published in 2008 and 2011 expand on the goal of the responsible development of nanotechnology by describing the state of science and research needed to ensure that nanotechnology provides maximum benefits to the environment and human well-being. The act also requires the NSTC to prepare an annual report to be submitted to congressional committees on the national nanotechnology program’s budget and an analysis of the progress made toward achieving the goals and priorities established for the program, among other things. In the NNI Supplements to the President’s Budget, the NSTC reports overall NNI spending and describes research efforts and investments within eight program component areas, as seen in table 1. These program component areas provide an organizational framework for categorizing the activities of the NNI. Data presented in the NNI annual supplements is collected from the NNI member agencies by OMB as part of the annual budget formulation process.

The act requires triennial external reviews of the national nanotechnology program. Specifically, the act requires the NNCO to contract with the NRC to conduct the triennial evaluations of the national nanotechnology program. The NRC draws on expertise from outside government, including from academia, companies, and NGOs. The NRC completed reviews in 2002, 2006, and 2009 based on the work of 15 to 23 panelists chosen by the NRC and identified in the review. In addition, the NRC also published an independent research strategy to address EHS aspects of nanomaterials in 2012.

The act also requires the President to establish or designate a National Nanotechnology Advisory Panel. The advisory panel, by statute, must consist primarily of members from academic institutions and industry, and panel members must be qualified to provide advice and information on nanotechnology research, development, demonstrations, education, technology transfer, commercial application, or societal and ethical concerns. Since 2004, the President has designated the President’s Council of Advisors on Science and Technology (PCAST) to function as the advisory panel. PCAST members are generally senior managers in major corporations and academia selected for diverse expertise in science and technology issues. The advisory panel is required to report not less frequently than once every 2 fiscal years on its assessment of the national nanotechnology program and recommendations for ways to improve the program. The advisory panel has produced three assessments to date, in 2005, 2008, and 2010. The first and second assessments were authored by a subset of PCAST membership, and the third assessment was authored by a working group of three PCAST members and additional external experts, but PCAST as a whole approved the assessments. The first and second assessments created Nanotechnology Technical Advisory Groups, which were comprised of approximately 40 members who provided written responses to questionnaires developed by PCAST. The membership of these groups is not identified in the assessments. The external members of the third assessment’s working group were selected by PCAST members and are identified in PCAST’s assessment.

Funding of EHS Research by NNI Member Agencies Has Increased, but Quality of Funding Data Is Uncertain

NSTC reported more than a doubling of funding for nanotechnology EHS research by NNI member agencies from fiscal years 2006 to 2010. Reported EHS funding also rose as a percentage of total NNI funding during this period, ending up at about 5 percent in 2010. We also identified several reporting problems related to the continued absence of detailed guidance on how agencies should report funding for their nanotechnology research, raising concerns about the quality of EHS funding data reported.

EHS Research Funding by NNI Member Agencies More Than Doubled From 2006 to 2010 and Amounted to About 5 Percent of Total NNI Funding in 2010

From fiscal years 2006 to 2010, NSTC reported more than a doubling of NNI member agencies’ funding for nanotechnology EHS research in the NNI Supplements to the President’s Budget––from approximately $38 million to $90 million.reported to be by NSF, NIH, and EPA, and the largest increases in reported EHS research funding over this period were at NIH and EPA.

EHS Research Funded in 2010 Focused On Metal- and Carbon-Based Nanomaterials

In fiscal year 2010, EHS research at the seven NNI member agencies we reviewed focused on two categories of nanomaterials more than others— metal- and carbon-based nanomaterials, which are used in a variety of applications, including electronics, consumer products such as sunscreens, medical products, and protective coatings (see fig. 4). NNI has not prioritized nanomaterials for EHS research, but in October 2011 it outlined criteria for its member agencies to use in doing so. It is not yet clear what effect these criteria will have on how agencies prioritize the nanomaterials they focus on in their EHS research.

As shown in figure 5, we found that in the 193 fiscal year 2010 research projects we determined were primarily directed at EHS, the nanomaterials that were most frequently the focus of this research were carbon nanotubes (70 projects), nanosilver (60 projects), and nanoscale titanium dioxide (48 projects).

Carbon nanotubes are nanoscale cylinders consisting of seamlessly “rolled” sheets of graphene, a form of carbon. They are extraordinarily strong, flexible, lightweight, heat resistant, and have high electrical conductivity. Carbon nanotubes are currently used in a variety of applications including conductive coatings for touchscreens and solar cells, and sporting goods such as bicycle frames and baseball bats. According to a 2009 NIOSH report, carbon nanotubes have been shown to produce adverse effects in the respiratory systems of rats.

Nanosilver is a metal-based nanomaterial that, according to a 2010 EPA report, is currently used in an increasing number of consumer and medical products because of its remarkably strong antimicrobial properties. For example, EPA reported that nanosilver is being incorporated into clothing, food contact materials such as packaging and cutting boards, household appliances, cosmetics, and children’s toys. It is also used in industrial processes because of its ability to catalyze many reactions. According to EPA’s 2010 report, there is clear evidence that nanosilver is toxic to aquatic and terrestrial organisms and may be detrimental to human health. In this report, EPA noted that several studies have shown that nanosilver can be released into the wastewater stream during washing, such as from socks containing nanosilver. The nanosilver released may disrupt the helpful bacteria used in wastewater treatment processes or be released into the environment.

Nanoscale titanium dioxide is a metal-based nanomaterial used in sunscreens, protective coatings, and other materials to manage heat and light by blocking UV light from the sun’s rays. It is also being added to paints, cements, windows, tiles, and other products for its sterilizing and deodorizing properties and is being used for antifogging coatings and self-cleaning windows. In addition, it is being investigated for use in removing contaminants from drinking water. According to a 2011 NIOSH report, nanoscale titanium dioxide is a potential occupational carcinogen when inhaled. Regarding exposure to this nanomaterial in sunscreens, a 2010 FDA publication found that nanoscale titanium dioxide included in a formulation similar to currently marketed sunscreens is unlikely to significantly penetrate human skin.

Particles,” Toxicological Sciences, vol. 115, no. 1 (2010):156-66.

Effect of Recent NNI Criteria for Agencies’ Prioritization of Nanomaterials Is Not Yet Clear

NNI has not prioritized nanomaterials for EHS research, but the 2011 NNI EHS research strategy outlines criteria to assist its member agencies in doing so. The main criteria NNI proposed are the extent to which (1) study of a particular nanomaterial may provide a major contribution to the research knowledge base, and (2) nanomaterials and nanotechnology- enabled products may pose a safety concern to workers, consumers, and the environment. NNI further identified five criteria for assessing whether a material may pose a safety concern: (1) the nanomaterial’s potential for hazard, (2) likelihood of exposure, (3) high reactivity, (4) biological novelty, and (5) the involvement of the material in an event that produced health or environmental impacts. However, because the criteria were published in October 2011, it is too soon to tell how they will influence NNI member agencies’ decisions about which nanomaterials to prioritize.

Furthermore, it is unclear if the information needed to use the NNI criteria is available. As we reported in May 2010, predicting and assessing the potential hazards, exposures, and resulting risks from nanomaterials is difficult, and current understanding of nanomaterial toxicity and exposure is limited. For example, the findings from completed toxicity studies of a nanomaterial constructed in one manner may not be applicable to understanding the risks posed by the same nanomaterial constructed in a different manner and, therefore, studies of similar nanomaterials may not be comparable. Also, according to the NRC’s 2012 draft strategy for nanotechnology EHS research, there is incomplete information on the effects of the array of nanomaterials used in products and a lack of information on effects of chronic exposures. NRC reported that most toxicity studies test a single material and usually focus on effects of acute exposures.

In addition, reliable, comprehensive information is not readily available on the likelihood of exposure to nanomaterials. Consumers and workers are more likely to be exposed to nanomaterials that are already produced in large quantities or incorporated into a larger number of products, but information on the prevalence or production volumes of nanomaterials or nanotechnology-enabled products is currently limited. As previously noted in this report, FDA, CPSC, and EPA may not receive complete information on what nanomaterials are already in use in the case of certain products. CPSC officials told us that one way the agency collects information on the use of nanomaterials in products to better understand potential exposures is by purchasing information from commercial vendors. However, we found that obtaining such information from commercial vendors such as market research companies can be costly, and it is difficult to assess the reliability of such research because it can involve data and methods, including modeling based on various assumptions, that may be proprietary.

Another source of information on nanomaterials in commerce is the Woodrow Wilson International Center for Scholars’ Project on Emerging Nanotechnologies, which maintains an inventory of nanotechnology- based products. As of the March 2011 update to the list, the inventory contained approximately 1,300 nanotechnology-based products. However, the Wilson Center’s list is not comprehensive, and it consists of consumer products. As we reported in May 2010, occupational exposure is a particular concern because the exposure and risk to workers is potentially greater than the risk to consumers. At present, though, there is little information on the exposure of workers to nanomaterials in the workplace.

NNI Member Agencies Have Collaborated Extensively on Nanotechnology EHS Research

The NNI member agencies have collaborated extensively with each other and nonfederal stakeholders on EHS research and strategies. The NNI member agencies participated in interagency efforts to develop joint strategies related to EHS research. They have also undertaken over 40 collaborative nanotechnology EHS research projects in recent years, signing agreements with both federal and nonfederal stakeholders. Many of the nonfederal stakeholders responding to our questionnaire rated their collaborative activities with NNI agencies as very or generally useful, although they have identified some challenges.

NNI Member Agencies Have Collaborated to Develop Joint Strategies

NNI member agencies have collaborated through the NSTC to develop joint EHS research strategies. The NSTC primarily coordinates nanotechnology EHS research through the NSET Subcommittee and the NEHI, an interagency working group. The NEHI’s purpose includes providing for an exchange of information among agencies and with nonfederal stakeholders; facilitating the identification, prioritization, and implementation of nanotechnology EHS research; and managing the EHS interagency research strategy and facilitating its implementation. The 2011 NNI EHS research strategy was developed by NNI member agencies working through the NEHI working group. The strategy sets a common vision for nanotechnology EHS research, EHS research categories and needs, and key principles intended to assist the NNI member agencies make strategic decisions about research programs that will advance the NNI EHS research agenda while meeting their respective missions. According to several NNI member agency officials, individual agencies’ implementation efforts of the 2011 NNI EHS research strategy are discussed at NEHI meetings. However, since the strategy was finalized in October 2011, it is too soon to tell to what extent it will be used to integrate the NNI member agencies’ implementation plans. The 2011 NNI EHS research strategy updated and replaced the 2008 NNI Strategy for Nanotechnology-Related EHS Research, which had established common priority EHS research needs. The 2008 NNI Strategy for Nanotechnology-Related EHS Research was also developed by the NEHI working group and informed by earlier publications of NNI EHS research needs.

Officials from five of the NNI member agencies told us that they use the NEHI working group as a forum to collaborate on nanotechnology EHS research projects. For example, CPSC officials told us that they identify agencies with expertise on testing nanomaterials found in consumer products through discussions in the NSET Subcommittee and NEHI working group, which has led to collaborative testing of exposure and release of selected nanomaterials. This collaboration has resulted in data that support the respective missions of the collaborative agencies. Officials from the National Toxicology Program at NIH noted that they selected nanomaterials for study based in part on discussions in the NEHI working group and NNI sponsored workshops. Officials from NIST told us that NNI member agencies are also working through the NEHI working group to develop an inventory of collaborative EHS research activities. Collaborative activities related to EHS research have been reported in the 2011 NNI EHS research strategy and annual NNI Supplements to the President’s Budget.

NNI Member Agencies Have Engaged In Numerous Collaborative EHS Research Projects

The NNI member agencies have recently initiated numerous nanotechnology EHS research projects in collaboration with other federal agencies and nonfederal stakeholders. We reviewed formal collaborative nanotechnology EHS research projects initiated from February 2008 to October 2011, while the 2008 NNI Strategy for Nanotechnology-Related EHS Research was in effect. The 2008 NNI Strategy for Nanotechnology- Related EHS Research called for the NEHI and the NNI member agencies to coordinate agency efforts to address priority research needs and, among others, identify opportunities for collaboration and joint development and use of resources where appropriate, facilitate partnerships with industry, and coordinate and support international efforts, support development of consensus-based standards, and facilitate wide dissemination of research results and other nonproprietary EHS information. During this period, NNI member agencies initiated 43 formal collaborative projects related to EHS research––24 interagency collaborations and 19 collaborations that included nonfederal stakeholders. Most of the interagency collaborations were among the NNI member agencies included in our review, but a few were with other federal agencies such as the Department of Defense, the Department of Labor’s Occupational Safety and Health Administration, and the Department of Agriculture. Nonfederal stakeholders included foreign governments, such as the United Kingdom (UK) and China; universities, such as the University of Massachusetts, Lowell and Rice University; and NGOs, such as the International Alliance for NanoEHS Harmonization (IANH) and the International Life Sciences Institute Research Foundation.

NNI member agencies used these collaborative projects to extend their capability to achieve their individual missions. For example, CPSC collaborated with four agencies to conduct research on nanomaterials found in consumer products. NIOSH, an agency within the Centers for Disease Control and Prevention that conducts research and makes recommendations for the prevention of work-related injury and illness, partnered with entities that operate nanomanufacturing facilities to better understand the potential for occupational exposure to nanomaterials. NIST, which aims to advance measurement science, standards, and technology, signed an agreement with the IANH to jointly create protocols for toxicological tests of selected representative nanoparticles, confirming inter-laboratory reproducibility and verifying the predictability of certain procedures.

Collaborative EHS research projects have resulted in transfers of funding as well as sharing of expertise, facilities, and other resources, as shown in the following examples:

An interagency agreement between CPSC and NIOSH to study the pulmonary effects of titanium dioxide nanoparticles released from aerosol spray products involved CPSC providing the product to be tested and transferring funding to NIOSH to construct the testing equipment, with NIOSH providing expertise and staff time to run the tests and produce a report.

An agreement among the NIH’s National Cancer Institute, the National Institute of Environmental Health Sciences (NIEHS), and the National Institute of Biomedical Imaging and Bioengineering to develop, maintain, and operate a web-based nanomaterial registry specified that the National Institute of Biomedical Imaging and Bioengineering will develop the registry to provide consistent and curated information on the biological and environmental interactions of well-characterized nanomaterials, as well as links to associated publications, modeling tools, computational results, and manufacturing guidance from existing databases. The National Cancer Institute and the NIEHS provided funding. NIEHS also committed to provide input to the constitution of an advisory board and attend its meetings, as well as be represented in the nano registry project management team.

Agencies have also issued joint solicitations for grant applications, such as one between EPA and a number of UK agencies to solicit proposals for research on environmental and health implications of nanotechnology. EPA and the UK agencies signed an agreement that notes (1) the funding of grants will be made consistent with the budget priorities of each party and (2) they are to work jointly to define the scientific priorities, issue the request, assist with peer review and selection of grantees, and disseminate the research results.

Details on these and other collaborative EHS research projects are provided in appendix II.

Nonfederal Stakeholders We Surveyed Benefited from NNI Collaboration but Identified Some Challenges

The 138 nonfederal stakeholders who responded to our questionnaire reported that they benefited from collaboration with NNI member agencies but faced some challenges. Most respondents indicated that the 2011 NNI EHS research strategy was very or somewhat effective at addressing the EHS research needs. Respondents reported that they also obtained information on nanotechnology EHS risks from NNI member agencies and nongovernmental sources.

The following three formal mechanisms for collaboration with NNI member agencies were most frequently identified by respondents as generally or very useful to them: (1) joint data gathering and sharing; (2) joint research solicitations or funding of research consortia; and (3) competitive grants. See figure 6 for respondents’ ratings of seven formal collaborative mechanisms we identified for them to rate. Some respondents who provided comments in response to optional open-ended questions also identified public-private partnerships, such as joint academic, government, and industry information exchange and research programs, as collaboration mechanisms the NNI member agencies should consider. A few respondents who commented in response to optional open-ended questions also cited benefits to informal collaboration with NNI agencies, such as discussions during workshops and conferences.

Since both the 2011 NNI EHS research strategy and its predecessor, the 2008 NNI Strategy for Nanotechnology-Related EHS Research, called for partnerships with industry and other nonfederal stakeholders, we included questions about collaboration with NNI member agencies in our questionnaire. When asked about the usefulness of their collaboration with the NNI member agencies, more than half of respondents rated their collaborative EHS research or related activities for each of the NNI member agencies as generally or very useful, as seen in figure 7.

Stakeholders responding to our questionnaire indicated that certain challenges to collaboration with NNI member agencies we identified on EHS research apply to them. As seen in figure 8, more than two-thirds of those who rated the challenge of lack of funding and the challenge of limited awareness of collaboration opportunities indicated that each of these challenges apply to them. This is consistent with what we heard from representatives of a nanotechnology trade association and an NGO, who stated that there is limited funding for corporate EHS research and development efforts because they do not generate revenues. Some respondents who provided comments in response to optional open-ended questions stated that the private sector needs government funding for EHS research. A number of respondents also commented that government funding for EHS research was inadequate or an “after- thought” to other purposes. Some respondents also commented that funding seems to be targeted toward large, multiyear centers rather than smaller targeted projects that can help address near-term EHS needs.

More than half of those responding noted that each of the challenges of regulatory uncertainty, lack of standardization—such as different terminology or protocols—and concerns regarding disclosure of proprietary or confidential business information, apply to them. Some respondents cited regulatory uncertainty—that is, lack of a clear regulatory environment to enable commercialization and protect consumers and the general public—as a challenge to collaboration with their work with NNI member agencies on nanotechnology EHS research. For example, a number of respondents who provided comments in response to optional open-ended questions pointed to regulatory uncertainty as limiting the ability of companies to determine their EHS responsibilities. Some respondents also commented on difficulties in collaborating with NNI member agencies resulting from differences in expertise and regulatory approaches across different government agencies or even within the same agency. With regard to lack of standardization as a challenge to collaboration on nanotechnology EHS research, some respondents provided examples related to the lack of scientific consensus on the definition of nanomaterials, the testing methods for toxicology research, and common terminology across research and regulatory agencies.

When asked to rate the effectiveness of mechanisms we identified that NNI member agencies can use for obtaining input from nonfederal stakeholders on the development of the NNI EHS research strategies, respondents most frequently identified the following three mechanisms as being used somewhat effectively or effectively: (1) workshops, conferences, and other public speaking engagements; (2) advisory councils; and (3) the NNI web portal.

When asked to rate the 2011 NNI EHS research strategy, most of those responding rated this strategy as somewhat or very effective at addressing nanotechnology EHS research needs. As seen in figure 10, while just over half of those responding to these questions indicated the 2008 NNI Strategy for Nanotechnology-Related EHS Research addressed these needs somewhat or very effectively, 79 out of 98 indicated the 2011 NNI EHS research strategy addressed these needs somewhat or very effectively. Some of the respondents who commented in response to optional open-ended questions commended the 2011 NNI EHS research strategy for accurately capturing the input of the workshops’ participants and summarizing the research needs, but others pointed to shortcomings, including a lack of prioritization and insufficient focus on implementation. For example, one respondent commented that the strategy would only be effective if it was implemented, and that it may not be sufficient to rely on the agencies’ voluntary implementation efforts.

Additionally, respondents identified various sources, including the NNI member agencies, from which they obtained information on the potential EHS risks of nanotechnology. When asked to estimate the frequency with which they obtained nanotechnology EHS information from NNI member agencies in recent years, respondents reported obtaining information from the NNI member agencies in varying frequencies. More than half of the respondents to these questions reported occasionally or often obtaining information from EPA and NIOSH, as seen in figure 11.

The 2011 NNI strategic plan calls for providing a “one-stop” Internet- based portal for nanotechnology information, including, among other things, scientific data such as characterization and toxicity measurements. About 80 percent of the 131 stakeholders responding to a question about how useful such a portal would be indicated that an Internet-based portal for nanotechnology EHS information would be generally or very useful to them. The NNI does not currently provide a portal that contains this information, but an NNCO official told us that the NNCO is implementing a web-based map that identifies the instrumentation and facilities for nanotechnology research at major universities and centers in the U.S. and federal labs.

When asked to rate the frequency with which they obtained information on the potential EHS risks of nanotechnology from nongovernmental sources, as seen in figure 12, more than two-thirds of respondents reported that they occasionally or often obtained information from each of the following four sources: (1) peer-reviewed scientific publications; (2) in- house research; (3) online databases; and (4) news outlets.

NNI Strategy Documents Address or Partially Address Desirable Characteristics of National Strategies

NNI strategy documents address two and partially address four of the six desirable characteristics of effective national strategies that we identified in prior work and that offer policymakers and implementing agencies a management tool that can help ensure accountability and more effective results. The three NNI strategy documents––the 2011 NNI strategic plan, the 2011 NNI EHS research strategy, and the NNI Supplement to the President’s 2012 Budget––compose a national strategy for nanotechnology EHS research and create a framework for achieving NNI program goals and priorities.

We rated the NNI strategy documents to determine how well they jointly addressed the six characteristics that we have identified for effective national strategies, as described in table 3.

Ideally, effective national strategies should fully address all of these characteristics. However, we recognize that by their nature, national strategies are intended to provide broad direction and guidance—rather than be prescriptive, detailed mandates—to the relevant implementing agencies. Thus it is unrealistic to expect all national strategies to provide details on each and every characteristic we identified. Moreover, the NNI member agencies have different statutory authority and functions that may significantly affect their research and research priorities. Nonetheless, we believe the more detail a strategy provides, the easier it is for the responsible parties to implement it and achieve its goals.

We reviewed the three NNI strategy documents for elements related to these characteristics, and based on the inclusion of these elements, rated how well the strategic documents address the six characteristics. According to our methodology, the strategy documents “address” a characteristic when they explicitly include all, or nearly all, elements of the characteristic and have sufficient specificity and detail. The documents “partially address” a characteristic when they include some or most of the elements of the characteristic with sufficient specificity and detail. The documents “do not address” a characteristic when they do not include any elements of a characteristic or references are too vague or general to be useful. Additional details about our ratings and the elements that make up these characteristics are available in appendix I.

As described in table 4, we found, when reviewed as a whole, that the strategy documents address or partially address all of the desirable characteristics of a national strategy.

Taken as a whole, the NNI strategy documents address the first two characteristics—purpose, scope, and methodology and problem definition and risk assessment—by including all, or nearly all, of their elements. These characteristics describe the scope of the strategy, describe how and why it was produced, define the problems the strategy intends to address, discuss causes and the operating environment, and provide a risk assessment or broad description of potential risks. Elements of these characteristics that are included in the documents are providing a clear statement of purpose, defining key terms, delineating what major functions or mission areas are covered, discussing problems with the current understanding of EHS implications of nanotechnology, and the quality of currently available data, among others.

Goals, Subordinate Objectives, Activities, and Performance Measures (Desirable Characteristic 3)

The NNI strategy documents partially address the third characteristic regarding goals, subordinate objectives, activities, and performance measures. This characteristic describes the overall desired results, hierarchy of strategic goals and subordinate objectives, priorities, milestones, outcome-related performance measures, and process for monitoring and reporting on progress, among others. The NNI strategy documents include a number of elements such as desired results, strategic goals, and subordinate objectives, but do not include, or do not fully develop, other elements such as priorities, milestones, or outcome- related performance measures. Performance information—such as outcome-related performance measures and milestones comprised of targets and time frames for meeting those measures—allows managers to identify performance problems and develop approaches that improve results. For the purposes of this report, we define performance information to mean data collected to measure progress toward achieving an established goal. A wide range of information can be relevant to program performance. Performance information can focus on various dimensions of performance such as outcomes, outputs, quality, timeliness, customer satisfaction, or efficiency. It can inform key management decisions such as setting program priorities, allocating resources, identifying program problems, and taking corrective action to solve those problems; or it can help determine progress in meeting the goals of programs or operations. The NNI strategy documents contain detailed research needs for nanotechnology EHS and report the annual funding of EHS research and the number of projects supported for selected years, but the documents do not prioritize among these needs or include outcome-related performance measures, targets, or time frames that allow for monitoring and reporting on progress toward meeting the research needs.

Independent reviews of the prior NNI strategy documents also noted an absence of performance information. In 2010, the National Nanotechnology Advisory Panel recommended that the NNCO monitor metrics that assess, among other things, the NNI’s progress on developing methodologies to assess plausible risks of nanotechnology. The advisory panel also recommended that the NEHI working group develop and implement a strategy that links EHS research activities with knowledge gaps and decision-making needs. The 2009 NRC review concluded that the 2008 NNI Strategy for Nanotechnology-Related EHS Research could be an effective tool, but did not include a number of elements, including measures of research progress. We previously reported that, ideally, a national strategy would set clear desired results and priorities, outcome-related performance measures, and specific milestones while giving implementing parties flexibility to pursue and achieve those results within a reasonable time frame.

The NNI strategy documents give agencies wide latitude to develop research programs to meet the goals and research needs of the NNI, but we found that NNI member agencies vary in their identification and reporting of agency-specific performance information for nanotechnology EHS research that could align with the NNI research needs. The 2011 NNI EHS research strategy states that prioritization, timing, and staging of the research needs identified by the strategy are components of an implementation plan and should be developed within agency missions and appropriations. Similarly, the 2011 NNI strategic plan states that the document serves as a guide for individual agency implementation. Six of the seven NNI agencies we reviewed have documented performance measures, targets, or time frames for their EHS research. Three of the NNI member agencies—CPSC, FDA, and NIH—annually report EHS- related performance information through their publicly available annual performance reports. Three other NNI member agencies—EPA, NIOSH, and NIST—identify strategic goals and performance measures and targets in their program-level nanotechnology EHS strategies, but have not collected, or have not publically reported results of these performance measures. NIOSH has previously published research summaries in its nanotechnology research progress reports. NIOSH officials told us that a forthcoming progress report will also document the results of performance measures identified in their nanotechnology EHS strategy. NSF reports planned EHS research activities as part of its annual budget request, but does not identify performance targets or measures for its nanotechnology EHS research beyond funding requested. According to an NNCO official, the NEHI working group is piloting an effort to gather information from the NNI member agencies to assess high-level progress toward meeting NNI research needs, but has not released the results of this work.

Resources, Investments, and Risk Management and Organizational Roles, Responsibilities, and Coordination (Desirable Characteristics 4 and 5)

The NNI strategy documents partially address the fourth and fifth characteristics describing resources, investments, and risk management and organizational roles, responsibilities, and coordination. We found that the documents include some, but not all, of the elements of these characteristics. With respect to the fourth characteristic, we found that the documents include elements relating to agency resources associated with the strategy such as descriptions of current activities of NNI member agencies and current investment levels. However, we found that the documents do not include, or do not fully develop, other elements such as the costs and types of resources—for example human capital or research and development costs—associated with implementing the strategy. An NNCO official told us that the NNI does not analyze the costs of resources to meet the EHS research needs because it does not control the funding for member agencies, and that each agency allocates funding and prioritizes its activities based on its mission area and available budget. According to this official, the 2011 NNI EHS research strategy identifies research goals, but it is up to the agencies to determine how their funding should be spent. Similarly, a NIST official told us that the NNI cannot dictate to agencies how or what research to fund. With respect to the fifth characteristic pertaining to organizational roles, responsibilities, and coordination, we found that the documents include elements such as lead, support, and partner roles for agencies and processes for coordination and collaboration, but do not include, or do not fully develop, other elements such as an accountability and oversight framework. The strategy documents include detailed descriptions of the NNI structure, describe coordination mechanisms such as the NEHI working group, and provide examples of collaborative activities related to EHS research. As we reported above, we found that NNI member agencies have engaged in numerous collaborative projects with federal and nonfederal partners and nonfederal partners generally rated these collaborations as generally or very useful. However, the strategy documents do not provide details on oversight of agency roles or how agencies will be held accountable to the goals and research needs of the NNI strategy documents.

Integration and Implementation (Desirable Characteristic 6)

The NNI strategy documents also partially address the sixth characteristic describing integration and implementation. This characteristic describes how the national strategy documents relate to subordinate levels of government and their plans to implement the strategy. As we have previously reported, to achieve a common outcome, collaborating agencies need to establish strategies that work in concert with those of their partners or are joint in nature. We found that the NNI strategy documents include limited information regarding the elements of this characteristic. For example, the 2011 NNI EHS research strategy describes a framework for implementation, but does not provide detailed implementation guidance. In addition, we found that the NNI strategy documents generally do not include information on the NNI member agencies’ nanotechnology EHS strategies or plans or their integration with the national strategy.

Five of the seven NNI member agencies we reviewed have developed such agency-level nanotechnology EHS research strategies. As noted above, EPA, NIOSH, and NIST have developed strategic plans specifically for EHS research. In addition, FDA has published a strategic plan for its regulatory science research efforts, which include nanotechnology, and developed a research plan specifically targeting Furthermore, NSF officials told us that the agency nanotechnology. describes its strategic direction for nanotechnology, including current and proposed activities and funding by PCA, in its annual budget request.CPSC includes nanotechnology as an external risk factor for its agencywide strategic plan, but the plan does not address EHS research directly. NIH officials told us they do not have agency or institute-specific strategic plans that specifically address nanotechnology EHS research, but nanotechnology EHS research plans developed by its institutes conform to the NNI strategy. Of the five agencies with nanotechnology EHS-related strategies, two—FDA and NIOSH—explicitly align agency research goals in their strategic plans for nanotechnology to NNI EHS research needs. The FDA and NIOSH strategic plans describe the agencies’ EHS research goals, their framework for meeting those goals, and which NNI EHS research needs will be advanced. NIST’s Nano-EHS program plan, EPA’s Nanomaterial Research Strategy, and NSF’s budget request to Congress describe agency research objectives. The documents do not explicitly identify NNI research needs; however, agency officials told us that they reflect the NNI strategic documents. The 2011 NNI strategic plan reports that NNI member agencies have initiated a mapping exercise to evaluate how this strategic plan relates to NNI member agencies’ strategic plans and to the priorities of the administration. In addition, according to NNI member agency officials, individual agencies’ implementation efforts of the 2011 NNI EHS research strategy are discussed at NEHI meetings. However, the results of this mapping exercise and proceedings of the NEHI are not publically available.

Conclusion

The increasing commercialization of nanotechnology-enabled products and information gaps related to nanotechnology EHS impacts underscore the importance of ensuring progress toward meeting EHS strategic goals and research needs. The NNI strategy documents help shape the policies, programs, priorities, resource allocations, and standards related to nanotechnology EHS research and activities. The NNI strategy documents address or partially address the desirable characteristics of effective national strategies, including overarching strategic goals and objectives such as the EHS research needs. However, the documents do not include some elements of the characteristics, which could make it more difficult for federal agencies and other stakeholders to implement the strategy and achieve the identified results. For example, progress toward achieving the strategy documents’ goals and research needs cannot be tracked because the NNI does not report specific performance information, such as performance measures, targets, or time frames. In addition, the NNI strategy documents do not include estimates of the costs and types of resources associated with these goals and research needs.

Developing and coordinating performance information and cost estimates for the NNI’s nanotechnology EHS research may be challenging because of the varied missions and priorities of the NNI member agencies. However, not having performance information that is aligned with the strategic goals and research needs of the NNI makes it difficult for agencies, policymakers, and stakeholders to determine the collective progress of the national nanotechnology program. Furthermore, developing performance information and including it in NNI strategy documents could help to strengthen two other desirable characteristics that the documents partially address. Specifically, this information could also support an accountability and oversight framework—that is currently not well-developed—by linking agency activities to specific measures of performance and creating consistent benchmarks to judge the overall progress of the NNI member agencies. In addition, developing this information could facilitate better integration between the NNI strategy documents and the NNI member agencies’ nanotechnology EHS research strategies and goals. Providing cost estimates related to the NNI EHS research needs would allow the NNI member agencies, policymakers, and stakeholders to assess if investments are commensurate with costs of the identified needs. The NSTC, which is administered by the OSTP, does not direct funding to meet the NNI research needs; however, rough estimates of their costs could serve as guidance to the NNI member agencies as they individually determine their own budgets and priorities. In addition, the NSTC is well-positioned to develop these estimates due to its coordinating role across the member agencies of the NNI and corresponding access to expertise at these agencies. Developing performance information and cost estimates could also support the analysis of the progress made toward achieving the goals and priorities established for the program, as required by the 21st Century Nanotechnology Research and Development Act.

In addition, we are reiterating our 2008 recommendation that the Director of OSTP, in consultation with the Directors of the NNCO and OMB, provide better guidance to agencies on how to report their nanotechnology EHS research. The continued absence of detailed guidance on how to report research under PCA 7 has contributed to the data reporting problems we identified. Therefore, agencies, policymakers, and stakeholders do not have access to accurate, consistent, and complete data on the federal government’s investment in nanotechnology EHS research. Without reliable and up-to-date data, it will be difficult for agencies to accurately assess and report their progress toward their own performance measures, as well as the EHS research goals and needs identified in the 2011 NNI EHS research strategy.

Recommendations for Executive Action

To better offer policymakers and implementing agencies a management tool that can help ensure accountability and more effective results, we are making two recommendations to the Director of OSTP: We recommend that the Director of OSTP coordinate development by the NNI member agencies of performance measures, targets, and time frames for nanotechnology EHS research that align with the research needs of the NNI, consistent with the agencies’ respective statutory authorities, and include this information in publicly available reports.

We also recommend that, to the extent possible, the Director of OSTP coordinate the development by the NNI member agencies of estimates of the costs and types of resources necessary to meet the EHS research needs.

Agency Comments and Our Evaluation

We provided a draft of this report to the Director, OSTP; Secretary, Commerce; Chairman, CPSC; Administrator, EPA; Secretary, Health and Human Services; and Director, NSF. OSTP and the agencies neither agreed nor disagreed with the recommendations. OSTP and Health and Human Services provided technical and clarifying comments, which we incorporated as appropriate. The Department of Commerce, CPSC, EPA, and NSF indicated they had no comments on the report.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Director, OSTP; Secretary, Commerce; Chairman, CPSC; Administrator, EPA; Secretary, Health and Human Services; Director, NSF; and other interested parties. The report also will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our review examines (1) changes in federal funding for nanotechnology environmental, health, and safety (EHS) research from fiscal years 2006 to 2010; (2) the nanomaterials that National Nanotechnology Initiative (NNI) member agencies focused on in their EHS research in fiscal year 2010; (3) the extent to which NNI member agencies collaborate with stakeholders on nanotechnology EHS research and related strategies; and (4) the extent to which NNI strategy documents address desirable characteristics of national strategies.

To conduct this work, we reviewed EHS research efforts funded by seven NNI member agencies, which collectively funded 93 percent of EHS research dollars in fiscal year 2010: the National Science Foundation (NSF), National Institutes of Health (NIH), Environmental Protection Agency (EPA), National Institute for Occupational Safety and Health (NIOSH), Food and Drug Administration (FDA), National Institute of Standards and Technology (NIST), and the Consumer Product Safety Commission (CPSC). The first six of these agencies represent the top six providers of EHS research funding from fiscal year 2006 to fiscal year 2010, and CPSC has an important role in ensuring the safe use of nanotechnology in consumer products.

To examine recent changes in federal funding for nanotechnology EHS research, we reviewed information published by the National Science and Technology Council (NSTC) in the NNI Supplements to the President’s Budget. Specifically, we reviewed the actual agency investments reported in each program component area for fiscal years 2006 through 2010 for all NNI member agencies funding nanotechnology research. For the dollar amounts that we adjusted for inflation, we used the Biomedical Research and Development Price Index to report funding in constant 2010 dollars. We consulted with the Office of Management and Budget (OMB) and officials at the seven selected agencies to determine the type of budget information reported as actual agency investments in these documents. For fiscal year 2010, the most recent year for which actual agency investment data were available, we also collected quantitative and qualitative project-level data (such as project funding amounts and project abstracts or progress reports) on all research projects that the seven selected agencies had categorized as Program Component Area (PCA) 7 (EHS research). The agencies do not report project-level data to OMB annually but report their total EHS research funding to OMB annually for inclusion in the NNI budget supplements published by NSTC. Therefore, we reviewed data on the individual projects included in the agencies’ total EHS research funding in 2010. We provided the agencies with a spreadsheet template to use in listing their projects, which we based on the spreadsheet OMB had used in its data calls to collect project-level data from agencies on their 2006 and 2009 EHS research projects. Consistent with the approach used in OMB’s data calls, we did not ask the agencies to use a standardized definition of what constitutes a “project;” instead, we deferred to each agency to identify their projects and consulted with agency officials as needed.

For some projects at NIH, NIOSH, and NIST, we grouped together some of the agencies’ data entries and counted them together for the purposes of our analyses. Consequently, the numbers of projects we are reporting may not match the number reported by these agencies elsewhere. Specifically, for NIH, we grouped the 14 supplement grants together with their main grants and counted each of those groups as a single project. We also grouped together the individual subprojects of each of NIH’s multiproject “parent” grants reported as EHS and counted each of those groups as a single project, for a total of seven groups of multiproject grants. In addition, we grouped together two grants made by different NIH institutes for the same project. For NIOSH, we identified two project entries with identical project descriptions. After consulting with NIOSH officials, we grouped those two entries together and counted them as a single project. For NIST, the agency reported a separate entry for each of its Project Tracking Numbers and explained that a single technical project is often associated with more than one Project Tracking Number. NIST officials instructed us to group together the entries with identical titles and count each of those groups as a single project.

To assess the reliability of the agencies’ fiscal year 2010 data, we sent each of the seven selected agencies a set of questions regarding the data and the information systems used to produce and store them. We also reviewed related supporting documentation, such as user manuals and data dictionaries for information systems, and, for some projects, copies of the associated interagency agreements. We examined the data for obvious errors, compared each agency’s total fiscal year 2010 PCA 7 funding reported to us to the totals reported in the NNI Supplement to the President’s 2012 Budget, and consulted with agency officials to understand the reasons for any differences. We determined that the data were sufficiently reliable for the purposes of our analyses. However, we did not attempt to verify the funding amounts reported for each project— for example, by reviewing documentation of agencies’ actual expenditures for each project.

We reviewed the qualitative project data, including abstracts and project reports, to assess whether the agencies had appropriately categorized the research projects they reported as EHS research—that is, whether the projects met the definition of program component area (PCA) 7. As described in this report, the definition of PCA 7—research primarily directed at understanding the EHS impacts of nanotechnology development and corresponding risk assessment, risk management, and methods for risk mitigation—is the only written guidance available to agencies on how to report their nanotechnology research in this program component area. Determining whether research meets this definition is an inherently subjective process. Therefore, rather than definitively conclude that any projects did not meet the definition, we assigned projects a designation of “not clearly PCA 7” if we were ultimately unable to identify any EHS research, or it was not clear that EHS was the primary focus of the research. For example, some projects studied various nanotechnology-enabled drugs, but it was not clear to what extent the research was directed at the safety of the drugs versus their efficacy. Some other projects appeared to be closely related to program component areas other than PCA 7, such as PCA 4 (instrumentation research, metrology, and standards for nanotechnology) or PCA 8 (education and societal dimensions), and it was not clear that PCA 7 was the most appropriate category. To minimize bias and to ensure the consistency of our evaluation, two analysts independently analyzed the project data and used their professional judgment to assess whether each project met the PCA 7 definition. The two analysts discussed those projects where they did not agree on whether the research was primarily directed at EHS and reached agreement. For categorization of projects that appeared questionable to us, we generally asked agencies to explain why they had reported those projects as EHS research. We reviewed agencies’ responses and modified our determinations for projects as appropriate given the additional support provided by the agencies. For two projects funded by NIST and 18 by NSF, the agencies had reported only a portion of the total project funding under PCA 7. In those instances, we assessed whether at least some of the project was primarily directed at EHS. However, we did not verify that the funding amounts reported as EHS research were correct because that information is, according to these agencies’ officials, based on the discretion of the knowledgeable agency program staff. We also reviewed our 2008 report in this area, which found that 22 of 119 fiscal year 2006 projects reported as EHS research were miscategorized, and recommended that the Director of the Office of Science and Technology Policy (OSTP), in consultation with the Directors of the National Nanotechnology Coordinating Office and OMB, provide better guidance to agencies on how to report nanotechnology EHS research.

GAO, Nanotechnology: Better Guidance Is Needed to Ensure Accurate Reporting of Federal Research Focused on Environmental, Health, and Safety Risks, GAO-08-402 (Washington, D.C.: Mar. 31, 2008). experience in nanotechnology helped identify the categories we used and verified that the materials were placed in the appropriate categories for each project. After reviewing the fiscal year 2010 EHS research by the selected agencies, we selected the five categories of nanomaterials we used, based on the primary composition of the materials, to encompass the range of materials identified in the projects. To inform our selection of the categories we used for grouping nanomaterials, we reviewed literature related to nanotechnology EHS issues from federal agencies (NIH and EPA), nonprofit organizations (the Woodrow Wilson International Center for Scholars’ Project on Emerging Nanotechnologies and the World Technology Evaluation Center, Inc.), and a market research firm that tracks nanotechnology (Lux Research). We found that various classes or categories were used to present information on nanomaterials. For materials identified as composites, we assigned them to categories based on the primary nanomaterials from which they were formed—for example, we assigned carbon nanotube nanocomposites to the carbon-based nanomaterials category.

To determine the extent to which the NNI member agencies collaborate with stakeholders on nanotechnology EHS research and related strategies, we (1) discussed with agency officials how their agencies collaborate on nanotechnology EHS research and NNI’s role in facilitating that collaboration, and (2) we obtained documentation on these collaborative efforts. We conducted a review of formal collaborative efforts (i.e., those that are documented in written agreements) that focused on nanotechnology EHS research initiated from February 2008 to October 2011 while the 2008 NNI Strategy for Nanotechnology-Related EHS Research was in effect. We chose this time frame because this strategy—the first NNI strategy to focus on EHS research—established collaboration as necessary for its implementation efforts. The strategy was in effect for a number of years, which allowed us to get a more complete picture of the number of collaborative projects completed during its implementation. We only included collaborative efforts that were formalized by written agreements because those provided a clearer description of the agencies’ contributions of resources. We obtained an initial list of such efforts from published NNI sources as well as an OMB data call used to compile EHS-related nanotechnology projects for the 2011 NNI EHS research strategy. We sent our initial list to the selected NNI agencies for their review and revision, and analyzed supporting documents provided by the agencies. We provide relevant information from the supporting documents for each of the projects within our scope in appendix II.

We administered a web-based questionnaire to a nonprobability sample of nonfederal stakeholders, including those affiliated with academia, companies, nongovernmental organizations (NGO), and state and local governments, to obtain their views on collaboration with the NNI member agencies on EHS research and the EHS research strategies. Because the population of nonfederal stakeholders is diverse, wide-ranging, and difficult to reliably define, we chose to use a nonprobability sample of individuals who (1) had expertise in the field of nanotechnology, and (2) had interacted with the NNI in the past few years, or who were representatives of organizations and companies suggested to us through our scoping interviews. We identified relevant potential respondents from several sources, including the list of participants in NNI workshops that were used to solicit input for the 2011 NNI EHS research strategy and participants in a review of the NNI by the President’s Council of Advisors on Science and Technology. We also included members of nanotechnology-related trade associations and other organizations through interviews with nanotechnology experts using an iterative process, often referred to as “snowball sampling.” We excluded federal employees and contractors, individuals or organizations that are not located in the United States, and those for whom we could not obtain contact information. In addition, individuals who were not considered to have expertise in the nanotechnology field based on their job titles were excluded.chosen as potential respondents. Of these 228 individuals, five subsequently were excluded from our sample because they had invalid e- mail addresses, were duplicates, or were not available for an extended period, giving us a final sample of 223 potential respondents. We categorized the population surveyed by affiliation type, usually based on information provided on each entity’s website, as follows: (1) “academic,” which includes universities or university-affiliated programs or centers; (2) “company,” which includes companies, corporations, or other for-profit entities whose primary purpose is to sell products, services, or information; (3) “organization,” which includes NGOs or other entities whose primary purpose is public interest or mutual benefit; and (4) “state and local government.”

These sources provided us with a list of 228 stakeholders The questions included in the questionnaire were developed using issues raised in our scoping interviews and background literature search. Most of our questions provided respondents with answer options identified through our interviews and background literature searches. In certain cases, we also provided respondents with an “other” category which allowed them to write in additional responses we did not identify. However, because few respondents chose this option, we presented only some of these responses as illustrative examples. The questionnaire went through internal reviews, including independent GAO survey experts, as well as four pretests with two stakeholders who work for companies, one academic, and one stakeholder from an NGO. We chose the pretesters to reflect the diversity of affiliations of the nanotechnology experts included among our respondents. We revised and clarified the questions and introductory material to the questionnaire based on comments obtained in the internal reviews and from the pretesters.

We administered the questionnaire through a secure server. When we completed the final survey questions and format, we sent an e-mail announcement of the questionnaire to the nonfederal stakeholders in our sample on September 23, 2011. Stakeholders were notified that the questionnaire was available online and were given unique passwords and user names on October 11, 2011. We sent follow-up e-mail messages on October 18, 2011, October 25, 2011, and November 2, 2011 to those who had not yet responded. Then we contacted all nonrespondents by telephone, starting on October 21, 2011. The questionnaire was available online until December 6, 2011.

Because this was not a sample survey, it has no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in interpreting a particular question, sources of information available to respondents, or entering data into a database or analyzing them can introduce unwanted variability into the survey results. We took steps in developing the questionnaire, collecting the data, and analyzing them to minimize such nonsampling errors. For example, we worked with social science survey specialists to design the questionnaire. We pretested the draft questionnaire with four nonfederal stakeholders to ensure that the questions were relevant, clearly stated, and easy to understand. When we analyzed the data, an independent analyst checked all computer programs. Since this was a web-based questionnaire, respondents entered their answers directly into the electronic questionnaire, eliminating the need to key data into a database, minimizing error.

Of the 223 potential respondents, 138 completed the questionnaire during our time frame, for an overall response rate of about 62 percent. The numbers responding by affiliation type as categorized by us were (1) “academic”—42 out of 62; (2) “company”—65 out of 114; (3) “organization”—24 out of 39; and (4) “state and local government”— 7 out of 8.

To determine the extent the NNI strategy documents address desirable characteristics of national strategies, we compared three key NNI strategic documents against criteria for desirable characteristics of national strategies. Prior GAO reports have identified six desirable characteristics for national strategy documents that would help shape the policies, programs, priorities, resource allocations, and standards that would enable federal agencies and other stakeholders to implement the strategies and achieve the identified results. National strategies that address these characteristics offer policymakers and implementing agencies a management tool that can help ensure accountability and more effective results. Table 5 provides the desirable characteristics and the elements we looked for.

To assess whether the documents addressed these desirable characteristics, two analysts independently reviewed the three NNI strategy documents for the elements of each characteristic. We examined the documents for inclusion of each element and for sufficient specificity and detail. The two analysts compared their assessments, discussed any discrepancies, and agreed upon a determination for each element. Each characteristic was then given a rating of “address,” “partially address,” or “does not address.” According to our methodology, the strategy documents “address” a characteristic when it explicitly includes all, or nearly all, elements of the characteristic and has sufficient specificity and detail. The documents “partially address” a characteristic when it includes some or most of the elements with sufficient specificity and detail. The “partially address” category includes a range that varies from explicitly including most of the elements to including as few as one of the elements of a characteristic. A strategy “does not address” a characteristic when it does not include any elements of a characteristic or references are too vague or general to be useful.

Because the three NNI documents have different purposes and audiences, an individual document need not address each element of each desirable national strategy characteristic. Therefore, we reviewed the three NNI strategy documents as a whole. In the case of the 2011 NNI strategic plan and Supplement to the President’s 2012 Budget, we focused on sections related to EHS research, where applicable.

We identified agency documents related to nanotechnology EHS research strategies and performance. Documents were identified based on interviews with agency officials, review of agency websites, and a data collection instrument sent to agencies requesting any documentation related to the planning or performance of their nanotechnology EHS research programs. In order to assess the extent of linkage between agencies’ nanotechnology EHS research and national priorities of the NNI, we evaluated agency management documents for explicit referrals to NNI activities or guidance, including strategic documents such as the NNI strategic plan or EHS research strategy, or activities of the NNI’s planning and coordinating bodies. We also reviewed agencies’ Performance Reports in order to assess agencies' use of performance information related to nanotechnology EHS research efforts.

We conducted this performance audit from February 2011 through May 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Collaborative Nanotechnology Environmental, Health, and Safety Research Agreements

Table 6 identifies collaborative nanotechnology EHS research project agreements initiated by the NNI member agencies included in our review from February 2008 to October 2011. The table identifies only those projects specifically focused on nanotechnology. All the information in the table is based on the written agreements, which were provided by NNI member agencies’ officials.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Dan Haas (Assistant Director), Krista Anderson, Nirmal Chaudhary, Elizabeth Curda, Lorraine Ettaro, Alison O’Neill, Tind Shepper Ryen, Jeanette Soares, Ruth Solomon, Hai Tran, and Jack Wang made key contributions to this report.