Federal Research: NASA Should Better Inform Researchers about How to Appeal Dissemination Decisions
Published: 2009-12-03T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: National Aeronautics and Space Administration (NASA) researchers generally disseminate their agency-funded research results through publications, presentations, agency releases, and media interviews. In 2007, GAO reviewed dissemination policies at NASA and two other agencies and found that NASA's policies were generally clear, but GAO's survey of NASA researchers raised concerns that many of them did not understand some of the policies and were generally unaware of how to appeal dissemination decisions. Congress in 2008 directed GAO to determine whether NASA is implementing its policies in a clear and consistent manner. To meet that requirement, GAO determined (1) what changes have been made to NASA's policies since 2007, (2) the views of NASA researchers on whether the policies have been more effectively communicated since 2007, and (3) what changes have occurred since 2007 in NASA's processes for researchers to follow if they wish to appeal decisions about the dissemination of their research results. GAO conducted a Web-based survey of all 2,790 NASA researchers and had a 57.5 percent response rate.
section_title: What GAO Found
paragraphs: Since May 2007, NASA has changed some of its policies on the dissemination of research results through publications and presentations, but has not changed its policy on dissemination through news releases and media interviews, although it has made a key leadership change in its main public affairs office. Regarding its policies for dissemination through publications and presentations, in 2007 and 2009, NASA clarified the roles and responsibilities of officials who participate in the review and approval process that is required before agency-funded research is released. The changes also required managers to notify researchers when their requests to release research results have been approved or denied, and they required researchers to refrain from releasing results until they received approval. In contrast, although NASA has not made any changes to its policy that guides the dissemination of research through the media, in May 2007, it converted a key management position in the Headquarters Office of Public Affairs from a political appointment to a career civil service position. This was done to address allegations that an official in this office had suppressed climate change science and denied media access to scientists during 2004 and 2005. The change resulted from a management review conducted in early 2006 by NASA's leadership looking into these allegations. While the majority of researchers are familiar with the dissemination policies, GAO's survey indicates that in 2009, 87 percent of researchers were familiar with NASA's dissemination policies, compared with 92 percent in 2007, and slightly less were confident they understood the policies well enough to follow them for certain kinds of dissemination routes, such as publications, presentations, and agency releases. As in 2007, researchers continue to learn about the policies mainly through on-the-job training and e-mails from NASA officials. GAO's 2009 survey, like the 2007 survey, also indicates that researchers remain unclear about when they may discuss research results that have policy implications, including when they may discuss their own views, even though NASA has clarified its policy to allow researchers to do so as long as they do not attribute their views to the agency. Since 2007, NASA has not adopted any new procedures for appealing dissemination decisions. As in 2007, in 2009 relatively few NASA researchers were aware of and familiar with the agency's appeals process. Specifically, GAO's survey found that about one-third of NASA researchers were aware that the agency has a process to appeal decisions related to dissemination, but only about 8 percent said they were familiar with it. Fewer researchers in 2009 than in 2007 said they had sought to disseminate their research in the past 5 years (83 percent compared with 91 percent), and more researchers had their requests denied for other than technical reasons (12 percent in 2009 compared with 7 percent in 2007). Nonetheless, 85 percent of researchers continue to believe that the agency generally supports the dissemination of research results and that the agency's efforts to inform them about policies for all dissemination routes are generally effective.

section_title: Letter
paragraphs: The federal government supports a wide range of scientific and  engineering research and development efforts to advance general  knowledge, support science-related public policy, and enhance the  nation’s economic competitiveness, among other things. Much of federally  funded research is designed to serve the goals and missions of over a  dozen federal agencies, including the National Aeronautics and Space  Administration (NASA). Open communication and the exchange of  research results with the broader scientific community are critical to  advancing the state of scientific understanding. While the results of some  research cannot be shared for national security or intellectual property  reasons, dissemination of most federally funded research helps to ensure a  return on the public’s investment in research. Federally sponsored  research may be conducted by federally employed researchers or by  university and other researchers who receive grants and contracts  awarded by various federal agencies.
From its beginning, NASA has been responsible for the nation’s  aeronautical and space activities, including conducting and sponsoring  research related to its four mission areas—aeronautics research,  exploration systems, science, and space operations. The statute that  established NASA in 1958 calls for the agency to “provide the widest  practicable and appropriate dissemination” of the scientific and technical  information resulting from its research efforts. Research results can be  disseminated through a number of different routes to reach a variety of  audiences. Some dissemination routes, such as publication through peer- reviewed journals and agency scientific and technical reports, share  information with the scientific community. Similarly, researchers can  present their findings to colleagues at scientific conferences, workshops,  symposia, or professional society meetings. Other dissemination routes  are intended to reach the public at large. For example, agency press  releases or postings on agency Web sites can help target media or public  attention to particular research findings and agency events, and media  interviews can provide the general public with access to information about  scientific activities and results.
In recent years, concerns arose that NASA researchers were prevented  from sharing the results of their work, particularly with respect to the  sensitive topic of climate change and the impact of global warming. In  light of these concerns, in 2006 Congress requested that we review the  dissemination policies and practices for federally funded research at three  agencies—NASA, the National Oceanic and Atmospheric Administration,  and the National Institute of Standards and Technology. In our May 2007  report, we found that NASA’s policies were generally clear and should  help facilitate the dissemination of research results. However, we raised a  number of concerns, including that many NASA scientists did not  understand their agency’s policies related specifically to media interviews  or press releases and were generally unaware of NASA’s policy for  appealing dissemination decisions. We also found that NASA generally did  not offer formal training to its researchers on its dissemination policies.
Through the National Aeronautics and Space Administration Authorization  Act of 2008, Congress stated that “NASA should not dilute, distort,  suppress, or impede scientific research or the dissemination thereof” and  directed us to determine whether NASA is implementing the regulations  governing the dissemination of research results in a clear and consistent  manner. In response to the mandate and as agreed with your offices, this  report, which follows up on our 2007 report, discusses (1) what changes, if  any, have been made since 2007 to the policies that guide the  dissemination of federally funded research results at NASA; (2) the extent  to which NASA researchers believe that the agency’s dissemination  policies have been more effectively communicated since 2007, and their  experiences in using the process; and (3) what processes, if any, NASA has  adopted since 2007 for researchers to follow if they have concerns about  decisions regarding the dissemination of their research results, and how  those concerns have been addressed.
To identify and evaluate the changes that may have been made since 2007  to the policies that guide the dissemination of federally funded research  results at NASA, we obtained, reviewed, and analyzed NASA’s current  research dissemination policies, and compared them with the prior  policies. We contacted scientific managers and public affairs officials who  are responsible for the dissemination of research results at NASA’s  headquarters and all 10 centers and used a standard set of questions to  ensure that we obtained consistent information about each aspect of  NASA’s dissemination policies and practices. In addition, we obtained  copies of the relevant dissemination policies and discussed with NASA  officials how these policies were implemented and how they have  changed, if at all, since 2007. To identify the extent to which NASA  researchers believe that the agency’s dissemination policies have been  more effectively communicated since 2007, what their experiences have  been in disseminating research, and how their concerns about  dissemination decisions have been addressed, we sent a Web-based survey  to all 2,790 researchers at NASA. We defined researchers to be included in  our population as federally employed scientists, engineers, or other  researchers who are in a position to disseminate their research results to a  wider audience. To identify these researchers, we obtained from center or  program managers at NASA a list of researchers in their organization who  would meet our defined criterion. Overall, we received a response rate of  57.5 percent to our survey, for a total of 1,605 respondents. We compared  the respondents with nonrespondents on available administrative data,  such as the center where they work, but did not find any significant  differences on the available variables. Although our survey was intended  to be a census, for the purposes of analyzing the results, we decided to  treat our survey as a random sample. We used largely the same survey  instrument that was the basis for our May 2007 report on dissemination  policies at selected agencies. Through our survey, we sought the  researchers’ views on NASA’s research dissemination policies, the level of  agency support for dissemination, and their experiences with  disseminating research results. Unless otherwise noted, point estimates  we report for 2009 have a 95 percent level of confidence of plus or minus 4  percent. An expanded explanation of our scope and methodology can be  found in appendix I. Appendix II contains selected results from our 2009  survey.
We conducted this performance audit from January 2009 to December  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: NASA’s work is performed at 10 research and flight centers across the  United States. Each of these centers conducts a variety of research,  engineering, construction, and support functions related to the agency’s  mission, and each has a separate management structure, including a  Center Director, to guide its activities. Furthermore, each center has its  own public affairs office responsible for managing its media contacts and  communications. NASA’s Washington, D.C., headquarters provides overall  guidance for each of the mission areas and public affairs oversight of  agencywide programs and activities to coordinate and maintain  communication channels to the news media and public.
Federally employed NASA researchers must adhere to both agencywide  and center-specific policies and procedures when disseminating  information outside of the agency. The requirements for dissemination  depend on the type of information to be released, the method used for  dissemination, and the intended audience. NASA has (1) policies for  disseminating scientific and technical information through presentations  and publications to the scientific community and also (2) a media policy  that governs the release of information to the general public through  media interviews and press releases. Scientific and technical  information—defined as the results of basic and applied scientific,  technical, and related engineering research and development—is typically  disseminated through NASA publications, outside journals, presentations  at meetings or workshops, and Web sites. NASA’s policies for the  dissemination of such research results establish a series of reviews to  determine whether the information is suitable for public release based on  national security or intellectual property concerns, is technically accurate,  and meets data quality standards. NASA’s media policy governs the release  of information about the agency’s activities and results to the general  public through such methods as interviews, press releases, media  advisories, news features, and Web postings. This policy specifically  defines the roles and responsibilities of managers, researchers, and public  affairs officials in deciding what information can be released. For  example, the policy directs public affairs officials at both NASA  headquarters and the relevant center to ensure the timely release of public  information by obtaining review and clearance by appropriate officials. It  also includes an appeals process for researchers to use when they disagree  with the agency’s decision regarding whether to release information.
In recent years, allegations have been made regarding interference with  the efforts of NASA researchers to disseminate their research results to  external audiences. A number of these concerns surrounded politically  sensitive topics, such as climate change, where research was allegedly  being suppressed. For example, in December 2005, the NASA  Headquarters Office of Public Affairs denied a request from National  Public Radio (NPR) to speak with a NASA scientist known for expertise  on the subject of climate change. However, not long before NPR’s request  for an interview, this scientist had delivered a presentation on climate  change at a scientific conference, prompting concerns that the NASA  Headquarters Office of Public Affairs had improperly denied NPR’s  interview request.
The concerns over NASA’s alleged suppression of research resulted in  reviews of the agency’s dissemination practices by the NASA Office of  Inspector General (OIG) in addition to our review in 2007. In June 2008,  NASA’s OIG issued two reports on the dissemination of research results at  NASA. Specifically, the OIG evaluated NASA’s process for reviewing,  approving, and releasing research results and investigated allegations that  NASA had suppressed the dissemination of climate change research and  information through the media. In the first report, the OIG found that  NASA’s guidance adequately defined the roles and responsibilities for the  review, approval, and release of NASA research results through  publications and presentations but that the guidance had not been  effectively implemented at the four centers it reviewed—Goddard Space  Flight Center, Johnson Space Center, Langley Research Center, and  Marshall Space Flight Center. The OIG determined that, among other  things, researchers were not always notified when the review process had  been completed, and center management had not adequately informed  researchers about the process to follow when they wanted to disseminate  their research results through publications or presentations. While the OIG  did not identify instances in which research results had been  inappropriately released, it noted that the absence of timely notice to  researchers regarding dissemination decisions increases the risk of  releasing research results that should be restricted for export control,  national security, or other reasons. Finally, the OIG surveyed researchers  who had sought to disseminate their research results and found no  evidence that NASA purposefully used the review process to suppress  scientific research at those four centers.
In its second report, the OIG found that during the fall of 2004 through  early 2006, the NASA Headquarters Office of Public Affairs reduced,  marginalized, or mischaracterized climate change science made available  to the general public. The OIG found that the suppression of climate  change research was localized within the NASA Headquarters Office of  Public Affairs and that senior NASA officials stated they were unaware of  the situation until the allegations appeared in a national newspaper.  According to the report, once NASA leadership was made aware of the  suppression, it aggressively implemented corrective actions. For example,  NASA revised its policy that governs dissemination of research results  through the media and reaffirmed its policy that researchers may discuss  their opinions on the implications of research results provided they  attribute such views to themselves and not the agency. The OIG  determined that the suppression had occurred, in part, because political  appointees controlled key positions responsible for deciding which  research results NASA would disseminate to the media.
section_title: NASA Has Made Some Changes to Improve Dissemination of Research Results
paragraphs: Since May 2007, NASA has changed some of its policies on the  dissemination of research results through publications and presentations,  and it has made a key leadership change in its main public affairs office to  address allegations that an official in this office had played a key role in  suppressing climate change science and media access to climate scientists  in the past. Since 2007, NASA has made no changes to its policy on  dissemination through news releases and media interviews.
subsections: section_title: NASA Has Clarified the Roles and Responsibilities of Reviewing Officials for Dissemination of Research Results through Publications and Presentations
paragraphs: In late 2007 and early 2009, NASA changed one of its policies on  dissemination of research results through publications and presentations.  Specifically, in contrast to the policy in place prior to December 2007, the  revised policy holds researchers responsible for ensuring that the results  they seek to disseminate have been properly reviewed and approved prior  to releasing them. In addition, the revised policy directs reviewing officials  in the centers to include training in their efforts to ensure that NASA  researchers are aware of the requirements and procedures for the review,  approval, and dissemination of research results. In January 2009, in  response to the OIG’s review of dissemination practices at four centers,  NASA further revised this policy to strengthen the earlier changes by  including additional responsibilities for officials in the centers and in  NASA headquarters. For example, the revised policy directed center  managers to (1) establish plans to ensure that researchers are aware of the  review requirements, and (2) notify researchers in a timely manner  whether their research has been approved or rejected for release. The  revised policy also directed NASA officials responsible for overseeing the  dissemination of research results to annually review the compliance of  NASA’s centers with the updated policy, and evaluate the overall  effectiveness of the agency’s review and approval process.
As part of its efforts to ensure compliance by the centers with the revised  policies, headquarters officials told us they have assessed compliance at  the four centers reviewed by the OIG and, as of October 2009, are  finalizing a report on their findings. These officials also said they will begin  annual compliance reviews for all centers in 2010. In addition, NASA  officials have taken steps to implement an automated system to enhance  managers’ and researchers’ ability to monitor requests through the review  and approval process. Specifically, during its review, the OIG found that  managers who use paper-based systems were unable to readily determine  when research may have been inappropriately released, whether requests  had been denied, and how many requests were undergoing review. In  fiscal year 2009, NASA approved funding to test an automated tracking  system in two centers; the system is in operation at one of the centers, and  is set to become operational at the other center early in fiscal year 2010.  The use of such systems will be voluntary at other centers, since  headquarters will provide only limited funding to maintain them. NASA  officials told us that several centers have expressed interest in developing  automated tracking systems but, as of October 2009, have not secured the  necessary funding.
Officials at the four centers reviewed by the OIG have also taken a number  of steps to implement the policy changes made in 2007 and 2009. For  example, all four centers reviewed their center-specific policies and  procedures—which generally reinforce or elaborate on NASA-wide  policies—and two centers revised their policies. For example, one center  revised a flow chart contained in its procedures to more accurately reflect  the center’s review and approval process.
Centers have also taken steps toward establishing a process to ensure  timely notification of researchers about dissemination decisions,  according to NASA officials. Currently, some centers have electronic  systems that provide notification automatically, while others provide  notification through e-mails and telephone calls. NASA headquarters  officials said that the automated tracking system, once implemented  agencywide, would also help ensure that notification occurs in a timely  manner at the conclusion of the review and approval process.
section_title: NASA Changed a Key Public Affairs Position to Help Ensure Dissemination of Research Results to the Public through the Media
paragraphs: Since 2007, NASA has not made any changes to its media policy, but in  May 2007 it converted a key management position in the Headquarters  Office of Public Affairs from a political appointment to a career civil  service position. This change followed a management review—conducted  in early 2006 by NASA’s leadership—of allegations that its Headquarters  Office of Public Affairs had suppressed climate change science and denied  media access to scientists in 2004 and 2005. In a June 2008 report on these  concerns, the OIG concluded that inappropriate political posturing was  the proximate cause of at least some of these actions and that a political  appointee who was the Deputy Assistant Administrator for Public Affairs  at the time of the alleged suppression was a central figure in the  allegations of censorship and denial of media access. According to the  OIG report, NASA intended that the conversion of the position to a career  civil service position would facilitate communications within the  Headquarters Office, where the suppression of information had been  localized. NASA officials told us that the agency placed a career civil  servant in the position after the incumbent political appointee, who had  held the position during the time that climate science had been  suppressed, left NASA.  The career civil servant took over the position in  May 2007.
section_title: Fewer Researchers Are Familiar with NASA’s Dissemination Policies, but Most Believe the Policies Effectively Ensure Access to Research Results
paragraphs: Although NASA has taken some steps to increase researchers’ awareness  of its dissemination policies, our survey results in 2009 indicate that while  most researchers are familiar with the dissemination policies, fewer  researchers were familiar with them than in 2007, and slightly fewer were  confident they understood the policies well enough to follow them.  However, most researchers reaffirmed their belief that NASA’s policies  effectively ensure access to research results and that the agency’s efforts  to help them understand the various dissemination policies are generally  effective.
subsections: section_title: NASA Has Undertaken Various Efforts to Increase Researchers’ Awareness of Dissemination Policies
paragraphs: In response to a review by its OIG, NASA tasked headquarters officials  responsible for oversight of dissemination of research results through  publications and presentations with ensuring that NASA researchers are  aware of the agency’s review and approval requirements. However, as of  October 2009, headquarters officials said they have relied on the centers to  create and implement plans to increase awareness. Thus far, officials at  some centers have taken actions to increase researchers’ awareness of the  requirements for review, such as sending a center-wide e-mail emphasizing  the importance of completing the review process prior to releasing  research results and increasing outreach to researchers who work in areas  that produce significant amounts of research. According to NASA  headquarters officials, they may take additional steps in the future to  increase awareness, such as having the centers include information on the  review and approval process in their new employee orientation programs.
NASA has also taken steps since 2007 in response to our report to help  raise awareness of its media policy. Specifically, in May 2007 we  recommended that NASA provide formal training to inform, reinforce, and  update managers, researchers, and public affairs staff on dissemination  policies. As a result of our recommendations, NASA undertook a number  of formal training activities with regard to its media policy. Specifically, a  senior public affairs official conducted presentations at each of the centers  regarding the media policy. As part of these meetings, officials presented  an overview of the policy, answered questions from employees, and  distributed copies of the policy to each public affairs officer and to senior  managers. In December 2007, NASA distributed materials on the policy to  public affairs staff and discussed the policy in meetings that included  headquarters and center public affairs officials. Finally, a discussion of the  media policy was incorporated into the orientation for new public affairs  officers.
Results from both surveys, as well as agency officials we spoke with,  indicated that most NASA researchers receive information about  dissemination policies through means other than formal training, although  fewer researchers said in 2009 that they learn about policies through e- mails from center and NASA managers and more said they had received  formal training. Our 2009 survey results show that researchers get  information through a variety of methods, most commonly through on-the- job training (about 70 percent) and e-mails or other correspondence from  center managers to all center staff (68 percent). Additionally, more than  half of researchers cited Administrator-level e-mails or correspondence  sent to all NASA staff. Although e-mails from both center and  Administrator-level management represent the second and third most  common method of receiving information about NASA policies, about 10  percent fewer researchers cited these in 2009 than in 2007. Furthermore,  roughly half of researchers in both surveys responded that they learned  about policies through notifications on intranet sites and staff or town hall  meetings. Finally, in 2009 more researchers reported learning about  dissemination policies through formal training—42 percent compared with  32 percent in 2007. These results appear consistent with the fact that  NASA headquarters and public affairs officials told us that the agency  largely relies on methods other than formal training to inform staff about  the media policy and appeals process.
section_title: Fewer Researchers Are Familiar with NASA’s Dissemination Policies than in 2007, and Slightly Fewer Are Confident That They Understand the Policies
paragraphs: Although our surveys from 2007 and 2009 indicate that about 90 percent of  researchers are familiar with NASA’s dissemination policies, fewer  researchers in 2009 compared with 2007 said they are very or moderately  familiar with NASA’s dissemination policies. Specifically, according to our  2009 survey results, 87 percent of researchers were very or moderately  familiar with NASA’s dissemination policies, compared with 92 percent in  2007. See figure 1.
From 2007 to 2009, we also found a small decline in the percentage of  researchers who said they were very or moderately confident that they  understood the policies for dissemination of research results through  publications and presentations well enough to follow them (about 85  percent in 2009 compared with about 90 percent in 2007). Similarly, a  small decline occurred in the percentage of researchers who were very or  moderately confident they understood the policies related to agency  releases (55 percent in 2009 compared with 61 percent in 2007). We found  no change in the confidence researchers have about their understanding of  the policies related to media interviews. See figure 2.
As would be expected, those researchers who thought NASA’s efforts to  educate them about dissemination policies were extremely to moderately  effective were more likely to feel confident that they understood the  policies than those who thought the agency’s efforts were less effective.  For example, 76 percent of researchers who reported being very to  moderately confident that they understood the dissemination policies  related to media interviews also responded that NASA’s efforts to educate  them about these policies were extremely to moderately effective. In  contrast, 17 percent of researchers who reported being very to moderately  confident about how to apply the policies for dissemination through media  interviews also responded that NASA’s efforts to educate them about these  policies were slightly to not at all effective. In addition, researchers who  said they had tried to disseminate their results in the past 5 years were  more likely than other researchers to be confident that they understood  the policies related to the dissemination route they used. For example, 69  percent of researchers who sought to disseminate research results through  agency releases in the past 5 years were very or moderately confident that  they understand NASA’s policies for dissemination through agency  releases, whereas 55 percent of all researchers, regardless of whether they  had sought to use agency releases, reported being very or moderately  confident they understand dissemination policies related to agency  releases.
Our 2007 and 2009 surveys also indicate that many researchers remain  unclear about when they may discuss research results that have potential  policy implications. Specifically, according to our 2009 survey, when  research results are consistent with NASA’s policy, 66 percent of  researchers believe they are allowed to discuss the results, while 15  percent believe they may not, and 19 percent were uncertain. However,  fewer researchers believe they may discuss research results when the  results differ from NASA’s policy. Specifically, 43 percent believe they may  discuss such results, 19 percent believe they may not, and 38 percent were  uncertain. Additionally, just over half of researchers (54 percent) believe  they may discuss their personal views related to research results that have  potential policy implications, while 13 percent believe they may not and 34  percent were uncertain. This uncertainty about discussing research results  and expressing personal views exists despite clear statements in NASA’s  revised media policy that employees may discuss their results and express  personal views as long as they attribute such views to themselves and not  the agency.
section_title: Most Researchers Believe NASA’s Policies Ensure Access to Research Results
paragraphs: In our 2007 and 2009 surveys, almost 90 percent of researchers reported  that overall the agency’s policies are extremely to moderately effective in  ensuring access to research results. In addition, over 85 percent of  researchers believed NASA’s efforts to inform them about policies for  dissemination through publications and presentations were extremely to  moderately effective, and about 68 percent reported the same regarding  media interviews. However, in 2009, fewer researchers believed that  NASA’s efforts to inform them about the policies for disseminating  research through agency releases were extremely to moderately effective  (72 percent said so in 2009 compared with 81 percent in 2007). See figure  3.
section_title: Many Researchers Remain Unfamiliar with How to Appeal Decisions, and More Dissemination Requests Have Been Denied
paragraphs: NASA has not adopted any new processes or procedures since 2007 for  researchers to use if they wish to appeal decisions regarding their requests  to disseminate their research results. Nevertheless, we found in 2009,  relatively few NASA researchers are aware of and familiar with the  agency’s process to appeal dissemination decisions. In addition, fewer  researchers said they had sought to disseminate their research results in  the past 5 years, and more said their requests to disseminate research  results had been denied. However, most NASA researchers continue to  believe that the agency supports the dissemination of research results and  that NASA and its centers apply policies for all routes of dissemination  consistently.
subsections: section_title: NASA’s Policy on Dissemination through the Media Includes an Appeals Process, but Many Researchers Remain Unfamiliar with It
paragraphs: NASA has not adopted any new processes or procedures since 2007 for  researchers to use if they wish to appeal decisions regarding their requests  to disseminate their research results. As in 2007, researchers may follow  the appeals process outlined in NASA’s media policy if they wish to appeal  a decision about dissemination through press releases and interviews. For  other types of dissemination, such as through publications and  presentations, there is no formal appeals process, although officials told us  that researchers may raise concerns through their supervisors or through  other agency dispute resolution processes, such as those for personnel  disputes. These officials also said they are not aware of any disputes  regarding decisions to release research results to the media and thus have  not used the appeals process described in the policy. For other types of  dissemination, such as through publications and presentations, for which  there is no appeals process, most NASA center managers told us that  disputes over disseminating research results are rare, and some managers  said they have never needed a formal process to resolve disputes.
Nonetheless, our 2009 survey shows that, as in 2007, about one-third of  NASA researchers reported that they were aware of NASA’s process to  appeal decisions about dissemination of research results, but only 8  percent responded that they were familiar with the process. About 60  percent of the 8 percent of researchers who were both aware of and  familiar with the appeals process reported that it was extremely to very  effective. In both 2007 and 2009, about 20 percent of researchers reported  that they had appealed decisions when their requests to disseminate  research results were denied.
section_title: Fewer Researchers Sought to Disseminate Research Results, and More Have Had Their Dissemination Requests Denied since 2007
paragraphs: According to our 2009 survey, fewer researchers have tried to disseminate  their research results than in 2007, and more researchers have had their  requests to disseminate denied. Specifically, there was an 8 percentage  point decline in the number of researchers who responded that they tried  to disseminate their research results outside of NASA in the last 5 years  (83 percent in 2009 compared with 91 percent in 2007). Researchers in  2009 largely sought to disseminate their research results through the same  routes—presentations, publications, agency releases, and media  interviews—as did researchers in 2007. However, fewer sought to  disseminate their results through agency releases. See figure 4.
According to our survey results, more researchers in 2009 had their  requests to disseminate research results denied for reasons other than  those stemming from standard technical reviews (about 12 percent of  researchers in 2009, compared with about 7 percent in 2007). Most  researchers whose requests had been denied were seeking to disseminate  research results through publications and presentations, as we found in  2007. Also, as in 2007, researchers took various actions in response to a  denial. While more than half of researchers gave up on efforts to  disseminate, many others successfully disseminated their research results  by making revisions that led to approval or by using a different  dissemination route.
Researchers commented that they received a variety of reasons for the  denials, although some indicated they were given no reason. As in 2007,  among the most common reasons reported for denials was that the  research was restricted for national security or other reasons or that the  topic or results were sensitive. Another commonly cited reason was the  lack of funding to attend conferences. Some researchers specifically  mentioned the restrictions on NASA’s spending related to conferences—a  key venue for presenting research results—contained in the NASA  Authorization Act of 2008. The act authorized $5 million for conference- related expenses in fiscal year 2009, a reduction of over two-thirds from  the previous fiscal year. Subsequently, in March 2009, the Omnibus  Appropriations Act of 2009 clarified that the restrictions did not apply to  scientific or technical conferences and other conferences at which NASA  science, technology, engineering, and mathematics are disseminated.  According to NASA officials, the spending limitation prevented many  researchers from disseminating their research results because decisions  about conference attendance are typically made early in a fiscal year. One  NASA official estimates that only one-third of NASA-funded research was  presented to the scientific community in 2009.
section_title: Researchers Continue to Believe That NASA Supports Dissemination, and Most Believe NASA and Its Centers Consistently Apply Dissemination Policies
paragraphs: As in 2007, most researchers believe that NASA is very to moderately  supportive of disseminating research results but that the agency is more  supportive of some dissemination routes than others. We estimate that 85  percent of researchers in 2009, compared with 89 percent in 2007, believe  that NASA either insists on or encourages the dissemination of research  results. When comparing 2007 and 2009 survey results for specific  dissemination routes, we found that fewer researchers now believe that  NASA is very to moderately supportive of dissemination through  presentations—92 percent of researchers said this in 2007 compared with  84 percent in 2009.  While we cannot be certain of the reasons for this  decline, they may also be related to the limitation on conference  attendance in 2009 mentioned earlier. We also found a slight increase in  the percentage of researchers who believe NASA is very to moderately  supportive of dissemination through media interviews. Specifically, we  estimate that 68 percent of researchers in 2009 believe that NASA is  supportive of dissemination through media interviews compared with 62  percent of researchers in 2007. We also found a slight decrease in the  percentage of researchers who believe NASA is very to moderately  supportive of dissemination through publications—89 percent in 2009  compared with 91 percent 2007. See figure 5.
In comparing the 2009 and 2007 survey results, we also found that, in  general, researchers believe that both the centers and NASA as a whole  apply dissemination policies very to somewhat consistently. However, a  greater number of researchers now believe the centers more consistently  apply policies for dissemination through agency releases (83 percent in  2009 compared with 71 percent in 2007) and media interviews (77 percent  in 2009 compared with 62 percent in 2007). In addition, more researchers  in 2009 believe that NASA consistently applies policies for media  interviews (68 percent compared with 57 percent). We also found an  increase that approaches statistical significance in the percentage of  researchers in 2009 who believe NASA consistently applies policies for  dissemination through agency releases (74 percent compared with 67  percent).
Our 2009 survey results indicate that researchers believe, across all  dissemination routes, that the centers more consistently apply  dissemination policies than NASA as a whole. Specifically, we estimate on  the basis of the 2009 survey results that 88 percent of researchers believe  their centers consistently apply policies related to publications and 81  percent believe their centers consistently apply policies related to  presentations. In contrast, 79 and 73 percent of researchers believe NASA  as a whole consistently applies publication and presentation policies,  respectively. Further, 83 percent of researchers believe their centers  consistently apply policies related to agency releases, and 77 percent  believe their centers consistently apply policies related to media  interviews, whereas fewer believe NASA as a whole consistently applies  policies related to agency releases and media interviews—74 and 68  percent, respectively.
section_title: Conclusions
paragraphs: Scientific research is critical to the ability of NASA and other agencies to  achieve their missions and is at the heart of many major policy decisions  that agencies face. Open communication and exchanging of research  results contribute to advances in the state of scientific knowledge and a  well-informed public. In 2007, while we found that NASA’s policies were  generally clear, we also raised a number of concerns, including that many  NASA researchers did not understand the full range of the agency’s  policies and were generally unaware of NASA’s policy for appealing  dissemination decisions. In response, NASA undertook various efforts to  increase the awareness of researchers about its media policy. These  efforts appear to have achieved some success because, as our survey  indicates, more researchers now believe that NASA is supportive of  dissemination through media interviews. In addition, more researchers  today believe that NASA and the centers consistently apply policies for  dissemination through agency releases and media interviews, and  researchers generally rate NASA’s efforts to help them understand the  policies as effective. However, with regard to dissemination through  publications and presentations, we found that fewer researchers are  confident they understand the policies well enough to apply them, and  fewer believe that NASA is supportive of using these routes to disseminate  research results. Finally, now, as in 2007, very few researchers are familiar  with how to appeal decisions made in regard to their requests to  disseminate their research results, and about half of those who had their  requests denied gave up trying to disseminate their results. We are  concerned that this may inhibit the full range of scientific research from  being shared with the scientific community and the public.
section_title: Recommendations for Executive Action
paragraphs: To strengthen NASA’s efforts to better inform its researchers and ensure  that the policies guiding researchers and public affairs officials in their  efforts to disseminate research results to other scientists and the public  are well understood, we recommend that the NASA Administrator direct  officials responsible for dissemination of research results through all  routes to include in their efforts to inform researchers a focus on the  processes researchers are to follow when they wish to appeal decisions.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided the NASA with a draft of this report for their review and  comment. NASA concurred with our recommendation and stated that the  NASA Administrator will direct the appropriate NASA officials, including  the Assistant Administrator for Public Affairs, to work with the  appropriate NASA entities to review the agency’s dissemination policies  and provide appropriate education and training for researchers to ensure  they clearly understand the processes for appealing dissemination  decisions.  The comment letter from NASA is presented in appendix III.
We are sending copies of this report to the appropriate congressional  committees, the Administrator of NASA, and other interested parties. The  report also will be available at no charge on the GAO Web site at  http://www.gao.gov. If you or your staff have any questions about this  report, please contact me at (202) 512-3841 or mittala@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The National Aeronautics and Space Administration (NASA) Authorization  Act of 2008 directed GAO to determine whether NASA is implementing the  regulations governing the dissemination of research in a clear and  consistent manner. In response, this report discusses (1) what changes, if  any, have been made since 2007 to the policies that guide the  dissemination of federally funded research results at NASA; (2) the extent  to which NASA researchers believe that the agency’s dissemination  policies have been more effectively communicated since 2007 and what  their experiences have been in using the process; and (3) what processes,  if any, NASA has adopted since 2007 for researchers to follow if they have  concerns about decisions regarding the dissemination of their research  results and how those concerns have been addressed.
To identify and evaluate what changes have been made since 2007 to  NASA’s policies that guide the dissemination of federally funded research  results, we obtained, reviewed, and analyzed the agency’s dissemination  policies, and compared them to prior policies. In addition, we contacted  managers and public affairs officials who are responsible for  dissemination of research results at NASA headquarters and at each of 10  research, space, and space flight centers. We used a standard set of  questions to ensure we obtained consistent information, confirmed that  we had copies of the relevant dissemination policies, and discussed at  length the ways in which these policies had changed since 2007 and how  these policies were put into practice. We also included questions about  what processes NASA has adopted since 2007 for researchers to follow if  they have concerns about decisions regarding the dissemination of their  research results.
To identify to what extent NASA researchers believe that the agency’s  dissemination policies have been more effectively communicated since  2007 and what their experiences have been in using the process for  disseminating their research results and how their concerns about  dissemination decisions have been addressed, we sent a Web-based survey  to all 2,790 researchers at NASA. This survey instrument was in large part  identical to the one used as the basis for GAO’S May 2007 report on  policies guiding dissemination of scientific research at NASA, the National  Institute for Standards and Technology, and the National Oceanic and  Atmospheric Administration. We sought the NASA researchers’ views on  their agency’s research dissemination policies, the level of agency support  for dissemination, and their experiences with dissemination. We defined  researchers to be included in our population as federally employed  scientists, engineers, or other researchers who are in a position to  disseminate their research results to a wider audience. We included  researchers from 9 of the 10 NASA centers and excluded the Jet  Propulsion Laboratory from our review because contract researchers  primarily perform research at this center and NASA does not maintain a  database that contains the names of individual contract researchers. In  addition, contract researchers might charge NASA for costs related to  participating in the survey. To identify researchers for our survey, we  obtained from center or program managers at NASA a list of researchers in  their organization who would meet our defined criterion. To mitigate the  risk that NASA center and program managers may not have provided the  complete universe of relevant researchers, we interviewed NASA public  affairs and scientific officials to identify the key characteristics of  individuals in a position to disseminate research. We then obtained from  NASA’s Office of Human Capital a list, broken out by center, of offices that  employed researchers who met these key characteristics. NASA obtained  this information from the Federal Personnel and Payroll System (FPPS)  Datamart, which is the official data source for personnel and payroll  information on NASA civil service employees. To assess the reliability of  the information in this database for the purposes of our report, we  submitted a set of questions to NASA related to how the data are managed  and maintained. On the basis of NASA’s responses to these questions, we  determined that the data are adequate to identify offices within NASA that  employ researchers who meet the key characteristics. After obtaining the  list of offices from NASA, we asked officials at each center to verify that  they had considered all the program offices on the list when they  developed their original list of researchers and, if not, we asked them to  contact those offices to determine if any researcher names should be  added to our survey universe.
Information about accessing the survey was provided via e-mail for all  survey participants. The survey was activated, and researchers were  informed of its availability, on July 14, 2009; it was available thorough  September 18, 2009. To ensure security and data integrity, we provided all  participants with user names and personal passwords that allowed them to  access and complete the survey. To reduce survey nonresponse, we sent  out multiple e-mail reminder messages and conducted follow-up telephone  calls to all nonrespondents to encourage them to complete the survey.  From initial notification, we identified 379 individuals who were outside  the target population. For example, some individuals had retired from the  agency. In all, we received a 57.5 percent response rate.
We compared the respondents with nonrespondents on available  administrative data, such as the center where they work, but did not find  any significant differences on the available variables. Although our survey  was intended to be a census, for the purposes of analyzing the results, we  decided to treat our survey as a random sample. Specifically, we took the  response rate within each of the 9 NASA centers and used this to derive a  statistical weight. We applied statistical weight to the respondents so that  their answers would represent the population.  This requires the  assumption that the nonrespondents are missing at random.
All sample surveys are subject to sampling error—that is, the extent to  which the survey results differ from what would have been obtained if the  whole population had been observed. Measures of sampling error are  defined by two elements, the width of the confidence intervals around the  estimate (sometimes called the precision of the estimate) and the  confidence level at which the intervals are computed. Because we treated  our survey as a stratified random sample, we assumed our sample was  only one of a large number that could have been drawn. Because each  sample could have provided different estimates, we expressed our  confidence in the precision of our particular sample’s results as a 95  percent confidence interval. This is the interval that would contain the  actual population value for 95 percent of the samples we could have  drawn. As a result, we are 95 percent confident that each of the confidence  intervals based on the survey includes the true values in the sample  population. The 95 percent confidence intervals for numeric estimates are  presented along with those estimates in the body of the report.  Unless  otherwise noted, any comparisons between point estimates for 2009 have  a 95 percent level of confidence of plus or minus 4 percentage points.
In addition to sampling errors, the practical difficulties of conducting any  survey may introduce errors, commonly referred to as nonsampling errors.  For example, differences in how a particular question is interpreted, the  information sources available to respondents, or the types of sample  members who do not respond can introduce unwanted variability into the  survey results. We assumed that the propensity to respond to our survey  did not depend upon the questions we asked, and we assumed that the  respondents were statistically similar to the nonrespondents. If  characteristics of respondents are different from those of nonrespondents  on key items, it could introduce a bias not accounted for in our analysis.  We took extensive steps in questionnaire development, data collection,  and the editing and analysis of the survey data to minimize nonsampling  errors. This survey instrument was in large part identical to the one used  as the basis for GAO’S May 2007 report on policies guiding dissemination  of scientific research at three agencies, including NASA. The 2007 survey  was developed by a GAO survey specialist in conjunction with subject  matter experts, and then reviewed by a second independent survey  specialist. In addition, we pretested the survey by telephone with  researchers from NASA. During these pretests, we asked the researchers  to complete the survey as they would when they received it. We then  interviewed the respondents to ensure that (1) the questions were clear  and unambiguous, (2) the terms we used were precise, (3) the survey did  not place an undue burden on the researchers completing it, and (4) the  survey was independent and unbiased. On the basis of the feedback from  the pretests, we revised the questions, as appropriate. In making minor  revisions to the 2007 survey to make it acceptable for the 2009 survey, we  also worked with survey specialists, who determined that the changes  made would not require additional pretesting. The SAS and SUDAN  programs that produced the 2009 survey estimates, including estimates of  categories derived from content analysis, were reviewed by a second,  independent programmer to ensure accuracy in the logic and syntax of the  program.
Regarding the survey data on instances in which researchers were denied  the opportunity to disseminate, the percentage of researchers reporting  such instances can be generalized to the population. However, because the  number of denials is small, the details concerning the reason for denial,  the researcher’s primary field of research, and the actions taken in  response to the denial cannot be generalized. Because the issue of a  researcher being denied is a salient piece of our analysis, we included this  nongeneralizable information to provide context to this important issue.  To analyze select, open-ended questions on our survey, including those  given as the reason researchers said they were denied, we conducted a  content analysis to develop our agreement statistics. Reviewers (two per  question) collaboratively developed content categories based on survey  responses, and then independently assessed and coded each survey  response into those categories. Intercoder reliability (agreement) statistics  were electronically generated in the coding process, and agreement  statistics for all categories were 90 percent or above. Coding  disagreements were resolved through reviewer discussion or a third-party  arbiter.
We conducted this performance audit from January 2009 through  December 2009 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Selected Results of the 2009 Survey of NASA Researchers
paragraphs: The following tables contain summary results of selected questions from  our survey of researchers at NASA in 2009. For each question reported  below, the estimated percentage is presented along with its 95 percent  confidence interval. These tables do not include summary-level data for  the demographic questions and do not include the results from any open- ended questions.
Q8. Overall how familiar or unfamiliar are you with the policies NASA has in place regarding the dissemination of research results  outside of the agency?
Q9. Has NASA used any of the following means to help you understand how to comply with its policies regarding the dissemination of  research results outside of the agency?
Q9. Has NASA used any of the following means to help you understand how to comply with its policies regarding the dissemination of  research results outside of the agency?  b. On-the-job training (including mentoring)
4.2-5.6  Q9. Has NASA used any of the following means to help you understand how to comply with its policies regarding the dissemination of  research results outside of the agency?
Q9. Has NASA used any of the following means to help you understand how to comply with its policies regarding the dissemination of  research results outside of the agency?  d. E-mails or other correspondence from Administrator-level offices sent to all staff  Q9. Has NASA used any of the following means to help you understand how to comply with its policies regarding the dissemination of  research results outside of the agency?  e. E-mails or other correspondence from center management sent to all center or office staff  Q9. Has NASA used any of the following means to help you understand how to comply with its policies regarding the dissemination of  research results outside of the agency?  29.8-32.8  Q9. Has NASA used any of the following means to help you understand how to comply with its policies regarding the dissemination of  research results outside of the agency?
Q10. Overall, how effective or ineffective have NASA’s efforts been to help you understand how to comply with its policies regarding  the dissemination of research results through each of the following routes?  a. Publications (such as peer-reviewed publications or non-peer-reviewed publications)
Q10. Overall, how effective or ineffective have NASA’s efforts been to help you understand how to comply with its policies regarding  the dissemination of research results through each of the following routes?  b. Presentations (such as keynote addresses, conferences, workshops, symposia, professional society meetings, or congressional  hearings)
Q10. Overall, how effective or ineffective have NASA’s efforts been to help you understand how to comply with its policies regarding  the dissemination of research results through each of the following routes?  c. Agency releases (such as press releases, posting on an agency intranet site, or agency reports or fact sheets)
26.4-29.9  Q10. Overall, how effective or ineffective have NASA’s efforts been to help you understand how to comply with its policies regarding  the dissemination of research results through each of the following routes?
Q10. Overall, how effective or ineffective have NASA’s efforts been to help you understand how to comply with its policies regarding  the dissemination of research results through each of the following routes?
Q11. Regardless of how effective or ineffective NASA’s efforts have been, how confident, if at all, are you that you understand the  agency’s policies regarding the dissemination of research results through each of the following routes well enough to follow them?
Q11. Regardless of how effective or ineffective NASA’s efforts have been, how confident, if at all, are you that you understand the  agency’s policies regarding the dissemination of research results through each of the following routes well enough to follow them?  13.6-15.9  Q11. Regardless of how effective or ineffective NASA’s efforts have been, how confident, if at all, are you that you understand the  agency’s policies regarding the dissemination of research results through each of the following routes well enough to follow them?
Q11. Regardless of how effective or ineffective NASA’s efforts have been, how confident, if at all, are you that you understand the  agency’s policies regarding the dissemination of research results through each of the following routes well enough to follow them?
Q11. Regardless of how effective or ineffective NASA’s efforts have been, how confident, if at all, are you that you understand the  agency’s policies regarding the dissemination of research results through each of the following routes well enough to follow them?
Q12. Overall, in your opinion, how consistently or inconsistently does your NASA Center apply its policies regarding the dissemination  of research results through each of the following routes?  6.4-8.2  Q12. Overall, in your opinion, how consistently or inconsistently does your NASA Center apply its policies regarding the dissemination  of research results through each of the following routes?
Q12. Overall, in your opinion, how consistently or inconsistently does your NASA Center apply its policies regarding the dissemination  of research results through each of the following routes?
Q12. Overall, in your opinion, how consistently or inconsistently does your NASA Center apply its policies regarding the dissemination  of research results through each of the following routes?
Q12. Overall, in your opinion, how consistently or inconsistently does your NASA Center apply its policies regarding the dissemination  of research results through each of the following routes?  23.7-44.3  Q13. Overall, in your opinion, how consistently or inconsistently does NASA as a whole apply its policies regarding the dissemination of  research results through each of the following routes?
Q13. Overall, in your opinion, how consistently or inconsistently does NASA as a whole apply its policies regarding the dissemination of  research results through each of the following routes?
Q13. Overall, in your opinion, how consistently or inconsistently does NASA as a whole apply its policies regarding the dissemination of  research results through each of the following routes?
Q13. Overall, in your opinion, how consistently or inconsistently does NASA as a whole apply its policies regarding the dissemination of  research results through each of the following routes?  14.5-18.4  Q13. Overall, in your opinion, how consistently or inconsistently does NASA as a whole apply its policies regarding the dissemination of  research results through each of the following routes?
Q14. Overall, do you believe NASA’s dissemination policies are effective or ineffective in ensuring access to the results of research  conducted at the agency?
Q15. How supportive, if at all, is NASA toward dissemination research results through each of the following routes?  a. Through publications, such as peer-reviewed journals or non-peer-reviewed journals   Q15. How supportive, if at all, is NASA toward dissemination research results through each of the following routes?  b. Through presentations, such as at conferences or at congressional hearings  Q15. How supportive, if at all, is NASA toward dissemination research results through each of the following routes?  c. Through agency releases, such as press releases, posting on an agency intranet site, or agency reports or fact sheets   20.5-23.9  Q15. How supportive, if at all, is NASA toward dissemination research results through each of the following routes?
Q16. Overall, which of the following statements best characterizes the extent to which NASA supports the dissemination of research  results?
NASA insists or encourages dissemination  NASA neither insists or encourages dissemination  NASA discourages or does not allow dissemination  Q17. Does your area of research have the potential to affect federal policy decisions?
Q18. When disseminating research results that have potential federal policy implications, does each of the following statements  correctly or not correctly represent NASA’s position on discussing these research results?  a. I can discuss research results with potential policy implications when the results are consistent with the agency’s policy or position  statements.   16.5-21.4  Q18. When disseminating research results that have potential federal policy implications, does each of the following statements  correctly or not correctly represent NASA’s position on discussing these research results?  b. I can discuss research results with potential policy implications when the results differ from the agency’s policy or position  statements.
Q18. When disseminating research results that have potential federal policy implications, does each of the following statements  correctly or not correctly represent NASA’s position on discussing these research results?  c. I can discuss research results with potential policy implications on issues for which the agency does not have a policy or position  statement.
Q18. When disseminating research results that have potential federal policy implications, does each of the following statements  correctly or not correctly represent NASA’s position on discussing these research results?  d. I can discuss potential policy implications of research results provided that I state the policy implications as my personal views and  not those of the agency.
Q18. When disseminating research results that have potential federal policy implications, does each of the following statements  correctly or not correctly represent NASA’s position on discussing these research results?  e. I am not allowed to discuss the potential policy implications of research results.
28.5-34.3  Q19. In the past 5 years, have you sought to disseminate the results of your own scientific research outside of NASA?
Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?  1. I don’t conduct research that requires dissemination.
Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?
Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?  3. Someone else disseminates research results on my behalf.
86.8-92.0  Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?  4. I sought to disseminate results in the past and was not allowed to.
Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?  5. My research is ongoing and is not ready to be released.
Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?  6. My research is not eligible for public dissemination.
Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?  7. Agency policies did not allow me to disseminate.
96.6-99.1  Q20. Which of the following reasons describe why in the past 5 years you have not sought to disseminate the results of your scientific  research outside of the agency?
Q21. Over the past 5 years, through which methods did you seek to disseminate your research results?  a. Publications (such as peer-reviewed publications or non-peer-reviewed publications)
Q21. Over the past 5 years, through which methods did you seek to disseminate your research results?  c. Agency releases (such as press releases, posting on an agency intranet site, or agency reports or fact sheets)
Q21. Over the past 5 years, through which methods did you seek to disseminate your research results?
Q21. Over the past 5 years, through which methods did you seek to disseminate your research results?
Q22. Other than for standard technical review reasons, over the past 5 years, have you ever encountered a situation when NASA did  not allow you to disseminate your research results?  87.0-89.3  Q23. Through which method(s) were you seeking to disseminate your research results when you were denied agency approval?  a. Publications (such as peer-reviewed publications or non-peer-reviewed publications)  (If Question 19 is Yes and Question 22 is Yes)
Q23. Through which method(s) were you seeking to disseminate your research results when you were denied agency approval?  b. Presentations (such as keynote addresses, conferences, workshops, symposia, professional society meetings, or congressional  hearings)  (If Question 19 is Yes and Question 22 is Yes)
Q23. Through which method(s) were you seeking to disseminate your research results when you were denied agency approval?  c. Agency releases (such as press releases, posting on an agency intranet site, or agency reports or fact sheets)  (If Question 19 is Yes and Question 22 is Yes)
Q23. Through which method(s) were you seeking to disseminate your research results when you were denied agency approval?  (If Question 19 is Yes and Question 22 is Yes)
93.4-97.3  Q23. Through which method(s) were you seeking to disseminate your research results when you were denied agency approval?  (If Question 19 is Yes and Question 22 is Yes)
Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  a. Appealed the decisions using established procedures.  (If Question 19 is Yes and Question 22 is Yes)
Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  (If Question 19 is Yes and Question 22 is Yes)
Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  c. Disseminated the results through a different route.  (If Question 19 is Yes and Question 22 is Yes)
1.4-5.1  Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  d. Added a disclaimer that the opinions expressed in the research results do not reflect the views of the agency.  (If Question 19 is Yes and Question 22 is Yes)
Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  e. Resubmitted the same or similar document.  (If Question 19 is Yes and Question 22 is Yes)
Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  (If Question 19 is Yes and Question 22 is Yes)
1.4-5.8  Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  g. Received approval following other types of revisions.  (If Question 19 is Yes and Question 22 is Yes)
Q25. Did you take any of the following actions when you did not receive approval to disseminate your research results?  (If Question 19 is Yes and Question 22 is Yes)
Q26. To the best of your knowledge, does NASA have a process or procedure in place to appeal decisions made regarding the  dissemination of research results?
section_title: Appendix III: Comments from the National Aeronautics and Space Administration
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact:
section_title: Staff Acknowledgments:
paragraphs: In addition to the contact person named above, Cheryl Williams (Assistant  Director), Carl Barden, Laura Erion, Richard Johnson, Amanda Miller, Ben  N. Shouse, Lisa Turner, Fatema Wachob, and Elizabeth Wood made key  contributions to this report.