Background

As identified and described in figure 1, we reported in October 2016 that  agencies have frequently used five open innovation strategies to  collaborate with citizens and other external parties, and encourage their  participation in agency initiatives.
As we described in our October 2016 report, agencies can use these  open innovation strategies singularly, or in combination as part of a larger  open innovation initiative, to achieve a number of different purposes.  Agencies can use them to efficiently engage a broad range of citizens  and external stakeholders in developing new ideas, solutions to specific  problems, or new products ranging from software applications to physical  devices. For example, DOE’s Wave Energy Prize competition was  designed to achieve a ground-breaking advancement in technology that  produces electricity by capturing energy from ocean waves. DOE’s goals  for the competition were to engage a diverse collection of developers to  create more efficient devices that would double the energy captured from  ocean waves. According to the Wave Energy Prize website, the  competition, which was concluded in November 2016, produced a  significant advancement in this technology. Ultimately, 4 of 9 finalist  teams—selected from 92 that registered to participate—developed  devices that surpassed DOE’s goal, while the winning team achieved a  five-fold improvement.
Agencies can also use these strategies to enhance collaboration among  citizens and external stakeholders or organizations interested in an issue,  and leverage the resources, knowledge, and expertise of citizens and  stakeholders to supplement that which is available within the agency.  These contributions enhance the agency’s capacity and its ability to  achieve goals that would be more difficult to reach without this additional  capacity or expertise. In another example highlighted in our October 2016  report, the Federal Highway Administration’s (FHWA) Every Day Counts  ideation initiative helps states implement innovations that improve the  efficiency and quality of their highway transportation and construction  projects. Every two years, FHWA engages a range of stakeholders to  identify innovative technologies and practices that merit more widespread  deployment. Once innovations are identified, FHWA creates deployment  teams comprised of experts from FHWA, state transportation agencies,  and industry to help states and other stakeholders implement these  innovations. According to an FHWA report, this open innovation initiative  has had significant and measurable effects in participating states. In one  case, FHWA reported that deploying Accelerated Bridge Construction as  an Every Day Counts innovation has allowed states to reduce the time it  takes to plan and construct bridges by years, significantly reducing traffic  delays, road closures, and often project costs.
In our October 2016 report, we also identified seven practices and 18  related key actions, described in figure 2, that federal agencies can use to  help effectively design, implement, and assess their open innovation  initiatives. To do so, we analyzed and synthesized suggested practices  from relevant federal guidance and literature, including public and  business administration journals, and publications from research  organizations, as well as interviews with experts and agency officials with  experience implementing open innovation initiatives. In that report, we  also illustrated how the application of these practices helped agencies  effectively implement open innovation initiatives in a way that achieved  intended results.
Three agencies with government-wide responsibilities—OMB, OSTP, and  GSA—have taken steps to support and encourage the use of open  innovation strategies by federal agencies.
OMB is the largest component of the Executive Office of the  President, and is responsible for helping agencies across the federal  government implement the commitments and priorities of the  President. Among other things, it develops policies and provides  direction on the use of Internet-based technologies that make it easier  for citizens to interact with the federal government.
OSTP is also a component of the Executive Office of the President.
Among other things, it has responsibility for leading interagency  efforts to develop and implement sound science and technology  policies.
GSA is responsible for helping federal agencies obtain the facilities,  products, and services that they need to serve the public. Among  other things, GSA builds, provides, and shares technology, platforms,  and processes that support initiatives to invite participation from  stakeholders and make data available to the public.

	OMB, OSTP, and GSA Developed Policies and Guidance, Dedicated Staff, and Established Websites to Support the Use of Open Innovation Strategies

		OMB, OSTP, and GSA Developed Policies and Implementation Guidance to Encourage Agencies to Use Open Innovation Strategies

OMB, OSTP, and GSA developed government-wide policies and  implementation guidance to encourage agencies to use open innovation  strategies to, among other things, advance their missions, improve  programs and services, and inspire new ideas to address specific  challenges. According to staff from these agencies, policies and guidance  do this by:  clarifying the legal authorities available to use specific strategies; highlighting the benefits and results that agencies can produce using  suggesting actions for agency staff to take when designing and  implementing an open innovation initiative.
Table 1 lists the government-wide policies and guidance that we, in  consultation with OMB, OSTP, and GSA staff, identified as key for  implementing each open innovation strategy.
As table 1 shows, the focus of the guidance varies across the different  types of open innovation strategies. For crowdsourcing and citizen  science, as well as prize competitions and challenges, toolkits provide  step-by-step guidance for agency staff to follow when implementing those  types of initiatives. OMB, OSTP, and GSA staff identified the U.S. Public  Participation Playbook as key guidance for a variety of approaches to  engage the public, including two open innovation strategies: ideation and  open dialogues. They also identified the Open Data Policy and Project  Open Data as the key policy and source of guidance, respectively, that  cover open data collaboration initiatives. They told us that these and  other resources have a broader focus, generally to help agencies  inventory, manage, and release their open data. These sources do  suggest some actions for agencies to take when implementing open data  collaboration initiatives. However, OMB, OSTP, and GSA staff have not  yet developed more detailed, step-by-step guidance for implementing  specific initiatives where agencies use events or websites to engage and  collaborate with outside stakeholders using open data.
As one example of policy and guidance for an open innovation strategy,  in September 2015, the Director of OSTP issued a memorandum  encouraging agencies to use crowdsourcing and citizen science to  enhance scientific research and address problems by drawing on the  voluntary participation of the public. The memorandum also directs  agencies to improve their ability to use crowdsourcing and citizen science  by identifying internal agency coordinators responsible for seeking  opportunities to use this strategy to meet agency goals. Crowdsourcing  and citizen science coordinators from four of the selected agencies— DOE, HHS, EPA, and NASA—told us that having this clear statement of  support has helped enhance awareness of and interest in the strategy.  The memorandum was supplemented by the Crowdsourcing and Citizen  Science Toolkit, which provides a step-by-step list of actions agencies  can take to carry out a specific initiative, as shown in figure 3.
These officials from DOE, EPA, HHS, and NASA told us that that the  toolkit is an important educational resource for agency staff, helping those  interested in using crowdsourcing and citizen science understand what  the strategy can be used to achieve and the full range of actions they can  take to develop and implement an initiative. An official from GSA’s  Technology Transformation Service (TTS) told us GSA is planning  updates to the toolkit to reflect the authorities provided by, and  requirements of, the American Innovation and Competitiveness Act  (AICA) for crowdsourcing and citizen science initiatives.

		Dedicated Staff Advise and Assist Agency Staff Implementing Initiatives and Open Innovation- Related Communities of Practice

In addition to developing relevant policies and implementation guidance,  staff from OMB, OSTP, and GSA told us they also provide ongoing  support for the use of open innovation strategies across the federal  government by: answering questions from agency staff, sharing lessons learned, and  providing advice and assistance; hosting training sessions and other events for agency staff to highlight  how agencies can use open innovation strategies and how agencies  have had success using them; and matching agency staff in need of assistance with mentors, advisors, or  partners in other agencies.
Table 2 provides additional information on the types of support they  provide.
One example of staff supporting the use of open innovation strategies is  GSA’s Challenge.gov program management team. According to an  official from GSA’s TTS, this team is the primary point-of-contact for staff  from other federal agencies, fielding their questions about the site or other  aspects of managing a prize competition or challenge. To help familiarize  agency staff with how to manage a prize competition or challenge and  use the Challenge.gov platform, the team conducts training sessions and  has developed on-demand videos and webinars that are available  through the website. In addition, to provide tailored assistance, the team  created the Challenge and Prize Mentorship Program, which matches  staff seeking support with experienced practitioners from other agencies  who can assist with all aspects of planning and implementing an initiative  using this strategy.
OMB, OSTP, and GSA staff also support communities of practice for  various open innovation strategies. According to staff from these  agencies, these communities provide venues to bring experienced and  knowledgeable agency staff together so that they can learn from one  another. Through regularly-scheduled meetings and e-mail lists, the  communities allow members to share experiences and lessons learned,  seek and provide advice and assistance, and ensure members are up to  date on relevant issues, such as the development and release of new  resources or upcoming initiatives. OMB, OSTP, and GSA staff also have  leveraged the expertise of community members to develop toolkits and  case studies, which capture leading practices and lessons learned to  ensure they are readily available to inform future initiatives. Collectively,  these communities, which are described in table 3, help staff across the  federal government communicate and collaborate with one another.
The federal Challenges and Prizes Community of Practice further  illustrates the support such communities can provide. According to a GSA  official, this community was created in 2010 to bring together staff from  across the federal government interested in that open innovation strategy.  According to information on the Challenge.gov website, the community is  comprised of federal employees representing a wide range of agency  perspectives and experience levels. Staff from OSTP’s Technology and  Innovation Division and GSA’s TTS told us that they have worked with  volunteers from the community to develop resources, including the  federal Challenges and Prizes Toolkit, to try to meet the diverse needs of  its members. The official from GSA’s TTS also told us that they plan to  work with members of the community to revise the toolkit to reflect the  updated authorities and requirements for prize competitions and  challenges in AICA. This official further stated that the Challenge.gov  program team provides ongoing support to this community by maintaining  its e-mail list, which allows for information sharing between members, and  working with members to identify topics and arrange speakers for the  community’s quarterly meetings. For instance, the Challenge.gov team  helped organize the March 2017 meeting, which focused on how AICA  updates agency authorities and requirements for prize competitions and  challenges.

		OMB, OSTP, and GSA Developed Websites to Improve Access to Guidance and Potential Participants for Open Innovation Strategies

According to staff from OMB, OSTP, and GSA, their agencies developed  several websites, described in table 4, that support the use of open  innovation strategies by: making relevant guidance and information easy to access in a single, providing agency staff access to additional resources, such as  applications, templates, and documents from other agencies, that they  can replicate to save time and effort; and providing online platforms for agencies to reach the public with  information on their open innovation initiatives.
As examples of such websites, OMB, OSTP, and GSA developed Project  Open Data and Data.gov to help agencies manage and release their  open data and meet requirements under OMB’s Open Data Policy.  According to officials from DOE, EPA, HHS, HUD, and NASA, Project  Open Data has made it easier for them to find the guidance they need by  making it accessible in one place. Project Open Data also encourages  agencies to hold open data events to engage with data users, and to use  these opportunities to expand awareness about their open data, and  collect feedback and ideas. The website provides an overview of the main  types of open data community events that federal agencies hold, as well  as instructions and templates to help agencies organize, publicize, and  carry out various types of events. To increase awareness about the open  data events that agencies are holding, and to help agencies increase  participation in these events, GSA has also added an “Events” page to  Data.gov. As shown in figure 4, this provides agencies with an additional  online platform to inform potential participants about open data events  they will be hosting.

	Key Guidance for Open Innovation Strategies Reflect Practices for Effective Implementation to Differing Extents

We determined that key government-wide guidance developed by OMB,  OSTP, and GSA to support the implementation of various open  innovation strategies reflects practices for effective implementation to  differing extents. Several factors led to these variances, including differing  scopes and methodologies used in their development and the dates when  they were issued. We identified key guidance for the implementation of  initiatives using each open innovation strategy in consultation with OMB,  OSTP, and GSA staff. As noted above, staff from these agencies  identified the U.S. Public Participation Playbook as key guidance for both  ideation initiatives and open dialogues. Therefore, we present our  assessment of guidance for those two strategies together below. OMB,  OSTP, and GSA staff also identified the Open Data Policy and Project  Open Data as key sources of guidance for the implementation of open  data collaboration initiatives. These sources suggest some actions for  agencies to take when implementing such initiatives. However, as noted  earlier, they have a broader focus to help agencies inventory, manage,  and release their open data. As such, our assessment found that this  guidance generally did not reflect practices for effective implementation of  individual open data collaboration initiatives.

		Practice 1: Select the Appropriate Strategy

Our October 2016 report identified two key actions agencies should take  when attempting to select the most appropriate strategy or strategies for  their initiatives:  1.  Clearly articulate the purpose(s) they hope to achieve by engaging the  public. As we found in that report, agencies have used open  innovation strategies for a number of purposes, including to develop  solutions to specific problems and to leverage the expertise of  external stakeholders to enhance an agency’s ability to achieve a  goal.  2.  Consider the agency’s capability to implement a strategy.
Considerations should include whether agency leadership supports  the use of a strategy, and whether they have legal authorities,  financial and technological resources, and staff available to support  an initiative.
As shown in figure 5, we determined that guidance for all but one of the  strategies—open data collaboration—fully reflects these key actions.
For example, the Challenges and Prize Toolkit encourages agencies to  clarify their purpose in engaging the public, including by developing a  detailed understanding of the specific problem they want to address  through the competition. In addition, when considering a prize competition  or challenge, the toolkit recommends that agency staff:  secure the approval of agency leadership to move forward; work with legal counsel to identify the most appropriate authority  under which to conduct the competition, and to address any other  potential legal issues; estimate the budget and resource needs of a competition; and ensure the availability of staff to monitor or run the challenge  throughout its life cycle.
Guidance for open data collaboration initiatives does not address the first  key action—defining the purpose—and partially reflects the second key  action. Although it encourages agencies to consider agency capacity and  applicable laws, regulations, and policies, and the availability of  resources, it does not address leadership support.

		Practice 2: Clearly Define Goals and Performance Measures

To guide agencies in designing and implementing initiatives, and to  provide those involved with a clear understanding of what they are  working to achieve, we identified three key actions that agencies should  take:  1.  Define specific and measurable goals for their initiatives.  2.  Identify performance measures to assess progress toward those  goals.  3.  Align the goals of their open innovation initiatives with the agency’s  broader mission. This final action helps to demonstrate the relevance  and value of an initiative to others in the agency, and reinforces the  connection between the agency’s goals and the day-to-day actions of  those carrying out the initiative.
As shown in figure 6, our assessment shows that the guidance for all but  one of the strategies fully reflects these three key actions.
For example, the Federal Crowdsourcing and Citizen Science Toolkit  encourages agencies to define their goals for an initiative while  considering how they will measure and evaluate these outcomes. In  addition, the toolkit encourages agencies to identify the specific measures  that they will use to track the initiative’s outputs and activities, such as the  number of samples collected or training sessions held, and to determine  whether the initiative is achieving its goals. Lastly, it states that those  managing initiatives should ensure that the initiative is aligned with their  agency’s mission, and be able to specify how it will help the agency meet  its goals.
As also shown in figure 6, we determined that guidance for the  implementation of open data collaboration initiatives partially addresses  the first key action. Specifically, the guidance identifies illustrative goals  for various types of open data events. For example, it states that the goal  of a “hackathon” is to build relationships with a community of developers  and designers, and to see immediate tools and prototypes built using  open data. However, as written, these illustrative goals are not  measurable, nor does the guidance directly encourage agencies to  develop such goals for their own specific initiatives. Furthermore, the  guidance does not reflect the other two key actions.

		Practice 3: Identify and Engage Stakeholders and Partners

To leverage the experience, insights, and expertise of those interested or  engaged in the area to be addressed by an initiative, agencies should:  1.  Identify and engage with external stakeholders. These are individuals  or organizations that share an interest in the issue being addressed  and may already be active in related efforts. For a federal agency,  external stakeholders can include representatives of relevant nonprofit  organizations and foundations, community or citizens’ groups,  universities and academic institutions, the private sector, members of  Congress and their staffs, other federal agencies, and state and local  governments.  2.  Look for opportunities to partner with other groups and organizations.
Partners are organizations and individuals that play a direct role in  designing and implementing an initiative. They provide staff capacity,  resources, administrative and logistical support, assistance with  communications and community building, or ongoing advice and  expertise. In considering potential partners, agencies should look for  groups and organizations that would be interested in, or could benefit  from, the results of an open innovation initiative.
As shown in figure 7, we determined that guidance for all of the open  innovation strategies fully reflects both key actions.
For example, guidance for implementing open data collaboration efforts in  OMB’s Open Data Policy encourages agencies to engage with various  stakeholders, including entrepreneurs and innovators in the private and  nonprofit sectors. This engagement could then lead to these stakeholders  participating in specific initiatives to use agency data to build products,  applications, and services. Guidance on the Project Open Data website  also suggests agencies use open data collaboration events to bring  together various stakeholders, such as entrepreneurs, technology  leaders, and policy experts, to explore available data and discuss new  ideas for tools that the private sector could create using agency data.  Project Open Data also encourages agencies to invite and partner with  other government entities and private companies when developing and  carrying out these events.

		Practice 4: Develop Implementation Plans

To ensure that tasks and time frames are clear for all involved in  implementing and managing an open innovation initiative, agencies  should document roles and responsibilities and develop implementation  plans for those initiatives. The plans should clearly identify specific  tasks, the parties responsible for completing them, and the time frames  for doing so. They should also outline when and how the agency will  reach out to various participant groups, and how data will be collected  and evaluated to determine results.
As shown in figure 8, we determined that guidance for crowdsourcing and  citizen science initiatives, and prize competitions and challenges, fully  reflects these key actions.
For example, the Challenges and Prizes Toolkit encourages agencies to  create an implementation plan that clearly outlines the roles and  responsibilities for those involved in an initiative. As the toolkit  emphasizes, managing a competition can involve a wide range of  government and contract staff with varying responsibilities, from project  management to providing subject matter expertise and technical,  communications, and legal support. Because of this, it is important to be  clear about the role of each team member, the duties each is assigned,  and how his or her work fits into the timeline for the competition. As part  of this, the toolkit also states that the plan should specify the procedures  that will be used to collect relevant data during the course of the  challenge. Lastly, the toolkit encourages agencies to develop a  communications plan that defines the audiences they want to reach with  information on the initiative; the websites, news outlets, social media, and  other outlets it will use to reach them; and the messages they will use to  reach potential participants and encourage them to participate.
In contrast, as also shown in figure 8, we found that guidance for open  data collaboration initiatives does not reflect either of these key actions,  and guidance for open dialogue and ideation initiatives partially reflects  these actions. Specifically, the guidance for open dialogue and ideation  initiatives encourages agencies to document roles and responsibilities,  and to develop a plan that specifies the tasks and time frames for  recruiting participants. However, the guidance does not discuss planning  for data collection.

		Practice 5: Engage Participants and Partners

To reach the right potential participants, motivate them to participate, and  keep partners and participants engaged throughout the implementation of  an initiative, we previously identified four key actions agencies should  take. They should:  1.  Use multiple outlets and venues to reach potential participants. In  their outreach, agencies should use the initiative’s website, social  media, press releases, journals, newsletters, and professional  conferences and networks to reach potential participants.  2.  Craft announcements in a way that motivates people to participate. In  doing so, agencies should address the interests of potential  participants and explain why it is important for them to participate.  3.  Engage with participants to provide answers to questions and any  necessary assistance. While an initiative is ongoing, agencies can use  websites, question-and-answer sessions, e-mails, and other forms of  communication to engage with participants and provide ongoing  support and assistance.  4.  Hold regular check-ins with those involved in implementation.
Agencies should hold regular meetings to help ensure that those  working to implement an initiative are aware of the status of efforts  and have an opportunity to raise and discuss any concerns.
As shown in figure 9, our assessment found that guidance for  crowdsourcing and citizen science fully reflects all of these key actions,  while guidance for the other strategies reflects some, but not all, of these  key actions.
For example, the Crowdsourcing and Citizen Science Toolkit encourages  agencies to find the best platforms for reaching communities with  information about their initiatives, and to reach out and communicate with  potential participants using media and messages that respond to their  interests. The toolkit also encourages agencies to let those participating  in an initiative know how they can engage with the agency to provide  information and feedback, and states that the agency should pay  attention to shifts in participant needs and interests over time. Lastly, the  toolkit instructs agency staff to hold regular meetings with an initiative’s  implementation team so that everyone can understand how the project is  progressing, and discuss new developments and concerns.
As also shown in figure 9, for the other strategies, we determined that  guidance fully reflected actively engaging participants, but none of them  reflected holding regular check-ins with those involved in implementation.  Guidance for ideation and open dialogue initiatives, and prize  competitions and challenges reflects two additional key actions—using  multiple outlets and crafting announcements to participants’ interests— while the guidance for open data collaboration initiatives does not.

		Practice 6: Collect, Assess, and Report Relevant Data

After an initiative has concluded, or at regular intervals if it is a long- standing or continuous effort, agencies should take three steps to assess  and report results and identify potential improvements:  1.  Assess the data collected during implementation to determine  whether the initiative met its goals.  2.  Analyze feedback from partners and participants. Agencies should  identify lessons learned about what went well and what would need to  be adjusted or improved for similar initiatives in the future.  3.  Report publicly on results achieved and lessons learned. Doing so  can demonstrate the value of an initiative and sustain a dialogue  within the community of interested organizations and individuals.
As shown in figure 10, we determined that guidance for crowdsourcing  and citizen science initiatives and prize competitions and challenges fully  reflected these key actions.
For example, the Challenges and Prizes Toolkit states that agencies  should assess the data they have collected to determine how well the  competition achieved its goals, and to identify other results and outcomes  it produced, such as quantifiable improvements to existing solutions and  technologies. The toolkit also states that agencies should conduct an  after-action assessment to capture feedback, lessons learned, and other  institutional knowledge so the agency can improve its challenges in the  future. Specifically, it suggests that agencies consider the following  questions when conducting this assessment:
What worked well?
What would you have done differently in challenge design looking  back?
How might agency clearance and coordination go more smoothly next  time?
What could have been improved in judging, communications, and  operations?
Did the evaluation process result in the selection of the best  submissions?
What were any unintended consequences, both positive and  negative?
Lastly, the toolkit reminds agencies to complete required public reporting,  sharing results, lessons learned, and success stories, which can be  critical to improving how challenges are designed and implemented.
As figure 10 also illustrates, our assessment found that guidance for open  data collaboration initiatives did not reflect any of these key actions, and  guidance for ideation and open dialogue initiatives reflected most of them.  Specifically, that guidance encourages agencies to collect and analyze  data to assess goal achievement and results, conduct an after-action  review, and report publicly on results. It does not, however, encourage  agencies to report publicly on lessons learned through their experience.

		Practice 7: Sustain Community of Participants and Partners

Given the time and resources that agencies may invest to build or  enhance communities of partners and participants for open innovation  initiatives, our October 2016 report identified two key actions agencies  should take to sustain these connections over time:  1.  Acknowledge and, as appropriate, reward the efforts and  achievements of partners and participants.  2.  Seek ways to maintain communication with members of the  community. Doing so can keep them informed of future initiatives and  other opportunities, and facilitate communication within the  community.
As figure 11 shows, we determined that none of the guidance fully reflects  the first action. The guidance for crowdsourcing and citizen science, and  prize competitions and challenges, both encourage agencies to  acknowledge the contributions of participants, and reward participants  with monetary and nonmonetary incentives (as appropriate). However,  guidance for these strategies does not also encourage agencies to  acknowledge the contributions of partner organizations, which can  provide critical resources, expertise, and capacity for open innovation  initiatives. The guidance for ideation, open dialogues, and open data  collaboration initiatives did not reflect this key action.
As also shown in figure 11, our assessment found that guidance for most  open innovation strategies fully reflect the second action; however,  guidance for open data collaboration does not. For example, the Federal  Crowdsourcing and Citizen Science Toolkit encourages agencies to  continue actively engaging partners and participants, and help direct  participants to other initiatives that might interest them. The toolkit also  encourages agencies to create opportunities for participants to socialize  and communicate with each other by supporting discussion forums, and  identifying group leaders that can help carry forward a discussion among  members of the community.

		Guidance Does Not Reflect Practices Due to Differing Approaches Used to Develop Them, among Other Factors

In instances where guidance does not fully reflect the practices and key  actions identified in our October 2016 report, agency staff may not be  aware of certain steps they should take to better ensure the success of  their open innovation initiatives. In part, the various guidance resources  developed by OMB, OSTP, and GSA do not fully reflect some practices  and key actions because those resources almost all pre-date our report.  We also used different methodologies and sources in pulling together our  practices and key actions than they did in developing their guidance. As  noted earlier, we developed our practices by analyzing and synthesizing  suggested practices and key actions from a wide range of relevant  literature, as well as interviews with nongovernmental experts and agency  officials. Our scope also was to identify practices applicable to  implementing individual initiatives using any open innovation strategy.  The guidance developed by OMB, OSTP, and GSA, by contrast, is  generally strategy specific and based on the experiences of and lessons  learned from individuals involved in the relevant communities of practice.  For example, as was highlighted earlier, OSTP and GSA staff worked  with volunteers from the Challenges and Prizes Community of Practice to  develop the Challenges and Prizes Toolkit.
Other factors contributed to the guidance for open data collaboration not  fully reflecting our practices and key actions for effectively implementing  open innovation initiatives. As was previously described, OMB, OSTP,  and GSA developed various resources, including websites and guidance,  to help agencies inventory, manage, and release their open data.  According to staff from these agencies, the focus of their efforts and  resources generally has been assisting agencies with the management  and release of open data, while providing some support for agencies  interested in mobilizing and collaborating with participants to use open  data (i.e., open data collaboration), which we identify as an open  innovation strategy. The existing guidance does suggest some actions for  agencies to take when implementing open data collaboration initiatives. In  addition, staff from OMB’s Office of the Chief Information Officer and  GSA’s TTS shared several ways in which they and members of the Open  Data Working Group provide support for staff interested in carrying out  open data collaboration initiatives. For example, they told us that when  agency staff ask for advice on how to conduct such initiatives, they often  match the individual making the request with experienced staff in another  agency, or rely upon members of the Open Data Working Group to  answer their questions. An official from GSA’s TTS stated that because  members of this community meet regularly, there are frequent  opportunities to share best practices and lessons learned to inform the  planning of any current or future agency open data collaboration  initiatives. In addition, the official told us that agency staff can also use  the open data community’s e-mail list to seek and receive assistance from  fellow members. Although OMB, OSTP, and GSA staff identified various  resources that are available to support agency staff interested in carrying  out open data collaboration initiatives, those resources do not include  detailed and consistent, step-by-step guidance for the implementation of  such initiatives.

	Selected Agencies Developed Open Innovation Resources Similar to Those Available Government Wide, Though Tailored to Their Specific Needs

The agencies we selected for this review—DOE, HHS, HUD, DOT, EPA,  and NASA—have put in place various resources to support the use of  open innovation strategies. These resources—policies and  implementation guidance, supporting staff and organizations, and  websites—complement what exists at the government-wide level. The  selected agencies have generally developed, or are developing,  resources for the open innovation strategies they use frequently, to  provide staff with tailored guidance and support. This helps ensure staff  carry out initiatives in a manner that is informed by the agency’s previous  experience and that is consistent with agency procedures. In some  instances, agency officials told us that they have not developed certain  agency-level resources, generally for one or more of the following  reasons:  they found government-wide resources to be sufficient;  they have not used a strategy, or use it infrequently, limiting the need  for agency-specific resources; or  they have not yet had sufficient experience using a strategy to be able  to craft policies or other resources informed by experiences and  lessons learned.

		Selected Agencies Developed Guidance to Clarify Procedures for Developing and Implementing Open Innovation Initiatives

Officials from the selected agencies told us that agency-level policies and  guidance help to raise awareness among agency staff of the value of  using open innovation strategies, and how these strategies can be an  effective tool for engaging the public to generate new ideas and solutions.  In instances where agency leaders have approved and issued these  policies, this also helps demonstrate to agency staff that leadership  supports using these strategies. The policies and guidance at these  agencies (examples provided in table 5) also clarify agency-specific steps  that staff should take when implementing an initiative, which, according to  agency officials, helps ensure that staff take actions necessary to meet  requirements and successfully carry out open innovation initiatives.
As illustrated in table 5, NASA’s Policy Directive on Challenges, Prize  Competitions, and Crowdsourcing Activities, which was signed by the  NASA Administrator and issued in February 2014, encourages agency  staff to use these open innovation strategies. The directive states that it is  NASA’s policy to encourage the use of challenge activities to obtain  solutions and stimulate innovation. According to an official from NASA’s  Center of Excellence for Collaborative Innovation (CoECI), the directive  was created to supplement government-wide policies and guidance. It  defines the roles that certain agency officials, such as the Associate  Administrator for the Space Technology Mission Directorate, are to play in  designing and implementing challenges. It also directs staff to develop  and maintain agency-level best practices and implementation guidance  for challenges. In keeping with this policy, in 2014, CoECI, which provides  advice and support for NASA teams to design, implement, and evaluate  challenges conducted through the NASA Tournament Lab, developed a  Challenge Coordinator Toolkit.
The toolkit provides a detailed checklist for CoECI challenge coordinators  to follow when supporting teams that are conducting those challenges.  According to a CoECI official, staff with varying levels of experience  working on challenges serve as challenge coordinators. The toolkit is  intended to ensure that coordinators consistently follow a set of specific  procedures for managing all phases of a challenge being conducted using  the NASA Tournament Lab, from completing necessary documentation  before a challenge is launched to holding a close-out interview to capture  feedback and lessons learned. See figure 12 for an overview of the steps  highlighted in the toolkit.

		Selected Agencies Created Staff Positions, Organizations, and Communities of Practice to Support Implementation of Open Innovation Initiatives

According to officials at the selected agencies, their agencies have  created staff positions and internal organizations to serve as a source of  information and guidance, sharing agency-specific and government-wide  resources with staff who are conducting open innovation initiatives. In  some instances, agencies have dedicated staff responsible for leading  efforts to use a certain strategy in their agency and providing internal  support to other staff interested in using the strategy. In other instances,  agency officials told us they also have staff who, given their interest and  experience using a strategy, have taken on responsibilities to support  others in their agency, but are not devoted full time to these efforts. Some  agencies also have brought together multiple staff to create internal  organizations responsible for supporting the use of open innovation  strategies. These staff and organizations—selected examples of which  are highlighted in tables 6 and 7, respectively—can provide tailored  advice and support based on their experience with these strategies.  According to agency officials, this can involve helping to ensure staff meet  requirements and take other actions to successfully propose, design, and  implement an open innovation initiative.
As noted in table 6, DOT’s Chief Data Officer (CDO) told us that he  provides technical support for the department’s open data efforts. As part  of his responsibilities, he works with staff to improve how the agency  collects, manages, and publishes data. He also has coordinated and  supported several open data events that have allowed DOT staff to  collaborate with the public, including private sector companies using  transportation data, state and local stakeholders, academic researchers,  and those working on new technologies. He told us that these events  have served to help the agency better understand how people are using  transportation data and what improvements that they would like to see  made. For example, in 2015, the CDO worked with staff from throughout  DOT to organize a Transportation “Datapalooza.” According to the CDO,  the event allowed DOT to both highlight its own data initiatives and learn  about the applications and products that private companies, developers,  and technologists are developing with DOT data. It also gave DOT staff  an opportunity to speak directly with data users, identifying areas where  they might work collaboratively to expand the use of the data and improve  quality.
According to the EPA memorandum that established it in July 2015,  EPA’s Challenge Review Team (ChaRT), mentioned above in table 7, is  intended to help ensure that agency prize competitions and challenges  meet applicable legal requirements, and have an adequate scientific  foundation, financial support, and a clear communications plan. As shown  in figure 13, ChaRT brings together officials from several EPA offices  who, according to an official from EPA’s Office of Research and  Development, need to be involved in prize competitions and challenges,  creating an efficient “one-stop” approach to reviewing and approving  proposals.
According to the official from EPA’s Office of Research and Development,  the officials who serve on ChaRT provide legal, financial, and scientific  expertise and support to EPA teams developing proposals for prize  competition or challenge initiatives. As part of the team’s review  processes, it requires EPA staff to complete a ChaRT review form to  ensure that (1) new initiatives will address agency needs, (2) leadership is  aware of and supports the challenge, and (3) teams have completed  required steps and addressed any concerns before they can proceed.
According to the memorandum that established it, ChaRT members must  concur before any prize competition or challenge can be announced.
In addition to dedicating staff and organizations, according to officials  from selected agencies, some of the agencies also created communities  of practice (COP) for those staff interested or involved in open innovation  initiatives, particularly citizen science. These include NASA’s Citizen  Science COP; EPA’s Citizen Science COP and recently-created Prize  and Challenge COP; and within HHS, citizen science working groups at  the Centers for Disease Control and Prevention and National Institutes of  Health (NIH). Like government-wide communities, those COPs and  working groups at the agency level can provide a venue for sharing  experiences and lessons learned from designing and implementing open  innovation initiatives, or can be used to develop guidance for agency  staff. For example, according to NIH officials, the agency’s Citizen  Science Working Group was established in 2012 to address interest  among NIH staff about how the agency could use crowdsourcing and  citizen science methods. NIH officials told us the working group brings  together, as of March 2017, approximately 60 staff from 14 of the 27 NIH  institutes and centers to discuss various topics and share their  experiences with and knowledge of implementing these initiatives. The  group also hosts presentations by outside speakers.

		Selected Agencies Developed Websites to Reach Specific Audiences about Their Open Innovation Initiatives

Agencies selected for this review have developed websites that, like the  government-wide websites described above, can provide a central,  publicly available location for stakeholders and participants to access  information, including agency data that can be used in open data  collaboration initiatives and details of specific projects in which they can  volunteer to participate. However, officials from DOE, EPA, HHS, and  NASA told us that websites developed by agencies, unlike the  government-wide websites, are often designed to reach specific  audiences and stakeholders that may be likely to participate in the  agencies’ open innovation initiatives and in some cases provide an online  forum for these audiences to collaborate. Table 8 provides examples of  websites that agencies have developed.
For example, DOE’s Open Energy Data website, as shown in figure 14,  provides energy data users and developers with access to DOE’s catalog  of open data sets that they can use to conduct research and develop  applications. In addition, it provides visitors with information on how to  engage with DOE to provide feedback on the department’s open data.  Open Energy Data also highlights and links to Open Energy Information  (OpenEI), a website sponsored by DOE and other partner organizations  that allows users to share datasets and collaborate on energy data  initiatives.
HUD’s Switchboard, an online ideation platform that was also highlighted  in our October 2016 report, allows employees, stakeholders, and other  members of the public to offer ideas for improving HUD’s processes,  programs, and administration. According to a HUD official, the platform  was originally created in 2009 to collect input for an update to the  agency’s strategic plan. Over time, however, HUD expanded the purpose  of the platform to make it a more general forum for the public and external  stakeholders, as well as HUD employees. Now Switchboard is intended  to ensure there is an avenue for anyone to reach HUD with any ideas or  feedback that could improve the department. Beyond that, the website  allows visitors to offer comments on other submissions, and vote in favor  of those ideas that they support. It also helps designated HUD staff  collect, review, and respond to the ideas submitted, after which they will  work to determine whether they can be implemented. Figure 15 illustrates  different aspects of Switchboard’s capabilities.

	Conclusions

OMB, OSTP, GSA, and selected agencies have developed various  resources to support the use of open innovation strategies. Of these,  government-wide guidance is particularly important, as it helps agency  staff understand the full range of actions they should take when  designing, implementing, and evaluating an open innovation initiative.  However, key government-wide implementation guidance, in particular for  open data collaboration, does not always fully reflect demonstrated good  practices and key actions. Better incorporating those practices and key  actions would help ensure agency staff are aware of, and can take, the  full range of steps to effectively design, implement, and assess their open  innovation initiatives. Consistently applying these practices can help  agencies ensure that their initiatives successfully achieve intended  results.

	Recommendations for Executive Action

To help ensure federal agencies effectively design, implement, and  assess open innovation initiatives in line with the practices and key  actions identified in our past report, we make 22 recommendations for  GSA, OMB, and OSTP to enhance relevant implementation guidance.
We recommend that the Director of the Office of Management and  Budget, the Director of the Office of Science and Technology Policy, and  the Administrator of the General Services Administration enhance key  guidance for open data collaboration initiatives to fully reflect the following  15 key actions:
Clearly define the purpose of engaging the public;
Consider the agency’s capability to implement a strategy, including  leadership support, legal authority, the availability of resources, and  capacity;
Define specific and measurable goals for the initiative;  Identify performance measures to assess progress;
Align goals of the initiative with the agency’s broader mission and
Document the roles and responsibilities for all involved in the initiative;
Develop a plan that identifies specific implementation tasks and time  frames, including those for participant outreach and data collection;
Use multiple outlets and venues to announce the initiative;
Craft announcements to respond to the interests and motivations of
Hold regular check-ins for those involved in the implementation of the
Collect and analyze data to assess goal achievement and results of
Conduct an after-action review to identify lessons learned and
Report on results and lessons learned publicly;
Acknowledge and, where appropriate, reward the efforts and  achievements of partners and participants; and
Seek to maintain communication with, and promote communication  among, members of the community.
We recommend that the Administrator of the General Services  Administration enhance key guidance for ideation and open dialogue  initiatives to fully reflect the following four key actions:
Develop a plan that identifies specific implementation tasks and time  frames, including those for participant outreach and data collection;
Hold regular check-ins for those involved in the implementation of the
Report on results and lessons learned publicly; and
Acknowledge the efforts and achievements of partners that have  contributed to the implementation of an initiative.
We recommend that the Director of the Office of Science and Technology  Policy and Administrator of the General Services Administration enhance  key guidance for crowdsourcing and citizen science initiatives to fully  reflect the key action to acknowledge the efforts and achievements of  partners that have contributed to implementing an initiative.
We recommend that the Director of the Office of Management and  Budget and the Administrator of the General Services Administration  enhance key guidance for prize competitions and challenges to fully  reflect the following two key actions:
Hold regular check-ins with those involved in the implementation of an  initiative; and
Acknowledge the efforts and achievements of partners who  contributed to the implementation of an initiative.

	Agency Comments

We provided a draft of the report to the Director of the Office of  Management and Budget, the Acting Director of the Office of Science and  Technology Policy, the Acting Administrator of the General Services  Administration (GSA), the Secretary of Energy, the Secretary of Health  and Human Services, the Secretary of Housing and Urban Development,  the Secretary of Transportation, the Administrator of the Environmental  Protection Agency, and the Acting Administrator of the National  Aeronautics and Space Administration for comment.
In its comments, reproduced in appendix II, GSA agreed with the  recommendations in this report.
Staff from OMB’s Office of the Federal Chief Information Officer and  Office of General Counsel provided oral comments on May 15, 2017,  stating that OMB generally agreed with the recommendations in this  report.
In comments provided by email, OSTP’s General Counsel stated that  OSTP neither agreed nor disagreed with the recommendations in this  report. She stated that, given their past and ongoing responsibilities  related to open innovation in the federal government, OMB and GSA are  best positioned to address these recommendations. She further stated  that OSTP may support OMB and GSA in these efforts to the extent that  OSTP has appropriate staff in the future.
EPA, NASA, and OMB provided technical comments, which we  incorporated as appropriate. DOE, DOT, HHS, and HUD informed us that  they had no comments.
We are sending copies of this report to interested congressional  committees, the heads of the agencies identified above and other  interested parties. This report will also be available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of our report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

As part of the federal performance management framework originally put  into place by the Government Performance and Results Act of 1993  (GPRA), and updated and expanded by the GPRA Modernization Act of  2010 (GPRAMA), agencies are to identify the strategies and resources  they will use to achieve their goals. GPRAMA also includes a provision  for us to periodically review how implementing the act’s requirements is  affecting agency performance. This report is part of our response to that  mandate, and builds on our October 2016 report that described how  agencies are using open innovation strategies, and identified practices to  ensure they are implemented effectively. Specifically, this report (1)  identifies key government-wide resources the Office of Management and  Budget (OMB), the Office of Science and Technology Policy (OSTP), and  the General Services Administration (GSA) put in place to support the use  of open innovation strategies; (2) examines the extent to which key  government-wide guidance reflects practices for effectively implementing  these strategies; and (3) identifies resources that selected agencies have  developed to support the use of open innovation strategies.
To identify the various government-wide resources that OMB, OSTP, and  GSA have developed, we reviewed relevant policy and guidance  documents related to each open innovation strategy (see table 9 below).  We also reviewed information available on relevant websites, including  Citizenscience.gov, Challenge.gov, Digitalgov.gov, Data.gov, and Project  Open Data, and corroborated information collected from these websites  through interviews with relevant agency staff. Lastly, we interviewed staff  from these agencies involved in efforts to support and encourage the use  of various open innovation strategies across the federal government. In  these interviews, we asked staff to identify key government-wide policies  and guidance for each type of strategy, and to describe their roles and  responsibilities in supporting and encouraging agency use of those  strategies. We also asked about their collaboration with staff in other  agencies, among other things.
To determine the extent to which government-wide policies and guidance  reflect practices and key actions for effectively implementing open  innovation strategies, we first, in consultation with OMB, OSTP, and GSA  staff, identified those that were considered key for each type of strategy.  These key policies and guidance are listed in table 9.
For each strategy type, we then compared key guidance contents to the  seven practices and 18 related key actions, listed in figure 16, that we  identified in October 2016. We identified those practices and key actions  by analyzing and synthesizing suggested practices from relevant federal  guidance and literature, including public and business administration  journals, and publications from research organizations, as well as  interviews with experts and agency officials with experience implementing  open innovation initiatives.
We then analyzed the content of the implementation guidance for each  strategy to determine the extent to which it reflects the practices and key  actions we previously identified. First, two analysts reviewed each source  of guidance to identify all excerpts suggesting actions in line with those  we identified. Next, a third analyst separately reviewed the guidance  documents to verify the accuracy of the initial determinations, or identify  those areas in need of additional discussion. The analysts involved in the  first and second stages of this analysis then made final determinations  about whether the actions suggested in the guidance reflected those we  identified. We considered guidance to fully reflect a key action when it  suggested steps in line with those outlined in our report. If guidance  suggested some but not all steps, we considered it to partially reflect the  key action. Finally, when guidance did not suggest any steps in line with a  key action, we considered guidance to not reflect it.
Lastly, to identify the resources that selected agencies have put in place,  we reviewed agency policies and guidance, and websites related to the  use and implementation of open innovation strategies from six agencies:  the Departments of Energy (DOE), Health and Human Services (HHS),  Housing and Urban Development (HUD), and Transportation (DOT); the  Environmental Protection Agency (EPA); and the National Aeronautics  and Space Administration (NASA). We selected these agencies based on  various criteria, including the number and variety of open innovation  strategies outlined in their individual agency Open Government Plans.  These selections were also in line with suggestions from knowledgeable  staff at OMB, OSTP, and GSA familiar with agencies that have actively  used such strategies.
In interviews with officials from these agencies, we asked them to identify  agency-specific policies, procedures, or other guidance that has been  developed to aid staff carrying out open innovation initiatives. Through  this consultation, we were able to identify relevant policy documents and  guidance developed by these agencies. For instance, we identified EPA’s  memorandum establishing the agency’s Challenge Review Team, which  is responsible for approving proposals for the agency’s prize competitions  and challenges. Similarly, we were able to use this consultation to identify  the Federal Highway Administration’s Public Involvement Techniques for  Transportation Decision-Making guide, which provides staff with guidance  on how to use various open dialogue approaches. We were also able to  use these interviews to identify relevant websites the agencies have  developed that support open innovation efforts. For instance, we  identified the openNASA website, which provides the public with access  to NASA’s datasets, code, and other tools, as well as information on  opportunities to collaborate with others on the use of NASA’s data.  Similarly, we also used this consultation to identify the National Institutes  of Health’s (NIH) Biomedical Citizen Science Hub website, which is an  online space for researchers and stakeholders interested in the use of  citizen science in biomedicine to collaborate.
In our interviews with officials supporting the use of various open  innovation strategies at these six agencies, we asked them to describe,  among other things:  the steps staff would go through to design, implement, and evaluate a  specific type of open innovation strategy;  the support available to staff throughout that process; and  the ways resources at the government-wide level have helped support  the use of open innovation strategies in their agency.
We conducted this performance audit from February 2016 to June 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Comments from the General Services Administration

Appendix III: GAO Contact and Staff Acknowledgments

	GAO Contact

	Staff Acknowledgements

In addition to the contact named above, Benjamin T. Licht (Assistant  Director) and Adam Miles supervised the development of this report.  Steven Putansu, Lauren Shaman, Erik Shive, Wesley Sholtes, and  Andrew J. Stephens made significant contributions to this report. John  Hussey and Donna Miller also made key contributions. James Cook, Eric  Gorman, Danielle Novak, and Jason Vassilicos verified the information in  the report.

Summary:
Why GAO Did This Study

To address the complex and crosscutting challenges facing the federal government, agencies need to effectively engage and collaborate with those in the private, nonprofit, and academic sectors; other levels of government; and citizens. Agencies are increasingly using open innovation strategies for these purposes.
The GPRA Modernization Act of 2010 requires agencies to identify strategies and resources they will use to achieve goals. The act also requires GAO to periodically review how implementation of the act's requirements is affecting agency performance. This report identifies the open innovation resources developed by GSA, OMB, OSTP, and six selected agencies, and examines the extent to which key guidance reflects practices for effective implementation.
To address these objectives, GAO identified various resources by reviewing relevant policies, guidance, and websites, and interviewing staff from each agency. GAO selected the six agencies based on several factors, including the number and type of open innovation initiatives outlined in their agency Open Government Plans. GAO also compared guidance to practices and key actions for effective implementation.

What GAO Found

Open innovation involves using various tools and approaches to harness the ideas, expertise, and resources of those outside an organization to address an issue or achieve specific goals. Agencies have frequently used several open innovation strategies—crowdsourcing and citizen science, ideation, open data collaboration, open dialogues, and prize competitions and challenges—to engage the public. Staff from the General Services Administration (GSA), Office of Management and Budget (OMB), and Office of Science and Technology Policy (OSTP) developed resources to support agency use of these strategies:
to encourage use, clarify legal authorities, and suggest actions for designing and implementing an open innovation initiative;
staff to advise and assist agency staff implementing initiatives and open innovation-related communities of practice; and
websites to improve access to relevant information and potential participants.
Six selected agencies—the Departments of Energy, Health and Human Services, Housing and Urban Development, and Transportation; the Environmental Protection Agency; and the National Aeronautics and Space Administration—also developed resources for those strategies they frequently use. These resources complement those at the government-wide level, providing agency staff with tailored guidance and support to help ensure they carry out initiatives consistent with agency procedures.
For the open innovation strategies identified above, GAO determined that key government-wide guidance developed by GSA, OMB, and OSTP reflect to differing extents practices GAO previously identified for effectively implementing specific initiatives (see table). Several factors led to these variances, including differing scopes and methodologies used in their development, and when they were issued. Better incorporating these practices could help ensure agency staff are aware of, and are able to take, steps to effectively design, implement, and assess their initiatives.

What GAO Recommends

GAO recommends GSA, OMB, and OSTP enhance key guidance for each open innovation strategy to fully reflect practices for effective implementation. GSA and OMB generally agreed with these recommendations. OSTP neither agreed nor disagreed with the recommendations.