Open Innovation: Practices to Engage Citizens and Effectively Implement Federal Initiatives
Published: 2016-10-13T00:00:00

Fast Facts:
How can the federal government better engage citizens?
Federal agencies are using "open innovation" tools to leverage the knowledge and skills of people outside government. Using dedicated websites and in-person outreach, agencies have worked with the public to rebuild communities after Hurricane Sandy, improve methods to find asteroids that could threaten the Earth, and reduce the amount of time required for highway construction projects.
We identified 7 practices that agencies can use to effectively engage the public when using open innovation tools.

Highlights:
section_title: Why GAO Did This Study
paragraphs: To address the complex and crosscutting challenges facing the federal government, agencies need to effectively engage and collaborate with those in the private, nonprofit, and academic sectors, other levels of government, and citizens. Agencies are increasingly using open innovation strategies for these purposes.
The GPRA Modernization Act of 2010 (GPRAMA) requires federal agencies to identify strategies and resources they will use to achieve their goals. GPRAMA also requires GAO to periodically review how implementation of its requirements is affecting agency performance. This report identifies and illustrates practices that help agencies effectively implement open innovation strategies, and how the use of those strategies has affected agency performance and opportunities for citizen engagement.
To identify these practices, GAO analyzed relevant federal guidance and academic literature, and interviewed open innovation experts. To refine and illustrate the practices, GAO reviewed documents and interviewed officials from the Office of Management and Budget, Office of Science and Technology Policy, General Services Administration, and six selected federal agencies. GAO selected the agencies and a sample of their initiatives based on several factors, including the number and type of initiatives outlined in their Open Government Plans.
section_title: What GAO Found
paragraphs: Open innovation involves using various tools and approaches to harness the ideas, expertise, and resources of those outside an organization to address an issue or achieve specific goals. GAO found that federal agencies have frequently used five open innovation strategies to collaborate with citizens and external stakeholders, and encourage their participation in agency initiatives.
GAO identified seven practices that agencies can use to effectively implement initiatives that involve the use of these strategies:
Select the strategy appropriate for the purpose of engaging the public and the agency’s capabilities.
Clearly define specific goals and performance measures for the initiative.
Identify and engage external stakeholders and potential partners.
Develop plans for implementing the initiative and recruiting participants.
Engage participants and partners while implementing the initiative.
Collect and assess relevant data and report results.
Sustain communities of interested partners and participants.
Aspects of these practices are illustrated by the 15 open innovation initiatives GAO reviewed at six selected agencies: the Departments of Energy, Health and Human Services, Housing and Urban Development, and Transportation (DOT); the Environmental Protection Agency; and the National Aeronautics and Space Administration (NASA). For example:
With the Asteroid Data Hunter challenge, NASA used a challenge and citizen science effort, beginning in 2014, to improve the accuracy of its asteroid detection program and develop an application for citizen scientists.
Since 2009, DOT’s Federal Highway Administration has used an ideation initiative called Every Day Counts to identify innovations to improve highway project delivery. Teams of federal, state, local, and industry experts then implement the ideas chosen through this process.

section_title: Letter
paragraphs: The complex and crosscutting nature of many challenges facing the  federal government has highlighted the need for agencies to engage and  collaborate with the public and different sectors of society to address  them. The advent of online technologies has enhanced the ability of  federal agencies to make these connections and provide opportunities to  address challenges through open innovation. Open innovation involves  using various tools and approaches to directly engage with people and organizations in the private, nonprofit,  and academic sectors; and harness their ideas, expertise, and resources to address an issue and  achieve specific goals.
The term “open innovation” was first widely used around 2003 to describe  efforts by companies to solicit external ideas, product designs, and  solutions. Since then, it has been used extensively in academia and the  private sector. In addition, the Executive Branch has used this term since  at least 2011 to characterize efforts to access the skills and contributions  of citizens and other external stakeholders.
In recent years, the Executive Branch and Congress have taken actions  aimed at encouraging and enhancing federal agency use of open  innovation. For example, the Presidential Memorandum on Transparency  and Open Government and the Office of Management and Budget’s  (OMB) Open Government Directive directed agencies to describe how  they would use new feedback mechanisms, technology platforms, and  other innovative methods to obtain ideas from, and increase collaboration  with, those outside the federal government. In December 2010, Congress  passed the America COMPETES Reauthorization Act of 2010. This  legislation, signed into law by the President in January 2011, provides,  among other things, government-wide authority for executive branch  agencies to use public prize competitions to advance their missions.  Additionally,  bills have been introduced in the 114th Congress related to  other types of open innovation strategies. For example, the  Crowdsourcing and Citizen Science Act of 2015, the American Innovation  and Competitiveness Act, and the Aeronautics Innovation  Act contain  provisions on crowdsourcing or citizen science.
As part of the federal performance management framework originally put  into place by the Government Performance and Results Act of 1993  (GPRA), and updated and expanded by the GPRA Modernization Act of  2010 (GPRAMA), agencies are to identify the various strategies and  resources they will use to achieve their goals. GPRAMA  also includes a  provision for us to periodically review how implementation of its  requirements is affecting agency performance. This report is part of our  response to that mandate. Our specific objective for this report is to  identify, and illustrate through selected agency examples, practices that  promote the effective implementation of open innovation strategies and  the effects, if any, the use of those strategies had on agency performance  and opportunities for citizen engagement.
To identify practices that can facilitate the effective implementation of  open innovation strategies, we analyzed and synthesized information  gathered from:  federal resources, including guidance with suggested practices for  implementing various open innovation strategies released by OMB,  the Office of Science and Technology Policy (OSTP), and the General  Services Administration (GSA); a review we conducted to identify literature with suggested practices  for implementing open innovation strategies, which covered public  and business administration journals, and publications from research  organizations;  interviews we conducted with 14 open innovation experts with  experience implementing open innovation initiatives, or with academic  or consultative expertise in this area; and  interviews we conducted with officials involved in implementing open  innovation initiatives at six selected federal agencies, as well as staff  from OMB, OSTP, and GSA.
To illustrate aspects of the practices we developed, and identify how open  innovation strategies can affect agency performance and citizen  engagement, we selected 15 initiatives that involved the use of open  innovation strategies at 6 agencies: the Departments of Energy (DOE),  Health and Human Services (HHS), Housing and Urban Development  (HUD), and Transportation (DOT); the Environmental Protection Agency  (EPA); and the National Aeronautics and Space Administration (NASA).  We selected these agencies based on various criteria, including the  number and variety of open innovation strategies outlined in their  individual agency open government plans. These selections also aligned  with suggestions from knowledgeable experts and staff at OMB, OSTP,  and GSA. We identified and selected initiatives that offered the greatest  potential to illustrate a range of practices based on our review of the open  government plans for the selected agencies, and input from  knowledgeable agency staff. These initiatives are listed below in table 1.
To develop the illustrative examples in this report, we obtained and  reviewed agency documentation related to the initiatives, and interviewed  relevant agency officials.
The scope of this review was to identify practices for effectively  implementing open innovation initiatives, and to describe actions  agencies took in carrying out open innovation initiatives that reflect  aspects of those practices. While we present information on the  implementation of agency open innovation initiatives, we did not assess  the success of the underlying agency programs and activities that these  initiatives were designed to support. See appendix I for additional details  about our scope and methodology.
We conducted this performance audit from July 2015 to October 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The federal performance management framework put into place by GPRA  and GPRAMA requires agencies to develop long-term strategic plans that  identify their missions, along with long-term goals and objectives (often  referred to as strategic goals and objectives) aimed at achieving their  missions. Agencies are to develop performance plans with near-term  goals annually, to show progress towards their long-term goals and  objectives. These near-term goals are called performance goals. In both  of these plans, agencies are directed to identify the various strategies and  resources they will use to achieve their goals.
In line with these requirements, the Open Government Directive instructs  federal agencies to develop Open Government Plans detailing the  strategies and initiatives they would use to improve public engagement  and collaboration on the agency’s core mission activities. It directs  agencies to describe how they would use innovative feedback  mechanisms, technology platforms, and such methods as prize  competitions to increase opportunities for public participation and  collaboration with those outside the agency and in other levels of  government. These outside parties include those in the private, nonprofit,  and academic sectors. Agencies were directed to release their initial  plans in 2010, and to update these plans every 2 years. In July 2016,  OMB released guidance for the development of 2016 Open Government  Plans, which were to be published in September 2016. The new  guidance instructs agencies to describe their activities to increase the use  of open innovation initiatives. In early 2010, OMB also created an  Interagency Open Government Working Group to provide a forum for  open government professionals to share best practices across agencies.  Representatives from 41 federal agencies made up the initial working  group.
OMB, OSTP, and GSA have taken additional steps to support and  encourage agency use of open innovation strategies. They have  developed specific policy and guidance documents, built websites that  facilitate their use, and supported knowledge sharing communities of  practice. For example:  In July 2010, GSA launched Challenge.gov. This site is designed to  help agencies find participants for prize competitions and challenges  by providing a centralized list of all competitions sponsored by federal  agencies. After the America COMPETES Reauthorization Act  authorized federal agencies to conduct prize competitions, OMB  issued guidance in August 2011 to help agencies use this authority.  GSA also hosts the Challenges and Prizes Community of Practice.  This group meets quarterly to discuss policies and procedures, and  share ideas and practices. According to information from  Challenge.gov, agencies have conducted more than 700 distinct prize  competitions or challenges since the site was first launched in 2010.
In May 2013, the President released an executive order requiring  OMB to issue an Open Data Policy. This policy, also released by  OMB in May 2013, directs agencies to collect or create information  using open formats that are non-proprietary and publicly available,  and to build or modernize information systems in a way that  maximizes the accessibility of information. The President’s  executive order also called for the creation of an Open Data Cross- Agency Priority Goal, which is designed, among other things, to  provide support to help agencies release high priority data sets and  facilitate the use of open data by those outside the agency. In May  2014, the administration also released an Open Data Action Plan.  This plan called on agencies to use online and in-person mechanisms  to engage with open data users and stakeholders to prioritize open  data sets for release, improve data based on feedback, and  encourage its use. OMB and OSTP have created a website called  Project Open Data to provide good practices and examples to assist  agencies. OMB, OSTP, and GSA also manage the Open Data  Working Group, which meets every 2 weeks to share best practices  and tools, and allow agencies to learn from one another.
In September 2015, OSTP released a memorandum that outlined  principles agencies should use when designing a crowdsourcing or  citizen science initiative. The memorandum also outlined actions the  agencies should take to build their respective agency capacity to use  that type of strategy. At the same time, OSTP released the  Crowdsourcing and Citizen Science Toolkit with practices, lessons  learned, and case studies to inform agency efforts to design,  implement, and sustain these initiatives. GSA has also launched  Citizenscience.gov, which is a centralized repository of information on  agency citizen science initiatives. As of September 2016, the  Crowdsourcing and Citizen Science Catalog on Citizenscience.gov  lists 303 active crowdsourcing and citizen science projects across 25  agencies. Lastly, practitioners from across the federal government  have come together to form the Federal Community of Practice for  Crowdsourcing and Citizen Science, which meets monthly to share  lessons learned and practices for implementing and evaluating  crowdsourcing and citizen science initiatives.
In some agencies, this government-wide infrastructure has been  supplemented by agency-level policies and organizations with dedicated  staff and resources. For instance, NASA has created the Center of  Excellence for Collaborative Innovation  (CoECI), which assists teams  from NASA and other agencies with implementing open innovation  strategies, particularly prize competitions and challenges.
section_title: Practices That Facilitate the Effective Implementation of Open Innovation Initiatives
paragraphs: Based on our review of agency Open Government Plans and other  sources, we found that agencies have frequently used the five open  innovation strategies shown below to collaborate with citizens and  external parties, and encourage their participation in agency efforts.  Figure 1 identifies and describes these strategies, and we provide further  information about them in appendix II.
Agencies can use these strategies singularly, or in combination as part of  a larger open innovation initiative. For example, an open innovation  initiative could primarily involve a prize competition or challenge that also  has an idea generation component focused on the identification of  promising new ideas or issues to be addressed. It could also have a  component where participants are asked to use open data to develop  new products or applications based on those ideas.
We identified seven practices that federal agencies can use to help  effectively design, implement, and assess open innovation initiatives.  These practices are detailed below. We drew from our analysis of federal  resources and relevant literature with suggested practices for the  implementation of open innovation strategies. We also interviewed  experts and agency officials with expertise in implementing such  initiatives. While we present these practices in a certain order, this is not  meant to imply they should be implemented in this sequence. Relevant  literature, agency officials, and an expert we consulted emphasized that,  in practice, agencies often take some of these actions concurrently or will  use an iterative approach.
subsections: section_title: Select the Strategy Appropriate for the Purpose of Engaging the Public and the Agency’s Capabilities
paragraphs: Through our analysis of relevant literature and interviews with experts, we  identified several factors agency officials should consider when selecting  the most appropriate open innovation strategy or strategies to use for an  initiative. First, agency officials considering the use of an open innovation  strategy should clearly articulate the purpose(s) they hope to achieve by  engaging the public. Through our literature review and interviews we  found that agencies generally used open innovation strategies to achieve  one or more of five high-level purposes. These purposes, which are not  mutually exclusive, are summarized below in table 2. implement  a strategy, including leadership  support, legal authority, the availability of  resources, and capacity.
Description  Agencies can collect the perspectives of a broad group of citizens and external stakeholders to  identify problems or challenges, gauge perceptions of a program or service, gather reactions to  proposed actions, or better understand their priorities, values and preferences. Agencies can  then use this information to inform decisions about policies, plans, and the allocation of  resources.  Agencies can efficiently engage a broad range of citizens and external stakeholders in  developing new ideas, solutions to specific problems, or new products ranging from software  applications to physical devices. Agencies can also have them evaluate the quality and feasibility  of the ideas and solutions proposed by others, or test the products that were developed. If it  uses a successive or iterative process, the agency can help build the capacity of participants in  these efforts to further develop or refine their ideas or products. Agencies can also use open  innovation initiatives to stimulate the creation of new markets and companies that will then  commercialize products and technologies developed for an initiative.  Agencies can leverage the time, resources, and expertise of citizens and external stakeholders  to supplement their own internal resources, data, and expertise. These contributions enhance  the agency’s capacity, and therefore, its ability to achieve goals that would be more difficult to  reach without this additional capacity or expertise. Open innovation initiatives may also allow  agencies to achieve goals more efficiently and effectively  than more traditional federal program  types, such as grants or contracts.  Agencies can establish or enhance collaboration among citizens and external stakeholders or  organizations interested in an issue. This can be done, in part, by developing relationships  among involved individuals and organizations. These relationships can then be leveraged to  achieve common or complementary goals. Agencies can also enhance previously-established  communities by using open innovation initiatives to strengthen existing relationships. This also  can be done to bring new individuals and organizations into the community.  An agency can provide participants or the broader public with balanced and objective information  and data to help them understand an issue or problem. Information can also be provided to help  them understand opportunities and various alternatives for addressing an issue or problem.
To determine how frequently agencies identified these as purposes for  each type of open innovation strategy, we identified both the primary  strategy and the purposes agencies articulated for each initiative  in their  most recent open government plans. The results of this analysis are  summarized below in figure 2.
Through this analysis, we found that agencies identified certain purposes  more frequently for different types of strategies. For example, we found  that, of the 26 prize competitions or challenges identified in agency plans,  agencies indicated that developing new ideas, products, or solutions was  a specific purpose for 25 (or 96 percent) of the initiatives. Similarly, of the  74 open dialogue initiatives we identified in agency plans, agencies  indicated that collecting information and perspectives was a specific  purpose for 57 (77 percent) of them.
In addition to the purpose(s) agency officials hope to achieve through  open innovation, through our literature review and interviews we identified  additional factors agency officials should consider when selecting the  strategy or strategies that will be used:
Leadership support: The support and approval of agency leaders for  the potential use of an open innovation strategy is particularly  important. Such leadership support can lend credibility and visibility,  help generate support from others throughout the agency, and  increase the likelihood an initiative will receive necessary approvals  and resources.
Legal authorities: Agency officials should work with their respective  agencies’ legal staff to ensure that they have appropriate legal  authority to use a strategy, and are aware of any relevant  requirements that need to be met as they work to implement a  strategy. For instance, the legal requirements that an agency must  meet when conducting a prize competition or challenge can be more  detailed and specific than those that apply to certain other open  innovation strategies. Those considering a strategy should also be  aware of any government-wide and agency-specific policies or  guidance that can help guide planning and implementation of these  tools.
Resource needs and availability: Agency officials should also work  with other relevant staff to understand what financial and information  technology resources are necessary and available to support the use  of various open innovation strategies. For example, agency officials  could work with staff to understand whether they can design or  leverage an existing website or other tool to engage and manage a  community of widely-dispersed participants. Assessing resource  needs and availability helps determine the costs and feasibility of  implementing the selected strategy.
Capacity to implement the strategy: Agency officials should  consider whether their staff has sufficient time and expertise to design  and implement a strategy. Agency officials could work with staff with  prior experience developing and implementing open innovation  initiatives. Such staff can help ensure successful practices from  previous initiatives are replicated and previously-identified problems  avoided. Similarly, officials can also work with agency contracting and  acquisition staff to contract for additional capacity and expertise to  support implementation.
Below we provide illustrative examples of how NASA, EPA, and DOT  selected and used various open innovation strategies to achieve specific  purposes.
subsections: section_title: NASA Used a Challenge and Citizen Science Initiative to Develop Software and Enhance Agency Capacity
paragraphs: As part of NASA’s strategic goal to expand the frontiers of knowledge,  capability, and opportunity in space, the agency is examining near-Earth  asteroids to determine whether any of these objects threaten Earth. In  June 2013, NASA also announced its Asteroid Grand Challenge, which is  a large-scale effort to use partnerships and collaboration to find all  asteroid threats to human populations. NASA officials also reported that  the algorithm that astronomers have been using to analyze images of  space to detect asteroids can produce false detections, and the process  to screen out those false detections is labor intensive and inefficient.
According to NASA officials, beginning in January 2014, staff working on  the Asteroid Grand Challenge began working with staff from NASA’s  CoECI, who have expertise in executing prize competitions and  challenges and are responsible for managing competitions launched  through the NASA Tournament Lab. To ensure they could access  necessary technical expertise to develop an improved algorithm to  identify asteroids in images captured by ground-based telescopes,  officials decided to leverage an existing NASA contract with Harvard  University to carry out a series of competitions. These competitions were  conducted by Harvard University’s subcontractor Topcoder, a private- sector company that administers contests in computer programming and  has an existing community of expert developers and data scientists.
In March 2014, NASA officially announced the Asteroid Data Hunter  challenge and citizen science effort to develop the more accurate  algorithm. The effort was also designed to develop a software application  that would allow citizen scientists to genuinely contribute to asteroid  detection, supplementing the efforts of professional astronomers.  According to an April 2015 report from OSTP on the implementation of  federal prize competitions and challenges, through the challenge’s 10  months, more than 1,200 participants submitted 700 potential solutions.  This resulted in the development of a new algorithm and software  package. Figure 3 provides a screenshot from the website where  interested members of the public can download the application.
According to NASA, the improved algorithm has led to a faster, more  accurate asteroid detection process. NASA and Planetary Resources,  Inc., a private-sector company also involved in the initiative, analyzed the  results and found that the new algorithm resulted in a 15 percent increase  in the positive identification of new asteroids in the main belt of asteroids  that orbit between Mars and Jupiter. Furthermore, NASA also stated that  the software application could increase the number of new asteroids  discovered by citizen astronomers. According to NASA officials, the  application has been downloaded over 8,000 times as of March 2016.  NASA obtained these results with a total project cost of less than  $200,000, which OSTP reported and NASA officials confirmed is less  than the fully loaded cost of employing an engineer for the same time  period.
section_title: EPA Collaborated with Partners on a Challenge to Produce Visualizations Raising Public Awareness about Water Pollution
paragraphs: Excessive levels of nutrients such as nitrogen and phosphorus can harm  aquatic environments, according to EPA. Governments, academic  research organizations, environmental organizations, utilities, and the  agriculture community are collecting data on nutrient levels. However,  EPA and its partners say the general public cannot easily access or  understand these data. To raise public awareness and identify new and  innovative ways to communicate data on nutrient pollution to the public,  EPA collaborated with the United States Geological Survey (USGS) and  Blue Legacy International  to conduct the Visualizing Nutrients Challenge.  USGS is a scientific organization within the Department of the Interior that  collects and distributes scientific data and information on the health of  ecosystems and the environment. Blue Legacy International  is a non- profit organization focused on the protection of water resources.
The goal of the challenge, which ran from April to June 2015, was to  invite participants to design innovative and compelling web applications,  images, and videos to help individuals and communities understand the  causes and consequences of, and solutions to, nutrient pollution.  According to EPA officials, as the idea came together for an effort to  identify innovative ways to translate and communicate information about  nutrient pollution, EPA staff reached out to colleagues at USGS to gauge  their interest in partnering. EPA officials said they did this because of the  role USGS plays in collecting data on the nation’s surface and ground  waters, and their interest in seeing those data communicated and used  more broadly. According to EPA officials, this relationship with USGS was  important because of the additional expertise and capacity USGS staff  provided, as well as their support in publicizing the challenge. EPA  officials explained that because of Blue Legacy International’s  mission  and interest in using digital media to build public awareness about the  importance of local watersheds and more sustainable stewardship of  water resources, it approached EPA about becoming involved. EPA  officials also stated that Blue Legacy International  provided $10,000 to  fund its own independently-selected awards to help incentivize  participation.
According to EPA officials, they determined that conducting a challenge  with an open call for submissions would be the preferred approach to  achieve the goals established for the effort. Before the challenge could  move forward it had to be reviewed and approved by all members of  EPA’s Challenge Review Team. EPA officials explained that the review  team consists of representatives from key offices throughout EPA. This  includes individuals from the Office of General Counsel, who determine  whether there is sufficient statutory authority to carry out a challenge, and  the Office of the Chief Financial Officer, who ensure there are sufficient  financial resources available to support the challenge.
To secure additional capacity to implement the challenge, EPA contracted  with InnoCentive,  a private-sector contractor that manages prize  competitions and challenges. According to EPA officials, they also used  the contract to access InnoCentive’s large existing network of potential  challenge participants with expertise in relevant disciplines, including  design, physical science, and data analysis. InnoCentive played a central  role by recruiting potential participants, assisting with design and  development, and prioritizing issues that needed to be addressed each  week by EPA, USGS, and Blue Legacy International.
According to EPA officials, a competition was selected because it offered  a superior cost-benefit ratio to more traditional federal contracting.  According to an August 2016 report from OSTP, using this approach,  EPA and USGS were able to collect 20 submissions. EPA officials said  these submissions provided a wide range of examples for how to present  and communicate data on nutrient pollution. They also said they achieved  this in approximately 3 months and at the cost of staff time—with  responsibilities shared among EPA and USGS, and Blue Legacy  International—and  $16,500 that EPA paid to InnoCentive  to administer  the competition. By contrast, EPA officials estimated that using traditional  procurement processes to produce a single visualization would have cost  significantly more and taken longer. In addition, they said a more  traditional procurement may not have resulted in a product of the quality  that was received through the competition.
section_title: DOT Used Online and In- Person Open Dialogues to Collect Information from Stakeholders and Increase Awareness about Freight Transportation
paragraphs: The Moving  Ahead for Progress in the 21st Century Act (MAP-21), signed  into law in July 2012, required DOT to develop a National Freight  Strategic Plan in consultation with stakeholders. As we have reported,  involving stakeholders in strategic planning can help ensure that efforts  and resources are targeted at the highest priorities, and that stakeholders  appreciate how competing demands and resource limitations require  careful balancing.
To inform the development of the freight strategic plan, DOT officials, led  by staff from the Office of the Secretary, the Office of Public Engagement,  and the Federal Highway Administration (FHWA) Office of Freight  Management and Operations, decided to engage a broad range of  stakeholders through a series of both online and in-person open  dialogues. For example, beginning in 2012, DOT used an online platform  called IdeaScale to launch an online dialogue and roundtables to  leverage web-based communications technology to engage with  stakeholders. According to DOT officials, the online dialogue session and  online roundtables allowed stakeholders to comment and provide  suggestions on various topics, including developing guidance for state  freight plans and potential measures of conditions and performance for a  national freight system. FHWA has also continued to conduct monthly  webinars to provide information on freight issues, technical assistance,  and training for those in the freight and transportation planning  communities. Since 2012 these webinars have been used to cover a  range of topics, including freight-related provisions in MAP-21 and other  legislation, state freight planning, and improving freight system  performance in metropolitan areas.
In addition to its web-based outreach, DOT also used in-person meetings  to engage with and collect recommendations from a range of  stakeholders. For example, in May 2013 the then-Secretary of  Transportation chartered the National Freight Advisory Committee  (NFAC), which was comprised of 47 stakeholders from different  organizations and groups with an interest in freight policy. It included  representatives from state and local governments, port and transportation  authorities, transportation-related companies and associations, unions,  and public interest groups. DOT officials emphasized that NFAC was  created to advise the department on matters related to freight  transportation. They added that it was critical to ensure a wide range of  perspectives would be represented. NFAC met in person 7 times between  June 2013 and November 2015, and ultimately provided DOT with nearly  100 recommendations. DOT leaders also conducted nearly 60  roundtables and public meetings across the country to collect the  perspective of stakeholders at the regional and local levels on various  freight policy issues.
According to DOT officials, the insights collected through this outreach  had a large influence on the development of the draft National Freight  Strategic Plan, which was released in October 2015. DOT officials told us  that they received substantial public input on issues such as freight  transportation safety, the adoption of new technologies, workforce  development, opportunities to strengthen connections between different  modes of transportation, and the need for reliable funding for freight  infrastructure. Each of these issues was then addressed in specific  sections of the draft freight strategic plan. DOT officials stated that these  insights also informed recent action by Congress. Specifically, in  December 2015, Congress enacted and the President signed into law the  Fixing America’s Surface Transportation (FAST) Act which created a new  grant program for nationally  significant freight and highway projects and  authorized appropriations for this new program as well as existing grant  programs through fiscal year 2020, among other things.
section_title: Clearly Define Goals and Performance Measures for the Open Innovation Initiative
paragraphs: According to relevant literature and our interviews with experts and  agency officials, once the agency has identified the high-level purposes it  wants to achieve through an open innovation initiative and selected the  strategy or strategies it will use, it should clearly define specific and  measurable goals for the initiative. Specific goals can help guide the  design and implementation of an initiative. They also can help those  involved maintain a sense of direction by providing a clear understanding  of what they are working to achieve.
Define specific and measurable goals for  the initiative.  Identify performance measures to  assess progress.  Align the goals of the initiative w ith the  agency’s broader mission and goals.
Relevant literature, experts, and agency officials we consulted highlighted  that the agency should also identify the performance measures it will use  to assess progress towards the goals and overall results. For open  innovation initiatives, measures can be used to assess the achievement  of specific outcomes, participation and engagement, and resources  invested in the initiative. Outcome measures could include the successful  achievement of a goal, improvements in the quality of a policy or process,  or the improved delivery of a service. Participation and engagement  measures could include the number or diversity of participants engaged in  the initiative;  the number of ideas submitted; the amount of time it takes to  respond to participant questions, comments, or feedback; and the  satisfaction of participants with their experience. Measures of resources  invested (input measures) could include the money, staff resources, and  time dedicated to implementing the initiative. This information can also  help an agency determine whether it would be appropriate to expand—or  “scale”—an approach if it is found to be successful.
Lastly, the literature and experts also emphasized that the agency should  seek to align the specific goals of an open innovation initiative with the  agency’s broader mission and goals. Aligning initiative-specific goals  with agency priorities can help ensure the relevance and value of an  initiative, by showing how its successful implementation could advance  progress on the agency’s mission and goals. This alignment also  reinforces the connection between the agency’s mission and goals and  the day-to-day activities of those carrying out an initiative.
The following two examples illustrate how DOE and EPA defined goals  and performance measures for selected open innovation initiatives.
subsections: section_title: DOE Aligned Its Goal for a Challenge to Capture Energy from Ocean Waves with Broader Departmental Goals
paragraphs: According to an official in DOE’s Wind and Water Power Technologies  Office (WWPTO), its Wave Energy Prize (WEP) competition is designed  to dramatically improve devices that produce electricity by capturing  energy from ocean waves. WEP began in April 2015 and is scheduled to  conclude in November 2016. WWPTO specified in its contest  documentation that the effort could stimulate private sector innovation  and contribute to energy security and international competitiveness in the  wave energy conversion sector. This was aligned with DOE’s strategic  objective to support a more economically competitive, environmentally  responsible, secure, and resilient U.S. energy infrastructure.
During the planning phase, WWPTO established a specific, measurable  goal in its rules for the competition. The goal required that devices  developed for the competition at least double the energy capture of  current technology. According to a DOE National Laboratories analysis,  the average rate of wave energy capture for a group of current devices is  1.5m/$M (or 1.5 meters per million dollars). To be eligible for a monetary  prize, which will range from $1.5 million for the winning team to $250,000  for the third place team, participants would have to develop a device that  would achieve 3m/$M. WWPTO officials told us that this target gave  participants a clear, achievable goal for which to strive. They added that  the goal also was aggressive enough to represent a ground-breaking  advancement over current technology. Although the competition is still  ongoing, according to information on the contest website, WEP has  demonstrated early success as a number of the teams are proposing  innovative technologies and have demonstrated a potential to achieve or  exceed WWPTO’s stated goal.
To help guide its outreach efforts, WWPTO also established a goal to  alert potential participants about the WEP, and have them take action by  registering to participate. WWPTO officials and the prize administration  team developed a detailed Communications and Outreach Plan for the  competition. The plan outlined the types of metrics that could be tracked  to determine the effectiveness of its outreach efforts. These metrics  include the number of registered teams, and traffic to the competition  website and social media pages. According to WWPTO officials, 92  teams registered to participate in the competition thanks to their  aggressive communications and outreach strategy. This number was  three times more than they had initially expected.
section_title: EPA Established Goals and Measures of Success for a Challenge to Develop Improved Water Pollution Sensors
paragraphs: EPA has a strategic objective to protect and restore watersheds and  aquatic ecosystems, and has reported that it is working with external  partners and stakeholders to spur technological innovations to reduce  costs and pollution through improved and less-expensive monitoring. In  2013, OSTP convened the Challenging Nutrients Coalition (CNC). CNC is  a group of federal agencies, including EPA, nongovernmental  organizations, and academia, working together to address the issue of  nutrient pollution. In November 2013, OSTP hosted a meeting of  agencies and experts familiar with nutrient pollution. According to EPA,  experts found that more affordable and reliable sensors are needed to  collect more data on nutrient levels to inform decisions about how to  manage and reduce these levels.
In December 2014, the Nutrient Sensor Challenge was announced, led by  EPA and supported by the National Oceanic and Atmospheric  Administration (NOAA) and other agencies. The goal of the Nutrient  Sensor Challenge is to accelerate the commercial development of  accurate, reliable, and affordable devices that will meet user needs and  be available for purchase by 2017. According to EPA officials, EPA  aligned the goals of the challenge with EPA’s strategic objective. The  challenge offers participants non-monetary rewards and incentives like  visibility in an emerging market and access to testing services and other  resources. In June 2014, the Partnership on Technology Innovation and  the Environment, another member of the CNC, conducted a study to  clarify the specific needs of potential sensor users. Through this study  they identified standards for accuracy, precision, and cost that the vast  majority of potential users would look for in devices. These became the  technical requirements that devices developed for the challenge must  meet to be eligible for awards. For example, most of the study’s  participants identified the $1,000-to-$5,000 price range as affordable for  their purposes. For this reason, EPA required that the devices built for the  competition have a purchase price of less than $5,000. As of August  2016, EPA and its partners are conducting final testing on the devices  submitted by participants to determine if any meet the technical  requirements, and plan to announce final awards in December 2016.  However, EPA officials stated that preliminary results indicate that the  devices developed through the competition will meet the technical  requirements that have been established. They added that several  companies are developing instruments of similar capabilities and price  outside of the challenge.
Another goal of the competition is to produce an identified, mobilized  market of community organizations, state and federal agencies, and  researchers. According to EPA officials, EPA and other CNC partners,  including USGS, NOAA, and the National Institute for Standards and  Technology, are creating pilot programs that will allow organizations to  deploy and test these sensors following the completion of the competition  in late 2016. According to EPA officials, as of March 2016, 14  organizations have expressed interest in participating in EPA’s pilot  program. EPA officials also stated that this pilot program will help identify  organizations that may want to purchase and deploy the sensors in a  more widespread way in the future.
EPA officials stated that having these specific goals has been critical  given the focus that they have provided. For example, the goals will help  ensure that the devices developed through the challenge serve as the  reliable and affordable devices necessary to stimulate the market, and to  expand how widely they are deployed.
section_title: Identify and Engage External Stakeholders and Potential Partners
paragraphs: Identify and engage outside stakeholders  interested in the issue addressed by the  initiative.  Look for opportunities to partner w ith  organizations on the design and  implementation  of the initiative.
Our literature review and agency officials highlighted the importance of  identifying and engaging with external stakeholders who share an interest  in the issue being addressed and may already be active in related efforts.  For a federal agency, external stakeholders can include representatives  of relevant non-profit organizations and foundations, community or  citizens’ groups, universities and academic institutions, the private sector,  members of Congress and their staffs, other federal agencies, and state  and local governments. By engaging with outside stakeholders, agencies  can gain their support for the initiative, gain insights from their prior  experience working on an issue, and see how they might use the results  (e.g., products) of an initiative.  This can help clarify the goals and design  of an initiative. This engagement can also be used to determine what  motivates stakeholders to get involved in an effort, and to identify  additional stakeholders, partners, or potential participants to engage in  the initiative.
The literature, experts, and agency officials also emphasized that  agencies should look for opportunities to partner with other groups and  organizations that would be interested in, or could benefit from, the  results of an open innovation initiative. Partners are organizations and  individuals that play a direct role in designing and implementing an  initiative. They provide staff capacity, resources, administrative and  logistical support, assistance with communications and community  building, or ongoing advice and expertise. Partner organizations provide  these resources and assistance because they have missions or goals that  overlap or align with what the agency wants to achieve through an open  innovation initiative. Agencies can also consider the most appropriate and  effective mechanism for formalizing these partnerships, such as  collaboration agreements, contracts, or interagency agreements. Agency  officials can identify partner organizations through discussions with  external stakeholders, professional contacts, or research into  organizations with complementary goals.
Finally, agency officials we interviewed emphasized the especially  important role that agency leaders can play with respect to this practice.  The support of agency leaders can be particularly important, as their  involvement can lend credibility and visibility to an initiative to those  outside the agency. It can also help mobilize a broader community of  external stakeholders and partner organizations.
Below we provide illustrative examples of how DOT, HUD, EPA, and HHS  identified and engaged external stakeholders and partners for three open  innovation initiatives.
subsections: section_title: DOT Involves Stakeholders in an Ideation Initiative to Identify and Implement Improvements in Highway Transportation
paragraphs: The Federal Highway Administration’s (FHWA) Every Day Counts (EDC)  is an example of an ideation initiative. EDC is designed to identify  effective, market-ready innovations states could implement to improve  highway project delivery. According to an FHWA official, from the  beginning of the initiative in 2009, the then-FHWA Administrator and  Deputy Administrator (who are now Deputy U.S. Secretary of  Transportation and FHWA Administrator respectively) established and  supported EDC as a state-based, stakeholder-driven program. They  established the Center for Accelerating Innovation  (Center) to implement  the program, and worked with internal and external stakeholders to  promote the idea of using innovative practices to improve how highway  construction projects are performed.
Every 2 years, FHWA works with various stakeholders to identify  innovative technologies and practices that merit more widespread  deployment through EDC. The process begins when FHWA publishes a  Request for Information inviting suggestions for new innovations to  consider from state, local, tribal, and industry experts. According to  FHWA officials, the agency typically receives more than 100 suggestions  and comments. FHWA staff review these submissions to develop a list of  those innovations that are market ready, could be implemented across  the country, and have the greatest potential to improve efficiency and  quality in highway transportation and construction.
According to an FHWA official, once this list of EDC innovations is  finalized, the Center works with FHWA program offices to identify leaders  for Innovation  Deployment Teams. The deployment team leaders identify  other team members, such as communication specialists, subject matter  and technical experts from state transportation agencies, and key  stakeholders like industry representatives. The deployment teams work  with state transportation agencies and other stakeholders to implement  the innovations that best fit their needs by providing technical assistance,  training, and outreach.
Once the EDC innovations are selected, transportation leaders from  across the country gather at regional summits to learn about and discuss  the innovations. According to a March 2015 report from FHWA, the  summits are used to disseminate information on innovations so states  can identify those that best fit the needs of their highway programs. The  summits include interactive working sessions to foster connections  among regional transportation professionals, and encourage longer-term  collaboration on the deployment of innovative  practices. In 2014, the  summits introduced online broadcasts of the presentations and  discussions so that a wider audience could participate.
section_title: HUD Used a Prize Competition to Develop Plans to Rebuild Following a Disaster
paragraphs: The President’s Hurricane Sandy Rebuilding Task Force launched  Rebuild by Design (RBD), a prize competition overseen by HUD, in June  2013 to generate innovative and implementable design ideas to rebuild  communities affected by Hurricane Sandy. According to HUD officials,  HUD searched for external organizations and foundations with  complementary missions to partner with on implementing RBD. In  particular, it sought established organizations with resources, capabilities  to administer a design competition, and the ability to engage local  residents and stakeholders in affected communities. Several philanthropic  organizations, including the Rockefeller Foundation, provided financial  support to fund the administration of the competition, $200,000 cash prize  awards to finalist design teams, and project evaluation. According to a  2014 evaluation of RBD conducted by the Rockefeller Foundation and  HUD officials, direct outreach to potential philanthropic partners by the  then-Secretary of HUD played a key role in securing their financial  commitments. To help administer the competition, HUD also partnered  with four local research and advocacy organizations to support the work  of RBD design teams at the local level. Figure 4 summarizes the  network of organizations involved in RBD.
According to HUD officials, each administering partner organization was  chosen for its complementary resources and expertise in research,  design competitions, community outreach, regional planning and design,  and local ties to the region. HUD staff also established a management  plan early in the process that outlined roles and responsibilities for how  these partner organizations would work together through each stage of  the competition.
According to HUD officials, this partnership with local organizations  supporting the competition’s implementation was critical to RBD’s  success. HUD officials were unfamiliar with local networks of community  groups and other relevant organizations in each region, so the ability to  partner with those that had knowledge, networks, and skills that HUD  could leverage was valuable. These networks helped facilitate community  engagement by design teams, who used meetings, community design  workshops, site visits, and social media to engage hundreds of local  stakeholder groups from communities affected by Hurricane Sandy.  According to HUD officials, this outreach was critical to meet HUD’s  expectation that projects receiving support be co-designed with  communities, have local support, and be financially viable. They also said  that RBD demonstrated the value that external partnerships can bring in  providing expertise, capacity, and connections that help an agency  achieve its mission and goals.
section_title: EPA and HHS Partnered on a Challenge to Develop Affordable and Wearable Air Sensors
paragraphs: According to EPA and HHS officials, both agencies shared an interest in  developing affordable, wearable sensors that would provide wearers with  information on air quality and the body’s reaction to it. The agencies  jointly sponsored the My Air, My Health challenge, asking participants to  develop a device that would do these things in tandem. The challenge  was held in two phases, and ran from June 2012 to June 2013.
According to EPA officials, EPA and HHS created a cross-agency design  team that included experts from EPA’s Offices of Air and Radiation and  Research and Development, and the National Institutes of Health (NIH), a  medical research agency within HHS. Within that design team, one cross- agency work group focused on identifying the air pollutants and health  concerns the competition would target, while another work group focused  on the technology and how the devices would communicate health data.  According to EPA officials, creating this collaborative design team helped  ensure key subject matter experts from each agency could guide the  development of technical requirements for the competition in a way that  would address the shared goals of each agency. According to an HHS  official, for example, during the development of these technical  requirements, EPA staff identified what air quality data would need to be  collected, while HHS staff identified what would need to be measured to  determine the health effects of exposure to air pollution.
According to EPA officials, the agencies shared responsibilities for  implementing the competition’s phases. EPA implemented the first phase  of the competition, which was focused on developing plans and proposals  for prototypes. HHS then implemented the second phase, in which  finalists developed and validated proposed prototypes.
EPA and HHS officials told us that the agencies used the competition to  communicate their shared interest in the technology and encourage  further private-sector development. The agencies used My Air, My Health  to demonstrate that open innovation initiatives involving partnerships  between agencies were feasible, and that collaboration between agencies  and with the private sector can allow agencies to achieve goals that they  may not have the capability to achieve alone.
section_title: Develop Plans for Implementing the Initiative and Recruiting Participants
paragraphs: Relevant literature and agency officials highlighted how important it is for  agencies to ensure that roles, responsibilities, expectations, and time  frames are clear for all involved in implementing and managing an  initiative. The agency and any of its partners can do this by establishing  and documenting a governance structure for the initiative  that clarifies the  processes that will be used to ensure regular communication; raise,  discuss, and resolve any pressing issues; and make decisions. According  to our literature review and interviews with experts and agency officials,  the agency and any partners should develop a detailed implementation  plan for the initiative  that clearly identifies  the specific tasks and actions needed to carry out the initiative, the  parties responsible for completing them, and the timeframes for doing  so; potential participant groups to engage in the initiative, including when  and how the agency and any partners will reach out to various  participant groups and encourage them to participate, and how they  will engage with participants during and after the initiative’s  implementation; and what data will be collected, and how, during and after implementation,  and how the data will be evaluated to determine overall results and  progress towards the initiative’s stated goals.
The following two examples show how HUD and HHS developed plans  for implementing and recruiting participants for selected open innovation  initiatives.
subsections: section_title: HUD Developed a Governance Structure, Processes, and Procedures to Manage Its Switchboard Ideation Platform
paragraphs: Switchboard is an online idea generation initiative that HUD uses to  collect ideas from citizens, stakeholders, and HUD staff on how the  agency can improve its processes, programs, and administration. HUD  officials can then consider these ideas for potential implementation.
HUD drafted a charter in 2011 to guide the initiative’s implementation that  describes the overall team structure, defines the roles and responsibilities  of each staff member involved in reviewing and responding to ideas  submitted through the website, and names liaisons for program offices  throughout HUD to review and respond to ideas that fall within their  programmatic jurisdiction. See table 3 for a summary of the roles and  responsibilities from the Switchboard charter.
Table 3. Information on Roles and Responsibilities  from HUD’s Switchboard Charter  Responsibility  Champion of the project.  Approval and sign off of project components and  requirements. Overall ownership of project from an  organizational perspective; management of budget.  Overall management of the project timelines and scope.
Oversight of internal and external communications; sets  direction for messaging.  Manages day-to-day activities of project.
Provide input into process, manage ideas and responses.
The charter also explains the process and criteria used to evaluate an  idea, and determine whether it should be elevated for consideration and  potential implementation. HUD supplemented this charter with a  document outlining policies and procedures for investigating, responding  to, and implementing an idea. Figure 5 summarizes these procedures.
According to HUD staff, Switchboard has become a tool for more effective  customer service by providing an easy way for anyone to contact HUD  with ideas for how the agency could do things more effectively. It has also  provided the agency with a platform to host specific issue forums that are  sponsored by various HUD program offices and targeted toward specific  segments of the public. For example, in 2011, the HUD Office of  HIV/AIDS  Housing used Switchboard (then called HUD Ideas in Action) to  ask for public input on how HUD should update the Housing Opportunities  for Persons with AIDS program funding formula to better target resources  to need. In response to this request, HUD received 17 submissions with  ideas—many of which generated additional comments from participants  in the forum—and a total of more than 500 votes. HUD then selected four  of these submissions for further review, and incorporated  recommendations from one of them into the department’s fiscal year 2013  budget request.
section_title: HHS and Its Partners Developed an Implementation Plan for a Challenge to Commercialize NIH Inventions
paragraphs: The Neuro Startup Challenge was created by NIH and the Center for  Advancing Innovation (CAI), a non-profit organization with a mission to  accelerate knowledge and technology transfer, and entrepreneurship.  Conducted from April 2014 to August 2015, the challenge was designed  to generate promising start-up companies with business plans to  commercialize NIH inventions for use in treating brain and neurological  disorders. According to the collaboration agreement between NIH and  CAI, the challenge supported NIH’s mission to advance research,  innovation, and education to protect public health. It also aligned with  CAI’s goals to encourage the commercialization of new technologies.
NIH and CAI used this collaboration agreement to outline a detailed  governance structure that specified the roles each organization would  play in implementing the competition. The agreement also identified the  respective tasks each would be responsible for completing during the  various phases of the competition, along with the timeframes for each  phase. For example, the agreement specified that during the planning  phase of the competition, which was scheduled to run from April to  August 2014, CAI would be responsible for identifying and engaging  stakeholders and potential participants, as well as other deliverables,  including the development of an advertising and marketing plan for the  competition. The agreement also specified that NIH  would provide input  on the rules and criteria for the competition, the selection of inventions,  and the identification of potential participants. According to an NIH official,  this delineation of responsibilities was particularly important to help frame  and focus efforts at the beginning of the project.
In the agreement, NIH and CAI also identified the potential participants  they wanted to reach through the competition. Participants included  graduate and post-doctoral students and experienced entrepreneurs.  According to an NIH official, NIH and CAI particularly focused on  engaging those affiliated with universities, given the focus on connecting  university students with real-world experience in business planning. Prior  to launching the initiative, CAI planned for extensive contact with  university faculty and students to get feedback on the concept and to  make them aware of the challenge. CAI then conducted an extensive  series of phone conversations and in-person meetings to connect with  stakeholders and potential participants at 37 universities in 14 states.  Through this outreach they reached approximately 1,500 people with  information on the challenge. According to NIH  officials, many of the more  than 70 teams that participated in the competition were from those  universities contacted through this outreach. CAI also reached out to local  economic development groups and universities to identify entrepreneurs  and business developers who would be interested in supporting  participating teams.
section_title: Engage Participants and Partners while Implementing the Initiative
paragraphs: Relevant literature, experts, and agency officials emphasized that when  agencies are ready to move forward with implementation, they should  announce the initiative in a way that generates interest among potential  participants. This involves using multiple outlets and venues—including  the initiative  website, social media, press releases, press conferences,  journals, newsletters, and professional conferences and networks—to  ensure they reach the right potential participants and make them aware of  the initiative.  The participants that an agency and any partners seek to  engage, and how they decide to solicit participation, will vary depending  on the purposes of the initiative. For instance, if an agency wants to use  an initiative to address a very specific technical issue it may attempt to  identify and engage individuals with the requisite skills through an existing  network of experts. However, if an agency intends to use an initiative to  collect a wide range of perspectives on an issue, it will likely need to be  much more open and inclusive in its outreach and encourage diverse  groups to participate.
Efforts to promote the initiative are important because reaching the right  participants and motivating them to participate is critical to the overall  success of an initiative. According to the literature and our interviews, the  initial outreach to potential participants should be crafted and  communicated in a way that responds to the interests and motivations of  potential participants, and explains why it is important for them to  participate. In addition, the agency should also establish clear  expectations for participants, describing in detail what they will be  expected to contribute; how and when their contributions will be collected,  evaluated, and used; and what participants must do to receive any  monetary or non-monetary incentives that may be provided.
Once the initiative begins, the agency and any partners should use  websites, question-and-answer sessions, emails, and other forms of  communication to keep participants apprised of progress. Through the  literature and our interviews we also found that agencies and their  partners can actively engage participants to solicit and respond to any  questions, comments, and feedback, and provide any necessary  assistance. These actions can increase the likelihood that participants will  have a positive experience, and can help show that their participation and  contributions are valued. According to experts and agency officials with  whom we consulted, however, doing this can be a very resource-intensive  activity, particularly if the initiative has a large number of participants and  there is a high volume of communication from participants. Therefore,  during the planning phase, the agency and any partners should work  together to ensure that the party responsible for this aspect of  implementation has sufficient capacity to respond in a timely fashion.
Agency officials highlighted that the agency and any partners should also  use regular check-ins to discuss the progress of the initiative. Such  check-ins can help ensure those involved in implementation know the  status of specific implementation tasks against established time frames,  and any decisions that may be needed. The agency and partners should  also review the data and feedback that are being collected during  implementation. This will allow them to identify and make any necessary  adjustments to improve implementation and the experience of the  participants.
As illustrated below, HUD, DOE, and HHS engaged participants and  partners during the implementation of three open innovation initiatives.
subsections: section_title: HUD Used Targeted Outreach to Announce Its Disaster Recovery Challenge and Regularly Communicated with Participating Teams
paragraphs: HUD’s objective for its outreach to potential participants for Rebuild by  Design (RBD), according to an April 2015 report from OSTP on the  implementation of federal prize competitions and HUD officials, was to  recruit world class design talent to participate in the competition. It used  its network of project partners, professional associations, university  programs, as well as websites focused on planning, design, and urban  issues, to promote the competition. For example, the American Institute of  Architects launched a communications campaign urging its membership  to participate in RBD. According to the April 2015 OSTP report, this  outreach was successful, as HUD ultimately received high-quality  proposals from 148 teams representing top engineering, architecture, and  design firms.
According to HUD officials, after 10 design teams were selected to  participate in RBD, HUD and its partners regularly communicated with the  teams to identify challenges they faced and assistance that they needed.  HUD officials explained that RBD was designed to allow more than one  winner, as each finalist team worked to develop innovative  approaches  for rebuilding and resilience in a different community. As a result, the  RBD management team facilitated collaboration between the design  teams. This allowed the teams to share good practices and learn from  each other’s experiences.
According to a 2014 evaluation of RBD conducted by the Rockefeller  Foundation, HUD’s local administering partners supporting RBD’s  implementation also worked closely with the design teams and provided  logistical support and connections to community-based organizations and  public officials. To ensure clarity about reporting requirements and  deadlines, those managing RBD also instituted other means of  communication. This included biweekly memorandums for the design  teams and weekly phone and e-mail communications with partner  organizations providing support to teams at the local level. The  Rockefeller Foundation also reported that effective management  practices and regular communication allowed the design teams to meet  all procedural deadlines and milestones despite the initiative’s fast pace  and logistical challenges.
section_title: WWPTO Established Processes and Tools to Engage with Participants in Its Challenge to Develop Devices to Capture Energy from Ocean Waves
paragraphs: In the Communications and Outreach Plan developed for the Wave  Energy Prize (WEP), DOE’s Wind and Water Power Technologies Office  (WWPTO) set a goal to expand the community of developers involved in  wave energy conversion technology. It sought to do this by drawing in  both experienced energy device developers and newcomers representing  a diverse group of companies, universities, and individuals. According to  WWPTO officials, to generate a large pool of new and experienced  developers for the competition, which began in April 2015 and is  scheduled to conclude in November 2016, they used multiple outlets and  venues to encourage WEP participation. As outlined in the  Communications and Outreach plan for the competition, this included the  WEP website, social media, email marketing, presentations, and outreach  to various media outlets to reach a broad range of potential participants.  Communications used to recruit participants also emphasized several key  messages to motivate interested individuals and teams to participate.  These messages included the availability  of a monetary prize, the  opportunity to help solve a difficult technological problem, and the chance  to work on technologies that could contribute to the nation’s energy  independence. See figure 6 for examples of these communications.
According to WWPTO officials, to ensure there would be participants with  technical expertise in energy production technology, WWPTO officials  reached out to individuals who previously had contacted WWPTO  regarding other projects involving wind and water power. WWPTO also  promoted the competition through specific industry publications, outreach  to professional and academic organizations focused on relevant technical  specialties, and presentations at energy technology-oriented conferences.
According to WWPTO officials, through this outreach, they attracted both  new and experienced developers to participate in WEP. Of the 92 teams  that registered to participate in WEP, most were previously unknown to  WWPTO. Furthermore, out of the nine finalists and two alternates that  were chosen to participate in the final phases of the competition, only two  had received any prior funding from WWPTO. WWPTO officials also  reported that they were successful in reaching teams with sufficient  technical expertise to reach aggressive technical goals. According to  information on the competition website from March 2016, while the  devices of finalist teams are currently undergoing final building and  testing, preliminary evaluations indicate that many of them could achieve  or exceed WWPTO’s goals for the competition.
According to WWPTO officials, the prize administration team has also  created processes to regularly engage with teams participating in the  competition. For example, the prize administration team holds biweekly  calls with participating teams and technical experts. These calls prepare  them for the final testing program, solicit and respond to participants’  questions and comments, and provide any necessary technical  assistance. According to WWPTO officials, these interactions can be  time- and resource intensive, so they planned for them during the early  phases of the competition. This ensured that the prize administration  team allocated sufficient resources to fulfill their participant management  responsibilities. Furthermore, WWPTO and the prize administration team  also hold weekly conference calls to discuss progress on key tasks and  any adjustments that may be needed. These check-ins help ensure that  WWPTO and the prize administration team are working from a common  set of expectations. It also allows WWPTO to provide the prize  administration team with any necessary information it needs to  successfully implement WEP.
section_title: HHS Collected Feedback from Users to Make Improvements to the OpenFDA Open Data Platform
paragraphs: OpenFDA is an open data platform released by the Food and Drug  Administration (FDA) in June 2014. FDA, an agency within HHS  responsible for assuring the safety of drugs, medical devices, and food,  uses OpenFDA to make several key datasets available in a format that  allows researchers and developers to more easily use the data.  According to an August 2014 report from Iodine, a private health data  company that assisted FDA in the development of OpenFDA, as the  platform was developed and became available for testing, FDA officials  actively engaged potential users. The officials solicited input from a group  of individuals and organizations that had expressed interest in the  platform and were willing to contribute feedback. The report also stated  that FDA officials observed that some of those testing the platform had  difficulty using it. As a result, FDA took actions to make the platform more  user friendly. These actions included adding an interactive tool that allows  users to filter and visualize the data more intuitively. According to an FDA  official, these changes permitted OpenFDA users without technical  expertise to more easily use and benefit from the platform.
In addition, FDA officials actively monitored the online forums created for  users of OpenFDA, and responded to any requests for clarity or  information. According to FDA officials, engagement with users has been  a priority. Through direct contact with the community of users, the agency  has collected information to help ensure OpenFDA will serve their needs.  For instance, in December 2015, FDA made the data on OpenFDA  available for direct download as a result of requests from users. In June  2016, FDA also launched an updated version of OpenFDA that was  redesigned in response to user feedback. This feedback included the  need to improve the website’s layout.
section_title: Collect Relevant Data and Assess and Report Results
paragraphs: Relevant literature and agency officials emphasized that after the initiative  has concluded, or at regular intervals if it is a long-standing or continuous  effort, the agency should assess whether the initiative has achieved its  goals. By analyzing the data it has collected, including quantitative  performance data and qualitative data provided by participants on the  effects of an initiative, the agency can determine if it has met its goals.  When a goal is unmet, the agency should conduct additional analyses to  understand why. In addition, because some outcomes may not be  observable until months or years later, agencies can consider whether a  long-term monitoring or assessment plan is needed and appropriate.
According to relevant literature we reviewed, the agency should also  conduct an after-action review to analyze feedback from partners and  participants. Such a review can help identify lessons learned and process  improvements that could be applied in future initiatives. For example,  participant feedback may provide insights on parts of the process that  went well and others that could have been executed better. These can  then be replicated or adjusted, accordingly, for reoccurring or similar  initiatives in the future. The agency can also engage with partners to  review planning and implementation activities to identify what worked well  and any notable gaps or challenges that may need to be addressed in  future initiatives.
Lastly, relevant literature and experts emphasized that once the agency  has assessed the initiative it should publicly report on the results  achieved and lessons learned. This transparency can help build trust with  partners and participants, demonstrate the value of open innovation  initiatives to other stakeholders and the public, and build momentum for  future initiatives. Reporting results while partners and participants are still  engaged can also help sustain a dialogue and increase awareness within  the community of interested organizations and individuals.
For the following three open innovation initiatives, we present how DOT,  NASA, and DOE collected data, and assessed and reported results.
subsections: section_title: FHWA Has Regular Data Collection, Assessment, and Reporting Cycles for Its Ideation Initiative to Improve Highway Transportation
paragraphs: The Federal Highway Administration’s (FHWA) Every Day Counts (EDC)  initiative focuses on ensuring that proven innovations to improve highway  construction and safety are quickly and broadly deployed. FHWA  launched EDC in 2009. FHWA tracks progress toward this goal primarily  by measuring the number of states that are deploying specific innovations  being supported by EDC, along with whether the innovation is being  developed, tested, assessed, or adopted as a standard practice.  According to FHWA officials, staff from the Center for Accelerating  Innovation  (Center), which is responsible for implementing EDC, and  deployment teams use this data to track how the level of deployment  compares with goals established at the beginning of each 2-year cycle.  Figure 7 shows the January 2015 baseline data for the e-Construction  innovation, the progress made through December 2015, and the overall  goal the agency is working to achieve by December 2016.
According to an FHWA official, staff from the Center work with  deployment teams to develop implementation plans for each innovation,  which include identifying interim performance goals that will be used by  the team to track implementation progress. FHWA officials say setting  specific performance goals for deployment helps to ensure accountability  for the advancement of innovations. For instance, the Director of the  Center meets with the leader of each deployment team each quarter to  review progress toward established goals. According to an FHWA official,  these review meetings can result in the provision of additional resources  or assistance to deployment teams, or, in some circumstances,  adjustments to team leadership.
FHWA has also established regular reporting cycles for EDC. It releases  two progress reports each year that summarize the status of each  innovation. In addition, it has also produced a final report at the end of  previous two-year cycles summarizing the highway community’s  accomplishments and progress. The final report includes data on how  widely each innovation was deployed, accomplishments in states where  innovations were deployed, and explanations of benefits and lessons  learned through implementation.
According to FHWA officials, publicly reporting results increases  transparency and shows the effects of the EDC program. It also highlights  successes achieved by state and local agencies in deploying innovations  faster. For instance, FHWA reported in its July to December 2015 EDC  progress report that the program has accelerated the deployment of  innovations across the country. Every state implemented at least 8 of the  38 innovations promoted under the initiative since 2010, while some have  adopted over 20. Furthermore, in August 2014, FHWA released a report  with examples demonstrating that implementing EDC innovations has had  significant and measurable effects in participating states. For example,  FHWA reported that deploying Accelerated Bridge Construction as an  EDC innovation has allowed states to reduce the time it takes to plan and  construct bridges by years. This significantly reduces traffic delays, road  closures, and often project costs. In 2015, Congress enacted and the  President signed into law a requirement that FHWA continue to use EDC  to work with states, local transportation agencies, and industry  stakeholders to identify and deploy proven innovative practices and  products.
section_title: NASA and Its Partner Collected Feedback from Participants and Conducted an After-Action Review to Identify Potential Improvements to Future Open Dialogues
paragraphs: NASA worked with Expert and Citizen Assessment of Science and  Technology (ECAST), a network of institutions that encourages public  input on science and technology policy issues, to solicit the views of  citizens on options for defending the Earth against an asteroid strike and  exploring asteroids. These in-person and online forums, known  collectively as the Asteroid Initiative  Citizen Forums, took place in  November 2014 and February 2015. The forums were used to obtain  information on participant preferences, priorities, and values. NASA  officials used this input to inform, among other things, decisions about its  future mission and technology investment goals. This includes detecting  asteroids, mitigating asteroid threats, and exploring asteroids with  astronauts. For example, after the forums were held, relevant results  were shared with NASA managers to inform the selection of a specific  technology and approach that would be used for a future mission to  capture an asteroid. According to NASA officials, the results of these  forums provided NASA with insights into public understanding and views  on NASA’s asteroid work. Figure 8 illustrates the platforms used for both  the in-person and online forums.
According to NASA officials, NASA also wanted to use the forums to  identify lessons that could guide its future efforts to engage citizens. To  do this, ECAST had participants complete post-forum surveys and  provide written comments on their experience in the forum. ECAST then  analyzed this information. Observers at selected tables also helped  assess the meaning of written comments from participants. Through an  after-action review, ECAST identified a small number of issues to address  in preparation for any future forums. These included insights into the  ability of citizens to understand complex information, and the need to  provide clearer information to participants about how the forum results  would be used.
Members of ECAST involved in designing and implementing the forums  also summarized their observations on potential refinements in their final  report to NASA. For example, ECAST found that connecting attendee  background information to individual responses could have also provided  context for interpreting the written results. ECAST members also found  they needed more time to test background materials given to participants  to read before the event, and needed to take additional steps to increase  consistency across table facilitators.
section_title: DOE Established a Long-Term Monitoring Effort to Assess the Outcomes of Its Challenge to Develop Innovative Energy Technologies
paragraphs: DOE’s SunShot Catalyst initiative (Catalyst) was a series of competitions  first begun in May 2014. It was designed to engage entrepreneurs, solar  professionals, and software and data experts to help them rapidly develop  start-up companies with viable technologies to address identified  challenges in the solar and energy efficiency markets. By providing  intensive training and support to those with the most promising ideas,  DOE officials also wanted to ensure that teams would have market-ready  innovations and viable business plans at the end of the competition.  According to DOE officials, DOE selected 35 teams to participate in the  initiative.
According to a DOE official, in order to determine the initiative’s  effectiveness, DOE developed a long-term effort to monitor the status of  the companies created through the competition. DOE officials said that  they collected publicly-available  information on the status of the 35 teams  that participated in Catalyst. DOE officials reported in June 2016 that  through collecting this information they found that 28 teams were still  actively pursuing their startups. DOE officials also invited all 35 teams to  one-on-one discussions, and were able to meet with 24 of them. Through  these discussions, DOE collected information on the amount of capital the  teams had raised, projected annual revenues, and the benefits they  gained from participating in Catalyst. For example, the 24 teams reported  that they had collectively raised a total of $6.4 million in private capital or  public funding, had 95 full-time employees, and had total expected annual  revenue of $5.6 million.
As DOE also reported, officials also identified specific lessons learned at  each stage of the Catalyst process that can be used to inform how future  competitions are conducted, and improve and expand the Catalyst  program. For example, DOE reported that the most effective way to reach  potential Catalyst participants was through the networks of previous  participants, along with recruitment efforts involving local partners and  events. DOE also reported that the 60-day period provided for  participants to develop their prototypes was challenging, and that the  department should consider adding time to that phase of the competition.
DOE also compared the cost and time for product development under the  Catalyst approach to those supported through DOE’s traditional financial  assistance awards, including cooperative agreements and grants.  According to DOE officials, the agency had learned through earlier efforts  to engage developers that the application process for traditional funding  opportunities can create a barrier for those who may not be interested in  or able to go through what can be seen as an extensive review and  approval process. DOE wanted to use Catalyst to test a faster, more open  way of engaging developers and entrepreneurs. Through its assessment,  DOE found that, under a traditional funding opportunity, it typically takes 9  months to move from the announcement of the opportunity to the award  being made, with minimum awards ranging from $300,000 to $500,000  for software or applications. By contrast, for Catalyst, this process was  completed in 3 months, with $25,000 prizes awarded to rapidly test and  validate prototypes.
section_title: Sustain Community of Interested Partner Organizations and Participants
paragraphs: Given the time and resources that agencies may invest to build or  enhance communities of partners and participants for open innovation  initiatives, agencies can take steps to sustain these connections over  time. This is particularly important if one purpose of the initiative  is to build  a new, or bring greater coherence to an existing, community of interested  organizations and individuals to work together on an issue. However, this  may be less applicable when an initiative is discrete in scope and  intended to be a one-time occurrence.
Seek to maintain communication w ith, and  promote communication among, members  of the community.
According to relevant literature and agency officials, agencies should  acknowledge and, where appropriate, reward the efforts and  achievements of partners and participants so that they feel their  contributions are valued and appreciated. This can be done in conjunction  with reporting the results of and lessons learned from the initiative, or  through separate venues such as announcements, award ceremonies, or  recognition on the initiative website. As part of this effort, it is also  important for agencies to explain how the contributions of partners and  participants helped the agency achieve, or progress toward, its goals, and  to communicate the next steps that will be taken following an initiative.
Relevant literature and experts we consulted also highlighted that  agencies can seek ways to maintain communication with members of the  community to keep them informed of future initiatives and other  opportunities of interest, and facilitate communication within the  community. To ensure these activities receive sufficient attention over  time, an agency may need to assign staff the responsibility of maintaining  contact with these communities. Efforts to sustain a community over time  can help enhance collaboration to continue progress on addressing an  issue, and provide the agency with a network that could be more easily  mobilized again for future initiatives. At some point, these communities  may become self-sustaining, with members continuing to collaborate with  little or no involvement from the agency.
To illustrate how agencies have built and sustained communities of  interested partners and participants by implementing open innovation  initiatives, we provide the following three examples from EPA, NASA, and  HHS.
subsections: section_title: EPA Took Steps to Build and Has Continued to Engage a Community Interested in Air Pollution Sensor Technology
paragraphs: From 2012 to 2015, EPA’s Office of Research and Development held a  series of air pollution sensor workshops that were, according to EPA  officials, designed to better understand the needs of governments and  community groups interested in using these sensors, and to build a more  coherent community of users and developers. EPA held the first  workshop in March 2012. It provided a forum for the exchange of ideas  and collaboration among people who use and research air pollution  sensors to learn from their successes and challenges. Seventy people  representing federal agencies, state and local governments, academia,  private industry, and community-based organizations attended the  workshop. According to EPA officials, workshop attendees agreed that it  was helpful to have EPA convene these groups so that they could learn  from each other, and build greater trust and understanding through  collaboration and communication.
Subsequent workshops held from 2013 to 2015 focused on specific  issues, including data quality, citizen science, and community-based  monitoring. In addition to in-person attendance, the workshops were also  broadcast as webinars to allow those unable to attend in person to  participate. Each year, there was increased interest in the workshops.  More than 800 people participated in the 2015 workshop, both in person  and via the webinar. The 2015 event, which was used to provide training  on how to conduct community air monitoring, was, according to EPA  officials, designed to build on the three previous annual workshops,  whose participants requested more hands-on training opportunities. EPA  officials reported that these regular workshops helped sustain this  growing community, providing opportunities to build partnerships and  identify and address stakeholder needs.
In addition to these workshops, EPA officials continue to share  information and resources to keep individuals in the community engaged  in efforts to develop and deploy improved air sensors. For example, after  the 2015 workshop, EPA officials told us that they hosted regular follow- up conference calls with 30 in-person attendees chosen because of their  involvement in air-monitoring projects in local communities. EPA officials  also said they periodically e-mail past workshop participants to inform  them about webinars, funding opportunities, and other items of interest. In  addition, according to documentation from EPA, officials have regularly  given presentations to stakeholder groups on community air monitoring.  EPA has also made resources available to support sensor developers  and citizen scientists, establishing a sensor technology testing program to  provide feedback to developers and users, and an online “Air Sensor  Toolbox” that offers training videos and answers to frequently asked  questions about community air monitoring.
According to EPA officials, these efforts to build and sustain the  community interested in air sensor technology have contributed directly to  EPA’s strategic goal to improve air quality. Prior to the series of  workshops, EPA had little in terms of ongoing research related to the  development, testing, and use of air sensor technology. The workshops  identified a need for, and inspired, a concerted research effort to identify  promising technologies, evaluate technology performance in field and  laboratory tests, and explore the use of these new technologies and the  data they produce.
section_title: NASA Provided Support and Used Acknowledgments to Sustain a Community to Encourage Greater Use of Open Data
paragraphs: Each year since 2012, NASA has held an annual 2-day event called the  International  Space Apps Challenge. At this event, teams of scientists,  developers and students use publicly-available  data to design solutions to  identified challenges. According to NASA’s report on the 2015 event,  Space Apps is used to make the agency’s open data and assets available  to the public, with the aim of giving people new ways to produce relevant  open-source solutions to global challenges.
According to a NASA report on the Space Apps Challenge and a NASA  official, beginning with local Space Apps events in 25 cities across the  world in 2012, each year the number of local events has increased. In  2016, local events were held in 161 locations spanning 61 countries.  NASA relies on local volunteer hosts to secure venues, manage logistics,  and promote the events. Given the importance of sustaining relationships  with those at the local level experienced in hosting these events, NASA  has provided tools to help ensure local hosts have a positive experience.  For example, NASA created a toolkit that provides prospective hosts with  practical advice, guidelines, and best practices for hosting a local event.
According to the NASA Space Apps report, three months prior to the  event, NASA staff begins to actively engage with those organizing local  events. They provide weekly suggestions, reminders, and resources to  help hosts plan and manage their local events. NASA staff also convenes  periodic planning conference calls with local hosts to communicate new  information and answer questions. According to a NASA official, actively  engaging through planning calls makes a significant difference. New  hosts can ask questions of, and learn from, experienced hosts and NASA  staff. It also allows local hosts to share ideas and advice with one  another. This official also stated that, by engaging with the community,  NASA can learn more about the support that local hosts need and collect  their suggestions. For example, rather than having one planning call each  week, NASA holds three different calls to accommodate the varying  schedules of local hosts in different time zones.
According to the NASA Space Apps report, after the completion of each  year’s event, NASA also acknowledges and honors hosts and winning  participants by recognizing them on the Space Apps website, in public  reports, and through other venues, like invitations to launches. Figure 9  provides an excerpt from the website used to acknowledge finalists and  winners from previous challenges.
According to a NASA official, all of these elements combined have helped  them maintain a strong level of involvement by hosts at the local level.  For example, she said 78 percent of the hosts for 2016 local events had  returned after hosting events in previous years. She also stated that, by  regularly engaging with a community of people using the agency’s data,  Space Apps has helped NASA meet its open data goals and mandates.  For example, through feedback from Space Apps participants, NASA  officials learned how difficult it could be to use the agency’s open data.  This led to action to improve the usefulness of the datasets and house  them in one location to make them more accessible. NASA also used  feedback from Space Apps participants to redesign the agency’s websites  and make it easier for visitors to understand and use NASA data.
section_title: HHS Built Tools to Allow Communication among OpenFDA Users
paragraphs: According to FDA officials, one of the key goals of the OpenFDA initiative  is to build an open community of users around FDA data. OpenFDA  developers emphasized the importance of direct contact with, and  feedback from, external users. However, due to resource limitations, the  developers knew it would be difficult to actively monitor online feedback  boards and regularly address individual questions or requests. They  wanted to create infrastructure to both support users and make the  community somewhat self-sustaining. According to FDA officials, they  believed that an engaged community would help provide resources and  assist new users.
According to an August 2014 report from Iodine, the private health data  company that assisted FDA with the development of OpenFDA, the  infrastructure that FDA put in place relies upon two online forums,  StackExchange and GitHub. These forums facilitate communication and  information sharing among members of the community. They allow  developers and researchers who use OpenFDA to ask questions of the  broader community of users, and get answers to those questions. This  allows lessons learned and insights to be spread among the community.  According to FDA officials, these forums also allow users to recommend  fixes to problems with, and make improvements to, the OpenFDA source  code.
According to data available on the StackExchange and GitHub websites,  both forums have been actively used. For example, since OpenFDA’s  June 2014 launch, members of the community of users have submitted  more than 100 questions on the StackExchange forum. Nearly 90 percent  of those questions have been answered by other members of the  community. According to an FDA official, since OpenFDA’s launch GitHub  has also been used to identify nearly 50 issues with the OpenFDA  platform. As of June 2016, 39 of those issues have been addressed.
section_title: Agency Comments
paragraphs: We provided a draft of the report to the Office of Management and  Budget, the Office of Science and Technology Policy, the General  Services Administration, the Departments of Energy, Health and Human  Services, Housing and Urban Development, and Transportation, the  Environmental Protection Agency, and the National Aeronautics and  Space Administration for comment.
These nine agencies provided responses via emails transmitted between  September 16 and September 27, 2016. All nine agencies concurred with  the findings of the report.
In its response, provided in an email from the OSTP General Counsel  transmitted on September 22, 2016, OSTP raised a concern that the  report does not include an example of an initiative that only involved the  use of citizen science. Our primary objective for this report was to identify,  and illustrate through selected agency examples, practices that promote  the effective implementation of open innovation strategies. Therefore, our  focus was on selecting those initiatives with the greatest potential to  illustrate aspects of these practices. In making those selections, we  ensured that the sample covered the five types of open innovation  strategies frequently used by federal agencies. Although we did not  include an initiative that only used citizen science, we included initiatives  that involved the use of citizen science in combination with other  strategies, such as NASA’s Asteroid Data Hunter initiative and EPA’s  efforts to encourage the use of air pollution sensors.
In addition, DOE, HHS, HUD, NASA, and OSTP provided technical  comments, which we have incorporated as appropriate.
We are sending copies of this report to interested congressional  committees, the heads of the agencies identified above and other  interested parties. This report will also be available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of our report. Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The GPRA Modernization Act of 2010 includes a provision for us to  periodically review how implementation of its requirements is affecting  agency performance. This report is part of our response to that mandate.  Our specific objective for this report is to identify, and illustrate through  selected agency examples, practices that facilitate the effective  implementation of open innovation strategies and the effects, if any, the  use of those strategies have had on agency performance and  opportunities for citizen engagement.
To identify the various open innovation strategies federal agencies have  used to facilitate participation by, and collaboration with, citizens and  other non-profit, academic, and private sector partners, we reviewed  documents, reports, and resources from the Office of Management and  Budget (OMB), the Office of Science and Technology Policy (OSTP), and  the General Services Administration (GSA), and analyzed the Open  Government Plans of federal agencies. Through our review of these  reports, and the most recent open government plans from 35 agencies,  we identified 5 open innovation strategies that agencies have frequently  used to engage citizens and external stakeholders.
To identify practices that can facilitate the effective implementation of  open innovation strategies, we analyzed and synthesized information  gathered from a number of different sources. First, we collected relevant  federal resources, including guidance with suggested practices for  implementing various open innovation strategies developed by OMB,  OSTP, and GSA. Through a literature review of relevant publications from  public and business administration journals, and research organizations  we identified those with suggested practices for the design and  implementation of open innovation initiatives in the public sector. We then  analyzed and synthesized suggested practices in these sources to  identify areas of commonality between them. We interviewed 14 open  innovation experts with experience in implementing open innovation  initiatives or with academic or consultative expertise in this area. We also  interviewed officials involved in implementing open innovation initiatives  at six selected agencies, as well as staff from OMB, OSTP, and GSA. We  initially selected and interviewed experts based on the results of our  literature review (e.g., the authors of relevant articles or books with  suggested practices for the design and implementation of open innovation  initiatives). Based on suggestions from those individuals, we expanded  our list of experts and conducted additional reviews.
Through our analysis and expert interviews, we developed a broad set of  practices that facilitate the effective implementation of open innovation  initiatives. We refined the list of practices through our audit work at  selected agencies (see below); reviewing our body of work on  performance management and collaboration; and incorporating feedback  from the open innovation experts we had previously interviewed, and from  knowledgeable federal officials at OSTP, GSA, and other agencies.
To illustrate how actions that selected agencies have taken to carry out  open innovation initiatives have reflected effective practices, and the  effects the application of these practices had on agency performance and  citizen engagement, we selected six agencies for more in-depth review:  the Departments of Energy, Health and Human Services, Housing and  Urban Development, and Transportation (DOT); the Environmental  Protection Agency; and the National Aeronautics and Space  Administration. We selected these agencies based on several criteria,  including the number and variety of open innovation strategies outlined in  their individual agency open government plans. These selections were  also in line with suggestions we independently obtained from  knowledgeable staff at OMB, OSTP, and GSA that were familiar with  agencies that have actively used such strategies. We also identified and  selected 15 specific open innovation initiatives led by these 6 agencies  which would allow us to illustrate how these agencies have applied  effective practices for implementing open innovation initiatives. We  selected these initiatives based on our review of the open government  plans for the 6 selected agencies, and of OSTP reports on the  implementation of prize competitions and challenges. Suggestions from  knowledgeable agency staff also contributed to our selection process.  These initiatives are listed below in table 4.
At these agencies, we reviewed relevant agency documents and  interviewed knowledgeable agency officials responsible for designing and  implementing these selected initiatives. We asked these officials how they  defined goals and selected specific strategies, how they designed and  implemented their initiatives, and what steps they took to collect data and  assess results. These interviews allowed us to capture detailed  illustrations showing how agencies took actions that reflect aspects of  effective practices in the implementation of their initiatives.
The scope of this review was to identify practices for the effective  implementation of open innovation initiatives, and to describe actions  agencies took in carrying out open innovation initiatives that reflect  aspects of those practices. While we present information on the  implementation of agency open innovation initiatives, we did not assess  the success of the underlying agency programs and activities that these  initiatives were designed to support. For example, while we examined the  implementation of DOT’s open dialogues on freight transportation, we  have ongoing work reviewing various DOT activities related to issues  mentioned in the draft National Freight Strategic Plan and have not  evaluated the plan nor determined its effectiveness in helping DOT meet  its freight goals.
We conducted this performance audit from July 2015 to October 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Descriptions of Open Innovation Strategies
paragraphs: Based on our review of agency open government plans and other  sources, we found that agencies have frequently used the five open  innovation strategies below to collaborate with citizens and external  parties, and encourage their participation in agency efforts.
Crowdsourcing and Citizen Science. In crowdsourcing, agencies  submit an open call, generally through the Internet, for voluntary  assistance from a large group of individuals to complete defined  tasks. This can help the agency complete projects, such as  transcribing large numbers of historical documents, while also  producing usable products that benefit the broader community, like  searchable databases. Similarly, agencies can use citizen science to  encourage members of the public to voluntarily assist with science- related tasks. Such tasks can include conducting experiments, making  observations, collecting and analyzing data, and interpreting results.  This can supplement an agency’s own data collection efforts. It also  allows agencies to study complex issues by conducting research at  large geographic scales and over long periods of time in ways that  professional scientists working alone cannot easily duplicate.
Idea Generation (Ideation). In idea generation, or ideation, an  agency asks participants to submit ideas to address a specific issue  or problem, and may allow them to provide comments on ideas  submitted by other participants, and vote to express their support for  an idea.
Open Data Collaboration. In open data collaboration, an agency  mobilizes participants to share, explore and analyze publicly-available  data sets. Examples of open data collaboration may include using  open data to conduct research, design data visualizations, or create  web and mobile applications and websites that help people access  and use the data. Participants can also be mobilized through in- person or online events, often referred to as “data jams” or  “hackathons,” or through websites that provide access to open data  and facilitate ongoing communication.
Open Dialogue. In an open dialogue, an agency collects and  responds to information, observations, and perspectives provided by a  range of citizens and other external experts and stakeholders. They  can do this using online tools, including websites or interactive  webinars, and in-person meetings or forums. The agency can also  use open dialogues to request input and suggestions on a set of  options under consideration, and to better understand the values,  perspectives, and preferences of citizens and stakeholders.
Prize Competition or Challenge. When an agency identifies a  problem to solve or a specific goal it wants to achieve with the  assistance of members of the public, it can hold a prize competition or  challenge. In a competition or challenge, the agency invites interested  members of the public to submit potential solutions to this problem or  challenge. The agency then evaluates these proposals and provides a  monetary or non-monetary award for those that meet specific criteria  and are selected as winners.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgements
paragraphs: In addition to the contact named above, Benjamin T. Licht (Assistant  Director) and Adam Miles supervised the development of this report.  Theodore Alexander, Joyce Y. Kang, Steven Putansu, Lauren Shaman,  Erik Shive, Wesley Sholtes, and Andrew J. Stephens made significant  contributions to this report. Sarah Gilliland,  Robert Robinson and Stewart  Small also made key contributions. Shea Bader, Giny Cheong, Jeffrey  DeMarco, Alexandra Edwards, Anthony Patterson, and Timothy Shaw  verified the information in the report.
section_title: Selected Bibliography
paragraphs: Ballentyne, Perrie. Challenge Prizes: A Practice Guide. United Kingdom:  Nesta, 2014.
Brabham, Daren C. Crowdsourcing in the Public Sector. Georgetown  Digital Shorts. Washington, D.C.: Georgetown University Press, 2015.
Brabham, Daren C. Using Crowdsourcing in Government. Washington,  D.C.: IBM Center for the Business of Government Collaborating across  Boundaries Series, 2013.
Department of Health and Human Services IDEA Lab. The HHS  COMPETES Playbook, accessed on January 12, 2016,  http://www.hhs.gov/idealab/what-we-do/hhs-competes.
Desouza, Kevin. Challenge.gov: Using Competitions and Awards to Spur  Innovation. Washington, D.C.: IBM Center for the Business of  Government Using Technology Series, 2012.
Eggers, William D. and Paul MacMillan.  “A Billion to One: The Crowd  Gets Personal.” United Kingdom: Deloitte Review issue 16 (2015).
Federal Public Participation Working Group. U.S. Public Participation  Playbook, 2015, accessed on July 27, 2015, https://participation.usa.gov.
Goldhammer, Jesse, Kwasi Mitchell, Anesa “Nes” Parker, Brad Anderson,  and Sahil Joshi. “The Craft of Incentive Prize Design: Lessons from the  Public Sector.” Deloitte University Press, June 2014.
Kannan, P. K. and Ai-Mei  Chang. Beyond Citizen Engagement: Involving  the Public in Co-Delivering Government Services. Washington, D.C.: IBM  Center for the Business of Government Collaborating across Boundaries  Series, 2013.
King, Andrew and Karim R. Lakhani. “Using Open Innovation  to Identify  the Best Ideas.” MIT Sloan Management Review, September 11, 2013.
Lee, Gwanhoo. Federal Ideation Programs: Challenges and Best  Practices. Washington, D.C.: IBM Center for the Business of Government  Using Technology Series, 2013.
Lee, Gwanhoo and Young Hoon Kwak. An Open Government  Implementation Model: Moving to Increased Public Engagement.  Washington, D.C.: IBM Center for the Business of Government Using  Technology Series, 2011.
Luciano, Kay. Managing Innovation Prizes in Government. Washington,  D.C.: BM Center for the Business of Government Collaborating across  Boundaries Series, 2011.
Lukensmeyer, Carolyn J., Joe Goldman, and David Stern. Assessing  Public Participation in an Open Government Era: A Review of Federal  Agency Plans. Washington, D.C.: IBM Center for the Business of  Government Fostering Transparency and Democracy Series, 2011.
McKinsey & Company. “And the Winner is…” Capturing the Promise of  Philanthropic Prizes. McKinsey & Company, July 2009.
Mergel, Ines. “Opening Government: Designing Open Innovation  Processes to Collaborate with External Problem Solvers.” Social Science  Computer Review vol. 33, no. 5 (2015): 599-612.
Mergel, Ines and Kevin Desouza. “Implementing Open Innovation  in the  Public Sector: The Case of Challenge.gov.” Public Administration Review  vol. 73, no. 6 (November/December 2013): 882–890.
Nabatchi, Tina and Matt Leighninger. “Participation Scenarios and  Tactics,” Public Participation for 21st Century Democracy. Hoboken, NJ:  2015, 241–285.
Nambisan, Satish. Transforming Government through Collaborative  Innovation. Washington, D.C.: IBM Center for the Business of  Government Innovation  Series, 2008.
Nambisan, Satish and Priya Nambisan. Engaging Citizens in Co-Creation  in Public Services: Lessons Learned and Best Practices. Washington,  D.C.: IBM Center for the Business of Government Collaborating across  Boundaries Series, 2013.
Noveck, Beth Simone. Smart Citizens, Smarter State: The Technologies  of Expertise and the Future of Governing. Cambridge, MA: Harvard  University Press, 2015.
Office of Management and Budget, Executive Office of the President of  the United States. The Common Approach to Federal Enterprise  Architecture. Washington, D.C.: May 2, 2012.
Office of Science and Technology Policy, General Services  Administration, and Federal Crowdsourcing and Citizen Science  Community of Practice. “Federal Crowdsourcing and Citizen Science  Toolkit,” adaptation of Bonney et al., “Citizen Science: A Developing Tool  for Expanding Science Knowledge and Scientific Literacy.” BioScience  59(11), 977-984 (2009), accessed on January 26, 2016.  https://crowdsourcing-toolkit.sites.usa.gov/howto.
Tong, Raymond and Karim R. Lakhani. Public-Private Partnerships for  Organizing and Executing Prize-Based Competitions, Research  Publication no. 2012-13. Cambridge, MA: The Berkman Center for  Internet and Society at Harvard University, June 2012.