Office of Personnel Management: Agency Needs to Improve Outcome Measures to Demonstrate the Value of Its Innovation Lab
Released: 2014-04-30T00:00:00

Letter

Background

Innovation is a dynamic process through which problems and challenges are defined, new and creative ideas are developed, and new solutions are selected and implemented. It is also a complex process that involves taking iterative steps to solve problems. Innovation requires an environment that encourages participants to challenge traditional practices without fear of repercussions. Ideally, innovation participants are empowered to be creative and make mistakes, and appropriate risk- taking is not only tolerated but encouraged.

Some federal leaders are trying various innovation tools, including on-line idea submission programs, competitions, and prizes, as ways of unleashing employee creativity. For example, in a memorandum issued in March 2010, the administration urged federal agencies to use challenges and prizes to crowdsource innovative approaches to government initiatives and programs. At relatively low costs, crowdsourcing initiatives can garner valuable and creative solutions that may not have come through traditional means. As another example, the Presidential Innovation Fellows program pairs top innovators from the private sector, nonprofit organizations, and academia with top innovators in government to collaborate during focused six- to thirteen-month periods. The program aims to develop solutions that can save lives, save taxpayer money, and fuel job creation. For example, the goal of one of the program’s projects is to identify information critical to saving lives and mitigating damage in a disaster.

Even with the efforts of some federal leaders to encourage innovation, federal government-wide scores tracking how agencies foster and reward employee innovation dropped in 2013 for the second year in a row. OPM’s 2013 Federal Employee Viewpoint Survey, released in November 2013, found that only 35 percent of federal workers believe that creativity and innovation are rewarded, with positive responses in this area showing a steady decline of six percentage points over the past three years. Research suggests that half of all innovations are not initiated by organizational leaders. Instead, research shows that it is important to have processes for gathering stakeholders’ and front-line workers’ views to identify areas for possible improvement.

As an innovation tool, labs are based on the idea that the competencies needed for systematic innovation—such as intelligent risk-taking to develop new services, products and processes—are not the same as those required for daily operations. Innovation labs seek to provide approaches, skills, models, and tools beyond those that most employees are trained in and use to do their work. In addition, public sector innovation labs can be viewed as attempts to create an organizational response to a range of challenges to innovation, as innovation efforts face unique obstacles in the public sector. For example, funding for new public ventures is limited, and the risks of innovation are high in government. A defining characteristic of the public sector is that it is subject to broad scrutiny, so that when an innovation fails or is less than a complete success, there is the prospect of political consequences.

With constrained budgets expected to continue into the foreseeable future, innovation in our public services is a necessity. In the last decade, many public sector organizations around the globe have set up facilities with the explicit purpose of supporting innovation efforts. For example, Denmark’s MindLab, started in 2002, is a cross-governmental innovation unit that is part of the country’s Ministry of Business and Growth, the Ministry of Education, the Ministry of Employment, and Odense Municipality, and which collaborates with the Ministry for Economic Affairs and the Interior. The group covers broad policy areas including areas such as entrepreneurship, digital self-service, education, and employment. OPM’s lab was modeled, in part, on Denmark’s MindLab.

OPM officials, consistent with other innovation lab representatives we interviewed, maintained that, unlike a typical conference room, innovation labs can be easily reconfigured for large groups and smaller breakout sessions. They allow users to write on walls and preserve visual artifacts more easily than typical cubicles and traditional office space. This can be done with very low-tech tools such as markers and a whiteboard. Figure 1 shows a view of OPM’s lab.

Organizations with different missions are pursuing a lab-based strategy to foster innovation. For example, organizations—including OPM—use their labs as a space where participants can conceptualize and prototype new products or processes outside their normal environment. Many also use their labs as a teaching space where participants can exchange ideas and information through classes, workshops, presentations, or other events. Figure 2 shows how innovation labs we surveyed from the public, private, and nonprofit sectors share common design elements, and how these different organizations generally use their labs for multiple and similar purposes.

OPM Lab Activities Are Intended to Build Capacity for Innovation and Support Project- Based Problem Solving

Based on OPM documents, the innovation lab’s start-up costs totaled approximately $1.1 million including facility upgrades and construction, equipment, and training and other personnel costs. (See table 1 for a breakdown of costs.) In building the lab, OPM worked with the General Services Administration (GSA) and contracted with both design and architectural firms to renovate a former storage room in the sub-basement of its headquarters building. The 3,000 square foot renovated space presents an open layout with a meeting area for up to two dozen people and is surrounded by breakout areas and team rooms. The physical renovation of the facility was completed in March 2012, after the installation of final technology equipment, asbestos abatement, and enhancements to ventilation and life-safety systems. According to OPM, to make the space useful for any purpose, much of the funding for the improvements and construction of the space would have been required.

OPM officials said that in fiscal year 2013, the lab’s total operating budget, including all contracting costs, was $476,000, which supported a build up to 5.5 full-time equivalent (FTE) employees over the last seven months of fiscal year 2013. Officials expect this amount will remain stable in the coming fiscal years, proportional to a full fiscal year.

OPM’s Lab Is Managed and Operated By a Core Group of Staff

Operational responsibility for the innovation lab has been assigned to OPM’s Employee Services Division and is managed by the agency’s Deputy Associate Director of Strategic Workforce Planning. According to OPM officials, since February 2013, the lab has grown from 1 FTE to roughly 6 FTEs. Specifically, as of the end of summer 2013, day-to-day operations in the lab are carried out by 4 FTE staff members, 1 FTE intern, 1 part-time intern, and 1 part-time staff member whose time is divided between the innovation lab and OPM’s Resource Management Office. A core group of staff from OPM’s Employee Services Division have been trained in human-centered design and also contribute up to 15 percent of their time in the lab. According to OPM officials, the lab reached its maximum fiscal year 2013 funding level of 5.5 FTEs in July 2013. A brief description of each position is provided in table 2.

OPM Employed a Phased Approach to Lab Development Initially Focusing on Capacity Building

OPM has taken a phased approach to developing the lab programming (activities taking place in the lab) and the policies governing lab use (such as priority-setting policies for lab projects). According to OPM documents, each phase has incorporated an element of experimentation, review, and a shift in strategy based on lessons learned.

Phase I lasted from March through June 2012. During these first 4 months after the lab was built, OPM made the space available to the OPM workforce for meetings and events. OPM leadership also used this time to investigate an appropriate problem-solving approach to pair with the lab that would be consistent with approaches used by other labs; they determined that a human-centered design approach and curriculum would complement OPM employees’ technical expertise and analytic competencies. OPM also began to recruit interested staff from the Employee Services Division to be trained in human-centered design fundamentals: this staff would then support project sessions in the lab as part of their collateral duties.

Phase II lasted from July 2012 through March 2013 and consisted mostly of facilitated sessions with OPM project teams. During these sessions, Employee Services staff worked with project teams to generate ideas to long-standing problems through exercises such as project or strategic planning, brainstorming sessions, or stakeholder mapping aimed at discussing and testing potential solutions. Topics discussed during these sessions involved a variety of initiatives directed at improving OPM processes and addressing government-wide human resources challenges. OPM officials noted that innovation lab projects have included, among others, designing an implementation plan with other federal agencies to collect valid, accurate, and timely data on the federal cyber security workforce; updating the government-wide strategy for veterans recruitment; attracting and retaining individuals with talent in science, technology, engineering, and mathematics disciplines; and specific challenges unique to individual agencies.

Phase III lasted from April through November 2013. In Phase III, OPM continued to provide facilitated design sessions and in some cases, follow-on coaching to program offices from within OPM and OPM-led projects. For example, a facilitated design session included lab staff working with the Food and Drug Administration’s (FDA) Battery Working Group to more effectively engage with the group’s external stakeholders. According to an OPM case study about the lab, eight FDA employees attended the Fundamentals of Human-Centered Design course. Following the course, lab staff provided planning support for a public workshop with over 200 participants from stakeholder groups including medical device and battery manufacturers, other regulatory groups, and hospital staff; lab staff also attended the Battery-Powered Medical Device workshop to support the FDA team in their use of design methods. According to an FDA participant, their collaboration with the lab helped them engage in stakeholder dialogue that would not have been otherwise possible. OPM lab staff also began to offer classes in the lab designed to develop mission-critical skills federal workers need to become better problem solvers. The OPM lab offers courses such as Human-Centered Design Fundamentals, Prototyping in the Public Sector, and Communicating Visually, among other topics. These classes are available to OPM staff and other federal workers. Staff also made the lab available for federal communities of practice to convene. According to lab staff, the lab is becoming a hub for a number of standing meetings of a growing community of federal innovators and innovation communities of practice.

Table 3 presents a summary of OPM’s human-centered design lab activities since its inception.

Lab staff report that in the future they intend to expand from their session- based work and targeted design support projects, such as those consultative sessions that took place in the lab during Phase III. While some of these more episodic projects may continue to occur, the lab’s focus will be on creating and establishing large-scale projects typically involving stakeholders from either wholly within OPM or across different agencies that are working on crosscutting issues. As discussed later in the report, projects appropriate for this design method would have diverse users, be more complex, and be called immersion projects. These would be the most structured activities undertaken in the lab, characteristically being longer-term activities that could take up to six months of intense collaboration with project owners and a diverse group of stakeholders.

OPM’s Lab Is Similar in Mission and Design to Other Innovation Labs, but OPM Needs to Systematically Evaluate the Lab’s Performance

We identified a common set of challenges that can undermine organizations’ efforts to use innovation labs and a set of prevalent practices that the organizations employ to address these challenges and support their labs’ success and sustainability. OPM has incorporated some of these practices, such as pairing a distinct space with a structured approach to problem-solving, but has not implemented others, such as developing meaningful performance measures. Although OPM has begun reaching out to other federal innovators, the agency has not fully leveraged the experience of other agencies employing similar approaches.

OPM Has Dedicated Physical Space, a Problem-Solving Approach, and Plans to Undertake Long-Term Immersion Projects

As a prevalent practice for encouraging and supporting greater innovation, both the literature and representatives from the organizations we reviewed stressed the benefits of pairing a dedicated physical space with a structured framework rooted in design-thinking principles. Many of the lab representatives said building or establishing a distinct space carries an important symbolic value as it signals an organization-level commitment to a culture that supports innovation. However, simply building a lab is not sufficient to change an organization’s culture; it is necessary to also introduce a new framework for problem solving. Although these organizations use different terminology to describe their selected frameworks, such as agile development and human-centered design, the general principles are similar. They include placing users at the center of the desired solution—research on successful innovation practices shows the importance of engaging customers and understanding their needs. Further, they include extensive collaboration with relevant stakeholders, experimentation, prototyping, and iterative steps to find a solution. A primary objective of this approach is to allow for failure in the beginning of the design cycle, so that organizations can manage and learn from early mistakes, rather than try to recover from an expensive, comprehensive failure upon implementation.

For example, Census and HUD have similar problem-solving frameworks for lab use. As figure 3 shows, the Census Center for Applied Technology and the HUD Innovation Lab rely on a five-step framework to guide innovation in their labs.

As another example, although CFPB does not have a dedicated physical space, it used a similar framework to develop its on-line mortgage disclosure form. According to CFPB’s Creative Director for Technology and Innovation, designers interacted with end-users including mortgage applicants, prototyped different forms, and made refinements based on continual feedback before launching the new form. On its website, CFPB describes its design process in detail, including prototyping and feedback sessions with consumers, lenders, and mortgage brokers.

OPM’s lab provides a menu of design services to meet the specific needs of various projects. The lab’s larger-scale immersion project work will involve taking a complex problem through OPM’s problem-solving framework, which encompasses steps for problem framing to learning about users to analysis to concept development, testing, and rapid iterative steps. This problem-solving framework is similar to those employed by other innovation labs.

As discussed earlier in this report, similar to other organizations with labs, OPM is using its lab for a variety of purposes including as a learning space for classes on human-centered design principles and techniques and as a meeting space for interagency task teams and communities of practice. As originally envisioned in its strategic and performance plans, the lab was designed to host a mix of activities rooted in the human- centered design approach, including longer-term design challenges. Moreover, OPM lab staff asserted that to gain an organization’s confidence and to instill a culture of innovation, it is necessary for a successful innovation lab to have an array of sufficiently compelling projects that demonstrate how the lab approach can lead to performance improvements.

Based on our interviews with public and private sector organizations with similar innovation facilities, larger-scale problem-solving projects were common activities in their innovation labs’ service portfolios. As an example, Denmark’s MindLab has contributed to tackling several pressing social issues including simplifying the process for managing claims related to industrial accidents and shortening the time before injured workers return to the market. Opportunities to showcase a new approach to problem solving reduce the likelihood that the lab might lose its distinction as different from a traditional meeting space or classroom usually associated with training facilities.

Consistent with what staff from other labs told us, OPM officials said they needed the past two years to first introduce agency staff to human- centered design concepts and applications before they could initiate an immersion project. Lab staff said this phased approach was necessary for several reasons. Targeted design support sessions allowed lab staff to expose lab users to design methods and provided opportunities for collaboration. These sessions also allowed lab staff to quickly show value for program offices in response to a specific need. For example, OPM lab staff members were able to help FDA staff plan and engage with over 200 different stakeholders at a conference. They also said targeted design support is a critical way for emerging design practitioners to develop and hone their own skills before applying them to a longer term, and higher stakes, project engagement. Lab staff said overall these sessions benefited both the users of the lab, who developed new skills to take to their home offices, such as problem framing and engaging with stakeholders, as well as lab employees, who continue to grow and refine their human-centered design skills. In a December 2013 document, OPM staff stated they intend to create and establish these longer-term immersion projects and evaluate their impact during the next phase of the lab’s development.

OPM Has Not Fully Developed Meaningful Performance Measures

Measuring the long-term outcomes of innovation labs is a prevalent practice for building acceptance and demonstrating the value of the labs. Consistent with our literature review, several representatives we interviewed from other innovation labs concurred with the director of innovation at Denmark’s innovation lab, MindLab, who said that innovation labs need to know how much they are spending and their outcomes. According to the director, the labs must also be able to attribute where the change happens based on their work. In addition, lab staff must be prepared to present a narrative of their work. He acknowledged that innovation labs are risky because they look different, and they have a different focus than other government entities. The director said that, as a result, innovation lab officials need to show where the funds are going along with the benefits and results of those investments.

Representatives from newer labs—i.e. those operating less than three years—stated they primarily rely on output measures to gauge their initial efforts such as number of users, ways in which the lab is being used, classes or events held in the lab, and anecdotal evidence. Developing outcome measures is more challenging for several reasons. Appropriate outcome measures are often not obvious at the onset of a project. Moreover, agencies may not have appropriate measures or baseline data when they start using an innovation lab as a problem-solving tool, and the role of the lab in driving a successful innovation may not always be clear.

Given these challenges to accurately measuring innovation and the value of an innovation lab, lab managers from labs that have been operating for a longer period of time told us they focus on developing meaningful milestones and measures applicable to different phases of the innovation lifecycle, such as problem generation, idea generation, and skills development. For example, the UNICEF Innovation Unit—which has been helping member-country offices set up innovation labs since 2006—and several European initiatives developed a set of benchmarks intended to help them measure the value of public sector labs and identify ways in which the lab’s performance can be improved. The benchmarks UNICEF developed span across six categories, such as problem definition and idea generation, internal and external collaboration, and secondary effects. Within each category, they include a list of questions intended to assess their strengths and weaknesses. For example, they want to know whether labs are helping employees define problems and generate ideas, strengthen internal collaborations, and build external partnerships. They also measure the extent to which work done in the labs results in new team or staff capacity, excitement and goodwill toward the organization, and an increase in leverage and influence in their field. OPM is undertaking a similar effort to establish benchmarks that will help lab staff gauge the extent to which lab users are learning and applying many of these same skills, but the lab is not mature enough to have results.

OPM documents state that the goal of OPM’s innovation lab is to provide federal workers with 21st century skills in design-led innovation, and the intended purpose of the lab is to provide a physical space for project- based problem solving. The documents also note that the value of the lab can be measured, in part, by how well it helps develop the mission-critical competencies to improve the federal workforce’s ability to solve problems and deliver results.

In its strategy document, OPM laid out the following high-level goals for the innovation lab:

Employees assigned to the innovation lab should go back to their home organization with an understanding of, and an appreciation for, the power of innovative approaches to problem solving.

Employees should be equipped to implement similar methodologies in their home organizations on future projects.

As the innovation lab matures, and as more and more projects are completed, the notion of using innovation to tackle complex problems will gain traction across the organization.

Eventually, leaders and employees across OPM will vie to get their issues sent to the innovation lab for resolution.

This in turn will contribute to a decrease in organizational silos, and a concurrent increase in cross-organizational teams addressing one organization’s issues.

In the same document, OPM officials also described an evaluation strategy resembling an agile approach. Specifically, OPM described these goals as moving targets which would be achieved through an evolving and self-correcting process. Lab staff immediately started to track lab activities and outputs, such as number of participating people and agencies, and how participants used the lab, such as consultative sessions, follow-on coaching, training classes, or as a meeting space. Five months after the lab opened, they also started to survey users who participated in day-long facilitated sessions. For example, there was a one-page evaluation that asked respondents to rate the appropriateness of the environment and quality of the facilitators. The surveys also asked whether users would recommend the lab to colleagues and whether human-centered design problem-solving tools can be used as an effective tool government-wide. The responses were generally positive—about 82 percent of respondents (84 out of 103) said they would recommend the lab to someone else, providing a baseline for subsequent survey findings. According to lab staff, they periodically reviewed the available data and adjusted their strategy for operating the lab.

Starting in March 2013, a year after the lab opened, OPM lab staff began work on a program evaluation framework to more systematically measure the lab’s progress toward meeting their overarching goals. To evaluate the extent to which lab participants are learning and applying innovative approaches, lab staff intends to measure the lab’s performance along three overarching categories: service experience, skill development, and project outcome. According to the framework, resources dedicated to evaluation efforts will reflect the resources needed to host lab-sponsored events. Episodic events such as consulting sessions will correspond to a “light-touch” follow-up effort, such as immediately surveying all participants on their session experience and skills development. More long-term, resource-intensive efforts such as immersion projects will employ a more robust follow-up effort that, in addition to assessing the session experience and skills development, will also address project- specific outcomes. Collection of assessment data in all three areas will include the administration of surveys to participants both before and right after a session, and some services will involve the administration of surveys to participants before a service and subsequent periodic check- ins. Depending on the nature of the lab session, information on skill development and outcomes will also be obtained from session clients in pre-session scoping conversations and periodic, post-session check-ins using either surveys, or interviews. For one type of session, assessment of participant skill development will also include a survey of participant supervisors.

Lab staff has used a series of surveys to measure participant experience and skills development, and to capture specific project-related outcomes for the different services they offer. However, the survey instruments are unlikely to yield data that would be of sufficient capacity, credibility, and relevance to indicate the nature and extent to which the lab is achieving what it intends to accomplish or its value to those who use the lab space. Although there are several items across all surveys that are reasonably aligned with generally accepted questionnaire and item design principles, there are limitations associated with many items where language is ambiguous, where the intent of the question is not clear, and directions are lacking. For example, phrases such as “changed behavior” or “tangible outputs you can move forward” are open to numerous interpretations and are likely to engender an array of responses that range from being relevant to not at all relevant or relatable to the purposes or objectives of the session. In addition, some of the items may be more likely to engender responses with a greater likelihood of being subject to a respondent’s social desirability bias. For example, the respondent may want to provide answers that are socially desirable, maintain the status quo, or make a good impression.

While some customization is to be expected, the surveys did not indicate any approach to evaluate some core aspects of the lab and its value using a consistently presented set of the same questions. For example, the question asking participants about the likelihood that they would recommend the lab to someone else is the type of item that could, with revision, be incorporated in all of the surveys. Analyses of a core set of items by type of lab event or service would enable lab staff to discern and compare where participants were more and less engaged in lab activities and curricula. Consequently, these survey instruments and the items on them may be susceptible to various types of question and respondent bias and could, when the responses are analyzed, produce results that would be difficult to interpret or link to expected participant effects, or to the intent or activities of the workshop session. Moreover, lab staff has not developed outcome measures or milestones related to customer experience and skills development.

The evaluation framework being developed by OPM does not include interim performance targets or measures. Best practices state that new initiatives benefit when managers set time-bounded, quantifiable interim goals, establish related performance measures, collect data, and use that information to assess and adjust their performance. To evaluate the overall performance of MindLab, the director said he develops an annual work plan, which describes the number and types of projects and other activities the lab will undertake, as well as the relative resource allocation to those projects and activities. He said his staff also conducts an annual review of the budget and actual expenditures with the board. OPM lab staff has been tracking outputs—such as number of participants and number and type of activities—meaning that they have baseline data which could inform realistic, meaningful targets and measures related to lab use and activities for the upcoming year. Although they continue to refine their surveys, they could use the results from earlier versions to establish targets and measures related to customer experience and skills development. Meaningful measures or milestones could help them assess their progress toward improving participants’ ability to solve problems and accurately measure the effect of working in the lab on services, products, and processes.

As mentioned previously, the lab plans to host the more resource- intensive immersion projects. To demonstrate that the lab is operating as originally intended, evaluation plans will be needed for specific immersion projects that can help track cost-benefits and performance improvement outcomes. OPM stated that evaluation plans will be prepared for each immersion project to account for project outcomes. They indicated that they wanted to host their first immersion project within the next several months.

OPM Has Started Efforts to Connect with the Federal Innovation Community, but Has Not Fully Leveraged Other Agencies’ Efforts

Another prevalent practice we identified included leveraging other innovation labs’ efforts to try to increase the value of the lab approach. Studies show that information sharing and interorganizational networks can be a powerful driver supporting innovation. One study showed that interorganizational networks of innovators help members develop new products at a faster rate with lower investment commitments, due in large part to the information sharing that takes place. Sharing information can help mitigate the risks and uncertainty that typically characterize innovation ventures. Best practices state the importance of establishing channels of communication and other mechanisms that facilitate knowledge-sharing and building networks of like-minded communities to help agencies achieve crosscutting objectives. For example, the Census Bureau’s Chief Technology Officer suggested a way in which innovation leaders could share information and pool resources. Specifically, instead of each agency creating its own technology innovation lab with its own hardware, software, and associated maintenance, they could use a common innovation infrastructure service in the public cloud. Every agency could still have their own branded offering and could still provide access in their own facility or at their regional offices. However, an outside vendor could provide the infrastructure. For example, if an agency wanted to experiment with some unique visual analytic tools, they could purchase what they need on a subscription service; this would eliminate agencies buying all the tools themselves.

While labs provide a physical space where innovators can convene, federal agencies are not fully aware of their growing community. As of June 2013, OPM was unaware that other agencies such as Census, HUD, and NASA were pursuing a lab approach to promote innovation. Moreover, the lab directors at these agencies were not aware or only marginally aware of OPM’s lab and its resources or other federal innovation labs. OPM’s efforts to develop an innovation lab occurred around the same time or pre-dated those of other agencies we interviewed.

According to OPM officials, during its first year of operations, OPM lab staff focused their efforts on promoting awareness of the lab and its resources internally to OPM staff. In their second year, OPM lab staff planned more activities intended to promote the lab and its resources externally to connect with federal agencies’ innovation efforts. Staff noted the OPM lab is the hub of various interagency networks of innovation practitioners. For example, an interagency community of practice on idea generation meets in the lab on a monthly basis. OPM’s lab staff also reported that they host weekly trainings in the lab on best practices, including webinars about measuring the success of enterprise-level design efforts and the value of visualizing information. These training sessions include case study presentations from other federal agencies, such as GSA, and non-federal entities. In addition, OPM has shared best practices with other public sector design labs across the globe by participating in a number of conferences. OPM is also collaborating with a current Presidential Innovation Fellow, who is building an innovation toolkit. Although projects in the lab are currently managed and for the most part delivered by OPM employees, staff noted that they are increasingly looking to leverage detailees, short-term assignments, and other ways to harness the potential of talent from other agencies. OPM staff said they also give regular tours of their innovation lab for other government entities already supporting innovation initiatives or developing them. In addition to these activities, OPM hosted a convening of federal innovators to compare various agencies’ innovation communication efforts across the agencies.

Several federal officials we interviewed said they would welcome the opportunity to communicate as the need arose with a community of peers to exchange information and ideas and trouble-shoot problems related to the start-up and maintenance of their labs. For example, CFPB’s Creative Director for Technology and Innovation said it would be helpful to find out what other bureaus and departments are doing to incorporate design principles, so that she could exchange ideas and information. An official from NASA’s Swamp Works noted that it would be beneficial to show that others in the federal sector are also looking at innovation labs. To that end, simply knowing the innovation community exists and how agency staff leading innovation efforts can initiate a conversation related to a specific topic would likely be beneficial and would help avoid the risk of a fragmented innovation community.

OPM’s 2014 through 2018 Strategic Plan Indicates It Intends to Use the Lab to Advance OPM Priorities

Because innovation necessarily entails culture change, experimentation, periodic setbacks, and often resource investments, another prevalent practice necessary to sustain a lab includes leadership support. Innovation labs are one tool that agencies can use to foster innovation. Agency officials and lab directors we interviewed said leaders must be willing to embrace experimentation within the lab and understand that smart failures—failures that result from trial and error, where the alternative would be to do something truly risky due to lack of evidence— are part of the design process. For example, the Census Bureau’s Chief Technology Officer noted that support by Census Bureau leadership is critical to ensure staff participation and the continued availability of funds to drive innovation in its Center for Applied Technology lab. Other lab directors highlighted several strategies they use to balance the risks and failures that accompany a problem-solving methodology rooted in a more experimental approach. These include accelerated timelines of three to six months, which allows organizations to quickly shelve projects that are without merit. Some lab leaders also noted that a quick win or early success can give new labs the underlying support they need to take on riskier projects.

In March 2014, OPM released its 2014 through 2018 strategic plan, which states that the agency plans to seek new, innovative ways to accomplish its work of advancing human resource management in the federal government. In the strategic plan, OPM indicates that, among other things, it intends to use the innovation lab and human-centered design methods to address OPM’s operational challenges.

Conclusions

For OPM and the rest of the federal government, finding more efficient and effective ways of doing business to help meet rising citizen demands for public services is critical, particularly in an era of continued fiscal and budgetary constraints. OPM’s innovation lab is one such tool intended to give rise to solutions of complex problems facing the federal government.

Consistent with other innovation labs, development of performance and outcome measures, tools to assess performance, and further leveraging the experience of other organizations undertaking similar efforts will also be critical. Having clear and specific outcome measures will help OPM track and evaluate the extent to which the lab is meeting its original intent and over time, to make any necessary adjustments. Otherwise, OPM’s innovation efforts may not be able to demonstrate the types of results initially envisioned.

Recommendations for Executive Action

We recommend that the Director of OPM take the following actions to help substantiate the lab’s original goals of enhancing skills in innovation and supporting project-based problem solving:

Direct lab staff to develop a mix of performance targets and measures to help them monitor and report on their progress toward lab goals. Output targets could include number and type of lab activities over the next year. Outcome targets and measures should correspond to the lab’s overarching goals to build organizational capacity to innovate and achieve specific innovations in concrete operational challenges.

Direct lab staff to review and refine the set of survey instruments to ensure that taken as a whole, they will yield data of sufficient credibility and relevance to indicate the nature and extent to which the lab is achieving what it intends to accomplish or is demonstrating its value to those who use the lab space. For example, lab staff should consider the following actions:

Developing a standard set of questions across all service offerings.

Revising the format and wording of existing questions related to skills development to diminish the likelihood of social desirability bias and use post-session questions that ask, in a straight- forward way, about whether, or the extent to which, new information was acquired.

Replacing words or phrases that are ambiguous or vague with defined or relevant terminology (e.g., terms actually used in the session) so that the respondent can easily recognize a link between what is being asked and the content of the session.

Direct lab staff to build on existing efforts to share information and knowledge within the federal innovation community. For example, OPM lab staff could reach out to other agencies with labs such as Census, HUD, and NASA’s Kennedy Space Center to share best practices and develop a credible evaluation framework.

Agency Comments and Our Evaluation

We provided a draft of this report to the Director of OPM for review and comment. The director provided written comments, which we have reprinted in appendix IV. In summary, OPM generally concurred with our recommendations and described ongoing and planned steps to refine evaluation efforts and further leverage other federal innovation labs.

For the recommendations on evaluating performance, the director described a competency-based skills gap pilot the lab is undertaking, based on targets from pre- and post-testing of participants in lab activities. We acknowledge that this is an important step in developing performance measures, and OPM will also need targets and measures to demonstrate the lab’s value in achieving specific innovations in concrete operational challenges.

For the recommendation on leveraging other federal agency innovation efforts, the director noted OPM’s work seeking out information and contacts from other innovation endeavors, including lab-based ones. We acknowledge OPM’s more recent emphasis in this area, including participating in an interagency community of practice on innovation. Federal officials we interviewed said they would welcome the opportunity to communicate as the need arose with a community of peers.

To clarify, the report recognizes sustained organizational leadership as a prevalent practice for the success of innovation labs. However, this was not a specific report recommendation, but an acknowledgement regarding the general role leadership plays in ensuring the success of innovation labs. In her response, the director stated that OPM recently released its 2014 through 2018 strategic plan, which she said demonstrates OPM leadership’s commitment to the advancement of work in the lab. Accordingly, we updated the report to reflect the most current information available at the time of our publication.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Director of OPM and appropriate congressional committees. This report will also be available at no charge on our website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-4749 or bagdoyans@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

This appendix provides information on the scope of work and the methodology used to (1) describe the Office of Personnel Management (OPM) innovation lab’s start-up and operating costs, staffing and organization, activities, and policies governing the lab’s use, and (2) assess how OPM’s innovation lab compares to other organizations’ innovation labs, including how it uses benchmarks and associated metrics and how it addresses potential challenges to innovation.

To address the first objective, we reviewed documentation and met several times with OPM staff overseeing the lab and its activities. We reviewed the lab’s construction and operations’ budget, including the funding sources for the lab, and interviewed agency officials knowledgeable about the lab’s budget. Based on interviews and e-mail exchanges with knowledgeable OPM and General Services Administration staff and reviewed documents, we found OPM’s lab expense data to be sufficiently reliable for the purposes of our report. We reviewed spreadsheets maintained by lab staff tracking lab outputs, such as workshops hosted in the lab and number of attendees. We also reviewed lab performance materials such as the lab’s performance plan, user surveys, and the results of those surveys. Survey specialists in our Center for Design, Methods, and Analysis reviewed the lab user surveys using internal review guidance that is typically performed on draft GAO surveys as part of our development process and required before deployment of a survey. In addition to reviewing with lab staff the documents they provided us, we interviewed them about OPM’s process for identifying and selecting a lab strategy, lab staff’s approach to implementing a human-centered design curriculum, and their goals for the lab.

To address the second objective, we conducted a detailed literature search of material from academic institutions, global management consultants, professional associations, think tanks, news outlets, and various other organizations. We also reviewed literature documenting public, private, and academic innovation efforts and associated positive and negative outcomes. Our literature search helped us identify benchmarks and associated metrics applicable to the development and use of innovation facilities in the public, private, and nonprofit sectors. We also interviewed OPM lab staff on how they intend to identify outcomes— such as cost reductions, performance improvements, or other results— from projects undertaken by OPM since the inception of the lab.

We used the findings from our literature review to identify organizations with innovation facilities having a dedicated physical space and using problem-solving methods similar to OPM’s lab. We selected a mix of 11 public, nonprofit, and private organizations to visit or interview. In addition, we met with an official from the Consumer Financial Protection Bureau (CFPB). While CFPB lacks a dedicated innovation lab, the agency has a reputation among federal agencies as a leader in innovative website development. Table 4 lists the organizations we visited in person or interviewed their representatives by telephone.

At every lab we visited or contacted, we interviewed lab representatives about the history of the lab, including why they decided to pursue a lab strategy; how the lab is used; the protocols for engaging participants; how lab directors measure the performance of the lab; challenges to promoting innovation within the organization; and practices for addressing those challenges. Based on our literature search, we identified common challenges that can hamper organizations’ efforts to use labs as innovation vehicles and prevalent practices that can support labs’ success and sustainability. In addition, we reviewed our interview records to identify commonly recurring challenges and prevalent practices that can support labs’ success and sustainability. We verified that the challenges and prevalent practices we identified during our literature search were also those more often cited during the interviews.

We also interviewed representatives from two management consultancies that promote problem-solving approaches rooted in design-thinking principles, IDEO and Luma. OPM contracted with Luma to help design the lab and implement human-centered design programming. We spoke with their representatives to understand the challenges their clients face in changing organizational culture and the benchmarks and metrics they advise their clients to adopt to measure the performance of new labs and problem-solving methods.

In addition, we interviewed officials from three public-sector organizations—San Francisco Mayor’s Office of Civic Innovation, Canada’s Public Policy Forum, and United Kingdom Behavioural Insights Team—that are pursuing strategies to promote innovation in their organizations but opted not to build innovation labs. They spoke to us about the challenges that prevented them from building labs and the steps they are taking to incorporate human-centered design-like problem- solving methods without a physical lab.

We conducted this performance audit from July 2013 to March 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Innovation Labs Share Similar Design Elements and Problem-Solving Methods

This table contains the same text portrayed in figure 2 and shows how innovation labs we surveyed from the public, private, and nonprofit sectors generally use their labs for multiple and similar purposes.

Purpose of lab To help deliver on President Obama’s vision of a more effective and efficient government for the American people by supporting a government-wide community of innovators.

How is it used Dedicated space where OPM and interagency teams can take a problem through a full design cycle. This includes problem framing, learning about users, analysis, concept development, testing, and rapid iteration. Capacity building—lab hosts classes and workshops. Meeting space for OPM staff and other federal workers and interagency communities of practice.

To provide a “safe zone” where Census staff can explore new technology solutions without impact to production operations.

Dedicated space where lab and other Census staff can develop, test, and implement ideas into production outside the standard production environment Capacity building—lab hosts presentations of new technologies and solutions.

To provide a dedicated space where NASA KSC engineers can quickly resolve problems related to deep-space exploration.

Space where 20 NASA engineers and scientists prototype and test emerging technologies. Swamp Works has contracts with other NASA centers worth about $7 million a year.

Provide a space that looks different from the traditional office environment, where HUD employees can accelerate the development of solutions more efficiently than other available approaches.

Dedicated space where HUD lab staff and mission area leads can develop, test, and implement ideas into production within a compressed timeframe.

Provide a neutral space for government ministries to work with citizens and businesses to create new solutions for society.

Ministry officials from Danish Ministries of Business and Growth, Education, and Employment use the MindLab space and resources to take a problem through the full design cycle. Capacity building—lab hosts classes, conferences, and workshops.

Nonprofit Help the organization become more flexible, agile, and better prepared for global changes by providing spaces where 135 country offices can collaborate with local partners.

Dedicated spaces where UNICEF country office staff and their local partners can take a problem through a full design cycle.

Sector Nonprofit Provides space, practical skill building, and programming for Harvard students, faculty, staff, alumni, and others engaged in new ventures, nonprofit creation, product or service innovation, small business development, and related educational and research activities.

How is it used Incubator, workspace, and programming for start-up ventures involving Harvard students, and their partners. Teaching space for Harvard students. Meet-up space for local community.

Nonprofit To tackle tough social issues, such as family breakdown and social inequality, by building Australia’s social innovation capability.

Laboratory for co-designing new social programs for vulnerable populations—this includes generating ideas, conducting ethnographic research, and prototyping potential solutions. Meet-up space: TACSI hosts events, workshops, and conferences for social change community. Capacity building for other social innovators.

Provide a dedicated physical space in the heart of the Cambridge, MA tech sector that can accelerate the speed of product development, increase collaboration, and attract new talent.

Office space for moving new and emerging technologies through the development pipeline. Meet-up space for local tech community. Recruitment tool for top tech talent.

To build a strong and permanent research and development presence in Cambridge, MA where Microsoft researchers and programmers can build relationships with local universities, biotech, and healthcare companies.

Office space for moving new and emerging technologies through the development pipeline. Meet-up space for local tech community. Recruitment tool for top tech talent.

Provide a dedicated physical space where Fidelity executives explore how emerging technologies can improve products and processes for internal business units.

Laboratory where FCAT staff can identify solutions and develop new products and processes for Fidelity business units. Hosts conferences, workshops, and social events related to technological and social innovation.

Provide a dedicated physical space that exemplifies how environment can promote creativity and collaboration where Deloitte’s future leaders can hypothesize, research, and test new ideas.

Leadership development institute for Deloitte’s highest performing consultants A think tank where fellows develop innovative yet practical strategies governments can use to transform the way they deliver their services and prepare for the challenges ahead Capacity building—GovLab educates Deloitte account teams on these emerging trends.

Appendix III: Office of Personnel Management’s Innovation Lab Non-Design Programmed Activities, Calendar Years 2012-13

Quarter (calendar year) Q1 2012

Number of participants per meeting (ranges)

Appendix IV: Comments from the Office of Personnel Management

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Seto J. Bagdoyan, (202) 512-4749 or bagdoyans@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Thomas Gilbert, Assistant Director, and Judith Kordahl, Analyst-in-Charge, supervised the development of this report. Jessica Nierenberg and Anthony Patterson made significant contributions to all aspects of this report. Other important contributors included Thomas Beall, Karin Fangman, Donna Miller, and Robert Robinson.