Office of Personnel Management: Agency Needs to Improve Outcome Measures to Demonstrate the Value of Its Innovation Lab
Published: 2014-03-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Organizations from around the globe are emphasizing that strategies promoting innovation are vital to solving complex problems. To try to instill a culture of innovation in its agency, OPM followed the lead of a number of private sector companies, nonprofit organizations, and government bodies by creating an innovation lab. GAO was asked to examine the lab.
Specifically, GAO 1) described the lab's start-up costs, staffing and organization, activities, and policies governing the lab's use, and 2) assessed how OPM's innovation lab compares to other organizations' innovation labs, including how it uses benchmarks and metrics and how it addresses challenges to innovation. GAO reviewed cost, staffing, and performance information. GAO also reviewed relevant literature on innovation and interviewed officials from public, private, and nonprofit organizations with innovation facilities similar to OPM's lab.
section_title: What GAO Found
paragraphs: In March 2012, the Office of Personnel Management (OPM) opened its innovation lab, a distinct physical space with a set of policies for engaging people and using technology in problem solving. The goals of OPM's innovation lab are to provide federal workers with 21st century skills in design-led innovation, such as intelligent risk-taking to develop new services, products, and processes. OPM's lab was built at a reported cost of $1.1 million, including facility upgrades and construction, equipment and training, and other personnel costs. The lab employs approximately 6 full-time equivalents, including a director, and in fiscal year 2013, the lab's operating costs were approximately $476,000, including salaries.
OPM's innovation lab is similar in mission and design to other innovation labs GAO reviewed, and OPM has incorporated some of the prevalent practices that other labs use to sustain their operations. Specifically, OPM is using its lab for a variety of projects, including as a classroom for building the capacity to innovate in the federal government. Lab staff indicated that they plan to begin long-term immersion projects—complex projects with diverse users—within a few months. OPM plans to develop and implement evaluation plans specific to each immersion project that will help them track cost benefits or performance improvement benefits associated with the projects.
Starting in March 2013, OPM lab staff began work on a program evaluation framework to more systematically measure the lab's progress toward meeting its overarching goals. In addition, lab staff members are tracking lab activities, such as classes and workshops, and are surveying lab users about the quality of their experience in the lab. However, they have not developed performance targets or measures related to project outcomes, and without a rigorous evaluation framework that can help OPM track the lab's performance, it will be hard to demonstrate that the lab is operating as originally envisioned.
While labs provide a physical space where innovators can convene, federal agencies are not fully aware of their growing community. However, OPM is taking steps to ensure work done in the lab is shared across OPM and with other federal innovators—for example, by hosting weekly training sessions in the lab on best practices. Studies show that information sharing and interorganizational networks can be a powerful driver supporting innovation.
section_title: What GAO Recommends
paragraphs: Among other things, GAO recommends that the Director of OPM should direct lab staff to 1) develop a mix of performance targets and measures to help them monitor and report on progress toward lab goals, and 2) build on existing efforts to share information with other agencies that have innovation labs. OPM generally concurred with GAO's recommendations; in addition, they described the steps being taken and planned to refine their ongoing evaluation efforts and to further leverage other federal innovation labs.

section_title: Letter
paragraphs: Organizations from around the globe are emphasizing that strategies  promoting innovation are vital to solving complex problems and that future  growth will increasingly depend on productivity-raising innovation. To try  to instill a culture of innovation in its agency, in March 2012, the Office of  Personnel Management (OPM) followed the lead of a number of private  sector companies, nonprofit organizations, and U.S. and foreign  government bodies and created an innovation lab. An innovation lab is  generally defined as a distinct physical space with a set of policies for  engaging people and using technology in problem solving. OPM’s lab,  located in the sub-basement of its Washington, D.C. headquarters, uses  a problem-solving approach called “human-centered design,” which is  described as the discipline of generating concrete solutions and ideas  driven by the needs, behaviors, and context of the people for whom they  are designed. This approach also uses iterative testing of new services,  products, and processes and is intended to encourage faster learning at  the front end of the design process rather than large, expensive failures  upon implementation. The premise behind OPM’s lab was that the new  space and a new approach to problem solving would support efforts to  address some complex issues and eventually lead to a more innovative  agency culture.
You requested that we review OPM’s innovation lab. This report (1)  describes the OPM innovation lab’s start-up and operating costs, staffing  and organization, activities, and policies governing the lab’s use, and (2)  assesses how OPM’s innovation lab compares to other organizations’  innovation labs, including how it uses benchmarks and associated metrics  and how it addresses potential challenges to innovation.
To address our first objective, we reviewed documentation on OPM’s  start-up and operating costs, staffing, and policies regarding the lab and  interviewed OPM staff overseeing the lab and its activities. To address  the second objective, we conducted a comprehensive literature search— including research conducted by global consulting firms and academic  institutions—and interviewed representatives familiar with innovation  facilities. We reviewed documentation and interviewed officials from the  U.S. Census Bureau (Census), the Department of Housing and Urban  Development (HUD), and the National Aeronautics and Space  Administration (NASA) innovation labs. In addition, we met with an official  from the Consumer Financial Protection Bureau (CFPB). While CFPB  lacks a dedicated innovation lab, the agency has a reputation among  federal agencies as a leader in innovative website development. We also  interviewed representatives from other government, private sector, and  nonprofit organization innovation facilities, such as Denmark’s MindLab,  the Fidelity Center for Applied Technology, and UNICEF’s innovation  labs. To select organizations for comparison, we identified those with  innovation facilities similar to OPM’s lab, including those with a dedicated  physical space and using similar problem-solving methods. We examined  documentation and interviewed OPM lab staff regarding identifying any  outcomes—such as cost reductions, performance improvements, or other  results from projects—undertaken by the lab since its inception. We also  conducted a detailed literature search to identify benchmarks and  associated metrics applicable to the development and use of innovation  facilities in the public and private sectors. Based on our literature search,  we identified common challenges that can hamper organizations’ efforts  to use labs as innovation vehicles and prevalent practices that can  support labs’ success and sustainability. In addition, we reviewed our  interview records to identify commonly recurring challenges and prevalent  practices that can support labs’ success and sustainability. We verified  that the challenges and prevalent practices we identified during our  literature search were also those more often cited during the interviews.
We conducted this performance audit from July 2013 to March 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. More detailed information on  our scope and methodology appears in appendix I.
subsections: section_title: Background
paragraphs: Innovation is a dynamic process through which problems and challenges  are defined, new and creative ideas are developed, and new solutions are  selected and implemented. It is also a complex process that involves  taking iterative steps to solve problems. Innovation requires an  environment that encourages participants to challenge traditional  practices without fear of repercussions. Ideally, innovation participants  are empowered to be creative and make mistakes, and appropriate risk- taking is not only tolerated but encouraged.
Some federal leaders are trying various innovation tools, including on-line  idea submission programs, competitions, and prizes, as ways of  unleashing employee creativity. For example, in a memorandum issued in  March 2010, the administration urged federal agencies to use challenges  and prizes to crowdsource innovative approaches to government  initiatives and programs. At relatively low costs, crowdsourcing initiatives  can garner valuable and creative solutions that may not have come  through traditional means. As another example, the Presidential  Innovation Fellows program pairs top innovators from the private sector,  nonprofit organizations, and academia with top innovators in government  to collaborate during focused six- to thirteen-month periods. The program  aims to develop solutions that can save lives, save taxpayer money, and  fuel job creation. For example, the goal of one of the program’s projects is  to identify information critical to saving lives and mitigating damage in a  disaster.
Even with the efforts of some federal leaders to encourage innovation,  federal government-wide scores tracking how agencies foster and reward  employee innovation dropped in 2013 for the second year in a row.  OPM’s 2013 Federal Employee Viewpoint Survey, released in November  2013, found that only 35 percent of federal workers believe that creativity  and innovation are rewarded, with positive responses in this area showing  a steady decline of six percentage points over the past three years.  Research suggests that half of all innovations are not initiated by  organizational leaders. Instead, research shows that it is important to  have processes for gathering stakeholders’ and front-line workers’ views  to identify areas for possible improvement.
As an innovation tool, labs are based on the idea that the competencies  needed for systematic innovation—such as intelligent risk-taking to  develop new services, products and processes—are not the same as  those required for daily operations. Innovation labs seek to provide  approaches, skills, models, and tools beyond those that most employees  are trained in and use to do their work. In addition, public sector  innovation labs can be viewed as attempts to create an organizational  response to a range of challenges to innovation, as innovation efforts face  unique obstacles in the public sector. For example, funding for new public  ventures is limited, and the risks of innovation are high in government. A  defining characteristic of the public sector is that it is subject to broad  scrutiny, so that when an innovation fails or is less than a complete  success, there is the prospect of political consequences.
With constrained budgets expected to continue into the foreseeable  future, innovation in our public services is a necessity. In the last decade,  many public sector organizations around the globe have set up facilities  with the explicit purpose of supporting innovation efforts. For example,  Denmark’s MindLab, started in 2002, is a cross-governmental innovation  unit that is part of the country’s Ministry of Business and Growth, the  Ministry of Education, the Ministry of Employment, and Odense  Municipality, and which collaborates with the Ministry for Economic Affairs  and the Interior. The group covers broad policy areas including areas  such as entrepreneurship, digital self-service, education, and  employment. OPM’s lab was modeled, in part, on Denmark’s MindLab.
OPM officials, consistent with other innovation lab representatives we  interviewed, maintained that, unlike a typical conference room, innovation  labs can be easily reconfigured for large groups and smaller breakout  sessions. They allow users to write on walls and preserve visual artifacts  more easily than typical cubicles and traditional office space. This can be  done with very low-tech tools such as markers and a whiteboard. Figure 1  shows a view of OPM’s lab.
Organizations with different missions are pursuing a lab-based strategy to  foster innovation. For example, organizations—including OPM—use their  labs as a space where participants can conceptualize and prototype new  products or processes outside their normal environment. Many also use  their labs as a teaching space where participants can exchange ideas  and information through classes, workshops, presentations, or other  events. Figure 2 shows how innovation labs we surveyed from the public,  private, and nonprofit sectors share common design elements, and how  these different organizations generally use their labs for multiple and  similar purposes.
section_title: OPM Lab Activities Are Intended to Build Capacity for Innovation and Support Project- Based Problem Solving
paragraphs: Based on OPM documents, the innovation lab’s start-up costs totaled  approximately $1.1 million including facility upgrades and construction,  equipment, and training and other personnel costs. (See table 1 for a  breakdown of costs.) In building the lab, OPM worked with the General  Services Administration (GSA) and contracted with both design and  architectural firms to renovate a former storage room in the sub-basement  of its headquarters building. The 3,000 square foot renovated space  presents an open layout with a meeting area for up to two dozen people  and is surrounded by breakout areas and team rooms. The physical  renovation of the facility was completed in March 2012, after the  installation of final technology equipment, asbestos abatement, and  enhancements to ventilation and life-safety systems. According to OPM,  to make the space useful for any purpose, much of the funding for the  improvements and construction of the space would have been required.
OPM officials said that in fiscal year 2013, the lab’s total operating  budget, including all contracting costs, was $476,000, which supported a  build up to 5.5 full-time equivalent (FTE) employees over the last seven  months of fiscal year 2013. Officials expect this amount will remain  stable in the coming fiscal years, proportional to a full fiscal year.
subsections: section_title: OPM’s Lab Is Managed and Operated By a Core Group of Staff
paragraphs: Operational responsibility for the innovation lab has been assigned to  OPM’s Employee Services Division and is managed by the agency’s  Deputy Associate Director of Strategic Workforce Planning. According to  OPM officials, since February 2013, the lab has grown from 1 FTE to  roughly 6 FTEs. Specifically, as of the end of summer 2013, day-to-day  operations in the lab are carried out by 4 FTE staff members, 1 FTE  intern, 1 part-time intern, and 1 part-time staff member whose time is  divided between the innovation lab and OPM’s Resource Management  Office. A core group of staff from OPM’s Employee Services Division  have been trained in human-centered design and also contribute up to 15  percent of their time in the lab. According to OPM officials, the lab  reached its maximum fiscal year 2013 funding level of 5.5 FTEs in July  2013. A brief description of each position is provided in table 2.
section_title: OPM Employed a Phased Approach to Lab Development Initially Focusing on Capacity Building
paragraphs: OPM has taken a phased approach to developing the lab programming  (activities taking place in the lab) and the policies governing lab use (such  as priority-setting policies for lab projects). According to OPM documents,  each phase has incorporated an element of experimentation, review, and  a shift in strategy based on lessons learned.
Phase I lasted from March through June 2012. During these first 4  months after the lab was built, OPM made the space available to the  OPM workforce for meetings and events. OPM leadership also used this  time to investigate an appropriate problem-solving approach to pair with  the lab that would be consistent with approaches used by other labs; they  determined that a human-centered design approach and curriculum  would complement OPM employees’ technical expertise and analytic  competencies. OPM also began to recruit interested staff from the  Employee Services Division to be trained in human-centered design  fundamentals: this staff would then support project sessions in the lab as  part of their collateral duties.
Phase II lasted from July 2012 through March 2013 and consisted mostly  of facilitated sessions with OPM project teams. During these sessions,  Employee Services staff worked with project teams to generate ideas to  long-standing problems through exercises such as project or strategic  planning, brainstorming sessions, or stakeholder mapping aimed at  discussing and testing potential solutions. Topics discussed during these  sessions involved a variety of initiatives directed at improving OPM  processes and addressing government-wide human resources  challenges. OPM officials noted that innovation lab projects have  included, among others, designing an implementation plan with other  federal agencies to collect valid, accurate, and timely data on the federal  cyber security workforce; updating the government-wide strategy for  veterans recruitment; attracting and retaining individuals with talent in  science, technology, engineering, and mathematics disciplines; and  specific challenges unique to individual agencies.
Phase III lasted from April through November 2013. In Phase III, OPM  continued to provide facilitated design sessions and in some cases,  follow-on coaching to program offices from within OPM and OPM-led  projects. For example, a facilitated design session included lab staff  working with the Food and Drug Administration’s (FDA) Battery Working  Group to more effectively engage with the group’s external stakeholders.  According to an OPM case study about the lab, eight FDA employees  attended the Fundamentals of Human-Centered Design course. Following  the course, lab staff provided planning support for a public workshop with  over 200 participants from stakeholder groups including medical device  and battery manufacturers, other regulatory groups, and hospital staff; lab  staff also attended the Battery-Powered Medical Device workshop to  support the FDA team in their use of design methods. According to an  FDA participant, their collaboration with the lab helped them engage in  stakeholder dialogue that would not have been otherwise possible. OPM  lab staff also began to offer classes in the lab designed to develop  mission-critical skills federal workers need to become better problem  solvers. The OPM lab offers courses such as Human-Centered Design  Fundamentals, Prototyping in the Public Sector, and Communicating  Visually, among other topics. These classes are available to OPM staff  and other federal workers. Staff also made the lab available for federal  communities of practice to convene. According to lab staff, the lab is  becoming a hub for a number of standing meetings of a growing  community of federal innovators and innovation communities of practice.
Table 3 presents a summary of OPM’s human-centered design lab  activities since its inception.
Lab staff report that in the future they intend to expand from their session- based work and targeted design support projects, such as those  consultative sessions that took place in the lab during Phase III. While  some of these more episodic projects may continue to occur, the lab’s  focus will be on creating and establishing large-scale projects typically  involving stakeholders from either wholly within OPM or across different  agencies that are working on crosscutting issues. As discussed later in  the report, projects appropriate for this design method would have diverse  users, be more complex, and be called immersion projects. These would  be the most structured activities undertaken in the lab, characteristically  being longer-term activities that could take up to six months of intense  collaboration with project owners and a diverse group of stakeholders.
section_title: OPM’s Lab Is Similar in Mission and Design to Other Innovation Labs, but OPM Needs to Systematically Evaluate the Lab’s Performance
paragraphs: We identified a common set of challenges that can undermine  organizations’ efforts to use innovation labs and a set of prevalent  practices that the organizations employ to address these challenges and  support their labs’ success and sustainability. OPM has incorporated  some of these practices, such as pairing a distinct space with a structured  approach to problem-solving, but has not implemented others, such as  developing meaningful performance measures. Although OPM has begun  reaching out to other federal innovators, the agency has not fully  leveraged the experience of other agencies employing similar  approaches.
subsections: section_title: OPM Has Dedicated Physical Space, a Problem-Solving Approach, and Plans to Undertake Long-Term Immersion Projects
paragraphs: As a prevalent practice for encouraging and supporting greater  innovation, both the literature and representatives from the organizations  we reviewed stressed the benefits of pairing a dedicated physical space  with a structured framework rooted in design-thinking principles. Many of  the lab representatives said building or establishing a distinct space  carries an important symbolic value as it signals an organization-level  commitment to a culture that supports innovation. However, simply  building a lab is not sufficient to change an organization’s culture; it is  necessary to also introduce a new framework for problem solving.  Although these organizations use different terminology to describe their  selected frameworks, such as agile development and human-centered  design, the general principles are similar. They include placing users at  the center of the desired solution—research on successful innovation  practices shows the importance of engaging customers and  understanding their needs. Further, they include extensive collaboration  with relevant stakeholders, experimentation, prototyping, and iterative  steps to find a solution. A primary objective of this approach is to allow for  failure in the beginning of the design cycle, so that organizations can  manage and learn from early mistakes, rather than try to recover from an  expensive, comprehensive failure upon implementation.
For example, Census and HUD have similar problem-solving frameworks  for lab use. As figure 3 shows, the Census Center for Applied Technology  and the HUD Innovation Lab rely on a five-step framework to guide  innovation in their labs.
As another example, although CFPB does not have a dedicated physical  space, it used a similar framework to develop its on-line mortgage  disclosure form. According to CFPB’s Creative Director for Technology  and Innovation, designers interacted with end-users including mortgage  applicants, prototyped different forms, and made refinements based on  continual feedback before launching the new form. On its website, CFPB  describes its design process in detail, including prototyping and feedback  sessions with consumers, lenders, and mortgage brokers.
OPM’s lab provides a menu of design services to meet the specific needs  of various projects. The lab’s larger-scale immersion project work will  involve taking a complex problem through OPM’s problem-solving  framework, which encompasses steps for problem framing to learning  about users to analysis to concept development, testing, and rapid  iterative steps. This problem-solving framework is similar to those  employed by other innovation labs.
As discussed earlier in this report, similar to other organizations with labs,  OPM is using its lab for a variety of purposes including as a learning  space for classes on human-centered design principles and techniques  and as a meeting space for interagency task teams and communities of  practice. As originally envisioned in its strategic and performance plans,  the lab was designed to host a mix of activities rooted in the human- centered design approach, including longer-term design challenges.  Moreover, OPM lab staff asserted that to gain an organization’s  confidence and to instill a culture of innovation, it is necessary for a  successful innovation lab to have an array of sufficiently compelling  projects that demonstrate how the lab approach can lead to performance  improvements.
Based on our interviews with public and private sector organizations with  similar innovation facilities, larger-scale problem-solving projects were  common activities in their innovation labs’ service portfolios. As an  example, Denmark’s MindLab has contributed to tackling several pressing  social issues including simplifying the process for managing claims  related to industrial accidents and shortening the time before injured  workers return to the market. Opportunities to showcase a new approach  to problem solving reduce the likelihood that the lab might lose its  distinction as different from a traditional meeting space or classroom  usually associated with training facilities.
Consistent with what staff from other labs told us, OPM officials said they  needed the past two years to first introduce agency staff to human- centered design concepts and applications before they could initiate an  immersion project. Lab staff said this phased approach was necessary for  several reasons. Targeted design support sessions allowed lab staff to  expose lab users to design methods and provided opportunities for  collaboration. These sessions also allowed lab staff to quickly show value  for program offices in response to a specific need. For example, OPM lab  staff members were able to help FDA staff plan and engage with over 200  different stakeholders at a conference. They also said targeted design  support is a critical way for emerging design practitioners to develop and  hone their own skills before applying them to a longer term, and higher  stakes, project engagement. Lab staff said overall these sessions  benefited both the users of the lab, who developed new skills to take to  their home offices, such as problem framing and engaging with  stakeholders, as well as lab employees, who continue to grow and refine  their human-centered design skills. In a December 2013 document, OPM  staff stated they intend to create and establish these longer-term  immersion projects and evaluate their impact during the next phase of the  lab’s development.
section_title: OPM Has Not Fully Developed Meaningful Performance Measures
paragraphs: Measuring the long-term outcomes of innovation labs is a prevalent  practice for building acceptance and demonstrating the value of the labs.  Consistent with our literature review, several representatives we  interviewed from other innovation labs concurred with the director of  innovation at Denmark’s innovation lab, MindLab, who said that  innovation labs need to know how much they are spending and their  outcomes. According to the director, the labs must also be able to  attribute where the change happens based on their work. In addition, lab  staff must be prepared to present a narrative of their work. He  acknowledged that innovation labs are risky because they look different,  and they have a different focus than other government entities. The  director said that, as a result, innovation lab officials need to show where  the funds are going along with the benefits and results of those  investments.
Representatives from newer labs—i.e. those operating less than three  years—stated they primarily rely on output measures to gauge their initial  efforts such as number of users, ways in which the lab is being used,  classes or events held in the lab, and anecdotal evidence. Developing  outcome measures is more challenging for several reasons. Appropriate  outcome measures are often not obvious at the onset of a project.  Moreover, agencies may not have appropriate measures or baseline data  when they start using an innovation lab as a problem-solving tool, and the  role of the lab in driving a successful innovation may not always be clear.
Given these challenges to accurately measuring innovation and the value  of an innovation lab, lab managers from labs that have been operating for  a longer period of time told us they focus on developing meaningful  milestones and measures applicable to different phases of the innovation  lifecycle, such as problem generation, idea generation, and skills  development. For example, the UNICEF Innovation Unit—which has been  helping member-country offices set up innovation labs since 2006—and  several European initiatives developed a set of benchmarks intended to  help them measure the value of public sector labs and identify ways in  which the lab’s performance can be improved. The benchmarks  UNICEF developed span across six categories, such as problem  definition and idea generation, internal and external collaboration, and  secondary effects. Within each category, they include a list of questions  intended to assess their strengths and weaknesses. For example, they  want to know whether labs are helping employees define problems and  generate ideas, strengthen internal collaborations, and build external  partnerships. They also measure the extent to which work done in the  labs results in new team or staff capacity, excitement and goodwill toward  the organization, and an increase in leverage and influence in their field.  OPM is undertaking a similar effort to establish benchmarks that will help  lab staff gauge the extent to which lab users are learning and applying  many of these same skills, but the lab is not mature enough to have  results.
OPM documents state that the goal of OPM’s innovation lab is to provide  federal workers with 21st century skills in design-led innovation, and the  intended purpose of the lab is to provide a physical space for project- based problem solving. The documents also note that the value of the lab  can be measured, in part, by how well it helps develop the mission-critical  competencies to improve the federal workforce’s ability to solve problems  and deliver results.
In its strategy document, OPM laid out the following high-level goals for  the innovation lab:
Employees assigned to the innovation lab should go back to their  home organization with an understanding of, and an appreciation for,  the power of innovative approaches to problem solving.
Employees should be equipped to implement similar methodologies in  their home organizations on future projects.
As the innovation lab matures, and as more and more projects are  completed, the notion of using innovation to tackle complex problems  will gain traction across the organization.
Eventually, leaders and employees across OPM will vie to get their  issues sent to the innovation lab for resolution.
This in turn will contribute to a decrease in organizational silos, and a  concurrent increase in cross-organizational teams addressing one  organization’s issues.
In the same document, OPM officials also described an evaluation  strategy resembling an agile approach. Specifically, OPM described  these goals as moving targets which would be achieved through an  evolving and self-correcting process. Lab staff immediately started to  track lab activities and outputs, such as number of participating people  and agencies, and how participants used the lab, such as consultative  sessions, follow-on coaching, training classes, or as a meeting space.  Five months after the lab opened, they also started to survey users who  participated in day-long facilitated sessions. For example, there was a  one-page evaluation that asked respondents to rate the appropriateness  of the environment and quality of the facilitators. The surveys also asked  whether users would recommend the lab to colleagues and whether  human-centered design problem-solving tools can be used as an effective  tool government-wide. The responses were generally positive—about 82  percent of respondents (84 out of 103) said they would recommend the  lab to someone else, providing a baseline for subsequent survey findings.  According to lab staff, they periodically reviewed the available data and  adjusted their strategy for operating the lab.
Starting in March 2013, a year after the lab opened, OPM lab staff began  work on a program evaluation framework to more systematically measure  the lab’s progress toward meeting their overarching goals. To evaluate  the extent to which lab participants are learning and applying innovative  approaches, lab staff intends to measure the lab’s performance along  three overarching categories: service experience, skill development, and  project outcome. According to the framework, resources dedicated to  evaluation efforts will reflect the resources needed to host lab-sponsored  events. Episodic events such as consulting sessions will correspond to a  “light-touch” follow-up effort, such as immediately surveying all  participants on their session experience and skills development. More  long-term, resource-intensive efforts such as immersion projects will  employ a more robust follow-up effort that, in addition to assessing the  session experience and skills development, will also address project- specific outcomes. Collection of assessment data in all three areas will  include the administration of surveys to participants both before and right  after a session, and some services will involve the administration of  surveys to participants before a service and subsequent periodic check- ins. Depending on the nature of the lab session, information on skill  development and outcomes will also be obtained from session clients in  pre-session scoping conversations and periodic, post-session check-ins  using either surveys, or interviews. For one type of session, assessment  of participant skill development will also include a survey of participant  supervisors.
Lab staff has used a series of surveys to measure participant experience  and skills development, and to capture specific project-related outcomes  for the different services they offer. However, the survey instruments are  unlikely to yield data that would be of sufficient capacity, credibility, and  relevance to indicate the nature and extent to which the lab is achieving  what it intends to accomplish or its value to those who use the lab space.  Although there are several items across all surveys that are reasonably  aligned with generally accepted questionnaire and item design principles,  there are limitations associated with many items where language is  ambiguous, where the intent of the question is not clear, and directions  are lacking. For example, phrases such as “changed behavior” or  “tangible outputs you can move forward” are open to numerous  interpretations and are likely to engender an array of responses that  range from being relevant to not at all relevant or relatable to the  purposes or objectives of the session. In addition, some of the items may  be more likely to engender responses with a greater likelihood of being  subject to a respondent’s social desirability bias. For example, the  respondent may want to provide answers that are socially desirable,  maintain the status quo, or make a good impression.
While some customization is to be expected, the surveys did not indicate  any approach to evaluate some core aspects of the lab and its value  using a consistently presented set of the same questions. For example,  the question asking participants about the likelihood that they would  recommend the lab to someone else is the type of item that could, with  revision, be incorporated in all of the surveys. Analyses of a core set of  items by type of lab event or service would enable lab staff to discern and  compare where participants were more and less engaged in lab activities  and curricula. Consequently, these survey instruments and the items on  them may be susceptible to various types of question and respondent  bias and could, when the responses are analyzed, produce results that  would be difficult to interpret or link to expected participant effects, or to  the intent or activities of the workshop session. Moreover, lab staff has  not developed outcome measures or milestones related to customer  experience and skills development.
The evaluation framework being developed by OPM does not include  interim performance targets or measures. Best practices state that new  initiatives benefit when managers set time-bounded, quantifiable interim  goals, establish related performance measures, collect data, and use that  information to assess and adjust their performance. To evaluate the  overall performance of MindLab, the director said he develops an annual  work plan, which describes the number and types of projects and other  activities the lab will undertake, as well as the relative resource allocation  to those projects and activities. He said his staff also conducts an annual  review of the budget and actual expenditures with the board. OPM lab  staff has been tracking outputs—such as number of participants and  number and type of activities—meaning that they have baseline data  which could inform realistic, meaningful targets and measures related to  lab use and activities for the upcoming year. Although they continue to  refine their surveys, they could use the results from earlier versions to  establish targets and measures related to customer experience and skills  development. Meaningful measures or milestones could help them  assess their progress toward improving participants’ ability to solve  problems and accurately measure the effect of working in the lab on  services, products, and processes.
As mentioned previously, the lab plans to host the more resource- intensive immersion projects. To demonstrate that the lab is operating as  originally intended, evaluation plans will be needed for specific immersion  projects that can help track cost-benefits and performance improvement  outcomes. OPM stated that evaluation plans will be prepared for each  immersion project to account for project outcomes. They indicated that  they wanted to host their first immersion project within the next several  months.
section_title: OPM Has Started Efforts to Connect with the Federal Innovation Community, but Has Not Fully Leveraged Other Agencies’ Efforts
paragraphs: Another prevalent practice we identified included leveraging other  innovation labs’ efforts to try to increase the value of the lab approach.  Studies show that information sharing and interorganizational networks  can be a powerful driver supporting innovation. One study showed that  interorganizational networks of innovators help members develop new  products at a faster rate with lower investment commitments, due in large  part to the information sharing that takes place. Sharing information can  help mitigate the risks and uncertainty that typically characterize  innovation ventures. Best practices state the importance of establishing  channels of communication and other mechanisms that facilitate  knowledge-sharing and building networks of like-minded communities to  help agencies achieve crosscutting objectives. For example, the Census  Bureau’s Chief Technology Officer suggested a way in which innovation  leaders could share information and pool resources. Specifically, instead  of each agency creating its own technology innovation lab with its own  hardware, software, and associated maintenance, they could use a  common innovation infrastructure service in the public cloud. Every  agency could still have their own branded offering and could still provide  access in their own facility or at their regional offices. However, an  outside vendor could provide the infrastructure. For example, if an agency  wanted to experiment with some unique visual analytic tools, they could  purchase what they need on a subscription service; this would eliminate  agencies buying all the tools themselves.
While labs provide a physical space where innovators can convene,  federal agencies are not fully aware of their growing community. As of  June 2013, OPM was unaware that other agencies such as Census,  HUD, and NASA were pursuing a lab approach to promote innovation.  Moreover, the lab directors at these agencies were not aware or only  marginally aware of OPM’s lab and its resources or other federal  innovation labs. OPM’s efforts to develop an innovation lab occurred  around the same time or pre-dated those of other agencies we  interviewed.
According to OPM officials, during its first year of operations, OPM lab  staff focused their efforts on promoting awareness of the lab and its  resources internally to OPM staff. In their second year, OPM lab staff  planned more activities intended to promote the lab and its resources  externally to connect with federal agencies’ innovation efforts. Staff noted  the OPM lab is the hub of various interagency networks of innovation  practitioners. For example, an interagency community of practice on idea  generation meets in the lab on a monthly basis. OPM’s lab staff also  reported that they host weekly trainings in the lab on best practices,  including webinars about measuring the success of enterprise-level  design efforts and the value of visualizing information. These training  sessions include case study presentations from other federal agencies,  such as GSA, and non-federal entities. In addition, OPM has shared best  practices with other public sector design labs across the globe by  participating in a number of conferences. OPM is also collaborating with a  current Presidential Innovation Fellow, who is building an innovation  toolkit. Although projects in the lab are currently managed and for the  most part delivered by OPM employees, staff noted that they are  increasingly looking to leverage detailees, short-term assignments, and  other ways to harness the potential of talent from other agencies. OPM  staff said they also give regular tours of their innovation lab for other  government entities already supporting innovation initiatives or  developing them. In addition to these activities, OPM hosted a convening  of federal innovators to compare various agencies’ innovation  communication efforts across the agencies.
Several federal officials we interviewed said they would welcome the  opportunity to communicate as the need arose with a community of peers  to exchange information and ideas and trouble-shoot problems related to  the start-up and maintenance of their labs. For example, CFPB’s Creative  Director for Technology and Innovation said it would be helpful to find out  what other bureaus and departments are doing to incorporate design  principles, so that she could exchange ideas and information. An official  from NASA’s Swamp Works noted that it would be beneficial to show that  others in the federal sector are also looking at innovation labs. To that  end, simply knowing the innovation community exists and how agency  staff leading innovation efforts can initiate a conversation related to a  specific topic would likely be beneficial and would help avoid the risk of a  fragmented innovation community.
section_title: OPM’s 2014 through 2018 Strategic Plan Indicates It Intends to Use the Lab to Advance OPM Priorities
paragraphs: Because innovation necessarily entails culture change, experimentation,  periodic setbacks, and often resource investments, another prevalent  practice necessary to sustain a lab includes leadership support.  Innovation labs are one tool that agencies can use to foster innovation.  Agency officials and lab directors we interviewed said leaders must be  willing to embrace experimentation within the lab and understand that  smart failures—failures that result from trial and error, where the  alternative would be to do something truly risky due to lack of evidence— are part of the design process. For example, the Census Bureau’s Chief  Technology Officer noted that support by Census Bureau leadership is  critical to ensure staff participation and the continued availability of funds  to drive innovation in its Center for Applied Technology lab. Other lab  directors highlighted several strategies they use to balance the risks and  failures that accompany a problem-solving methodology rooted in a more  experimental approach. These include accelerated timelines of three to  six months, which allows organizations to quickly shelve projects that are  without merit. Some lab leaders also noted that a quick win or early  success can give new labs the underlying support they need to take on  riskier projects.
In March 2014, OPM released its 2014 through 2018 strategic plan, which  states that the agency plans to seek new, innovative ways to accomplish  its work of advancing human resource management in the federal  government. In the strategic plan, OPM indicates that, among other  things, it intends to use the innovation lab and human-centered design  methods to address OPM’s operational challenges.
section_title: Conclusions
paragraphs: For OPM and the rest of the federal government, finding more efficient  and effective ways of doing business to help meet rising citizen demands  for public services is critical, particularly in an era of continued fiscal and  budgetary constraints. OPM’s innovation lab is one such tool intended to  give rise to solutions of complex problems facing the federal government.
Consistent with other innovation labs, development of performance and  outcome measures, tools to assess performance, and further leveraging  the experience of other organizations undertaking similar efforts will also  be critical. Having clear and specific outcome measures will help OPM  track and evaluate the extent to which the lab is meeting its original intent  and over time, to make any necessary adjustments. Otherwise, OPM’s  innovation efforts may not be able to demonstrate the types of results  initially envisioned.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Director of OPM take the following actions to  help substantiate the lab’s original goals of enhancing skills in innovation  and supporting project-based problem solving:
Direct lab staff to develop a mix of performance targets and measures  to help them monitor and report on their progress toward lab goals.  Output targets could include number and type of lab activities over the  next year. Outcome targets and measures should correspond to the  lab’s overarching goals to build organizational capacity to innovate  and achieve specific innovations in concrete operational challenges.
Direct lab staff to review and refine the set of survey instruments to  ensure that taken as a whole, they will yield data of sufficient  credibility and relevance to indicate the nature and extent to which the  lab is achieving what it intends to accomplish or is demonstrating its  value to those who use the lab space. For example, lab staff should  consider the following actions:
Developing a standard set of questions across all service  offerings.
Revising the format and wording of existing questions related to  skills development to diminish the likelihood of social desirability  bias and use post-session questions that ask, in a straight- forward way, about whether, or the extent to which, new  information was acquired.
Replacing words or phrases that are ambiguous or vague with  defined or relevant terminology (e.g., terms actually used in the  session) so that the respondent can easily recognize a link  between what is being asked and the content of the session.
Direct lab staff to build on existing efforts to share information and  knowledge within the federal innovation community. For example,  OPM lab staff could reach out to other agencies with labs such as  Census, HUD, and NASA’s Kennedy Space Center to share best  practices and develop a credible evaluation framework.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of OPM for review and  comment. The director provided written comments, which we have  reprinted in appendix IV. In summary, OPM generally concurred with our  recommendations and described ongoing and planned steps to refine  evaluation efforts and further leverage other federal innovation labs.
For the recommendations on evaluating performance, the director  described a competency-based skills gap pilot the lab is undertaking,  based on targets from pre- and post-testing of participants in lab  activities. We acknowledge that this is an important step in developing  performance measures, and OPM will also need targets and measures to  demonstrate the lab’s value in achieving specific innovations in concrete  operational challenges.
For the recommendation on leveraging other federal agency innovation  efforts, the director noted OPM’s work seeking out information and  contacts from other innovation endeavors, including lab-based ones. We  acknowledge OPM’s more recent emphasis in this area, including  participating in an interagency community of practice on innovation.  Federal officials we interviewed said they would welcome the opportunity  to communicate as the need arose with a community of peers.
To clarify, the report recognizes sustained organizational leadership as a  prevalent practice for the success of innovation labs. However, this was  not a specific report recommendation, but an acknowledgement regarding  the general role leadership plays in ensuring the success of innovation  labs. In her response, the director stated that OPM recently released its  2014 through 2018 strategic plan, which she said demonstrates OPM  leadership’s commitment to the advancement of work in the lab.  Accordingly, we updated the report to reflect the most current information  available at the time of our publication.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the Director  of OPM and appropriate congressional committees. This report will also  be available at no charge on our website at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-4749 or bagdoyans@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff members who made major  contributions to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This appendix provides information on the scope of work and the  methodology used to (1) describe the Office of Personnel Management  (OPM) innovation lab’s start-up and operating costs, staffing and  organization, activities, and policies governing the lab’s use, and (2)  assess how OPM’s innovation lab compares to other organizations’  innovation labs, including how it uses benchmarks and associated metrics  and how it addresses potential challenges to innovation.
To address the first objective, we reviewed documentation and met  several times with OPM staff overseeing the lab and its activities. We  reviewed the lab’s construction and operations’ budget, including the  funding sources for the lab, and interviewed agency officials  knowledgeable about the lab’s budget. Based on interviews and e-mail  exchanges with knowledgeable OPM and General Services  Administration staff and reviewed documents, we found OPM’s lab  expense data to be sufficiently reliable for the purposes of our report. We  reviewed spreadsheets maintained by lab staff tracking lab outputs, such  as workshops hosted in the lab and number of attendees. We also  reviewed lab performance materials such as the lab’s performance plan,  user surveys, and the results of those surveys. Survey specialists in our  Center for Design, Methods, and Analysis reviewed the lab user surveys  using internal review guidance that is typically performed on draft GAO  surveys as part of our development process and required before  deployment of a survey. In addition to reviewing with lab staff the  documents they provided us, we interviewed them about OPM’s process  for identifying and selecting a lab strategy, lab staff’s approach to  implementing a human-centered design curriculum, and their goals for the  lab.
To address the second objective, we conducted a detailed literature  search of material from academic institutions, global management  consultants, professional associations, think tanks, news outlets, and  various other organizations. We also reviewed literature documenting  public, private, and academic innovation efforts and associated positive  and negative outcomes. Our literature search helped us identify  benchmarks and associated metrics applicable to the development and  use of innovation facilities in the public, private, and nonprofit sectors. We  also interviewed OPM lab staff on how they intend to identify outcomes— such as cost reductions, performance improvements, or other results— from projects undertaken by OPM since the inception of the lab.
We used the findings from our literature review to identify organizations  with innovation facilities having a dedicated physical space and using  problem-solving methods similar to OPM’s lab. We selected a mix of 11  public, nonprofit, and private organizations to visit or interview. In  addition, we met with an official from the Consumer Financial Protection  Bureau (CFPB). While CFPB lacks a dedicated innovation lab, the  agency has a reputation among federal agencies as a leader in innovative  website development. Table 4 lists the organizations we visited in person  or interviewed their representatives by telephone.
At every lab we visited or contacted, we interviewed lab representatives  about the history of the lab, including why they decided to pursue a lab  strategy; how the lab is used; the protocols for engaging participants; how  lab directors measure the performance of the lab; challenges to  promoting innovation within the organization; and practices for addressing  those challenges. Based on our literature search, we identified common  challenges that can hamper organizations’ efforts to use labs as  innovation vehicles and prevalent practices that can support labs’  success and sustainability. In addition, we reviewed our interview records  to identify commonly recurring challenges and prevalent practices that  can support labs’ success and sustainability. We verified that the  challenges and prevalent practices we identified during our literature  search were also those more often cited during the interviews.
We also interviewed representatives from two management consultancies  that promote problem-solving approaches rooted in design-thinking  principles, IDEO and Luma. OPM contracted with Luma to help design  the lab and implement human-centered design programming. We spoke  with their representatives to understand the challenges their clients face  in changing organizational culture and the benchmarks and metrics they  advise their clients to adopt to measure the performance of new labs and  problem-solving methods.
In addition, we interviewed officials from three public-sector  organizations—San Francisco Mayor’s Office of Civic Innovation,  Canada’s Public Policy Forum, and United Kingdom Behavioural Insights  Team—that are pursuing strategies to promote innovation in their  organizations but opted not to build innovation labs. They spoke to us  about the challenges that prevented them from building labs and the  steps they are taking to incorporate human-centered design-like problem- solving methods without a physical lab.
We conducted this performance audit from July 2013 to March 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Innovation Labs Share Similar Design Elements and Problem-Solving Methods
paragraphs: This table contains the same text portrayed in figure 2 and shows how  innovation labs we surveyed from the public, private, and nonprofit  sectors generally use their labs for multiple and similar purposes.
Purpose of lab  To help deliver on President  Obama’s vision of a more  effective and efficient government  for the American people by  supporting a government-wide  community of innovators.
How is it used  Dedicated space where OPM and interagency  teams can take a problem through a full  design cycle. This includes problem framing,  learning about users, analysis, concept  development, testing, and rapid iteration.  Capacity building—lab hosts classes and  workshops.  Meeting space for OPM staff and other  federal workers and interagency communities  of practice.
To provide a “safe zone” where  Census staff can explore new  technology solutions without  impact to production operations.
Dedicated space where lab and other Census  staff can develop, test, and implement ideas  into production outside the standard  production environment  Capacity building—lab hosts presentations of  new technologies and solutions.
To provide a dedicated space  where NASA KSC engineers can  quickly resolve problems related  to deep-space exploration.
Space where 20 NASA engineers and  scientists prototype and test emerging  technologies.  Swamp Works has contracts with other NASA  centers worth about $7 million a year.
Provide a space that looks  different from the traditional office  environment, where HUD  employees can accelerate the  development of solutions more  efficiently than other available  approaches.
Dedicated space where HUD lab staff and  mission area leads can develop, test, and  implement ideas into production within a  compressed timeframe.
Provide a neutral space for  government ministries to work  with citizens and businesses to  create new solutions for society.
Ministry officials from Danish Ministries of  Business and Growth, Education, and  Employment use the MindLab space and  resources to take a problem through the full  design cycle.  Capacity building—lab hosts classes,  conferences, and workshops.
Nonprofit  Help the organization become  more flexible, agile, and better  prepared for global changes by  providing spaces where 135  country offices can collaborate  with local partners.
Dedicated spaces where UNICEF country  office staff and their local partners can take a  problem through a full design cycle.
Sector  Nonprofit  Provides space, practical skill  building, and programming for  Harvard students, faculty, staff,  alumni, and others engaged in  new ventures, nonprofit creation,  product or service innovation,  small business development, and  related educational and research  activities.
How is it used  Incubator, workspace, and programming for  start-up ventures involving Harvard students,  and their partners.  Teaching space for Harvard students.  Meet-up space for local community.
Nonprofit  To tackle tough social issues,  such as family breakdown and  social inequality, by building  Australia’s social innovation  capability.
Laboratory for co-designing new social  programs for vulnerable populations—this  includes generating ideas, conducting  ethnographic research, and prototyping  potential solutions.  Meet-up space: TACSI hosts events,  workshops, and conferences for social  change community.  Capacity building for other social innovators.
Provide a dedicated physical  space in the heart of the  Cambridge, MA tech sector that  can accelerate the speed of  product development, increase  collaboration, and attract new  talent.
Office space for moving new and emerging  technologies through the development  pipeline.  Meet-up space for local tech community.  Recruitment tool for top tech talent.
To build a strong and permanent  research and development  presence in Cambridge, MA  where Microsoft researchers and  programmers can build  relationships with local  universities, biotech, and  healthcare companies.
Office space for moving new and emerging  technologies through the development  pipeline.  Meet-up space for local tech community.  Recruitment tool for top tech talent.
Provide a dedicated physical  space where Fidelity executives  explore how emerging  technologies can improve  products and processes for  internal business units.
Laboratory where FCAT staff can identify  solutions and develop new products and  processes for Fidelity business units.  Hosts conferences, workshops, and social  events related to technological and social  innovation.
Provide a dedicated physical  space that exemplifies how  environment can promote  creativity and collaboration where  Deloitte’s future leaders can  hypothesize, research, and test  new ideas.
Leadership development institute for  Deloitte’s highest performing consultants  A think tank where fellows develop innovative  yet practical strategies governments can use  to transform the way they deliver their  services and prepare for the challenges  ahead  Capacity building—GovLab educates Deloitte  account teams on these emerging trends.
section_title: Appendix III: Office of Personnel Management’s Innovation Lab Non-Design Programmed Activities, Calendar Years 2012-13
subsections: section_title: Quarter (calendar year) Q1 2012
paragraphs: Number of  participants per  meeting (ranges)
section_title: Appendix IV: Comments from the Office of Personnel Management
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Seto J. Bagdoyan, (202) 512-4749 or bagdoyans@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Thomas Gilbert, Assistant  Director, and Judith Kordahl, Analyst-in-Charge, supervised the  development of this report. Jessica Nierenberg and Anthony Patterson  made significant contributions to all aspects of this report. Other important  contributors included Thomas Beall, Karin Fangman, Donna Miller, and  Robert Robinson.