Managing for Results: OMB Improved Implementation of Cross-Agency Priority Goals, But Could Be More Transparent About Measuring Progress
Released: 2016-05-20T00:00:00

Letter

Background

Several players have important roles in implementing the CAP goals, as required by GPRAMA:

OMB. OMB provides guidance to agencies and CAP goal teams in Circular A-11 on how to implement GPRAMA, including implementing and reporting on the CAP goals. Beginning in March 2014, OMB and the PIC also developed more detailed guidance, which they have updated periodically, to help ensure CAP goal teams are complying with GPRAMA requirements, including developing CAP goal implementation plans, identifying contributors, and reporting quarterly progress toward their goals on the public website, Performance.gov.

PIC. The PIC is chaired by OMB’s Deputy Director for Management and is composed of agency Performance Improvement Officers, other agency performance staff, and senior OMB staff to facilitate the exchange of useful practices to strengthen agency performance management, such as through cross-agency working groups. The PIC is supported by an Executive Director and a team of 8 full-time staff, that conduct implementation planning and coordination on crosscutting performance areas, including working with OMB, other government-wide management councils, and agencies on the CAP goals.

CAP Goal Leaders. CAP goal leaders are responsible for coordinating efforts to implement each goal. As shown in figure 1 above, the current CAP goals have at least two goal leaders—one from the Executive Office of the President and the other from a key responsible agency. OMB directs CAP goal leaders to engage officials from contributing agencies by leveraging existing interagency working groups, committees, and councils.

GPRAMA has a number of requirements for reporting on CAP goals (see text box). These reporting requirements are designed to ensure that relevant performance information is used to improve performance and results of the goals, that OMB and CAP goal leaders actively lead efforts to engage all relevant participants in collaborative performance improvement initiatives, and hold them accountable for progress on identified goals and milestones.

GPRA Modernization Act of 2010 Requirements for Reporting on Cross-Agency Priority Goal Progress Make available on the website Performance.gov: 1. A brief description of each of the federal government priority goals required by section 1120(a) of this title. 2. An identification of the lead government official for each federal government performance goal. 3. An identification of the agencies, organizations, program activities, regulations, tax expenditures, policies, and other activities that contribute to each federal government performance goal. 4. Plans to address major management challenges that are government-wide or crosscutting in nature, and describe plans to address such challenges, including relevant performance goals, performance indicators, and milestones. 5. Performance goals to define the planned level of performance for each goal for the year in which the plan is submitted and the next fiscal year for each of the federal priority goals. 6. Common federal government performance indicators. 7. Quarterly targets for each common performance indicator.assessing the overall progress toward each federal government performance goal. 11. How performance indicators with quarterly targets are being used in measuring or assessing whether relevant agencies, organizations, program activities, regulations, tax expenditures, policies, and other activities are contributing as planned. 12. Results achieved during the most recent quarter. 13. Overall trend data compared to the planned level of performance.

OMB established the first set of CAP goals for a 2-year interim period in February 2012. We issued reports in 2012 and 2014, evaluating implementation challenges faced during the interim CAP goal period. In May 2012, we reported that the interim CAP goals did not leverage all relevant parties, and therefore important opportunities for achieving these goals may have been missed. OMB responded by identifying and including additional relevant departments, agencies, and programs for the interim CAP goals. In June 2014, we identified a number of reporting and accountability gaps with the interim CAP goals. Specifically, we found that some goals did not report on progress towards a planned level of performance because the goals lacked either a quantitative target or the data needed to track progress. We also found that quarterly progress updates published on Performance.gov listed planned activities and milestones contributing to each goal, but some did not include relevant information, including time frames for the completion of specific actions and the status of ongoing efforts. We concluded that the incomplete information in the updates provided a limited basis for ensuring accountability for the achievement of targets and milestones. As a result, we made seven recommendations to OMB and the PIC to improve the reporting of performance information for CAP goals and quarterly progress reviews. OMB responded to these recommendations by updating its guidance to CAP goal teams to address the reporting and accountability gaps we identified. We closed these recommendations as implemented based on OMB’s updated guidance to CAP goal teams, our assessment of the quarterly progress updates, and interviews with CAP goal teams.

In March 2014, OMB issued the current set of CAP goals, which GPRAMA requires to be updated, at a minimum, every 4 years. As shown in figure 2, the current set of 15 CAP goals includes 7 related to crosscutting mission areas and 8 related to management. OMB reported that it worked to select CAP goals that represent high-priority issue areas with outcomes that could be enhanced through improved cross-agency implementation.

OMB will issue the next set of 4-year CAP goals in February 2018 along with the President’s fiscal year 2019 budget. OMB staff told us that they expect the last progress update posted on Performance.gov for this set of CAP goals will reflect progress from the second quarter of fiscal year 2018, which ends in March 2018. Each quarterly progress update for this set of CAP goals is available on Performance.gov.

OMB Incorporated Lessons from Interim CAP Goals into Governance of Current CAP Goals

OMB Updated Guidance to Improve Public Reporting on Implementation of CAP Goals

As we reported in September 2015, OMB improved CAP goal reporting and accountability by addressing recommendations we made in June 2014. Specifically, OMB developed updated guidance and a new reporting template for CAP goal teams that provide a consistent reporting format across all goal teams, and helps goal teams meet GPRAMA reporting requirements. According to OMB staff managing the CAP goals, this revised reporting structure and template is intended to promote engagement from goal leaders, facilitate performance monitoring, ensure transparency of the CAP goal initiative, and provide a framework for CAP goal teams to articulate cross-agency goals using milestones and measures. The template outlines information that GPRAMA requires goal teams to report on Performance.gov, such as goal leaders; contributing agencies, organizations, and programs; performance measures and targets; key milestones; and plans to address government-wide management challenges. The updated guidance directs CAP goal teams to include additional performance information—not required by GPRAMA—in their quarterly progress updates, such as the status of milestones, associated time frames for their completion, performance indicators (or measures) under development, and an organizational chart depicting the goal’s governance team.

According to the seven CAP goal teams we spoke with and OMB staff, the quarterly progress updates provide useful information for goal leaders to track progress over time and to make timely management decisions that affect goal implementation. PIC guidance states that the update should provide succinct, high-level information that is understandable to all involved in implementation (i.e., program staff to senior leaders). OMB staff managing the CAP goals told us that the quarterly progress updates summarize the progress made on each CAP goal. They added that, at this point in the goal period, reviewing the milestone status for each CAP goal is the best way to determine whether goals are at risk or sufficient progress is being made quarterly. For example, the Lab-to-Market CAP goal leader in the Department of Energy told us that the progress updates enables the goal team to track progress on activities that support and expand technology between the national laboratories and the private sector. The goal leader also told us the quarterly reports are useful to implementation teams because they reinforce the need to coordinate across agency lines, and drive discussion about what can be done to achieve goals and milestones.

OMB Developed Strategies to Build CAP Goal Teams’ Capacity for Goal Implementation

OMB and the PIC have also implemented strategies to build agency capacity to work across agency lines. For example:

Assigned agency goal leader. For the current set of CAP goals, OMB changed the CAP goal governance structure to include agency co-leads for each CAP goal, in addition to entities within the Executive Office of the President, such as OMB, that were already serving as goal leaders (see figure 2 above). According to OMB staff, this new governance structure reflects agency leadership and expertise in CAP goal subject areas, more effectively leverages agency resources for crosscutting efforts, and promotes greater coordination across multiple agencies. For example, Science, Technology, Engineering and Mathematics (STEM) Education CAP goal team staff from the Office of Science and Technology Policy (OSTP) told us that they found contributing agencies to be more receptive to directives and efforts for implementing the CAP goal because they come jointly from the National Science Foundation (NSF) and are not solely based on OSTP’s policy perspective. They also told us that NSF’s leadership of CAP goal activities and its ability to secure agency buy-in, among other things, makes it an effective CAP goal leader.

Provide ongoing guidance and assistance to CAP goal teams.

The seven CAP goal teams we spoke with told us that OMB and the PIC staff are available regularly to provide them with ongoing support, such as assisting with the regular collection of performance data, leading seminars to develop useful milestones and priority actions, and improving teams’ ability to track and report progress. The PIC plans to continue to assist teams with tracking their progress and ensuring accountability across the goals, including supporting the quality and completeness of the regular quarterly progress updates through Performance.gov. For example, in August 2014, PIC staff met with the STEM Education CAP goal team to assist it in developing milestones and performance measures and to define actionable next steps. The STEM Education CAP goal leader from NSF told us the assistance provided by the PIC helped the goal team improve its implementation plan and develop relevant performance measures.

Holding senior-level reviews. Another way OMB and the PIC are increasing leadership attention of CAP goal implementation is by committing to hold regularly scheduled senior-level meetings to review CAP goal progress. OMB’s Deputy Director for Management leads implementation-focused meetings for the eight management CAP goals approximately three times a year, and OMB’s Deputy Director for Budget leads meetings to review the seven mission-focused CAP goals as necessary. According to OMB, as of March 2016, it held the senior-level review meetings for all 15 CAP goals as planned.

Obtained a means of funding cross-agency CAP goal activities.

As part of the fiscal year 2016 consolidated appropriations act, Congress provided authority for the heads of executive departments and agencies, with OMB approval, to transfer up to $15 million for purposes of improving coordination, reducing duplication, and other activities related to the implementation of the CAP goals. OMB staff told us that they proposed this means of funding crosscutting activities in response to lessons learned from the interim CAP goal process, feedback from agencies, and our work on enhancing collaboration in interagency groups. Table 1 lists the fiscal year 2016 interagency transfers for selected CAP goals. For example, OMB reported that the Lab-to-Market CAP goal team would use $1.9 million of transferred funds to develop an interface for all 17 DOE national laboratories to directly interact with external stakeholders, such as the investment community, and allow the public to access specific capabilities across the national laboratory network.

Launched a government-wide White House Leadership Development Program. OMB staff told us that leadership fellows were selected and assigned to one of the CAP goal teams and other cross-agency initiatives to expose emerging agency leaders to cross- agency issues and address the need for strong leadership on the CAP goals, while leveraging existing resources. In November 2015, 16 fellows began their 1-year rotations.

Goal Teams Reported CAP Goal Designation Led to Increased Leadership Attention and Collaboration

CAP goal leaders and their teams told us that the CAP goal designation has resulted in increased leadership attention within their agencies and access to management and performance expertise at OMB, the PIC, and government-wide councils such as the President’s Management Council (PMC) and the Chief Human Capital Officers Council. CAP goal teams we spoke with told us that the increased focus on CAP goal implementation enabled CAP goal teams to request and receive agency resources that helped sustain progress toward goals. For example, the Smarter IT Delivery CAP goal team worked with OPM to obtain a new hiring authority for digital service experts to assist with production of information technology and digital services. As of December 2015, five agencies and OMB reported hiring digital service experts. Goal team staff told us that obtaining the hiring authority was a direct result of the CAP goal.

According to OMB and PIC staff, the CAP goal process is a management tool to encourage interagency collaboration. OMB staff told us that they selected CAP goals, in part, that would focus on implementation challenges that could benefit from greater collaboration between multiple agencies. According to OMB staff, some CAP goal efforts are relatively new and agencies do not have much experience working together, whereas other CAP goal issues have long-standing relationships and interagency mechanisms in place. All of the CAP goal teams described approaches they are using to work across agency lines. For example, CAP goal teams established formal interagency groups, used information- sharing tools to communicate, and developed standardized guidance.

Leveraging expertise across agencies. CAP goal teams report that they are working with pre-existing interagency working groups or establishing new working groups or communities of practice to leverage expertise and experience across agencies. For example, the Customer Service CAP goal team established an interagency Community of Practice (COP) which meets monthly to share practices and provide feedback to CAP goal leadership. For example, the goal team reports that a group within the COP led the development of a draft customer service toolkit, which includes customer service principles and a model for gauging program-level progress toward those principles. Likewise, the Open Data CAP goal team established the Interagency Open Data Working Group, which includes the 24 Chief Financial Officers (CFO) Act agencies and smaller entities government-wide. The group meets biweekly and its members represent a diverse range of levels of employees, such as IT contractors and chief data officers. According to the goal team, the establishment of this working group is the best outcome from the CAP goal so far. The group shares best practices, allows agencies to learn from other agencies that may be farther along, and shares tools. For example, the goal team told us that it hosted a series of webinars on Data.gov explaining a new method to convert data from common format to machine readable. The goal team told us that this process was a challenge for agencies, but through the webinars and the working group, every agency now knows how to produce machine- readable documents, and is currently validating their processes.

Developing common reporting framework. The Job-Creating Investment goal team told us that the CAP goal designation has helped the Department of Commerce (Commerce) headquarters staff to engage with overseas staff and better leverage their resources and data. For example, the CAP goal team told us that they launched an interagency effort between State and Commerce to provide training and update guidance to U.S. overseas staff on standardized methods and procedures for reporting information on their activities to attract foreign direct investment. Overseas staff are now submitting updates that reflect Commerce’s activities abroad. The goal team reported that it now holds quarterly reviews, in part, to ensure the data is being collected and reported consistently by staff from multiple agencies and programs.

Outreach to the public and nonfederal stakeholders. The Open Data CAP goal team reported a number of White House-sponsored events to encourage public participation in using government data and to elicit feedback on Open Data CAP goal efforts. For example, the goal team reported that agencies hosted Open Data Roundtables to highlight success stories and get feedback from its stakeholders, including nonfederal data users. In another example, the STEM Education CAP goal team developed an online portal for graduate students to locate federally-funded STEM research opportunities. CAP goal team staff told us they recently completed a similar resource for undergraduate students.

Selected CAP Goal Teams Have Met Many GPRAMA Reporting Requirements, but Most Have Not Established Quarterly Targets

Based on GPRAMA requirements, we identified five broad actions that CAP goal teams follow to implement a goal: (1) establish the goal, (2) identify goal leaders and contributors, (3) develop strategies and performance measures, (4) use performance information to make decisions and improve performance, and (5) report results. The 13 GPRAMA requirements for reporting on CAP goal progress each relate to an action for goal implementation. These actions do not necessarily occur sequentially. For example, in the quarterly progress updates published on Performance.gov, CAP goal teams reported developing new strategies and performance measures, while simultaneously using performance information to report results and develop new strategies.

Action 1: Establish the Goal

Example of a Goal Statement: Lab-to- Market CAP Goal

The first action for goal implementation is to articulate the cross-agency goal. GPRAMA requires that Performance.gov contain a description of each CAP goal. To meet this requirement, CAP goal teams list the overall goal statement and provide other contextual information, such as the problem being addressed and a vision for what the goal might achieve in each quarterly progress update published on Performance.gov. As shown in figure 3, all of the selected CAP goal teams met this reporting requirement.

Action 2: Identify Goal Leaders and Contributors

Definition of common term: Contributors –The agencies, organizations, program activities, regulations, tax expenditures, policies, and other activities that contribute to each CAP goal.

The second action for goal implementation is to identify leaders and contributors that are accountable for progress toward the goal, and to establish the cross-agency team that will monitor and oversee implementation. GPRAMA requires that Performance.gov provide information on the lead government official for each CAP goal and the agencies, organizations, policies, and other activities—including tax expenditures—that contribute to the goal. As depicted in figure 4, we found that all of the selected CAP goals identified and clearly reported the goal leaders accountable for implementation of the CAP goal.

Also shown in figure 4, we determined that all of the selected CAP goal teams fully met the requirement to identify and report key contributors to the goal. To make this determination, we reviewed the CAP goal statements and related activities, and evaluated whether selected goal teams identified key contributors to the goal. For example, the Open Data CAP goal team identified contributing agencies and offices within the Executive Office of the President and the General Services Administration in addition to policies governing open data efforts at federal agencies, such as agency requirements to protect personal and confidential data.

For this review, we also evaluated whether OMB and the goal teams considered the extent to which tax expenditures contribute to the selected CAP goals. This issue relates to a recommendation we made in 2013 that OMB should review whether all relevant tax expenditures that contribute to a CAP goal have been identified, and as necessary, include any additional tax expenditures in the list of federal contributors for each goal. In September 2015, OMB staff told us that OMB had determined that there were no tax expenditures that were critical to support achievement of the current CAP goals. Goal teams we spoke with during our review told us that they did not identify any relevant tax expenditures related to their goals. As a result, we have recently closed this recommendation as implemented.

Action 3: Develop Strategies and Performance Measures

The third action for goal implementation is developing strategies and performance measures. GPRAMA requires that each CAP goal team publish on Performance.gov plans to address government-wide major management challenges, and establish clearly defined quarterly milestones and performance indicators (which we refer to as performance measures), and targets to measure or assess overall progress. GPRAMA also requires that teams develop strategies for performance improvement as they regularly review implementation and results.

As shown in figure 5, all CAP goal teams reported plans to address major management challenges that are government-wide or crosscutting in nature as reflected in the quarterly progress updates published on Performance.gov. In addition, all the goal teams’ quarterly progress updates established clearly defined quarterly milestones. Specifically, we found that all CAP goal teams we reviewed established a work plan that included a brief description of planned milestones, milestone due dates, and the agency or interagency group responsible for implementing milestones. If clearly linked to the outcome, milestones can be a way for goal teams to report and track progress quarterly.

We determined that all but one of the selected goal teams fully reported strategies for performance improvement, meaning that each goal team updated or revised its strategies or major actions from one quarter to the next, or identified challenges or barriers to the goal or a specific milestone. The Lab-to-Market CAP goal team partially met this requirement because while the team updated some of its milestones in the first quarter of fiscal year 2016, it did not report any associated barriers or challenges, nor did it identify any milestones for the sub-goal to assess the economic impact of federal technology transfer activities.

Definition of common terms: Performance measure - A measurable value that indicates the state or level of something (also called “indicator”). Sub-goal - A goal that contributes to the achievement of the broader CAP goal.

Five of the seven CAP goal teams reported common performance measures for each of their sub-goals. The Lab-to-Market CAP goal team reported a number of performance measures for three of its four sub- goals, partially meeting this requirement. The Customer Service CAP goal does not have any performance measures at this time, but the team reported a number of performance measures that are under development. According to OMB and PIC staff, some of the CAP goal teams may be working to develop cross-agency performance measures for the first time, or may have measures in place that require improvement. Because the scope of our review was to examine the implementation of quarterly reporting requirements, we did not evaluate whether these measures were appropriate indicators of performance, were sufficiently ambitious, or met other dimensions of quality.

Definition of common term: Target - Level of performance expressed as a tangible, measurable objective, against which actual achievement can be compared, including a goal expressed as a quantitative standard, value, or rate.

Five of seven goal teams fully or partially met the requirement to report on the planned level of performance because they reported an annual target or a target for the 4-year goal period for at least one sub-goal. The two CAP goal teams that fully met this requirement reported targets for all performance measures. For example, for one of its sub-goals, the People and Culture CAP goal team established a target that, by the end of the CAP goal period in 2018, there would be an increase in hiring managers’ satisfaction with the quality of applicants from 60 percent to 70 percent, as measured by the Chief Human Capital Officers Management Satisfaction Survey. The Customer Service CAP goal team did not report any targets because it does not yet have established performance measures. The Science, Technology, Engineering, and Mathematics (STEM) Education goal team did not identify a target—expressing both magnitude and direction—for 12 of its 15 performance measures. The most recent data available for these measures is from 2013, prior to the start of the current CAP goal period.

Of this set of actions, the selected goal teams had the most difficulty meeting the requirement to report quarterly targets for each performance measure. Five of the seven selected CAP goal teams did not report quarterly targets. The Open Data goal team reported a quarterly target for 1 of its 16 performance measures, partially meeting this requirement. The STEM Education goal team also partially met this requirement by reporting a 1 percent quarterly target increase in the number of views, downloads, and international downloads of a June 2015 webinar focused on implementing evidence-based practices in undergraduate STEM instruction. OMB and PIC staff told us that they would like to work toward requiring quarterly targets, as appropriate, going forward. However, they also noted it may not be possible or appropriate for a goal to have a quantitative quarterly target, as relevant data may not be available quarterly, particularly for new initiatives. In such a case, a qualitative target might be more appropriate. CAP goal teams we spoke with expressed similar concerns and told us that they are instead relying on milestones to track progress quarterly. For example, the People and Culture CAP goal team relies on results from the Federal Employee Viewpoint Survey to measure progress on employee engagement efforts. Since the survey is administered annually, the goal team reports annual targets for this measure. The Lab-to-Market CAP goal team told us that its goal to accelerate and improve the transfer of new technologies from the laboratory to the commercial marketplace can take years and is not measured quarterly.

Action 4: Use Performance Information

The fourth action related to goal implementation involves using performance information during regular progress reviews to assess if major actions and contributors are collectively achieving the desired results. We have long reported that agencies are better equipped to address management and performance challenges when managers effectively use performance information for decision making. OMB’s guidance to CAP goal teams states that frequent reviews provide a mechanism for CAP goal leaders to keep contributing agencies focused on crosscutting priorities. For public reporting, GPRAMA requires that CAP goal teams report on how they use performance measures (or indicators) and quarterly targets to assess overall progress toward the goal, and whether relevant agencies, organizations, program activities, and tax expenditures are contributing as planned.

Figure 6 shows that for six of the seven selected CAP goals, the teams partially met the requirement to report how they are using performance measures with quarterly targets to assess overall progress. This assessment is largely a result of the fact that while six of the seven goal teams have established and reported performance measures, they generally have not established quarterly targets for the measures, as shown in figure 5 above. OMB and PIC staff managing the CAP goals told us that they are aware that CAP goal teams can improve how they are using performance measures and targets.

Four of the seven selected CAP goals partially met the requirement to report how they are using performance measures and quarterly targets to assess key contributors to the goal—such as relevant agencies and programs—because some of their performance measures track contributors’ implementation of certain activities or requirements. For example, the Smarter IT Delivery CAP goal team reports an indicator tracking the number of agencies that have created “buyers’ clubs” to promote more efficient information technology (IT) contracting practices. The other three CAP goals did not meet this requirement because they either did not have measures in place, or did not have performance measures that assess contributions by agencies or programs responsible for implementing the goal. For example, some of the measures for these goals are focused on an outcome or output that is external to the government, such as number of STEM degrees as a performance measure for the STEM Education CAP goal. In such cases, goal teams told us they are tracking progress and key contributors toward the goal through milestones.

Action 5: Report Results

Reporting results is the next action related to goal implementation. GPRAMA requires that CAP goal teams report the most recent quarterly results on Performance.gov. They are also required to publish overall trend data compared to the planned level of performance or target. Our prior work has demonstrated the need to regularly report results to promote accountability and provide agency leaders with information they can use to inform their decisions. By adding trend information, OMB and decision makers can more easily track the goal teams’ quarterly progress. As shown in figure 7, we found that the seven selected CAP goal teams reported results achieved during the most recent quarter by reporting on the status of milestones, data on implementation, and narrative progress updates.

As of the first quarter of fiscal year 2016, six of the seven selected CAP goal teams are able to report overall trend data for at least some, if not all, of their performance measures. In some cases, goal teams reported new performance measures, noting that they are still working to establish a baseline and trend data. The Open Data, Smarter IT Delivery, and People and Culture CAP goal teams reported overall trend data compared to the planned level of performance for each performance measure. The Customer Service CAP goal team did not report trend data because it stated that it does not have a method to collect and report customer service performance information government-wide.

Selected CAP Goal Teams Reported Completing Milestones, but Could Improve Transparency by Reporting on the Development of Performance Measures

Selected CAP Goal Teams Reported Completing Quarterly Milestones

CAP goals are designed to focus on longer term or complex outcomes involving multiple agencies, programs, or entities. In such cases, there may not be a common measure of the desired outcome, the outcome is not frequently observed, or is influenced by external factors. As a result, determining if the goal team is making progress each quarter can be challenging. In such cases, the goal team may need to break the goal into pieces that can be more easily measured or assessed (e.g., sub-goal and milestone). All of the CAP goal teams we examined identified two or more sub-goals with associated milestones and performance measures, where available.

All of the selected CAP goal teams and OMB staff managing the goals told us that they are using milestones to track and report on progress quarterly. However, they are still working to improve the collection and reporting of performance information for the CAP goals. Our review of selected CAP goal quarterly progress updates—from March 2014 through December 2015—found that the seven CAP goal teams demonstrated progress in their ability to meet GPRAMA reporting requirements. Specifically, we found that in the most recent quarterly progress update, the selected CAP goal teams all reported prospects and strategies for performance improvement, and reported more performance information in the form of new performance measures, targets, baseline data, and trend information, if available. However, because most of the targets established are annual or 4-year targets or data for certain performance measures were unavailable, CAP goal teams and OMB staff told us that milestones provided the most relevant information on quarterly progress for the CAP goals.

According to OMB and PIC staff, milestones have also helped the goal teams reach agreement on respective roles and responsibilities, and have helped agencies align their activities with the strategies to implement the goal. Within the first two years of implementation, CAP goal teams reported having completed milestones that cover a range of activities, including those related to project management, piloting new programs, development of information technology, and other tools to support implementation, as shown in the examples below.

Smarter IT Delivery CAP goal. In October 2015, the CAP goal team launched a pilot to develop a Digital Service Contracting Professional Training and Development Program to improve the process of IT acquisition and digital services.

Lab-to-Market CAP goal. In the third quarter of fiscal year 2015, the Department of Energy launched the Small Business Voucher pilot, creating a single point of access for small business to access the agency’s laboratory resources for clean energy projects.

People and Culture CAP goal. In October 2015, OPM launched the web-based Hiring Toolkit, which provides policy and technical guidance, information on roles and responsibilities, and an inventory of hiring flexibilities for human resource staff and managers in federal agencies.

As directed by OMB, goal teams are also reporting milestones that are at risk of not meeting scheduled due dates or face barriers to completion. During the CAP goal period, goal teams reported delays for various reasons, including coordination across agency boundaries, availability of staffing, and challenges with information technology. Teams also reported changes in strategy or approach that altered time frames or required revisions to existing milestones. For example, in the first quarter of fiscal year 2015, the Open Data CAP goal team missed a milestone to refresh its metrics on data quality and machine readability due to technical challenges that delayed the launch of critical tools. The team reported completing this milestone in the following quarter. In another example, the Job-Creating Investment CAP goal team revised a milestone due date to allow more time to evaluate the effectiveness and impact of its processes to assist potential investors in meeting their objectives. The goal team reported in the fourth quarter of fiscal year 2015 that it completed this milestone according to the revised timeline.

Selected CAP Goal Teams Have Emphasized Alignment of Milestones with Strategies and Results

The use of milestones is a recognized approach for tracking interim progress toward a goal as long as they are clearly linked to the desired outcome. OMB Circular A-11 states that if milestones are used as a performance goal, they must be described in a way that makes it possible to discern if progress is being made toward the goal. Figure 8 provides examples of a performance measure and a milestone for two different CAP goals.

According to OMB and PIC staff managing the CAP goals, they are working with goal teams to develop strategies to improve performance on crosscutting issue areas and also develop concrete milestones. In its guidance to CAP goal teams, OMB and the PIC directed goal teams to design and report a high-level, goal-to-strategy crosswalk, highlighting the key elements of the goal or sub-goal, such as the related major actions to achieve impact, and key performance measures, if available. According to the guidance, once the team has identified sub-goals and developed a high-level strategy, the goal team should identify and report the quarterly status of related milestones, which should be linked to a sub-goal or major action to illustrate how the milestones contribute to the achievement of the goal. Our review of the selected CAP goal quarterly progress updates found that all seven CAP goal teams followed the guidance and identified sub-goals with major actions and performance measures, where possible. As designed, the reporting template provides flexibility for the CAP goal teams to organize and report this information, and the CAP goal teams generally reported on how their milestones were linked with their related sub-goals.

CAP goal teams that used their quarterly progress updates to clearly articulate a strategy to achieve the sub-goal, and any barriers or challenges along with associated actions and milestones, are better positioned to demonstrate progress. As shown in figure 9, the People and Culture CAP goal quarterly progress update demonstrated the link between the milestones, and the result it intends to achieve, by identifying specific milestones or major actions and performance measures for each sub-goal or outcome. OPM staff for the People and Culture CAP goal told us that aligning sub-goals and milestones allows them to see how all milestones are contributing to the overall needs of the CAP goal. It also allows them to review the information at the sub-goal level and roll the information up to the CAP goal level when necessary.

Taken together, the sub-goals, milestones, and performance measures, if available, should demonstrate how the goal team is measuring progress related to the broader CAP goal. An effective tool to demonstrate that the goal team’s activities are contributing to the goal’s desired long-term outcome is to develop a model that describes the logical relationship between an agency’s inputs, activities, outputs, and outcomes. By making those linkages explicit, decision makers can have more focused and meaningful discussions for how proposed strategies or milestones are tied to desired results, and how to measure the success of strategy execution and impact.

Identifying, agreeing with, and understanding the logical relationship between the goal team’s activities and the desired outcome is an important step in developing a sound and cohesive plan for implementation. As we have previously reported, the relationship between activities—such as milestones—and outcomes should be periodically assessed through program evaluations or other methods to determine if outcomes are being achieved as expected, and should be revised as necessary. The quarterly progress reviews of CAP goals required by GPRAMA is one vehicle for assessing implementation of the goals and adjusting strategies as needed. As directed by OMB, any adjustments to the goal strategy or next steps should be clearly reported in the subsequent quarterly progress updates published on Performance.gov to allow the public and decision makers to easily follow the implementation strategy and goal progress.

Selected CAP Goal Teams Can Be More Transparent about the Status of Their Efforts to Develop Performance Measures

All of the goal teams we reviewed reported efforts to develop performance measures as part of their strategy to improve performance of these complex and crosscutting issues. Performance goals and measures allow CAP goal teams and other stakeholders to track the progress they are making toward their goals and provide critical information on which to base decisions for improving programs or activities. OMB and PIC staff overseeing the CAP goals told us they are aware that improvement can be made in goal teams’ tracking and reporting of progress using performance measures, and they hope to continue to improve goal teams’ capacity to track performance. OMB and the PIC directed CAP goal teams to select or develop performance measures that are relevant, well defined, timely, reliable, and capable of being influenced by the actions of contributing organizations.

We found that three of the seven selected goal teams reported specific actions they have completed, or are completing, to develop performance measures. The actions included identifying existing and needed data, collecting new data, testing and validating the data for reliability, and setting the baseline for measurement. For example:

Throughout the goal period, the STEM Education CAP goal team consistently reported on the steps they were taking to develop performance measures, and we were able to track their efforts from one quarter to the next. For example, the goal team identified the need to collect additional data to develop performance measures through a long-term survey. In the fourth quarter of 2015, the goal team reported they had completed some milestones—and others were on track—related to conducting in-depth testing to add a new item on undergraduate mathematics instruction for the High School Longitudinal Survey (see figure 10). The goal team reported three related milestones that, according to the goal team, should provide survey results by December 2017. At that point the team expects to be able to test and validate a new performance measure and establish a baseline and targets for future measurement.

The Job-Creating Investment CAP goal team reported that they are collecting baseline data and trends to ensure the methodology supporting future performance measures is valid. The goal team reported that it conducted a feasibility study to capture data on foreign direct investment in the U.S. through a new evaluation system and database. It also reported that it is working to collect data on client satisfaction with the services it received. The CAP goal team plans to use these sources of data to measure the impact of its work.

The Customer Service CAP goal team reported steps it plans to take to develop a measure tracking the percentage of customer-facing federal programs showing improvements in their customer feedback data. For example, the goal team identified milestones for implementation into fiscal year 2017 to identify the core customer- facing federal programs that have a plan to improve customer service and collect customer service feedback data. The goal team reported that these programs would conduct a self-assessment and develop strategies to improve customer service followed by assessment of the data to identify its baseline customer feedback data.

For the other four CAP goals we reviewed, the goal teams told us about the actions they are taking to develop performance measures. However, their quarterly reports on Performance.gov did not contain this information. For example, the Lab-to-Market CAP goal leaders told us that they are conducting public literature reviews and commissioning studies that may help the goal team identify potential quantitative and qualitative performance measures to assess the economic impact of federal technology transfer activities. The goal team reported having completed milestones related to this effort in 2014 and 2015, but has not reported additional steps it has taken, or plans to take, to test or validate the results of the studies. As a result, we could not determine if the goal team is making progress towards establishing performance measures.

Conclusions

The CAP goals we reviewed are intended to drive progress in important and complex areas, such as improving information technology and customer service interactions across government, coordinating federal STEM education efforts, and improving federal hiring practices. We found that OMB’s updated guidance to CAP goal teams and the quarterly reporting template have assisted goal teams in managing implementation of the goals and in meeting the GPRAMA reporting requirements. Further, OMB and the PIC’s efforts to build the capacity of the CAP goal teams to implement the goals has resulted in increased leadership attention and improved interagency collaboration for these goals.

We found that CAP goal teams are meeting a number of GPRAMA reporting requirements, including identifying contributors and reporting quarterly results and milestones. However, most of the CAP goal teams we reviewed have not established quarterly targets for all performance measures. In an effort to assist each team with addressing their reporting gaps, OMB and the PIC are working to improve collection and reporting of performance information. We found that during the first two years of goal implementation, OMB and CAP goal teams made improvements to their quarterly progress updates by adding new performance measures, providing additional information on performance trends, and reporting on new prospects for performance improvement, among other improvements.

CAP goal teams told us that they are using milestones to track and report progress quarterly. We generally found that the goal teams are aligning their quarterly activities with longer-term strategies to achieve the desired goal outcomes. Further, all of the selected CAP goal teams reported that they are working to develop performance measures, and are at various stages of the process. But, they are not consistently reporting on their efforts to develop these measures. Given OMB’s, the PIC’s, and CAP goal teams’ emphasis on developing measures that are relevant and well defined, greater transparency is needed to track goal team’s efforts quarterly. Because the CAP goal teams are working to implement policies and activities that span multiple agencies, and in some cases government-wide, it is important that the goal teams clearly communicate the steps they are taking to develop performance measures to ensure that the measures will be aligned with major activities and clearly understood by contributors to the goals. With improved performance information, the CAP goal teams will be better positioned to demonstrate the progress that they are making, and will help ensure goal achievement at the end of the 4-year goal period in 2018.

Recommendation for Executive Action

To improve the transparency of public reporting on CAP goal progress, we recommend that the Director of OMB, working with the PIC, take the following action:

Report on Performance.gov the actions that CAP goal teams are taking, or plan to take, to develop performance measures and quarterly targets.

Agency Comments and Our Evaluation

We provided a draft of this report for review and comment to the Director of OMB, the Director of the Office of Science and Technology Policy (OSTP), and the Secretaries of the Departments of Commerce, Energy, State, Transportation, and Veterans Affairs, and the Directors of the National Science Foundation and the Office of Personnel Management, and the Commissioner of the Social Security Administration.

On May 3, 2016, OMB staff provided us with oral comments on the report. OMB staff generally agreed with the recommendation in the report, and provided us with technical clarifications, which we have incorporated as appropriate. Officials from the Departments of Energy and Commerce provided technical clarifications, which we incorporated, as appropriate.

OSTP and the Departments of State, Transportation, Veterans Affairs, the National Science Foundation, the Office of Personnel Management, and the Social Security Administration had no comments on the report. The written response from the Social Security Administration is reproduced in appendix III.

We are sending copies of this report to the Director of OMB and the heads of the agencies we reviewed as well as appropriate congressional committees and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Selected Cross-Agency Priority (CAP) Goals and Related Sub-goals

Selected CAP Goal Customer Service Increase citizen satisfaction and promote positive experiences with the federal government by making it faster and easier for individuals and businesses to complete transactions and receive quality services. Job-Creating Investment Encourage foreign direct investment, spurring job growth by improving federal investment tools and resources, while also increasing interagency coordination. Lab-to-Market Increase the economic impact of federally- funded research and development (R&D) by accelerating and improving the transfer of new technologies from the laboratory to the commercial marketplace. Open Data Fuel entrepreneurship and innovation, and improve government efficiency and effectiveness by unlocking the value of government data and adopting management approaches that promote interoperability and openness of these data. People and Culture Innovate by unlocking the full potential of the workforce we have today and building the workforce we need for tomorrow.

Sub-goals 1. Improve top customer service interactions 2. Develop and implement standards, practices, and tools 3. Feedback and transparency 4. Focus on the frontline 1. Promote and market the United States as the premier investment destination 2. 3.

Smarter IT Delivery Eliminate barriers and create new incentives to enable the federal government to procure, build, and provide world-class, cost-effective information technology (IT) delivery for its citizens, and hold agencies accountable to modern IT development and customer service standards.

STEM Education Improve Science, Technology, Engineering, and Mathematics (STEM) Education by implementing the Federal STEM Education 5- Year Strategic Plan, announced in May 2013.

Appendix II: Objectives, Scope, and Methodology

The GPRA Modernization Act of 2010 (GPRAMA) included a provision for us to periodically report on the implementation of the act, and this report is part of that series. The objectives of this report are to assess (1) the extent to which lessons learned from implementing the interim cross- agency priority (CAP) goals were incorporated into the governance of the current CAP goals; (2) the extent to which GPRAMA requirements for reporting on CAP goal progress are included in the selected CAP goal quarterly progress updates; and (3) the initial reported progress in implementing the selected CAP goals. In September 2015, we reported on lessons learned from the interim CAP goal period—which ended in March 2014—and provided an assessment of CAP goal teams’ initial reported progress implementing the current set of 2014-2018 CAP goals.

To conduct our assessment, we selected 7 of the 15 CAP goals for more in-depth review. We randomly selected four of the seven CAP goals including Open Data, STEM Education, Job-Creating Investment, and Lab-to-Market. We also included three goals in our review because we have completed and ongoing work related to the Customer Service, People and Culture, and Smarter IT Delivery CAP goals. Our sample of goals is nongeneralizable and is not representative of all CAP goals.

We interviewed Office of Management and Budget (OMB) and Performance Improvement Council (PIC) staff responsible for the management and implementation of the current CAP goals, as well as agency officials and staff, including CAP goal leaders and members of the seven CAP goal teams. We requested to meet with representatives from the Presidential Personnel Office and National Economic Council who have responsibility for leading the People and Culture and Job-Creating Investment CAP goals. Staff from those offices directed us to obtain feedback on these goals from responsible staff at OMB. We reviewed OMB and PIC guidance, relevant documentation, our prior related work, and quarterly progress updates published on Performance.gov from the second quarter of fiscal year 2014 through the first quarter of fiscal year 2016, published in March 2016.

To determine the extent to which selected CAP goal quarterly progress updates reflect GPRAMA requirements for assessing and reporting CAP goal progress, we conducted a content analysis—using NVivo software and Excel—of the quarterly updates from the second quarter of fiscal year 2015 through the first quarter of fiscal year 2016. Specifically, we reviewed each quarterly progress update and coded the content into one or more of the five major actions (as described below) to implement a CAP goal, which we developed from an initial review of the selected quarterly progress updates, and OMB and PIC guidance to CAP goal teams.

Purpose - establishing the goal;

Governance - identification of leaders and contributors;

Planning - development of strategies and performance measures;

Implementation - execution of plans and use of performance information; and

Results - reporting results on Performance.gov.

To improve the validity of results, each plan was coded independently by two of our analysts, who then compared their analysis in three stages. In cases where there was disagreement, the two coders met and either came to agreement on a code, or a third coder made a judgment about correct coding.

In the first stage of content analysis, the coders independently assigned the content in each of the select CAP goal quarterly progress updates into one or more of the five actions of goal implementation described above. For the second stage of content analysis, the coders assigned the relevant GPRAMA requirements to one or more of the five actions of goal implementation. During the first two stages of content analysis, we determined if an element or requirement was present in the plan (one or more coded entries). For the third and final stage of content analysis, we assessed the extent to which the progress update met the reporting requirements.

To determine the extent to which the seven selected CAP goals’ quarterly progress updates reflect GPRAMA requirements, we conducted a content analysis of the four quarterly updates published from March 2015 through December 2015. To fully meet each requirement, the quarterly progress updates had to provide all information required by GPRAMA. We analyzed 13 requirements – 6 that we determined were relevant at the CAP goal level, and 7 that we determined were relevant at the sub-goal level. For those requirements analyzed at the sub-goal level, we determined a goal had partially met a requirement if the required information was present in the quarterly progress update for at least one of the goal’s sub-goals. For requirements that were not met, we determined the required information was not present in the quarterly progress update. We shared our analysis with each of the selected CAP goal teams and interviewed agency staff to collect their feedback on our analysis. We also collected and reviewed relevant documentation provided by agencies, OMB, and the PIC.

One of the GPRAMA reporting requirements is for OMB, in coordination with CAP goal teams, to identify and publish the key agencies, organizations, program activities, regulations, tax expenditures, policies, and other activities that contribute to each CAP goal on Performance.gov. To determine the extent to which selected CAP goals met this requirement, we reviewed the published lists of contributors in the quarterly progress updates, and interviewed CAP goal teams and OMB staff responsible for managing the CAP goals. We did not evaluate the process that goal teams used to identify the contributors.

To assess the extent to which the selected CAP goal teams have made progress during the first 2 years of implementation from March 2014 through December 2015, we reviewed the quarterly progress updates published on Performance.gov and relevant agency documents, interviewed CAP goal team staff to discuss challenges, efforts to mitigate challenges, and key achievements. Because the scope of our review was to examine the implementation of quarterly reporting requirements, we did not evaluate whether these goals were appropriate indicators of performance, were sufficiently ambitious, or met other dimensions of quality.

We conducted our work from January 2015 to May 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Comments from the Social Security Administration

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

J. Christopher Mihm, (202) 512-6806 or mihmj@gao.gov.

Staff Acknowledgements

In addition to the contact named above, Sarah E. Veale, Assistant Director, and Peter Beck, Analyst-in-Charge, supervised the development of this report. Margaret Adams, Jehan Chase, Jillian Feirson, Jennifer Felder, and Steven Putansu made significant contributions to all aspects of this report. Lisette Baylor, Adam Cowles, Robert Gebhart, David Hinchman, Donna Miller, Christopher Murray, Nyree Ryder Tee, and MaryLynn Sergent provided additional assistance.