Managing for Results: OMB Should Strengthen Reviews of Cross-Agency Goals
Released: 2014-06-10T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: The federal government faces complex, high-risk challenges, such as protecting our nation's critical information systems. Effectively managing these challenges is essential for national and economic security and public health and safety. However, responsibility for addressing these challenges often rests with multiple agencies. To effectively address them, shared goals and cross-agency collaboration are fundamental.

This report responds to GAO's mandate to evaluate the implementation of GPRAMA. It assesses (1) what is known about progress made towards the interim CAP goals; and (2) how, if at all, quarterly progress reviews reflected GPRAMA requirements and leading practices for reviews, as well as how reviews contributed to improved cross-agency performance and collaboration. To address these objectives, GAO analyzed CAP goal status updates and other documents from OMB and CAP goal progress-review meetings, and interviewed OMB staff and CAP goal representatives. GAO compared this information to GPRAMA requirements and to leading practices for performance reviews previously reported on by GAO.

section_title: What GAO Found
paragraphs: CAP Goal Progress. The GPRA Modernization Act of 2010 (GPRAMA) requires the Office of Management and Budget (OMB) to coordinate with agencies to: (1) establish outcome-oriented, federal government priority goals (known as cross-agency priority, or CAP, goals) with annual and quarterly performance targets and milestones; and (2) report quarterly on a single website now known as Performance.gov the results achieved for each CAP goal compared to the targets. In February 2012, OMB identified 14 interim CAP goals and subsequently published five quarterly updates on the status of the interim CAP goals on Performance.gov. While updates for eight of the goals included data that indicated performance towards an overall planned level of performance, only three also contained annual or quarterly targets that allowed for an assessment of interim progress. Updates for the other six goals did not report on progress towards a planned level of performance because the goals lacked either a quantitative target or the data needed to track progress. The updates on Performance.gov also listed planned activities and milestones contributing to each goal, but some did not include relevant information, including time frames for the completion of specific actions and the status of ongoing efforts. The incomplete information in the updates provided a limited basis for ensuring accountability for the achievement of targets and milestones.

OMB Quarterly Progress Reviews. GPRAMA also requires that OMB—with the support of the Performance Improvement Council (PIC)—review CAP goal progress quarterly with goal leaders. OMB instituted processes for reviewing progress on the goals each quarter, which involved the collection of data from goal leaders and the development of a memorandum for the OMB Director. However, the information included in these memorandums was not fully consistent with GPRAMA requirements. For example, GPRAMA requires OMB to identify strategies for improving the performance of goals at risk of not being met, but this was not consistently done. Without this information, OMB leadership and others may not be able to adequately track whether corrective actions are being taken, thereby limiting their ability to hold officials accountable for addressing identified risks and improving performance.

Leading Practices for Reviews. At the CAP-goal level, goal leaders for two CAP goals and one sub-goal instituted in-person progress reviews with officials from contributing agencies that were broadly consistent with the full range of leading practices for reviews, such as leadership involvement in reviews of progress on identified goals and milestones, and rigorous follow-up on issues identified through these reviews. In these cases, goal managers reported there were positive effects on performance, accountability, and collaboration. In contrast, review processes used by other goal leaders did not consistently reflect the full range of leading practices. Effective review processes consistently engage leaders and agency officials in efforts to identify and address performance deficiencies, and to ensure accountability for commitments. Thus, not using them may result in missed opportunities to hold meaningful performance discussions, ensure accountability and oversight, and drive performance improvement.

section_title: What GAO Recommends
paragraphs: GAO is making seven recommendations to OMB to improve the reporting of performance information for CAP goals and ensure that CAP goal progress reviews meet GPRAMA requirements and reflect leading practices. OMB staff generally agreed to consider GAO's recommendations.

section_title: Letter
paragraphs: Collaboration across agencies is fundamental to effectively addressing  many complex, high-risk challenges facing the federal government,  including protecting the nation’s critical information systems, reducing  billions of dollars lost through improper payments, and better managing  the risks of climate change. Effectively managing these challenges is  essential to national and economic security, public health and safety, and  the integrity of federal programs. However, because these challenges are  complex, responsibility for addressing them often rests with multiple  agencies. Both Congress and the executive branch have recognized the  need for improved collaboration across the federal government, and the  GPRA Modernization Act of 2010 (GPRAMA) established a new  framework and processes aimed at encouraging a more crosscutting and  integrated approach to focusing on results and improving government  performance and management.

Among other things, GPRAMA requires the Office of Management and  Budget (OMB) to coordinate with agencies to develop federal government  priority goals (now known as cross-agency priority, or CAP, goals), which  include outcome-oriented goals covering a limited number of crosscutting  policy areas, as well as goals to improve management across the federal  government. These goals are intended to cover areas where increased  cross-agency collaboration is needed to improve progress towards  shared, complex policy or management objectives, such as improving our  nation’s cybersecurity posture and addressing critical skills gaps in the  federal workforce. In addition, in an effort to drive improved performance,  transparency, and accountability across the federal government,  GPRAMA requires the use of specific processes to encourage the  application of leading management practices.

One GPRAMA requirement is that leaders regularly review progress  towards established agency and cross-agency goals. Specifically, the  Director of OMB, with the support of the Performance Improvement  Council (PIC), must review progress towards the CAP goals with the  appropriate lead government official at least quarterly to increase  management’s collection and use of performance information to achieve  crosscutting goals. Under GPRAMA, most large federal agencies have  been holding quarterly reviews to assess progress in achieving their  individual agency priority goals. The emphasis on the creation of shared  performance goals, the collection and analysis of data, and the regular  review of these data with involved officials is designed to foster  collaboration among agencies, ensure that interagency efforts are  informed by data on performance, identify successful practices, and  reinforce agency and collective accountability for the achievement of  shared outcomes.

This report is our response to a mandate that we evaluate the  implementation of GPRAMA.

Due to the timing of our work, we focused  on efforts surrounding the 14 interim goals established in February 2012.This report assesses (1) what is known about progress made towards the  interim CAP goals; and (2) how, if at all, quarterly progress reviews  reflected GPRAMA requirements and leading practices for reviews, as  well as how they contributed to improved cross-agency performance and  collaboration. The experience with the interim goals can provide lessons  for the implementation and management of the new CAP goals released  with the fiscal year 2015 budget.

To address these objectives, we interviewed representatives of 13 of the  14 interim CAP goals.  For 8 of the 13 goals, we spoke directly with the  goal leader or deputy goal leader, along with, in some cases, other staff  from agencies and OMB involved in supporting efforts related to the  goals. For the other five goals, we met with agency officials or OMB staff  playing a key role in the management of interagency efforts related to the  goal. We also interviewed the goal leaders for 11 agency priority goals  aligned with, or contributing to, a CAP goal.

To address the first objective, and assess what is known about progress  made towards the interim CAP goals, we analyzed performance  information included in the quarterly status updates for each CAP goal  published on Performance.gov, as well as relevant information collected  through our interviews with CAP goal representatives. We compared the  data and information made available through these updates with the  GPRAMA requirement that Performance.gov includes the results  achieved during the most recent quarter and overall trend data. To  assess the reliability of performance data and information available  through Performance.gov, we collected information from OMB and PIC  staff and CAP goal representatives about data quality control procedures.  We determined that the data and information were sufficiently reliable for  our analysis of what was reported on Performance.gov about progress  towards identified goals and milestones.

To address the second objective and assess how OMB CAP goal  progress reviews were conducted, we reviewed quarterly review  memorandums developed by OMB staff. We also conducted interviews  with staff from OMB and the PIC. We compared the contents of these  memorandums with information required to be reviewed by the Director of  OMB on a quarterly basis.

To further address the second objective, we reviewed documents created  for interagency meetings (where available), such as meeting agendas,  presentation materials, notes, and attendee lists. We also interviewed  sub-goal leaders from the Closing Skills Gaps and Entrepreneurship and  Small Business CAP goals to learn more about their involvement in  quarterly review processes for their CAP goals, and to learn of any impact  the CAP goal designation or review processes had at the sub-goal level.  We selected these goals to reflect one review process that involved  regular meetings between the goal leader and contributors dedicated to  reviewing progress (Closing Skills Gaps), and one that did not rely on  regular meetings at the CAP-goal level to review progress  (Entrepreneurship and Small Business). Our findings from these  interviews cannot be generalized to the other CAP goals. We then  compared what we learned about the processes instituted by each goal  leader to review progress to leading practices for performance reviews  previously identified by GAO.

Because the scope of our review was to examine the implementation of  quarterly reporting and reviews, we did not evaluate whether these goals  were appropriate indicators of performance, were sufficiently ambitious,  or met other dimensions of quality.

We conducted our work from May 2013 to June 2014 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives. More detailed information on our scope and  methodology appears in appendix I.
subsections: section_title: Background
paragraphs: GPRAMA requires OMB to coordinate with agencies to develop long- term, outcome-oriented federal government priority goals for a limited  number of crosscutting policy areas and management improvement areas  every 4 years. Furthermore, with the submission of the fiscal year 2013  budget, GPRAMA required OMB to identify a set of interim priority  goals.interim CAP goals, 9 of which were related to crosscutting policy areas  and 5 of which were management improvement goals.

The President’s 2013 budget submission included a list of 14  The CAP Goal Leader. As required by GPRAMA, each of the interim  CAP goals had a goal leader responsible for coordinating efforts to  achieve each goal. CAP goal leaders were given flexibility in how to  manage these efforts, and were encouraged by OMB to engage officials  from contributing agencies by leveraging existing inter-agency working  groups, policy committees, and councils. For information on the position  of the goal leader and the interagency groups used to engage officials  from agencies contributing to each interim CAP goal, see figure 1. For  more information on the interagency groups used to engage agency  officials in efforts related to each goal, see appendix III.

According to OMB and PIC staff, because CAP goal leaders were  responsible for managing efforts related to the achievement of the goals  as part of a larger portfolio of responsibilities, staff from the PIC, OMB,  and—in some cases—from agencies with project management  responsibilities, provided additional capacity for coordinating interagency  efforts and overseeing the process of collecting, analyzing, and reporting  data. Specifically, PIC staff provided logistical support, assisting with the  regular collection of data, updates to Performance.gov, and the  development of CAP goal governance structures and working groups.  They also provided support in the area of performance measurement and  analysis. For example, PIC staff supported the Exports goal leader by  informing discussions of how to measure the success and impact of  export promotion efforts, providing expertise in the development and  selection of appropriate performance measures, and assisting in the  collection and analysis of relevant data.

Progress Reviews. GPRAMA also requires that the Director of OMB,  with the support of the PIC, review progress towards each CAP goal with  the appropriate lead government official at least quarterly. Specifically,  the law requires that these should include a review of progress during the  most recent quarter, overall trends, and the likelihood of meeting the  planned level of performance. As part of these reviews, OMB is to  assess whether relevant agencies, organizations, program activities,  regulations, tax expenditures, policies, and other activities are  contributing as planned to the goal. The law also requires that OMB  categorize the goals by risk of not achieving the planned level of  performance and, for those at greatest risk of not meeting the planned  level of performance, identify strategies for performance improvement.

In an earlier evaluation of the implementation of quarterly performance  reviews at the agency level, we found that regular, in-person review  meetings provide a critical opportunity for leaders to use current data and  information to analyze performance, provide feedback to managers and  staff, follow up on previous decisions or commitments, learn from efforts  to improve performance, and identify and solve performance problems.

As part of this work we also identified nine leading practices that can be  used to promote successful performance reviews at the federal level. To  identify these practices, we conducted a review of relevant academic and  policy literature, including our previous reports. We refined these  practices with additional information obtained from practitioners at the  local, state, and federal level who shared their experiences and lessons  learned.

Nine Leading Practices That Can Be Used to Promote Successful Performance Reviews

Leaders use data-driven reviews as a leadership strategy to drive performance improvement.

Key players attend reviews to facilitate problem solving.

Reviews ensure alignment between goals, program activities, and resources.

Leaders hold managers accountable for diagnosing performance problems and identifying  strategies for improvement.

There is capacity to collect accurate, useful, and timely performance data.

Staff have skills to analyze and clearly communicate complex data for decision making.

Rigorous preparations enable meaningful performance discussions.

Reviews are conducted on a frequent and regularly scheduled basis.

Participants engage in rigorous and sustained follow-up on issues identified during reviews.

Reporting Requirements. In addition to requiring quarterly reviews,  GPRAMA requires that OMB make information available on “a single  website” (now known as Performance.gov) for each CAP goal on the  results achieved during the most recent quarter, and overall trend data  compared to the planned level of performance. In addition, information  on Performance.gov is to include an assessment of whether relevant  federal organizations, programs, and activities are contributing as  planned, and, for those CAP goals at risk of not achieving the planned  level of performance, information on strategies for performance  improvement.

New CAP Goals. As required by GPRAMA, in March 2014, OMB  announced the creation of a new set of CAP goals in the fiscal year 2015  budget. It then identified 15 CAP goals with 4-year time frames on  Performance.gov—7 mission-oriented goals and 8 management-focused  goals. Five goal areas—Cybersecurity; Open Data; Science, Technology,  Engineering, and Mathematics (STEM) Education; Strategic Sourcing;  and Sustainability (renamed Climate Change (Federal Actions))—were  carried over from the set of interim CAP goals, while the other 10 are new  goal areas. OMB stated on Performance.gov that more detailed action  plans for each of the goals, including specific metrics and milestones that  will be used to gauge progress, will subsequently be released. The new  CAP goals will also have co-leaders; one from an office within the  Executive Office of the President (EOP) and one or more from federal  agencies. According to OMB staff, this change was made to ensure that  CAP goal leaders can leverage the convening authority of officials from  the EOP while also drawing upon expertise and resources from the  agency level.

section_title: CAP Goal Leaders Reported Performance on Goals, but Many Lacked Key Information to Demonstrate Progress
paragraphs: GPRAMA Requirements for Establishing Planned Performance for CAP Goals  GPRAMA requires the Director of OMB to establish, in the annual federal government  performance plan, a planned level of performance for each CAP goal for the year in which the  plan is submitted and the next fiscal year, as well as quarterly performance targets for the goals.

GPRAMA Requirements for Reporting CAP Goal Performance Information  GPRAMA requires the Director of OMB to publish on Performance.gov information about the  results achieved during the most recent quarter and trend data compared to the planned level of  performance for each CAP goal.

OMB released the federal government performance plan on  Performance.gov concurrently with the fiscal year 2013 budget  submission that identified the 14 interim CAP goals. The information on  Performance.gov included a goal statement for each of the interim goals  that established an overall planned level of performance. During the two- year interim goal period, OMB addressed the requirement to report on  results achieved during the most recent quarter for each of the CAP goals  by publishing 5 sets of quarterly updates to the interim CAP goals on  Performance.gov. The first set of updates, for the fourth quarter of fiscal  year 2012, was published in December 2012 and the final set of updates,  for the fourth quarter of fiscal year 2013, was published in February  2014. These documents described general accomplishments made to  date, specific actions completed, or both. The updates to the Broadband  CAP goal, for instance, included short descriptions of general progress  made towards each of the five strategies identified for achieving the goal,  as well as specific milestones accomplished.

The quarterly updates did not, however, consistently identify required  interim planned levels of performance and data necessary to indicate  progress being made toward the CAP goals. Updates to eight of the goals  included quarterly, biannual, or annual data that indicated performance  achieved to date toward the target identified in the goal statement. Three  of the eight goals (Cybersecurity, Energy Efficiency, and Strategic  Sourcing) also contained the required annual or quarterly targets that  defined planned levels of performance, which allowed for an assessment  of interim progress. For example, the Cybersecurity goal’s updates stated  that the goal would not be met within its established time frame, and  provided quarterly performance data compared to quarterly targets for the  entirety of the goal period to support the statement. In contrast, the  updates for the other five goals did not contain annual or quarterly  targets, which made it difficult to determine whether interim progress  towards the goals’ overall planned levels of performance was being  made. For example, updates to the Exports goal included data on the  total amount of U.S. exports by quarter for calendar years 2012 and 2013  but did not include a target level of performance for those years or  quarters. Therefore, it was unclear whether the goal’s overall planned  level of performance of doubling U.S. exports by the end of 2014 is on  track to be met. Furthermore, updates to six interim CAP goals did not  include trend data to indicate progress being made towards the goals’  overall planned levels of performance.

Figure 2 below identifies the frequency with which data on CAP goal  performance were reported, as well as the overall performance CAP goal  leaders reported making compared to the goal’s planned level of  performance through the fourth quarter of fiscal year 2013.

Through our review of information on Performance.gov and interviews  with managers of the six interim CAP goals that did not report any data on  progress towards the stated goal, we identified reasons that included:

Lack of quantitative planned level of performance (targets). The  Entrepreneurship and Small Business CAP goal lacked a quantitative  performance target. The quarterly updates to the goal explained that  efforts were focused on the goal’s 10 sub-goals. Most of these sub- goals, however, also lacked quantitative performance targets. The  deputy goal leader told us that some of the sub-goals did not have  quantitative targets by design, as goal managers thought it more  appropriate to use qualitative milestones to track progress towards  them. The quarterly updates to the “Streamline immigration pathways  for immigrant entrepreneurs” sub-goal lacked a quantitative target but  had a range of qualitative milestones. For example, the Department of  Homeland Security and the Department of State established a  milestone to identify reforms needed to ease the application and  adjudication processes for visas available to certain immigrant  entrepreneurs.

Unavailable data. Some CAP goal managers told us that the data  needed to assess and report progress toward their goals’  performance targets were unavailable or not yet being collected. For  example, a manager of the Job Training CAP goal told us that staff  had not established a baseline number of participants served by  federal job training programs against which progress towards the goal  could be tracked. In addition, managers of the Real Property CAP  goal told us that they did not have data available for tracking progress  toward the goal of holding the federal real property footprint at its  fiscal year 2012 baseline level.

Where key data were not reported, some goal managers took actions to  obtain previously unavailable data or developed an alternative approach  for assessing progress.

Job Training CAP Goal. The first quarterly update for the Job  Training CAP goal, published on Performance.gov in December 2012,  stated that federal agencies were surveyed to compile a list of all job  training programs in the federal government, including the number of  participants served by those programs, and that a working group was  developing a baseline for measuring progress towards the goal of  preparing 2 million workers with skills training by 2015. A goal  manager told us that the deputy goal leader and staff from the PIC  gathered baseline information for most of the programs within the  scope of the CAP goal, but that they were unable to complete the  efforts by the end of the goal period.

Real Property CAP Goal. Managers of the Real Property CAP goal  told us that they worked to establish a baseline and metrics for  measuring future performance and would be able to report on  progress after the goal period ended.

Closing Skills Gaps CAP Goal. A manager of the Closing Skills  Gaps goal told us that the goal’s managers decided early on that it did  not make sense for each of the goal’s identified mission-critical  occupations to have the same skills gaps reduction target. Instead,  managers of the goal’s sub-goals identified efforts to reduce skills  gaps in their specific occupations. They identified an individual  targeted level of performance for that effort and collected and reported  data on progress made towards the target. For instance, managers of  the Acquisitions sub-goal established a target for increasing the  certification rate of GS-1102 contract specialists to 80 percent. The  final quarterly status update to the Closing Skills Gaps CAP goal  reported that the target was met and the certification rate increased to  81 percent.

Veteran Career Readiness CAP Goal. The leader of the Veteran  Career Readiness CAP goal told us that efforts were made to collect  data to assess the veteran employment situation. For instance, she  said that an interagency data-gathering working group reviewed  sources of available data, integrated those data – such as the  unemployment rate for various sub-populations of veterans – into  dashboards for senior leadership review, and made proposals to  improve data availability. In addition, the Army led a working group to  develop a more complete picture of veterans receiving unemployment  compensation. She said that these and other efforts led to a  concerted effort to improve the availability of data, and to develop and  implement metrics measuring career readiness and attendance in a  veteran career transition assistance program. However, no data to  track progress towards the overall goal were reported during the  interim goal period.

As we have previously reported, no picture of what the federal  government is accomplishing can be complete without adequate  performance information. However, OMB and CAP goal leaders did not  identify interim planned levels of performance or targets for most of the  interim CAP goals. Furthermore, they established a number of CAP goals  for which data necessary to indicate progress towards the goal could not  be reported. In so doing, they limited their ability to demonstrate progress  being made towards most of the CAP goals and ensure accountability for  results from those who helped to manage the goals.
subsections: section_title: CAP Goal Leaders Described What Contributed to Goal Achievement, but in Some Cases Information Was Incomplete
paragraphs: GPRAMA Requirement for Establishing Milestones  GPRAMA requires the Director of OMB to establish, in the federal government performance plan,  clearly defined quarterly milestones for the CAP goals.

GPRAMA Requirement for Reporting on Contributions towards Cross-Agency Priority  Goals  GPRAMA requires that OMB identify the agencies, organizations, program activities, regulations,  tax expenditures, policies, and other activities that contribute to each CAP goal on  Performance.gov. It also requires OMB to make available on the website an assessment of  whether relevant agencies, organizations, program activities, regulations, tax expenditures,  policies, and other activities are contributing as planned.

In the status updates that were published on Performance.gov, managers  of each of the CAP goals reported the general approaches, strategies, or  specific initiatives being employed to make progress towards the  achievement of the goal, as well as the departments, agencies, and  programs that were expected to contribute to goal achievement. For  example, the leader of the Science, Technology, Engineering, and  Mathematics (STEM) Education CAP goal identified a number of general  strategies for making progress towards the achievement of its goal of  increasing the number of graduates in STEM subjects by 1 million over  the next 10 years, such as “Address the mathematics preparation gap  that students face when they arrive at college” and “Identifying and  supporting the role of technology and innovation in higher education.” In  addition, the goal leader identified a number of programs and goals within  four departments and agencies that were likely to contribute in part or in  whole to the goal. Figure 3 below illustrates how this information was  presented in the update to the STEM Education CAP goal for the fourth  quarter of fiscal year 2013.

In a May 2012 report on our work related to the CAP goals, we noted that  information on Performance.gov indicated additional programs with the  potential to contribute to each of the CAP goals may be identified over  time. We then recommended that OMB review and consider adding to  the list of CAP goal contributors the additional departments, agencies,  and programs that we identified, as appropriate. OMB agreed with the  recommendation, and in the quarterly updates to the CAP goals  published in December 2012 and March 2013, OMB added some of the  departments, agencies, and programs we identified in our work to some  CAP goals’ lists of contributors. For example, we had noted that 12  member agencies of the Trade Promotion Coordinating Committee had  not been identified as contributors to the Exports CAP goal. OMB added  additional information about contributors to the Exports goal in the update  published in December 2012.

During our review, in some cases CAP goal managers told us about  additional organizations and program types that contributed to their goals,  but which were not identified on Performance.gov or in our previous  report. For example, the leader of the STEM Education CAP goal told us that representatives from the Smithsonian Institution led an  interagency working group that contributed to key efforts towards  achieving the goal. Although the CAP goal updates indicate that the  Smithsonian Institution is involved in federal STEM education efforts, it  was not identified in a dedicated list of contributors to the goal. We have  previously found that federal STEM education programs are fragmented  across a number of agencies. We continue to believe that the federal  government’s efforts to ensure STEM education programs are effectively  coordinated must include all relevant efforts. Furthermore, the leader of  the Broadband CAP goal told us that he is aware that tax deductions  available to businesses making capital investments contributed to the  goal by incentivizing investments in broadband. We have long referred to  such deductions, along with other reductions in a taxpayer’s liability that  result from special exemptions and exclusions from taxation, credits,  deferrals of tax liability, or preferential tax rates, as tax expenditures. As  we have previously reported, as with spending programs, tax  expenditures represent a substantial federal commitment to a wide range  of mission areas. We have recommended greater scrutiny of tax  expenditures. Periodic reviews could help determine how well specific tax  expenditures work to achieve their goals and how their benefits and costs  compare to those of programs with similar goals. As previously  mentioned, GPRAMA also requires OMB to identify tax expenditures that  contribute to CAP goals. However, tax expenditures were not reported as  contributors to the Broadband CAP goal in the quarterly status updates  published on Performance.gov.

section_title: CAP Goal Leaders Identified Milestones for Tracking Progress, but in Some Cases Milestones Were Missing Key Information
paragraphs: Leading practices state that a clear connection between goals and day-to- day activities can help organizations better articulate how they plan to  accomplish their goals. In addition, a clear connection between goals and  the programs that contribute to them helps to reinforce accountability and  ensure that managers keep in mind the results their organizations are  striving to achieve. Milestones—scheduled events signifying the  completion of a major deliverable or a set of related deliverables or a  phase of work—can help organizations demonstrate the connection  between their goals and day-to-day activities and that they are tracking  progress to accomplish their goals. Organizations, by describing the  strategies to be used to achieve results, including clearly defined  milestones, can provide information that would help key stakeholders  better understand the relationship between resources and results.

GAO-13-174; GAO-13-228; and GAO, Managing for Results: Critical Issues for  Improving Federal Agencies’ Strategic Plans, GAO/GGD-97-180 (Washington, D.C.: Sept.  16, 1997). actions, however, lacked clear time frames for completion. Figure 4  below illustrates the “next steps” identified for the Strategic Sourcing  CAP goal in the update for the third quarter of fiscal year 2013.

Completion status: The Real Property CAP goal update for the  second quarter of fiscal year 2013 identified two planned actions as  “next steps.” “After agencies submit their Revised Cost Savings and  Innovation Plans to OMB, OMB will evaluate agency plans to maintain  their square footage baselines, while balancing mission  requirements,” and “Updates on agency square footage baselines and  projects are forthcoming and will be posted on Performance.gov.”  These two actions were again identified as “next steps” in the update  for the third quarter of fiscal year 2013, but no update was provided  on the status of the actions.

By establishing planned activities that, in many of the CAP goal updates,  did not have information about their alignment with the strategies they  supported, their time frames for completion, or their completion status,  CAP goal leaders did not fully demonstrate that they had effectively  planned to support goal achievement or were tracking progress toward  the goal or identified milestones.

OMB did not issue formal guidance to CAP goal leaders on the types of  information that were to be included in the CAP goal updates, including  information about contributors and milestones. Standards for internal  control in the federal government emphasize the importance of  documenting policies and procedures to provide a reasonable assurance  that activities comply with applicable laws and regulations, and that  managers review performance and compare actual performance to  planned or expected results and analyze significant differences.staff told us they provided an implementation plan template to goal  leaders, which outlined the data elements to be reported in the quarterly  status updates. The template was also used to collect information for  internal and public reporting. Some CAP goal managers told us that OMB  or PIC staff, in their role supporting the collection, analysis, and  presentation of data on CAP goal performance, occasionally provided  feedback on the information that the individuals submitted in draft updates  that OMB reviewed before they were published on Performance.gov. For  example, one CAP goal manager told us that during a review of an  update submission PIC staff told him that he should develop additional  milestones to be completed during a specific future fiscal year quarter.

This is in contrast to the detailed guidance that OMB issued on the types  of information that agencies must provide for the updates for agency  priority goals (APG), which are also published quarterly on  Performance.gov. The APG guidance includes explicit instructions for  agencies to identify, as appropriate, the organizations, regulations, tax  expenditures, policies, and other activities within and external to the  agency that contribute to each APG, as well as key milestones with  planned completion dates for the remainder of the goal period. Because  guidance for the types of information that should have been included in  the CAP goal updates was never formally established, CAP goal leaders  were at a heightened risk of failing to take into account important  contributors to the goal and providing incomplete information about  milestones that could help demonstrate progress being made.

section_title: OMB and Goal Leaders Established Processes for Reviewing Cross- Agency Priority Goal Progress, but Not All Review Processes Were Consistent with Requirements and Leading Practices
subsections: section_title: OMB Established a Quarterly Process for Reviewing Progress on CAP Goals, but Did Not Consistently Outline Improvement Strategies Where Goal Achievement Was at Risk
paragraphs: GPRAMA Requirement for OMB Progress Reviews  GPRAMA requires that, not less than quarterly, the Director of OMB, with the support of the PIC,  shall review progress on the CAP goals, including progress during the most recent quarter,  overall trends, and the likelihood of meeting the planned level of performance. GPRAMA also  requires that, as part of these reviews, OMB categorize goals by their risk of not achieving the  planned level of performance and, for those goals most at risk of not meeting the planned level of  performance, identify strategies for performance improvement.

As required by GPRAMA, OMB reviewed progress on CAP goals each  quarter, beginning with the quarter ending June 30, 2012. This review  process consisted of the collection of updated information for each CAP  goal by OMB or PIC staff, and the development of a memorandum for the  Director of OMB with information on the status of the CAP goals. To  develop these memorandums, OMB staff told us that approximately 6  weeks after the end of each quarter, OMB and PIC staff worked with CAP  goal leaders to collect updated data and information on goal metrics and  milestones, and to update the narratives supporting the data. CAP goal  leaders, or staff assisting leaders with the management of efforts related  to the goal, would provide this information to OMB using a template for  the status updates ultimately published on Performance.gov. In addition  to the memorandums developed for the Director of OMB, OMB published  more detailed information through the quarterly status updates available  on Performance.gov.

OMB and PIC staff told us that to support OMB’s quarterly review efforts,  PIC staff were to conduct assessments rating the overall health of  implementation efforts and goal leader engagement. They were also to  assess the execution status of each goal, including the quality and trend  of performance indicators. One purpose of these assessments was to  identify areas where risks, such as goal leader turnover, could affect the  ability to achieve the planned level of performance. Consistent with this  intent, several of the quarterly OMB review memorandums we examined  highlighted turnover in goal leader or deputy goal leader positions as  risks, and suggested the need to find or approve replacements. Although  PIC staff have been tasked with assessing these elements of CAP goal  implementation, and said that there was a shared understanding between  involved staff as to how these assessments would be carried out, the PIC  has not documented its procedures or criteria for conducting these  assessments. Standards for internal control in the federal government  emphasize the importance of documenting procedures, including those  for assessing performance. Without clearly established criteria and  procedures, PIC staff lack a means to: consistently assess  implementation efforts and execution across all goals; bring any  deficiencies, risks, and recommended improvements identified to the  attention of leadership; and ensure consistent application of criteria over  time.

While these quarterly review memorandums identified one goal as being  at risk of not achieving the planned level of performance, and identified  other instances where progress on goals had been slower than planned,  the memorandums did not consistently outline the strategies that were  being used to improve performance or address identified risks. For  example, the Cybersecurity CAP goal was the one goal specifically  described as being at risk of not achieving the planned level of  performance, both in these memorandums and in the status updates on  Performance.gov. Specifically, the memorandum for the third quarter of  fiscal year 2012 identified the risk of not achieving the planned level of  performance, and outlined seven specific risks facing the goal and the  steps being taken to mitigate them. Similarly, the memorandum for the  second quarter of fiscal year 2013 also acknowledged that some  agencies were at risk of not meeting their Cybersecurity CAP goal  targets. However, in contrast to the earlier memorandums, no information  was included about the specific steps that were being taken to mitigate  these risks, although information on planned and ongoing actions to  improve government-wide implementation was included in the milestones  section of the status update for that quarter on Performance.gov.

The memorandum for the fourth quarter of fiscal year 2012 also  acknowledged that the pace of progress on the STEM Education and  Closing Skills Gaps goals had been slower than expected. While the  memorandum stated that additional OMB attention was needed to  support implementation and assure sufficient progress, no information on  the specific strategies being employed to improve performance was  mentioned. According to OMB staff, however, these memorandums were  used to inform subsequent conversations with OMB leadership, which  would build on the information presented in the memorandums.

Furthermore, because the data necessary to track progress for some  goals were unavailable, the Director of OMB would not have been able to  consistently review progress for all CAP goals, or make a determination  about whether some CAP goals were at risk of meeting their planned  levels of performance. This fact was acknowledged in the quarterly review  memorandums for quarters one and two of fiscal year 2013, which  acknowledged that progress on three goals (Entrepreneurship and Small  Business, Job Training, and STEM Education) was difficult to track, and  that additional work was needed on data collection. However, no  information on the specific steps that were being taken to address these  shortcomings was included. A lack of specific information about the steps  being taken to mitigate identified risk areas and improve performance  could hinder the ability of OMB leadership—and others—to adequately  track the status of efforts to address identified deficiencies or risks and to  hold officials accountable for taking necessary actions.

section_title: CAP Goal Leaders Established Processes to Review Progress, but Their Consistency with Leading Practices and Their Effects on Performance and Collaboration Varied
paragraphs: GPRAMA Requirement for Goal Leader and Agency Involvement in Progress Reviews  As part of the quarterly review process, GPRAMA requires that the Director of OMB review each  priority goal with the appropriate lead government official, and include in these reviews officials  from the agencies, organizations, and program activities that contribute to the achievement of the  goal.

According to OMB staff, to encourage goal leaders and contributing  agencies to take ownership of efforts to achieve the goals, OMB gave  goal leaders flexibility to use different approaches to engage agency  officials and review progress at the CAP-goal level. While guidance  released by OMB in August 2012 encouraged goal leaders to leverage  existing interagency working groups, committees, and councils in the  management of the goals as much as practicable, it did not include  information on the purpose of reviews, expectations for how reviews  should be conducted to maximize their effectiveness as a tool for  performance management and accountability, or the roles that CAP goal  leaders and agency officials should play in the review process. Again,  standards for internal control in the federal government emphasize the  importance of documenting procedures for reviewing performance against  established goals and objectives.

This is in contrast to the detailed guidance that OMB released for agency  priority goal and agency strategic objective reviews, which outlined the  specific purposes of the reviews, how frequently they should be  conducted, the roles and responsibilities of agency leaders involved in the  review process, and how the reviews should be conducted. We also  found that this guidance for reviews at the agency level was broadly  consistent with the leading practices for performance reviews that we  previously identified. While no official guidance was published to guide  how reviews involving goal leaders and staff from contributing agencies  could be conducted for the CAP goals, OMB staff said the principles of  the guidance released for agency reviews, which reflected many of the  leading practices, was referenced in conversations with CAP goal leaders  and teams.

OMB has emphasized that flexibility is needed to ensure that goal leaders  can use review processes that are appropriate given the scope of  interagency efforts, the number of people involved, and the maturity of  existing reporting and review processes. The guidance for agency  reviews gave agencies flexibility to design their performance review  processes in a way that would fit the agency’s mission, leadership  preferences, organizational structure, culture, and existing decision- making processes. In our previous work, we detailed how several federal  agencies had implemented quarterly performance reviews in a manner  consistent with leading practices, but which were also tailored to the  structures, processes, and needs of each agency. In this way, flexible  implementation of review processes is possible within a framework that  encourages the application of leading practices.

A lack of clear expectations for how progress should be reviewed at the  CAP-goal level resulted in a number of different approaches being used  by goal leaders to engage officials from contributing agencies to review  progress on identified goals and milestones, ranging from regular in- person review meetings led by the CAP goal leader to the review of  written updates provided to the goal leader by officials from contributing  agencies. See appendix IV for more detailed information on the various  processes used by goal leaders to collect data on, and review progress  towards, identified goals.
subsections: section_title: Some Goal Leaders Used Review Processes Broadly Consistent with Leading Practices, and Noted Their Positive Effects on Performance, Accountability, and Collaboration
paragraphs: Instituting review processes consistent with the leading practices we  previously identified can help ensure that reviews include meaningful  performance discussions, provide opportunities for oversight and  accountability, and drive performance improvement. Taken together,  these leading practices emphasize the importance of leadership  involvement in the review process, data collection and review meeting  preparation, participation by key officials, and rigorous follow-up.

Through our evaluation of how goal leaders and contributing agency  officials reviewed progress towards the interim goals, we identified two  CAP goals—Cybersecurity and Closing Skills Gaps—and one sub-goal— the Entrepreneurship and Small Business sub-goal on improving access  to government services and information (BusinessUSA sub-goal)—where  goal managers instituted in-person review processes with officials from  contributing agencies that were broadly consistent with the full range of  leading practices for reviews, which we have summarized in four  categories below. The processes used by other CAP goal leaders to  engage agency officials in the review of progress did not reflect the full  range of leading practices.

Leadership Involvement. Leading practices indicate that leaders should  use frequent and regular progress reviews as a leadership strategy to  drive performance improvement and as an opportunity to hold people  accountable for diagnosing performance problems and identifying  strategies for improvement. The direct and visible engagement of  leadership is vital to the success of such reviews. Leadership involvement  helps ensure that participants take the review process seriously and that  decisions and commitments can be made. The goal leaders managing  the Cybersecurity and Closing Skills Gaps goals, as well as the  BusinessUSA sub-goal, were directly involved in leading in-person  reviews for these goals, and in using them as opportunities to review  progress, identify and address performance problems, and hold agency  officials accountable for progress on identified goals and milestones, as  detailed in table 1.

Data Collection and Review Meeting Preparation. Leading practices  also indicate that those managing review processes should have the  capacity to collect, analyze, and communicate accurate, useful, and  timely performance data, and should rigorously prepare for reviews to  enable meaningful performance discussions. The collection of current,  reliable data on the status of activities and progress towards goals and  milestones is critical so that those involved can determine whether  performance is improving, identify performance problems, ensure  accountability for fulfilling commitments, and learn from efforts to improve  performance. The ability to assess data to identify key trends and areas  of strong or weak performance, and to communicate this to managers  and staff effectively through materials prepared for reviews, is also  critical. As detailed in table 2, those supporting the Cybersecurity and  Skills Gap goals, and the BusinessUSA sub-goal, instituted processes to  regularly collect and analyze data on progress towards identified goals  and milestones, and to ensure these data would be communicated  through materials prepared for review meetings.

Participation by Key Officials. Leading practices indicate that key  players involved in efforts to achieve a goal should attend reviews to  facilitate problem solving. This is critical as their participation enables  those involved to break down information silos, and to use the forum  provided by the review to communicate with each other, identify  improvement strategies, and agree on specific next steps. Reviews for  both the Cybersecurity and Closing Skills Gaps CAP goals, and the  BusinessUSA sub-goal, were structured so that relevant agency officials  playing a key role in efforts to carry out the goal were included, as  detailed in table 3.

Review Follow-Up. Leading practices indicate that participants should  engage in sustained follow-up on issues identified during reviews, which  is critical to ensure the success of the reviews as a performance  improvement tool. Important follow-up activities include identifying and  documenting specific follow-up actions stemming from reviews, those  responsible for each action item, as well as who will be responsible for  monitoring and follow-up. Follow-up actions should also be included as  agenda items for subsequent reviews to hold responsible officials  accountable for addressing issues raised and communicating what was  done. Goal managers for the Cybersecurity and Closing Skills Gap CAP  goals, as well as the BusinessUSA sub-goal, took steps to follow up on  action items identified in these meetings, and to ensure that steps were  taken towards their completion, as detailed in table 4.

Review Effects. Goal leaders and managers we interviewed said that  these review processes were valuable in driving improved performance,  establishing a greater sense of accountability for progress on the part of  contributors, and in providing a forum for interagency communication and  collaboration. For example, according to DHS staff involved in the  management of the Cybersecurity CAP goal, implementation of Personal  Identity Verification (PIV) requirements across the federal government  had been stagnant for several years prior to the introduction of  cybersecurity as a CAP goal. The review process was used to hold  agencies accountable for improved PIV implementation, which helped  bring an increased focus on the issue and drive recent progress. Since  the reviews were instituted in 2012, DHS has reported improved PIV  adoption in civilian agencies, which has increased from 1.24 percent in  fiscal year 2010, to 7.45 percent in fiscal year 2012, to 19.67 percent in  the fourth quarter of fiscal year 2013. According to data from DHS, while  still falling short of the target, this has contributed to the overall increase  in PIV adoption across the federal government—including both civilian  agencies and the Department of Defense—from 57.26 percent in fiscal   DHS  year 2012 to 66.61 percent in fourth quarter of fiscal year 2013.staff also added that agencies generally had not previously collaborated  on cybersecurity issues or worked to identify best practices. According to  DHS staff, the reviews have created an important point of collaboration  between DHS, OMB, National Security Staff, and agencies, and provided  an opportunity to inform agencies of best practices and connect them with  other agencies that are meeting their targets to learn from them.

Similarly, OPM officials and sub-goal leaders involved in the management  of the Closing Skills Gap CAP goal said that the quarterly review  meetings were a critical means to ensure sub-goal leaders and staff were  demonstrating progress. Having sub-goal leaders report out on progress,  and hear about the progress made in other sub-goal areas, provided  additional pressure for continuous improvement and the need to remain  focused on driving progress towards their goals. Having the goal leader  lead the review was also a way to demonstrate leadership commitment to  the achievement of each sub-goal. According to OPM officials and sub- goal leaders, the review meetings also served as an important forum for  discussing innovative approaches being taken to address skills gaps in  different areas, opportunities for collaboration to address challenges  shared by different sub-goals, and how leaders could leverage the efforts  of other sub-goals to drive progress on their own.

The BusinessUSA sub-goal leader said that having it as the basis for a  CAP sub-goal elevated the cross cutting nature of the initiative. In  addition to reviewing performance information and the status of  deliverables, discussions at inter-agency Steering Committee meetings  were used to discuss how contributors could work together to meet the  initiative’s performance goals. This communication and coordination led  to connections between agencies and to discussions about how programs  could be working in a more integrated way. For example, these  discussions were used to identify ways that programs could more  effectively integrate program information on the BusinessUSA website to  increase customer satisfaction.

section_title: Other Goal Leaders Did Not Use Review Processes Consistent with the Full Range of Leading Practices for Reviews
paragraphs: We found that the processes used by other CAP goal leaders to engage  agency officials in the review of progress, which are summarized in  appendix IV, did not reflect the full range of leading practices. For  example, the process for reviewing progress on the Job Training CAP  goal involved staff from the PIC collecting updates on recent milestones  from agencies, which were then compiled in the quarterly status update  and reviewed by the goal leader. This approach was used by the goal  leaders for the Broadband and STEM Education CAP goals to review  progress as well. While goal leaders and managers for these goals  indicated that they used the collection and review of information as an  opportunity to communicate with officials from contributing agencies, this  approach contrasts with OMB guidance for reviews of agency priority  goals, which states explicitly that performance reviews should not be  conducted solely through the sharing of written communications. As OMB  noted in its guidance, in-person engagement of leaders in performance  reviews greatly accelerates learning and performance improvement, and  personal engagement can demonstrate commitment to improvement,  ensure coordination across agency silos, and enable rapid decision  making.

While not employing the full range, goal leaders for a number of goals did  use processes that reflected one or more leading practices. For example,  many CAP goal leaders led or participated in interagency meetings with  representatives of contributing agencies. While these were used to  facilitate interagency communication and collaboration on the  development of plans and policies, it was unclear whether many of these  meetings were consistently used to review progress on identified CAP  goals and milestones. The goal leader for the Strategic Sourcing CAP  goal used processes that reflected leadership involvement, participation  by key officials, and the collection and analysis of relevant data.  Specifically, according to goal managers, the goal leader led regular  meetings of the Strategic Sourcing Leadership Council (SSLC), which  were attended by senior procurement officials from eight agencies that  combine to make up almost all of the federal government’s total  procurement spending.

To prepare for each SSLC meeting, staff from OMB’s Office of Federal  Procurement Policy (OFPP) held a meeting for supporting staff from each  agency, who would then prepare the SSLC member from their agency for  the issues to be discussed in the SSLC meeting. OFPP also established  a regular data collection process where each agency would report on its  adoption and spending rates for two strategic sourcing options, which  would then be used for the purposes of reporting on the CAP goal.  However, it was unclear how regularly, if at all, SSLC meetings were used  to engage agency officials in the review of data on agency adoption of,  and spending on, strategic sourcing options, or how regularly meetings  were used to review progress that was being made towards the CAP  goal. It was also unclear what mechanisms, if any, were used to ensure  rigorous follow-up on issues raised in these meetings, a key leading  practice, as there were no official meeting minutes maintained. The lack  of an official record could hinder follow-up and accountability for any  identified actions that need to be taken.

section_title: Some Goal Leaders Reported Minimal Effects on Performance and Collaboration
paragraphs: Representatives of some goals stated that it was difficult to isolate the  impact of the CAP goal designation, and its associated reporting and  review requirements, on performance and collaboration. According to  some goal managers, because their interim goals were based on  initiatives that had been previously established in executive orders or  Presidential memorandums, much of the interagency activity supporting  their efforts would have happened without the CAP goal designation and  its reporting and review requirements. For example, a manager for the  Data Center Consolidation CAP goal told us that the previously  established Federal Data Center Consolidation Initiative was used to  drive progress and that the CAP goal designation and quarterly reporting  and review requirements had little impact. Similarly, Job Training CAP  goal managers said that interagency collaboration on job training issues  had been established prior to the creation of the CAP goal, that the goal’s  reporting and review requirements were incidental to the contributors’  ongoing work, and that it did not add an additional level of accountability  for the completion of job training initiatives. However, this is a goal where  no data were reported to demonstrate its impact on federal job training  programs, and which was identified in multiple OMB reviews as having  slower than anticipated progress due, in part, to extended periods of time  in which there was no deputy CAP goal leader to provide support  necessary to improve coordination and collaboration.

While many CAP goal leaders and staff we interviewed noted the  progress they had made with their existing interagency meetings and  approaches, a lack of clear expectations or guidance for how review  processes at the CAP goal level should be carried out can lead to a  situation where reviews are implemented in a manner that is not informed  by, or fully consistent with, leading practices. This could result in missed  opportunities to realize the positive effects on performance and  accountability that can stem from the implementation of review processes  that regularly and consistently involve leaders and agency officials in the  analysis of performance data to identify and address performance  deficiencies, and use rigorous follow-up to ensure accountability for  commitments.

section_title: Conclusions
paragraphs: Many of the meaningful results that the federal government seeks to  achieve require the coordinated efforts of more than one federal agency.  GPRAMA’s requirement that OMB establish CAP goals offers a unique  opportunity to coordinate cross-agency efforts to drive progress in priority  areas. That opportunity will not be realized, however, if the CAP goal  reporting and review requirements and leading review practices are not  followed. The reporting and review requirements for the CAP goals, and  leading practices for the reviews, are designed to ensure that relevant  performance information is used to improve performance and results, and  that OMB and goal leaders actively lead efforts to engage all relevant  participants in collaborative performance improvement initiatives and hold  them accountable for progress on identified goals and milestones.

OMB reported performance information in the quarterly CAP goal status  updates it published on Performance.gov. While updates for most goals  reported data on performance towards the identified planned level of  performance, the information in the updates did not always present a  complete picture of progress towards identified goals and milestones. For  example, while updates for 8 of the 14 goals included data that indicated  performance towards the identified overall planned level of performance,  only 3 also contained annual or quarterly targets that allowed for an  assessment of interim progress. Updates for the other 6 of the 14 goals  did not report on performance towards the goal’s primary performance  target because the goal was established without a quantitative target or  because goal managers were unable to collect the data needed to track  performance. In other cases, planned activities that were identified as  contributing to the goal were sometimes missing important elements,  including alignment with the strategies for goal achievement they  supported, a time frame for completion, or information on their  implementation status. The incomplete picture of progress that many of  the updates gave limited the ability of goal leaders and others to ensure  accountability for the achievement of targets and milestones.

Holding regular progress reviews that are consistent with GPRAMA  requirements and the full range of leading practices can produce positive  effects on performance and collaboration. Engaging contributors in  regular reviews of data on performance can help ensure interagency  efforts are informed by information on progress towards identified goals  and milestones, which can be used to identify and address areas where  goal or milestone achievement is at risk. Reviews can also be used to  reinforce agency and collective accountability for the achievement of  individual and shared outcomes, helping to ensure that efforts to improve  performance or address identified risks are implemented. Lastly, reviews  can be used to foster greater collaboration, ensuring opportunities for  communication and coordination between officials involved in efforts to  achieve shared outcomes. While OMB and CAP goal leaders instituted  processes for reviewing progress on the interim CAP goals, if GPRAMA  requirements and leading practices for reviews are not consistently  followed, it may result in missed opportunities to improve performance,  hold officials accountable for achieving identified goals and milestones,  and ensure agency officials are coordinating their activities in a way that  is directed towards the achievement of shared goals and milestones.

section_title: Recommendations for Executive Action
paragraphs: We recommend that the Director of OMB take the following three actions:  Include the following in the quarterly reviews of CAP goal progress, as  required by GPRAMA: a consistent set of information on progress  made during the most recent quarter, overall trends, and the likelihood  of meeting the planned level of performance; goals at risk of not  achieving the planned level of performance; and the strategies being  employed to improve performance.

Work with the PIC to establish and document procedures and criteria  to assess CAP goal implementation efforts and the status of goal  execution, to ensure that the PIC can conduct these assessments  consistently across all goals and over time.

Develop guidance similar to what exists for agency priority goal and  strategic objective reviews, outlining the purposes of CAP goal  progress reviews, expectations for how the reviews should be carried  out, and the roles and responsibilities of CAP goal leaders, agency  officials, and OMB and PIC staff in the review process.

To ensure that OMB and CAP goal leaders include all key contributors  and can track and report fully on progress being made towards CAP  goals overall and each quarter, we recommend that the Director of OMB  direct CAP goal leaders to take the following four actions:  Identify all key contributors to the achievement of their goals;  Identify annual planned levels of performance and quarterly targets for  each CAP goal;

Develop plans to identify, collect, and report data necessary to  demonstrate progress being made towards each CAP goal or develop  an alternative approach for tracking and reporting on progress  quarterly; and

Report the time frames for the completion of milestones; the status of  milestones; and how milestones are aligned with strategies or  initiatives that support the achievement of the goal.

section_title: Agency Comments
paragraphs: We provided a draft of this report for review and comment to the Director  of OMB, the Secretaries of Commerce and Homeland Security, the  Director of the Office of Personnel Management, the Administrator of the  Small Business Administration, as well as the officials we interviewed to  collect information on the interim CAP goals from the Council on  Environmental Quality, Department of Education, Department of Labor,  Department of Veterans Affairs, National Science Foundation, and the  Office of Science and Technology Policy. OMB and PIC staff provided  oral comments on the draft, and we made technical changes as  appropriate. OMB staff generally agreed to consider our  recommendations. For example, while they said that OMB and PIC staff  will continue to work directly with CAP goal leaders to convey suggested  practices for reviewing performance, they will consider referencing  principles and practices for data-driven performance reviews in future  Circular A-11 guidance related to the management of CAP goals.  Furthermore, while they noted that quantitative performance data for  some key measures may not available on a quarterly basis, they said that  they will continue to work to develop more robust quarterly targets.  Officials or staff from the Departments of Commerce and Veterans  Affairs, and the Office of Science and Technology Policy provided  technical comments, which we incorporated as appropriate.

We are sending copies of this report to the Director of OMB as well as  appropriate congressional committees and other interested parties. The  report is also available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions concerning this report, please  contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix VI.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report is part of our response to a mandate that we evaluate the  implementation of the federal government priority goals under the GPRA  Modernization Act of 2010 (GPRAMA). Due to the timing of our work, we  focused on the implementation of the reporting and review requirements  for the 14 interim cross-agency priority (CAP) goals established in  February 2012.about progress made towards the interim CAP goals; and (2) how, if at  all, quarterly progress reflected GPRAMA requirements and leading  practices for data-driven reviews, as well as how they contributed to  improved cross-agency performance and collaboration.

Specifically, this report assesses (1) what is known  To address these objectives, we interviewed representatives of 13 of the  14 interim goals. For 8 of the 13 goals we spoke directly with the goal  leader or deputy goal leader, along with, in some cases, staff from Office  of Management and Budget (OMB) and agencies involved in supporting  efforts related to the goals. For the other five goals (Closing Skills Gaps,  Cybersecurity, Data Center Consolidation, Exports, and Job Training) we  met with agency officials or OMB staff playing a key role in the  management of interagency efforts related to the CAP goal. During these  interviews, we asked officials questions concerning how the goal leader  and officials from contributing agencies reviewed progress on the goal;  the interagency groups used to engage agency officials and manage  efforts related to the goal; the role that staff from OMB and the  Performance Improvement Council (PIC) played in the review process;  and any impact the CAP goal designation and review processes had on  performance, collaboration, and accountability. We also participated in  interviews with the goal leaders of 11 agency priority goals that were  aligned with, or identified as a contributor to, a CAP goal.

To further address the first objective, and assess what is known about  progress made toward the interim CAP goals, we analyzed information on  identified performance metrics and milestones included in the quarterly  status updates for each CAP goal published on Performance.gov. We  also analyzed relevant information collected through our interviews with  CAP goal leaders, deputies, and supporting staff. We compared the data  and information made available through the quarterly status updates with  requirements in GPRAMA that Performance.gov include information for  each goal on results achieved during the most recent quarter and overall  trend data. To assess the reliability of performance data and information  available through Performance.gov we collected information from OMB  and PIC staff, and CAP goal representatives, about data quality control  procedures. We determined that the data and information were  sufficiently reliable for our analysis of what was reported on  Performance.gov about progress towards identified goals and milestones.

To address the second objective, we reviewed quarterly review  memorandums developed for OMB leadership for five quarters, from the  third quarter of fiscal year 2012 to the third quarter of fiscal year 2013.We compared the contents of these review memorandums with  requirements for the OMB quarterly reviews established in GPRAMA. We  also interviewed staff from OMB and the PIC to discuss the various  approaches being used to review progress at the CAP-goal level, the data  collection and review process, and the role of the PIC in supporting the  quarterly review process.

To further address the second objective we reviewed (where available)  documents created for interagency meetings, such as meeting agendas,  presentation materials, meeting notes, and attendee lists. We also  observed one quarterly review meeting held for the Closing Skills Gap  goals, and conducted interviews with sub-goal leaders from the Closing  Skills Gaps and Entrepreneurship and Small Business CAP goals. These  interviews were used to learn more about the involvement of officials from  contributing agencies in the quarterly review process for each CAP goal,  the processes that had been established to review progress at the sub- goal level, and to gain a more complete picture of participating agency  officials’ perceptions of the impact of the CAP goals and review  processes.

We selected these sub-goals through a two-part process. Of the eight  CAP goals for which we had completed interviews through the end of  2013, the team selected one goal for which the goal leader held quarterly  meetings dedicated to reviewing progress toward the CAP goal with the  goal’s contributors (Closing Skills Gaps). The team also selected a  second goal for which the goal leader used a review process that did not  rely on quarterly meetings between the goal leader and contributing  agencies (Entrepreneurship and Small Business). To ensure that the  team would have at least one goal representing each type of goal, the  team also ensured that one goal would be an outcome-oriented policy  goal and one goal would be a management goal. For both the Closing  Skills Gaps and Entrepreneurship and Small Business CAP goals the  team then selected four sub-goals for interviews. For the Closing Skills  Gaps CAP goal the team interviewed the sub-goal leaders for the  Economist; Information Technology/Cybersecurity; Science, Technology,  Engineering, and Mathematics (STEM) Education; and Human  Resources sub-goals. For the Entrepreneurship and Small Business CAP  goal the team held interviews with the sub-goal leaders for the sub-goals  to “Accelerate commercialization of Federal research grants,” “Advance  federal small business procurement goals,” “Improve access to  government services and information,” and “Streamline immigration  pathways for immigrant entrepreneurs.” These were selected to ensure  that the team would capture sub-goals in which a range of approaches for  measuring and reviewing progress were being used. Specifically, sub- goals were selected to ensure the team would have some that did, and  did not, hold regular meetings, and some that did, and did not, track  quantitative measures of performance or milestones with time frames.  Our selection of these sub-goals was nonstatistical and therefore our  findings from these interviews are not generalizable to the other CAP  goals.

We compared what we learned about review processes at the CAP goal  and sub-goal levels, through interviews and the collection of  documentation, used by leaders from each goal against leading practices  for performance reviews previously identified by GAO.

Because the scope of our review was to examine the implementation of  quarterly progress reviews, we did not evaluate whether these goals were  appropriate indicators of performance, sufficiently ambitious, or met other  dimensions of quality.

We conducted our work from May 2013 to June 2014 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.

section_title: Appendix II: Interim Cross-Agency Priority Goals and Goal Statements
subsections: section_title: Goal Broadband
paragraphs: Goal Statement  As part of expanding all broadband capabilities, ensure 4G wireless broadband coverage  for 98 percent of Americans by 2016.

section_title: Closing Skills Gaps
paragraphs: Close critical skills gaps in the federal workforce to improve mission performance. By  September 30, 2013, close the skills gaps by 50 percent for three to five critical federal  government occupations or competencies, and close additional agency-specific high risk  occupation and competency gaps.

section_title: Cybersecurity
paragraphs: Executive branch departments and agencies will achieve 95 percent implementation of the  administration’s priority cybersecurity capabilities by the end of FY 2014. These  capabilities include strong authentication, trusted Internet connections, and continuous  monitoring.

section_title: Data Center Consolidation
paragraphs: Improve information technology service delivery, reduce waste, and save $3 billion in  taxpayer dollars by closing at least 2,500 data centers by fiscal year 2015.

section_title: Energy Efficiency
paragraphs: Increase energy productivity (amount of real gross domestic product in dollars/energy  demand) 50 percent by 2030.

section_title: Entrepreneurship and Small Business
paragraphs: Increase federal services to entrepreneurs and small businesses with an emphasis on      1) startups and growing firms and 2) underserved markets.

section_title: Exports Improper Payments
paragraphs: Double U.S. exports by the end of 2014.

The federal government will achieve a payment accuracy rate of 97 percent by the end of  2016.

section_title: Job Training
paragraphs: Ensure our country has one of the most skilled workforces in the world by preparing 2  million workers with skills training by 2015 and improving the coordination and delivery of  job training services.

section_title: Real Property
paragraphs: The federal government will maintain the fiscal year 2012 square footage baseline of its  office and warehouse inventory.

section_title: Science, Technology, Engineering, and Mathematics (STEM) education
paragraphs: In support of the president’s goal that the U.S. have the highest proportion of college  graduates in the world by 2020, the federal government will work with education partners  to improve the quality of STEM education at all levels to help increase the number of well- prepared graduates with STEM degrees by one-third over the next 10 years, resulting in  an additional 1 million graduates with degrees in STEM subjects.

section_title: Strategic sourcing
paragraphs: Reduce the costs of acquiring common products and services by agencies’ strategic  sourcing of at least two new commodities or services in both 2013 and 2014, that yield at  least a 10 percent savings. In addition, agencies must increase their use of Federal  Strategic Sourcing Initiative vehicles by at least 10 percent in both fiscal years 2013 and  2014.

section_title: Sustainability
paragraphs: By 2020, the federal government will reduce its direct greenhouse gas emissions by 28  percent and will reduce its indirect greenhouse gas emissions by 13 percent by 2020  (from 2008 baseline).

section_title: Veteran Career Readiness
paragraphs: By September 30, 2013, increase the percent of eligible service members who will be  served by career readiness and preparedness programs from 50 percent to 90 percent in  order to improve their competitiveness in the job market.

section_title: Appendix III: Interagency Group Membership and Meeting Frequency and Purpose
paragraphs: Goal leaders for 13 of 14 cross-agency priority (CAP) goals leveraged  interagency groups for the purposes of coordinating efforts designed to  contribute to progress on the cross-agency priority goal. This appendix  includes information on the membership of these interagency groups, the  frequency with which they met, and the purposes of those meetings.

Membership  Fourteen agencies with  federal property  management or  transportation funding  responsibilities, and  broadband or other related  expertise.

To discuss best practices on  broadband-related land  management issues, and  actions to implement an  executive order on  accelerating broadband  infrastructure deployment.

Senior officials from  agencies considered  major spectrum  stakeholders and users of  spectrum, including the  Departments of Defense,  Justice, Homeland  Security (DHS),  Commerce, and the  National Aeronautics and  Space Administration  (NASA).

To provide advice on  spectrum policy and  strategic plans, discuss  commercial transfer of  federal agency spectrum,  and resolve issues affecting  federal/non-federal users.
subsections: section_title: Closing Skills Gaps
paragraphs: To review progress on  performance metrics and  actions taken to close skills  gaps in each of the six sub- goal areas.

section_title: Cybersecurity
paragraphs: Officials from National  Institute of Standards and  Technology, General  Services Administration  (GSA), DHS, National  Security Staff, Office of  Management and Budget  (OMB) and Performance  Improvement Council.

Twice each quarter   Beginning in 2013, a  meeting was held each  quarter prior to the collection  of data on agency progress  on cybersecurity metrics.  Another was held after data  had been collected and  analyzed to review and  discuss agency progress.

section_title: Data Center Consolidation
paragraphs: Data center consolidation  program managers from  24 federal agencies.

To identify and disseminate  key information about  solutions and processes to  help agencies make  progress towards data  center consolidation goals.

section_title: Goal or sub-goal Energy Efficiency Entrepreneurship and Small Business Improve Access to Government Information and Services Sub-Goal
paragraphs: Interagency group  No interagency groups were used to manage efforts related to this goal.

Interagency groups were used to manage efforts at the sub-goal level.

Senior-level  representatives from 24  participating agencies.

To oversee strategy,  resources and timetables for  the development of the  BusinessUSA website,  resolve interagency issues  and ensure  department/agency  viewpoints are represented.

Mid-to-senior level  program, technology and  customer service  managers from 24  participating agencies.

To assist the BusinessUSA  program management office  coordinate the design,  development, and operation  of the BusinessUSA website,  and to track and monitor  performance metrics on  customer service and  outcomes.

section_title: Commercialization of Federal Research Grants Sub-Goal
paragraphs: SBIR/STTR program  managers from11  agencies, and  coordinating officials from  Small Business  Administration (SBA) and  Office of Science and  Technology Policy (OSTP)

To discuss the development  of SBIR/STTR program  policy directives, the  implementation of  requirements, outreach and  access to the programs, and  program best practices.

section_title: Streamlining Pathways for Immigrant Entrepreneurs Sub- Goal
paragraphs: To provide updates on  relevant agency activities  and identify opportunities for  interagency collaboration.

section_title: Small Business Procurement Sub-Goal
paragraphs: To share best practices for  expanding contracting to  small and disadvantaged  businesses, and reviewing  progress on agency  simplified-acquisition  threshold goals.

To provide officials from the  White House, SBA,  Commerce, and OMB with  an opportunity to meet with  senior agency leaders and  discuss the steps agencies  are taking to increase small  business contracting.

section_title: Goal or sub-goal Exports
paragraphs: Membership  Principals (cabinet  secretaries and deputies)  and staff from 20 agencies  involved in export policy,  service, finance, and  oversight.

To review progress on  deliverables supporting the  National Export Strategy,  communications, and the  status of individual export  promotion initiatives.

section_title: Improper Payments
paragraphs: Bi-weekly to monthly  To review the status of  agency implementation of  Do Not Pay requirements  and milestones, and  guidance for implementation.

Officials from agencies  with “high-priority”  programs, as designated  by OMB.

To discuss the government- wide improper payment  initiative and overall  strategy.

section_title: Job Training
paragraphs: To discuss expanding  access to job training  performance data, and  opportunities to promote its  use at the local, state, and  federal levels.

section_title: Real Property
paragraphs: Among other policy  discussions, to discuss the  development of agency  “Freeze the Footprint” plans.

section_title: Sustainability
paragraphs: To discuss policy to guide  the federal government on  sustainability issues, and to  discuss sustainability goals.

section_title: Science, Technology, Engineering, and Mathematics (STEM) Education
paragraphs: Every 4-6 weeks,  during the  development of the  5-year strategic plan.

To develop a 5-year  strategic plan for federal  support for STEM education.

section_title: Strategic Sourcing
paragraphs: Representatives from  Departments of Defense,  Energy, and Veterans  Affairs (VA), DHS, HHS,  GSA, NASA, and SBA.

To discuss the development  and adoption of strategic  sourcing options.

section_title: Goal or sub-goal Veteran Career Readiness
paragraphs: To review ongoing policy  initiatives and opportunities  for collaboration between  agencies.

To develop and implement a  redesigned veterans  transition program.

section_title: Appendix IV: Description of Interim Cross- Agency Priority Goal Review Processes
subsections: section_title: Goal Broadband

section_title: Energy Efficiency
paragraphs: meetings with officials from each agency. According to OMB staff, during these  reviews participants reviewed metrics from across the agency’s information  technology portfolio, which included, in some cases, those related to data center  consolidation.  Each quarter staff supporting the goal leader would collect updated information on  contributing agency priority goals for the purposes of updating the quarterly status  update.  Each quarter the deputy goal leader would collect updated information on goals and  milestones from the leaders of each of 10 sub-goals for the purposes of developing  the quarterly status update.  The deputy goal leader would follow-up with sub-goal leaders or agency officials, as  necessary, to address issues or questions about the status of efforts. The goal leader  would then review and approve the quarterly status update.  Some sub-goal leaders would hold in-person meetings with officials from contributing  agencies to, among other things, review progress on identified goals and milestones.  See appendix III for information on interagency groups that were used to manage  efforts for four of the sub-goals.  Each quarter the goal leader, with the assistance of staff from Commerce and the  PIC, would collect updated information on relevant agency metrics and activities for  the purposes of updating the quarterly status update.  Periodic meetings of the Export Promotion Cabinet/Trade Promotion Coordinating  Committee, and its Small Business and other working groups, were also used to  discuss the status of export promotion efforts and progress on specific deliverables.  Each year OMB would collect and report data on agency improper payment rates.  Staff from the OMB Office of Federal Financial Management led monthly meetings  with agency representatives to discuss the implementation of the Do Not Pay  initiative, which was designed to contribute to the reduction of improper payments.  The Department of Treasury, as the agency leading implementation of the Do Not  Pay initiative, would track agency progress on implementation milestones.

section_title: Science, Technology, Engineering, and Mathematics (STEM) Education
paragraphs: Each quarter staff from the PIC would collect updated information on progress  towards agency milestones, and work with the goal leader on the development of the  quarterly status update.   After this goal was revised in the second quarter of 2013, a new review process to  track agency adherence to the goal was under development by OMB.  Twice a year the Council on Environmental Quality (CEQ) would collect and review  quantitative and qualitative data on agency progress towards established  sustainability goals, including the reduction of agency greenhouse gas emissions.  Following the collection of these data, the goal leader hosted meetings of the  Steering Committee on Federal Sustainability, which were used to discuss federal  sustainability policy and progress on sustainability goals.  According to CEQ staff, the goal leader and CEQ staff would meet with  representatives from agencies about sustainability issues on an ad hoc basis. In  instances where there was a gap between an agency’s actual performance and the  target established in that area, the goal leader, or other staff from CEQ, would meet  with officials from that agency to discuss ways to address the performance gap.   Each quarter the goal leader would collect updated information on agency milestones  for inclusion in the quarterly status updates.  Progress on some identified strategies to achieve the goal, such as the National  Science Foundation’s efforts to improve undergraduate STEM education, were  reviewed at the agency level. After progress was reviewed at the agency level the  information was passed onto the goal leader and reported publicly in the quarterly  status update.  Each quarter the General Services Administration would collect data on agency  adoption and spending rates for the Federal Strategic Sourcing Initiative (FSSI)  solutions for domestic delivery and office supplies.  The Strategic Sourcing Leadership Council met bi-monthly to guide the creation and  adoption of new FSSI options, and, as part of that effort, might review quarterly data  on agency adoption and spending rates.  According to the goal leader, each month staff from the Departments of Defense and  Veterans Affairs, and the PIC, would provide data for “one-pagers” and other status  update documents with key pieces of relevant information, such as the veterans’  unemployment rate and the number of active employers on the Veteran’s Job Bank.  These one-pagers would be used to inform regular Interagency Policy Council (IPC)  discussions, along with more specific briefing memorandums, which were used to  cover the latest issues, keep stakeholders focused on overall outcomes, and to  inform discussion around specific outliers. Some of the data in these one-pagers  would also be incorporated into the quarterly status updates.

More frequently, issue papers and data analysis were provided to Veterans  Employment Initiative (VEI) Task Force and IPC members as needed to address  topical issues.

Ongoing milestone reviews held by the VEI Task Force and its associated working  groups on Education, Employment, Transition, and Entrepreneurship, provided an  opportunity to discuss strategies being employed to improve performance.

section_title: Appendix V: Full Text for Interactive Figure 2 on Frequency of Data Reporting for Cross- Agency Priority Goals’ Overall Planned Levels of Performance
paragraphs: This appendix includes the print version of the text and rollover graphics  contained in interactive figure 2.

Overall Planned Level of Performance  …achieve 95 percent implementation of the  Administration’s priority cybersecurity capabilities  by the end of fiscal year 2014.

Data reported for primary performance goal …save $3 billion in taxpayer dollars by closing at  least 2500 data centers by fiscal year 2015.
subsections: section_title: Data reported for primary performance goal
paragraphs: “Agencies have already closed 640 data centers…”

section_title: Cross-Agency Priority Goal Exports Data reported for primary performance goal
paragraphs: Overall Planned Level of Performance  Double U.S. exports by the end of 2014.

Frequency of Data Reporting for  Overall Goal  Quarterly  …agencies’ strategic sourcing of at least two new  commodities or services in both 2013 and 2014,  that yield at least a 10 percent savings… In  addition, agencies must increase their use of  Federal Strategic Sourcing Initiative vehicles by at  least 10 percent in both fiscal years 2013 and  2014.

section_title: Broadband
paragraphs: …ensure 4G wireless broadband coverage for 98  percent of Americans by 2016.

section_title: Cross-Agency Priority Goal Improper Payments
paragraphs: Overall Planned Level of Performance  …achieve a payment accuracy rate of 97 percent  by the end of 2016.

section_title: Data reported for primary performance goal
paragraphs: “Data Not Reported”

section_title: Entrepreneurship and Small Businesses
paragraphs: Increase federal services to entrepreneurs and  small businesses with an emphasis on 1) startups  and growing firms and 2) underserved markets.

section_title: Data reported for primary performance goal
paragraphs: “Data Not Reported”

section_title: Job Training

section_title: Data reported for primary performance goal
paragraphs: “Data Not Reported”

section_title: Real Property
paragraphs: The Federal Government will maintain the fiscal  year 2012 square footage baseline of its office and  warehouse inventory.

section_title: Data reported for primary performance goal
paragraphs: “Data Not Reported”

section_title: Science, Technology, Engineering, and Math (STEM) Education
paragraphs: …increase the number of well-prepared graduates  with STEM degrees by one-third over the next 10  years, resulting in an additional 1 million graduates  with degrees in STEM subjects.

section_title: Cross-Agency Priority Goal Data reported for primary performance goal
paragraphs: Frequency of Data Reporting for  Overall Goal  “Data Not Reported”

section_title: Veteran Career Readiness

section_title: Data reported for primary performance goal
paragraphs: “Data Not Reported”

section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Elizabeth Curda (Assistant  Director) and Adam Miles supervised the development of this report.  Virginia Chanley, Jehan Chase, Steven Putansu, Stacy Ann Spence, and  Dan Webb made significant contributions to this report. Deirdre Duffy and  Robert Robinson also made key contributions.


Question-summary annotations: questions: question: How has the government codified the needs of its CAP goals for OMB to implement?
answer: The GPRA Modernization Act of 2010 (GPRAMA) requires the Office of Management and Budget (OMB) to coordinate with agencies to: (1) establish outcome-oriented, federal government priority goals (known as cross-agency priority, or CAP, goals) with annual and quarterly performance targets and milestones; and (2) report quarterly on a single website now known as Performance.gov the results achieved for each CAP goal compared to the targets.

question: What did OMB do to implement GPRAMA?
answer: In February 2012, OMB identified 14 interim CAP goals and subsequently published five quarterly updates on the status of the interim CAP goals on Performance.gov.
child_questions: question: What were the findings of OMB in regards to their goals?
answer: While updates for eight of the goals included data that indicated performance towards an overall planned level of performance, only three also contained annual or quarterly targets that allowed for an assessment of interim progress. Updates for the other six goals did not report on progress towards a planned level of performance because the goals lacked either a quantitative target or the data needed to track progress.

question: How did OMB fail to fully implement the GPRAMA?
answer: The updates on Performance.gov also listed planned activities and milestones contributing to each goal, but some did not include relevant information, including time frames for the completion of specific actions and the status of ongoing efforts.
child_questions: question: How did these shortcomings affect the achievement of the goals?
answer: The incomplete information in the updates provided a limited basis for ensuring accountability for the achievement of targets and milestones.

questions: question: What does GPRAMA do to ensure the reviews are as effective as possible?
answer: GPRAMA also requires that OMB—with the support of the Performance Improvement Council (PIC)—review CAP goal progress quarterly with goal leaders.

question: How did OMB establish GPRAMA's requirements?
answer: OMB instituted processes for reviewing progress on the goals each quarter, which involved the collection of data from goal leaders and the development of a memorandum for the OMB Director. However, the information included in these memorandums was not fully consistent with GPRAMA requirements.
child_questions: question: How has the OMB failed to fully uphold GPRAMA guidelines in these cases?
answer: For example, GPRAMA requires OMB to identify strategies for improving the performance of goals at risk of not being met, but this was not consistently done.
child_questions: question: What are the ramifications of the failure of the OMB to fully implement GPRAMA's requirements?
answer: Without this information, OMB leadership and others may not be able to adequately track whether corrective actions are being taken, thereby limiting their ability to hold officials accountable for addressing identified risks and improving performance.

questions: question: How did the OMB evaluate their CAP goals?
answer: At the CAP-goal level, goal leaders for two CAP goals and one sub-goal instituted in-person progress reviews with officials from contributing agencies that were broadly consistent with the full range of leading practices for reviews, such as leadership involvement in reviews of progress on identified goals and milestones, and rigorous follow-up on issues identified through these reviews.
child_questions: question: What were the wide results of these reviews?
answer: In these cases, goal managers reported there were positive effects on performance, accountability, and collaboration.

question: Why were other review processes not as effective as the previous processes?
answer: In contrast, review processes used by other goal leaders did not consistently reflect the full range of leading practices.

question: What about these effective review processes work to evaluate the CAP goals?
answer: Effective review processes consistently engage leaders and agency officials in efforts to identify and address performance deficiencies, and to ensure accountability for commitments.
child_questions: question: Why are these components necessary for an effective review and what are the ramifications for not using them?
answer: Thus, not using them may result in missed opportunities to hold meaningful performance discussions, ensure accountability and oversight, and drive performance improvement.

questions: question: How does this report attempt to clarify the shortcomings of GAO?
answer: This report responds to GAO's mandate to evaluate the implementation of GPRAMA.
child_questions: question: What does this report assess about GAO's evaluation of GPRAMA's implementation?
answer: It assesses (1) what is known about progress made towards the interim CAP goals; and (2) how, if at all, quarterly progress reviews reflected GPRAMA requirements and leading practices for reviews, as well as how reviews contributed to improved cross-agency performance and collaboration.

question: How did GAO address the objectives of GPRAMA?
answer: To address these objectives, GAO analyzed CAP goal status updates and other documents from OMB and CAP goal progress-review meetings, and interviewed OMB staff and CAP goal representatives.

question: How did GAO use this data in their assessment?
answer: GAO compared this information to GPRAMA requirements and to leading practices for performance reviews previously reported on by GAO.