Managing for Results: OMB Improved Implementation of Cross-Agency Priority Goals, But Could Be More Transparent About Measuring Progress
Published: 2016-05-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Given the significance and complexity of the CAP goals, it is important that CAP goal contributors, Congress, and the public are able to track how CAP goal teams are making progress.
This report is one in a series in response to a statutory provision to review GPRAMA implementation. It assesses (1) the extent to which lessons learned from implementing the interim CAP goals were incorporated into the governance of the current CAP goals; (2) the extent to which GPRAMA requirements for reporting on CAP goal progress are included in the selected CAP goal quarterly progress updates; and (3) the initial progress in implementing the selected CAP goals.
GAO selected 7 of the 15 CAP goals to review (Customer Service, Job-Creating Investment, Lab-to-Market, Open Data, People and Culture, Smarter IT Delivery, and STEM Education). GAO assessed those goals' quarterly progress updates against relevant GPRAMA requirements, reviewed OMB and PIC guidance to CAP goal teams, and interviewed OMB, PIC, and CAP goal staff.
section_title: What GAO Found
paragraphs: The GPRA Modernization Act of 2010 (GPRAMA) requires the Office of Management and Budget (OMB) to coordinate with agencies to develop cross-agency priority (CAP) goals, which are 4-year outcome-oriented goals covering a number of complex or high-risk management and mission issues.
For the current set of CAP goals covering the period from 2014-2018, OMB and the interagency Performance Improvement Council (PIC) incorporated lessons learned from the 2012-2014 interim CAP goal period to improve the governance and implementation of these cross-cutting goals. For example, OMB and the PIC changed the CAP goal governance structure to include agency leaders, and holds regular senior-level reviews on CAP goal progress. They also provide ongoing assistance to CAP goal teams, such as by helping teams develop milestones and performance measures. Based in part on prior GAO recommendations, OMB and the PIC updated its guidance to assist CAP goal teams in managing the goals and in meeting GPRAMA reporting requirements. CAP goal teams told GAO that the CAP goal designation increased leadership attention and improved interagency collaboration on these issues.
GAO's assessment of the selected CAP goals' quarterly progress updates—published on Performance.gov—determined that CAP goal teams are meeting a number of GPRAMA reporting requirements, including identifying contributors, reporting strategies for performance improvement and quarterly results. However, most of the selected CAP goal teams have not established quarterly targets as required by GPRAMA, but are consistently reporting the status of quarterly milestones to track goal progress.
GAO found that the selected goal teams are aligning their quarterly milestones with strategies to achieve the desired goal outcomes. Further, all of the selected CAP goal teams reported that they are working to develop performance measures, and are at various stages of the process. However, the selected CAP goal teams were not consistently reporting on their efforts to develop performance measures. Given OMB, the PIC, and CAP goal teams' emphasis on developing measures that are relevant and well-defined, greater transparency is needed to track goal team's efforts on a quarterly basis. With improved performance information the CAP goal teams will be better positioned to demonstrate goal progress at the end of the 4-year goal period.
section_title: What GAO Recommends
paragraphs: GAO recommends that OMB, working with the PIC, report on Performance.gov the actions that CAP goal teams are taking to develop performance measures and quarterly targets. OMB staff generally agreed with GAO's recommendation and provided technical clarifications, which GAO incorporated as appropriate.

section_title: Letter
paragraphs: Many of the results the federal government seeks to achieve require the  coordinated effort of more than one federal agency, level of government,  or sector. These tasks include protecting the nation’s critical information  systems, responding to public health emergencies, ensuring food safety,  and better managing the risks of climate change. Effectively managing  these government-wide efforts requires coordination between multiple  federal agencies and transparent communications to the Congress and  decision makers about progress toward achieving results. Further, to  ensure that the federal government is achieving its crosscutting goals,  both top leaders and agency managers must have accurate and reliable  performance information to monitor and track the progress they are  making on achieving their goals. The GPRA Modernization Act of 2010  (GPRAMA) established a framework and processes aimed at  encouraging a more crosscutting and integrated approach to federal  performance, and improving transparency and accountability for these  efforts.
Among other things, GPRAMA requires the Office of Management and  Budget (OMB) to coordinate with agencies to develop federal government  priority goals (known as cross-agency priority or CAP goals) —4-year  outcome-oriented goals covering a number of crosscutting mission  areas—as well as goals to improve management across the federal  government. These goals are intended to cover areas where increased  cross-agency collaboration is needed to improve progress towards  shared, complex policy or management objectives, such as attracting  foreign investment to spur job growth and enabling agencies to recruit  and hire the best talent. GPRAMA requires OMB, in consultation with  CAP goal leaders, to publish progress updates quarterly on a public  website—Performance.gov. Performance.gov is a central government- wide website that provides program and performance information to  members and committees of Congress and to the public.
GPRAMA includes a provision for us to periodically report on the  implementation of the act. This report is part of that series. The  objectives of this report are to assess (1) the extent to which lessons  learned from implementing the interim CAP goals were incorporated into  the governance of the current CAP goals; (2) the extent to which  GPRAMA requirements for reporting on CAP goal progress are included  in the selected CAP goal quarterly progress updates; and (3) the initial  reported progress in implementing the selected CAP goals. In September  2015, we reported on lessons learned from the interim CAP goal period— which ended in March 2014—and provided an assessment of CAP goal  teams’ initial reported progress implementing the current set of CAP  goals.
To conduct our assessment, we selected 7 of the 15 CAP goals for more  in-depth review. We randomly selected four of the seven CAP goals  including Open Data, Science Technology Engineering and Mathematics  (STEM) Education, Job-Creating Investment, and Lab-to-Market. We also  included three goals in our review because we have completed and  ongoing work related to the Customer Service, People and Culture, and  Smarter IT Delivery CAP goals. Our sample of goals is nongeneralizable  and not representative of all CAP goals. See figure 1 for a list of the  seven selected CAP goals, CAP goal leaders, and a description of the  goals. Appendix I provides additional information on the seven selected  goals, including goal statements, agency goal leads, and sub-goals.
To assess the extent to which OMB incorporated lessons learned from  implementing the interim CAP goals into governance for the current CAP  goals, we reviewed GPRAMA, OMB, and Performance Improvement  Council (PIC) guidance for assessing and reporting CAP goal progress,  and relevant OMB and agency documentation. We also reviewed CAP  goal quarterly progress updates from the second quarter of fiscal year  2014 through the first quarter of fiscal year 2016, and our prior related  work. We interviewed OMB and PIC staff, and selected CAP goal leaders  and goal teams from among the agencies and entities shown in figure 1.  We also assessed the extent to which OMB addressed the  recommendations from our June 2014 report on the interim CAP goals.
To determine the extent to which the seven selected CAP goals’ quarterly  progress updates reflect GPRAMA requirements, we conducted a content  analysis of the four quarterly updates published from March 2015 through  December 2015 (quarter two of fiscal year 2015 through quarter one of  fiscal year 2016). We assessed the updates against 13 reporting  requirements—6 that we determined were relevant at the CAP goal level,  and 7 that we determined were relevant at the sub-goal level. To fully  meet each requirement, the quarterly progress update had to provide all  information required by GPRAMA, either for the overall CAP goal or for  each sub-goal, as appropriate. For those requirements analyzed at the  sub-goal level, we determined a goal had partially met a requirement if  the required information was present in the quarterly progress update for  at least one of the goal’s sub-goals. For requirements that were not met,  we determined the required information was not present in the quarterly  progress update. We shared our analysis with each of the selected CAP  goal teams and interviewed agency staff to collect their feedback on our  analysis. We also collected and reviewed relevant documentation  provided by agencies, OMB, and the PIC.
To describe the extent to which the selected CAP goal teams have made  progress during the first two years of implementation covering the period  from March 2014 through December 2015, we reviewed eight quarterly  progress updates published on Performance.gov and relevant agency  documents. We also interviewed CAP goal team staff to discuss  challenges, efforts to mitigate challenges, and key achievements.  Because the scope of our review was to examine the implementation of  quarterly reporting requirements, we did not evaluate whether these goals  were appropriate indicators of performance, were sufficiently ambitious,  or met other dimensions of quality.
We conducted our work from January 2015 to May 2016 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives. More detailed information on our scope and  methodology appears in appendix II.
subsections: section_title: Background
paragraphs: Several players have important roles in implementing the CAP goals, as  required by GPRAMA:
OMB. OMB provides guidance to agencies and CAP goal teams in  Circular A-11 on how to implement GPRAMA, including implementing  and reporting on the CAP goals. Beginning in March 2014, OMB and  the PIC also developed more detailed guidance, which they have  updated periodically, to help ensure CAP goal teams are complying  with GPRAMA requirements, including developing CAP goal  implementation plans, identifying contributors, and reporting quarterly  progress toward their goals on the public website, Performance.gov.
PIC. The PIC is chaired by OMB’s Deputy Director for Management  and is composed of agency Performance Improvement Officers, other  agency performance staff, and senior OMB staff to facilitate the  exchange of useful practices to strengthen agency performance  management, such as through cross-agency working groups. The  PIC is supported by an Executive Director and a team of 8 full-time  staff, that conduct implementation planning and coordination on  crosscutting performance areas, including working with OMB, other  government-wide management councils, and agencies on the CAP  goals.
CAP Goal Leaders. CAP goal leaders are responsible for  coordinating efforts to implement each goal. As shown in figure 1  above, the current CAP goals have at least two goal leaders—one  from the Executive Office of the President and the other from a key  responsible agency. OMB directs CAP goal leaders to engage officials  from contributing agencies by leveraging existing interagency working  groups, committees, and councils.
GPRAMA has a number of requirements for reporting on CAP goals (see  text box). These reporting requirements are designed to ensure that  relevant performance information is used to improve performance and  results of the goals, that OMB and CAP goal leaders actively lead efforts  to engage all relevant participants in collaborative performance  improvement initiatives, and hold them accountable for progress on  identified goals and milestones.
GPRA Modernization Act of 2010 Requirements for Reporting on Cross-Agency  Priority Goal Progress  Make available on the website Performance.gov:  1.  A brief description of each of the federal government priority goals required by  section 1120(a) of this title.  2.  An identification of the lead government official for each federal government  performance goal.  3.  An identification of the agencies, organizations, program activities, regulations, tax  expenditures, policies, and other activities that contribute to each federal government  performance goal.  4.  Plans to address major management challenges that are government-wide or  crosscutting in nature, and describe plans to address such challenges, including  relevant performance goals, performance indicators, and milestones.  5.  Performance goals to define the planned level of performance for each goal for the  year in which the plan is submitted and the next fiscal year for each of the federal  priority goals.  6.  Common federal government performance indicators.  7.  Quarterly targets for each common performance indicator.assessing the overall progress toward each federal government performance goal.  11.  How performance indicators with quarterly targets are being used in measuring or  assessing whether relevant agencies, organizations, program activities, regulations,  tax expenditures, policies, and other activities are contributing as planned.  12.  Results achieved during the most recent quarter.  13.  Overall trend data compared to the planned level of performance.
OMB established the first set of CAP goals for a 2-year interim period in  February 2012. We issued reports in 2012 and 2014, evaluating  implementation challenges faced during the interim CAP goal period. In  May 2012, we reported that the interim CAP goals did not leverage all  relevant parties, and therefore important opportunities for achieving these  goals may have been missed. OMB responded by identifying and  including additional relevant departments, agencies, and programs for the  interim CAP goals. In June 2014, we identified a number of reporting and  accountability gaps with the interim CAP goals. Specifically, we found  that some goals did not report on progress towards a planned level of  performance because the goals lacked either a quantitative target or the  data needed to track progress. We also found that quarterly progress  updates published on Performance.gov listed planned activities and  milestones contributing to each goal, but some did not include relevant  information, including time frames for the completion of specific actions  and the status of ongoing efforts. We concluded that the incomplete  information in the updates provided a limited basis for ensuring  accountability for the achievement of targets and milestones. As a result,  we made seven recommendations to OMB and the PIC to improve the  reporting of performance information for CAP goals and quarterly  progress reviews. OMB responded to these recommendations by  updating its guidance to CAP goal teams to address the reporting and  accountability gaps we identified. We closed these recommendations as  implemented based on OMB’s updated guidance to CAP goal teams, our  assessment of the quarterly progress updates, and interviews with CAP  goal teams.
In March 2014, OMB issued the current set of CAP goals, which  GPRAMA requires to be updated, at a minimum, every 4 years. As  shown in figure 2, the current set of 15 CAP goals includes 7 related to  crosscutting mission areas and 8 related to management. OMB reported  that it worked to select CAP goals that represent high-priority issue areas  with outcomes that could be enhanced through improved cross-agency  implementation.
OMB will issue the next set of 4-year CAP goals in February 2018 along  with the President’s fiscal year 2019 budget. OMB staff told us that they  expect the last progress update posted on Performance.gov for this set of  CAP goals will reflect progress from the second quarter of fiscal year  2018, which ends in March 2018. Each quarterly progress update for this  set of CAP goals is available on Performance.gov.
section_title: OMB Incorporated Lessons from Interim CAP Goals into Governance of Current CAP Goals
subsections: section_title: OMB Updated Guidance to Improve Public Reporting on Implementation of CAP Goals
paragraphs: As we reported in September 2015, OMB improved CAP goal reporting  and accountability by addressing recommendations we made in June  2014. Specifically, OMB developed updated guidance and a new  reporting template for CAP goal teams that provide a consistent reporting  format across all goal teams, and helps goal teams meet GPRAMA  reporting requirements. According to OMB staff managing the CAP goals,  this revised reporting structure and template is intended to promote  engagement from goal leaders, facilitate performance monitoring, ensure  transparency of the CAP goal initiative, and provide a framework for CAP  goal teams to articulate cross-agency goals using milestones and  measures. The template outlines information that GPRAMA requires goal  teams to report on Performance.gov, such as goal leaders; contributing  agencies, organizations, and programs; performance measures and  targets; key milestones; and plans to address government-wide  management challenges. The updated guidance directs CAP goal teams  to include additional performance information—not required by  GPRAMA—in their quarterly progress updates, such as the status of  milestones, associated time frames for their completion, performance  indicators (or measures) under development, and an organizational chart  depicting the goal’s governance team.
According to the seven CAP goal teams we spoke with and OMB staff,  the quarterly progress updates provide useful information for goal leaders  to track progress over time and to make timely management decisions  that affect goal implementation. PIC guidance states that the update  should provide succinct, high-level information that is understandable to  all involved in implementation (i.e., program staff to senior leaders). OMB  staff managing the CAP goals told us that the quarterly progress updates  summarize the progress made on each CAP goal. They added that, at  this point in the goal period, reviewing the milestone status for each CAP  goal is the best way to determine whether goals are at risk or sufficient  progress is being made quarterly. For example, the Lab-to-Market CAP  goal leader in the Department of Energy told us that the progress updates  enables the goal team to track progress on activities that support and  expand technology between the national laboratories and the private  sector. The goal leader also told us the quarterly reports are useful to  implementation teams because they reinforce the need to coordinate  across agency lines, and drive discussion about what can be done to  achieve goals and milestones.
section_title: OMB Developed Strategies to Build CAP Goal Teams’ Capacity for Goal Implementation
paragraphs: OMB and the PIC have also implemented strategies to build agency  capacity to work across agency lines. For example:
Assigned agency goal leader. For the current set of CAP goals,  OMB changed the CAP goal governance structure to include agency  co-leads for each CAP goal, in addition to entities within the Executive  Office of the President, such as OMB, that were already serving as  goal leaders (see figure 2 above). According to OMB staff, this new  governance structure reflects agency leadership and expertise in CAP  goal subject areas, more effectively leverages agency resources for  crosscutting efforts, and promotes greater coordination across  multiple agencies. For example, Science, Technology, Engineering  and Mathematics (STEM) Education CAP goal team staff from the  Office of Science and Technology Policy (OSTP) told us that they  found contributing agencies to be more receptive to directives and  efforts for implementing the CAP goal because they come jointly from  the National Science Foundation (NSF) and are not solely based on  OSTP’s policy perspective. They also told us that NSF’s leadership of  CAP goal activities and its ability to secure agency buy-in, among  other things, makes it an effective CAP goal leader.
Provide ongoing guidance and assistance to CAP goal teams.
The seven CAP goal teams we spoke with told us that OMB and the  PIC staff are available regularly to provide them with ongoing support,  such as assisting with the regular collection of performance data,  leading seminars to develop useful milestones and priority actions,  and improving teams’ ability to track and report progress. The PIC  plans to continue to assist teams with tracking their progress and  ensuring accountability across the goals, including supporting the  quality and completeness of the regular quarterly progress updates  through Performance.gov. For example, in August 2014, PIC staff met  with the STEM Education CAP goal team to assist it in developing  milestones and performance measures and to define actionable next  steps. The STEM Education CAP goal leader from NSF told us the  assistance provided by the PIC helped the goal team improve its  implementation plan and develop relevant performance measures.
Holding senior-level reviews. Another way OMB and the PIC are  increasing leadership attention of CAP goal implementation is by  committing to hold regularly scheduled senior-level meetings to review  CAP goal progress. OMB’s Deputy Director for Management leads  implementation-focused meetings for the eight management CAP  goals approximately three times a year, and OMB’s Deputy Director  for Budget leads meetings to review the seven mission-focused CAP  goals as necessary. According to OMB, as of March 2016, it held the  senior-level review meetings for all 15 CAP goals as planned.
Obtained a means of funding cross-agency CAP goal activities.
As part of the fiscal year 2016 consolidated appropriations act,  Congress provided authority for the heads of executive departments  and agencies, with OMB approval, to transfer up to $15 million for  purposes of improving coordination, reducing duplication, and other  activities related to the implementation of the CAP goals. OMB staff  told us that they proposed this means of funding crosscutting activities  in response to lessons learned from the interim CAP goal process,  feedback from agencies, and our work on enhancing collaboration in  interagency groups. Table 1 lists the fiscal year 2016 interagency  transfers for selected CAP goals. For example, OMB reported that the  Lab-to-Market CAP goal team would use $1.9 million of transferred  funds to develop an interface for all 17 DOE national laboratories to  directly interact with external stakeholders, such as the investment  community, and allow the public to access specific capabilities across  the national laboratory network.
Launched a government-wide White House Leadership  Development Program. OMB staff told us that leadership fellows  were selected and assigned to one of the CAP goal teams and other  cross-agency initiatives to expose emerging agency leaders to cross- agency issues and address the need for strong leadership on the CAP  goals, while leveraging existing resources. In November 2015, 16  fellows began their 1-year rotations.
section_title: Goal Teams Reported CAP Goal Designation Led to Increased Leadership Attention and Collaboration
paragraphs: CAP goal leaders and their teams told us that the CAP goal designation  has resulted in increased leadership attention within their agencies and  access to management and performance expertise at OMB, the PIC, and  government-wide councils such as the President’s Management Council  (PMC) and the Chief Human Capital Officers Council. CAP goal teams  we spoke with told us that the increased focus on CAP goal  implementation enabled CAP goal teams to request and receive agency  resources that helped sustain progress toward goals. For example, the  Smarter IT Delivery CAP goal team worked with OPM to obtain a new  hiring authority for digital service experts to assist with production of  information technology and digital services. As of December 2015, five  agencies and OMB reported hiring digital service experts. Goal team  staff told us that obtaining the hiring authority was a direct result of the  CAP goal.
According to OMB and PIC staff, the CAP goal process is a management  tool to encourage interagency collaboration. OMB staff told us that they  selected CAP goals, in part, that would focus on implementation  challenges that could benefit from greater collaboration between multiple  agencies. According to OMB staff, some CAP goal efforts are relatively  new and agencies do not have much experience working together,  whereas other CAP goal issues have long-standing relationships and  interagency mechanisms in place. All of the CAP goal teams described  approaches they are using to work across agency lines. For example,  CAP goal teams established formal interagency groups, used information- sharing tools to communicate, and developed standardized guidance.
Leveraging expertise across agencies. CAP goal teams report that  they are working with pre-existing interagency working groups or  establishing new working groups or communities of practice to  leverage expertise and experience across agencies. For example, the  Customer Service CAP goal team established an interagency  Community of Practice (COP) which meets monthly to share practices  and provide feedback to CAP goal leadership. For example, the goal  team reports that a group within the COP led the development of a  draft customer service toolkit, which includes customer service  principles and a model for gauging program-level progress toward  those principles. Likewise, the Open Data CAP goal team established  the Interagency Open Data Working Group, which includes the 24  Chief Financial Officers (CFO) Act agencies and smaller entities  government-wide. The group meets biweekly and its members  represent a diverse range of levels of employees, such as IT  contractors and chief data officers. According to the goal team, the  establishment of this working group is the best outcome from the CAP  goal so far. The group shares best practices, allows agencies to learn  from other agencies that may be farther along, and shares tools. For  example, the goal team told us that it hosted a series of webinars on  Data.gov explaining a new method to convert data from common  format to machine readable. The goal team told us that this process  was a challenge for agencies, but through the webinars and the  working group, every agency now knows how to produce machine- readable documents, and is currently validating their processes.
Developing common reporting framework. The Job-Creating  Investment goal team told us that the CAP goal designation has  helped the Department of Commerce (Commerce) headquarters staff  to engage with overseas staff and better leverage their resources and  data. For example, the CAP goal team told us that they launched an  interagency effort between State and Commerce to provide training  and update guidance to U.S. overseas staff on standardized methods  and procedures for reporting information on their activities to attract  foreign direct investment. Overseas staff are now submitting updates  that reflect Commerce’s activities abroad. The goal team reported that  it now holds quarterly reviews, in part, to ensure the data is being  collected and reported consistently by staff from multiple agencies  and programs.
Outreach to the public and nonfederal stakeholders. The Open  Data CAP goal team reported a number of White House-sponsored  events to encourage public participation in using government data and  to elicit feedback on Open Data CAP goal efforts. For example, the  goal team reported that agencies hosted Open Data Roundtables to  highlight success stories and get feedback from its stakeholders,  including nonfederal data users. In another example, the STEM  Education CAP goal team developed an online portal for graduate  students to locate federally-funded STEM research opportunities.  CAP goal team staff told us they recently completed a similar  resource for undergraduate students.
section_title: Selected CAP Goal Teams Have Met Many GPRAMA Reporting Requirements, but Most Have Not Established Quarterly Targets
paragraphs: Based on GPRAMA requirements, we identified five broad actions that  CAP goal teams follow to implement a goal: (1) establish the goal, (2)  identify goal leaders and contributors, (3) develop strategies and  performance measures, (4) use performance information to make  decisions and improve performance, and (5) report results. The 13  GPRAMA requirements for reporting on CAP goal progress each relate to  an action for goal implementation. These actions do not necessarily  occur sequentially. For example, in the quarterly progress updates  published on Performance.gov, CAP goal teams reported developing new  strategies and performance measures, while simultaneously using  performance information to report results and develop new strategies.
subsections: section_title: Action 1: Establish the Goal
subsections: section_title: Example of a Goal Statement: Lab-to- Market CAP Goal
paragraphs: The first action for goal implementation is to articulate the cross-agency  goal. GPRAMA requires that Performance.gov contain a description of  each CAP goal. To meet this requirement, CAP goal teams list the overall  goal statement and provide other contextual information, such as the  problem being addressed and a vision for what the goal might achieve in  each quarterly progress update published on Performance.gov. As shown  in figure 3, all of the selected CAP goal teams met this reporting  requirement.
section_title: Action 2: Identify Goal Leaders and Contributors
paragraphs: Definition of common term:  Contributors –The agencies, organizations,  program activities, regulations, tax  expenditures, policies, and other activities that  contribute to each CAP goal.
The second action for goal implementation is to identify leaders and  contributors that are accountable for progress toward the goal, and to  establish the cross-agency team that will monitor and oversee  implementation. GPRAMA requires that Performance.gov provide  information on the lead government official for each CAP goal and the  agencies, organizations, policies, and other activities—including tax  expenditures—that contribute to the goal. As depicted in figure 4, we  found that all of the selected CAP goals identified and clearly reported the  goal leaders accountable for implementation of the CAP goal.
Also shown in figure 4, we determined that all of the selected CAP goal  teams fully met the requirement to identify and report key contributors to  the goal. To make this determination, we reviewed the CAP goal  statements and related activities, and evaluated whether selected goal  teams identified key contributors to the goal. For example, the Open Data  CAP goal team identified contributing agencies and offices within the  Executive Office of the President and the General Services Administration  in addition to policies governing open data efforts at federal agencies,  such as agency requirements to protect personal and confidential data.
For this review, we also evaluated whether OMB and the goal teams  considered the extent to which tax expenditures contribute to the selected  CAP goals. This issue relates to a recommendation we made in 2013 that  OMB should review whether all relevant tax expenditures that contribute  to a CAP goal have been identified, and as necessary, include any  additional tax expenditures in the list of federal contributors for each  goal. In September 2015, OMB staff told us that OMB had determined  that there were no tax expenditures that were critical to support  achievement of the current CAP goals. Goal teams we spoke with during  our review told us that they did not identify any relevant tax expenditures  related to their goals. As a result, we have recently closed this  recommendation as implemented.
section_title: Action 3: Develop Strategies and Performance Measures
paragraphs: The third action for goal implementation is developing strategies and  performance measures. GPRAMA requires that each CAP goal team  publish on Performance.gov plans to address government-wide major  management challenges, and establish clearly defined quarterly  milestones and performance indicators (which we refer to as performance  measures), and targets to measure or assess overall progress. GPRAMA  also requires that teams develop strategies for performance improvement  as they regularly review implementation and results.
As shown in figure 5, all CAP goal teams reported plans to address major  management challenges that are government-wide or crosscutting in  nature as reflected in the quarterly progress updates published on  Performance.gov. In addition, all the goal teams’ quarterly progress  updates established clearly defined quarterly milestones. Specifically, we  found that all CAP goal teams we reviewed established a work plan that  included a brief description of planned milestones, milestone due dates,  and the agency or interagency group responsible for implementing  milestones. If clearly linked to the outcome, milestones can be a way for  goal teams to report and track progress quarterly.
We determined that all but one of the selected goal teams fully reported  strategies for performance improvement, meaning that each goal team  updated or revised its strategies or major actions from one quarter to the  next, or identified challenges or barriers to the goal or a specific  milestone. The Lab-to-Market CAP goal team partially met this  requirement because while the team updated some of its milestones in  the first quarter of fiscal year 2016, it did not report any associated  barriers or challenges, nor did it identify any milestones for the sub-goal  to assess the economic impact of federal technology transfer activities.
Definition of common terms:  Performance measure - A measurable value  that indicates the state or level of something  (also called “indicator”).  Sub-goal - A goal that contributes to the  achievement of the broader CAP goal.
Five of the seven CAP goal teams reported common performance  measures for each of their sub-goals. The Lab-to-Market CAP goal team  reported a number of performance measures for three of its four sub- goals, partially meeting this requirement. The Customer Service CAP goal  does not have any performance measures at this time, but the team  reported a number of performance measures that are under development.  According to OMB and PIC staff, some of the CAP goal teams may be  working to develop cross-agency performance measures for the first time,  or may have measures in place that require improvement. Because the  scope of our review was to examine the implementation of quarterly  reporting requirements, we did not evaluate whether these measures  were appropriate indicators of performance, were sufficiently ambitious,  or met other dimensions of quality.
Definition of common term:  Target - Level of performance expressed as a  tangible, measurable objective, against which  actual achievement can be compared,  including a goal expressed as a quantitative  standard, value, or rate.
Five of seven goal teams fully or partially met the requirement to report on  the planned level of performance because they reported an annual target  or a target for the 4-year goal period for at least one sub-goal. The two  CAP goal teams that fully met this requirement reported targets for all  performance measures. For example, for one of its sub-goals, the People  and Culture CAP goal team established a target that, by the end of the  CAP goal period in 2018, there would be an increase in hiring managers’  satisfaction with the quality of applicants from 60 percent to 70 percent,  as measured by the Chief Human Capital Officers Management  Satisfaction Survey. The Customer Service CAP goal team did not  report any targets because it does not yet have established performance  measures. The Science, Technology, Engineering, and Mathematics  (STEM) Education goal team did not identify a target—expressing both  magnitude and direction—for 12 of its 15 performance measures. The  most recent data available for these measures is from 2013, prior to the  start of the current CAP goal period.
Of this set of actions, the selected goal teams had the most difficulty  meeting the requirement to report quarterly targets for each performance  measure. Five of the seven selected CAP goal teams did not report  quarterly targets. The Open Data goal team reported a quarterly target for  1 of its 16 performance measures, partially meeting this requirement. The  STEM Education goal team also partially met this requirement by  reporting a 1 percent quarterly target increase in the number of views,  downloads, and international downloads of a June 2015 webinar focused  on implementing evidence-based practices in undergraduate STEM  instruction. OMB and PIC staff told us that they would like to work  toward requiring quarterly targets, as appropriate, going forward.  However, they also noted it may not be possible or appropriate for a goal  to have a quantitative quarterly target, as relevant data may not be  available quarterly, particularly for new initiatives. In such a case, a  qualitative target might be more appropriate. CAP goal teams we spoke  with expressed similar concerns and told us that they are instead relying  on milestones to track progress quarterly. For example, the People and  Culture CAP goal team relies on results from the Federal Employee  Viewpoint Survey to measure progress on employee engagement efforts.  Since the survey is administered annually, the goal team reports annual  targets for this measure. The Lab-to-Market CAP goal team told us that  its goal to accelerate and improve the transfer of new technologies from  the laboratory to the commercial marketplace can take years and is not  measured quarterly.
section_title: Action 4: Use Performance Information
paragraphs: The fourth action related to goal implementation involves using  performance information during regular progress reviews to assess if  major actions and contributors are collectively achieving the desired  results. We have long reported that agencies are better equipped to  address management and performance challenges when managers  effectively use performance information for decision making. OMB’s  guidance to CAP goal teams states that frequent reviews provide a  mechanism for CAP goal leaders to keep contributing agencies focused  on crosscutting priorities. For public reporting, GPRAMA requires that  CAP goal teams report on how they use performance measures (or  indicators) and quarterly targets to assess overall progress toward the  goal, and whether relevant agencies, organizations, program activities,  and tax expenditures are contributing as planned.
Figure 6 shows that for six of the seven selected CAP goals, the teams  partially met the requirement to report how they are using performance  measures with quarterly targets to assess overall progress. This  assessment is largely a result of the fact that while six of the seven goal  teams have established and reported performance measures, they  generally have not established quarterly targets for the measures, as  shown in figure 5 above. OMB and PIC staff managing the CAP goals told  us that they are aware that CAP goal teams can improve how they are  using performance measures and targets.
Four of the seven selected CAP goals partially met the requirement to  report how they are using performance measures and quarterly targets to  assess key contributors to the goal—such as relevant agencies and  programs—because some of their performance measures track  contributors’ implementation of certain activities or requirements. For  example, the Smarter IT Delivery CAP goal team reports an indicator  tracking the number of agencies that have created “buyers’ clubs” to  promote more efficient information technology (IT) contracting practices.  The other three CAP goals did not meet this requirement because they  either did not have measures in place, or did not have performance  measures that assess contributions by agencies or programs responsible  for implementing the goal. For example, some of the measures for these  goals are focused on an outcome or output that is external to the  government, such as number of STEM degrees as a performance  measure for the STEM Education CAP goal. In such cases, goal teams  told us they are tracking progress and key contributors toward the goal  through milestones.
section_title: Action 5: Report Results
paragraphs: Reporting results is the next action related to goal implementation.  GPRAMA requires that CAP goal teams report the most recent quarterly  results on Performance.gov. They are also required to publish overall  trend data compared to the planned level of performance or target. Our  prior work has demonstrated the need to regularly report results to  promote accountability and provide agency leaders with information they  can use to inform their decisions. By adding trend information, OMB and  decision makers can more easily track the goal teams’ quarterly progress.  As shown in figure 7, we found that the seven selected CAP goal teams  reported results achieved during the most recent quarter by reporting on  the status of milestones, data on implementation, and narrative progress  updates.
As of the first quarter of fiscal year 2016, six of the seven selected CAP  goal teams are able to report overall trend data for at least some, if not  all, of their performance measures. In some cases, goal teams reported  new performance measures, noting that they are still working to establish  a baseline and trend data. The Open Data, Smarter IT Delivery, and  People and Culture CAP goal teams reported overall trend data  compared to the planned level of performance for each performance  measure. The Customer Service CAP goal team did not report trend data  because it stated that it does not have a method to collect and report  customer service performance information government-wide.
section_title: Selected CAP Goal Teams Reported Completing Milestones, but Could Improve Transparency by Reporting on the Development of Performance Measures
subsections: section_title: Selected CAP Goal Teams Reported Completing Quarterly Milestones
paragraphs: CAP goals are designed to focus on longer term or complex outcomes  involving multiple agencies, programs, or entities. In such cases, there  may not be a common measure of the desired outcome, the outcome is  not frequently observed, or is influenced by external factors. As a result,  determining if the goal team is making progress each quarter can be  challenging. In such cases, the goal team may need to break the goal into  pieces that can be more easily measured or assessed (e.g., sub-goal and  milestone). All of the CAP goal teams we examined identified two or more  sub-goals with associated milestones and performance measures, where  available.
All of the selected CAP goal teams and OMB staff managing the goals  told us that they are using milestones to track and report on progress  quarterly. However, they are still working to improve the collection and  reporting of performance information for the CAP goals. Our review of  selected CAP goal quarterly progress updates—from March 2014 through  December 2015—found that the seven CAP goal teams demonstrated  progress in their ability to meet GPRAMA reporting requirements.  Specifically, we found that in the most recent quarterly progress update,  the selected CAP goal teams all reported prospects and strategies for  performance improvement, and reported more performance information in  the form of new performance measures, targets, baseline data, and trend  information, if available. However, because most of the targets  established are annual or 4-year targets or data for certain performance  measures were unavailable, CAP goal teams and OMB staff told us that  milestones provided the most relevant information on quarterly progress  for the CAP goals.
According to OMB and PIC staff, milestones have also helped the goal  teams reach agreement on respective roles and responsibilities, and have  helped agencies align their activities with the strategies to implement the  goal. Within the first two years of implementation, CAP goal teams  reported having completed milestones that cover a range of activities,  including those related to project management, piloting new programs,  development of information technology, and other tools to support  implementation, as shown in the examples below.
Smarter IT Delivery CAP goal. In October 2015, the CAP goal team  launched a pilot to develop a Digital Service Contracting Professional  Training and Development Program to improve the process of IT  acquisition and digital services.
Lab-to-Market CAP goal. In the third quarter of fiscal year 2015, the  Department of Energy launched the Small Business Voucher pilot,  creating a single point of access for small business to access the  agency’s laboratory resources for clean energy projects.
People and Culture CAP goal. In October 2015, OPM launched the  web-based Hiring Toolkit, which provides policy and technical  guidance, information on roles and responsibilities, and an inventory  of hiring flexibilities for human resource staff and managers in federal  agencies.
As directed by OMB, goal teams are also reporting milestones that are at  risk of not meeting scheduled due dates or face barriers to completion.  During the CAP goal period, goal teams reported delays for various  reasons, including coordination across agency boundaries, availability of  staffing, and challenges with information technology. Teams also reported  changes in strategy or approach that altered time frames or required  revisions to existing milestones. For example, in the first quarter of fiscal  year 2015, the Open Data CAP goal team missed a milestone to refresh  its metrics on data quality and machine readability due to technical  challenges that delayed the launch of critical tools. The team reported  completing this milestone in the following quarter. In another example, the  Job-Creating Investment CAP goal team revised a milestone due date to  allow more time to evaluate the effectiveness and impact of its processes  to assist potential investors in meeting their objectives. The goal team  reported in the fourth quarter of fiscal year 2015 that it completed this  milestone according to the revised timeline.
section_title: Selected CAP Goal Teams Have Emphasized Alignment of Milestones with Strategies and Results
paragraphs: The use of milestones is a recognized approach for tracking interim  progress toward a goal as long as they are clearly linked to the desired  outcome. OMB Circular A-11 states that if milestones are used as a  performance goal, they must be described in a way that makes it possible  to discern if progress is being made toward the goal. Figure 8 provides  examples of a performance measure and a milestone for two different  CAP goals.
According to OMB and PIC staff managing the CAP goals, they are  working with goal teams to develop strategies to improve performance on  crosscutting issue areas and also develop concrete milestones. In its  guidance to CAP goal teams, OMB and the PIC directed goal teams to  design and report a high-level, goal-to-strategy crosswalk, highlighting the  key elements of the goal or sub-goal, such as the related major actions to  achieve impact, and key performance measures, if available. According to  the guidance, once the team has identified sub-goals and developed a  high-level strategy, the goal team should identify and report the quarterly  status of related milestones, which should be linked to a sub-goal or  major action to illustrate how the milestones contribute to the  achievement of the goal. Our review of the selected CAP goal quarterly  progress updates found that all seven CAP goal teams followed the  guidance and identified sub-goals with major actions and performance  measures, where possible. As designed, the reporting template provides  flexibility for the CAP goal teams to organize and report this information,  and the CAP goal teams generally reported on how their milestones were  linked with their related sub-goals.
CAP goal teams that used their quarterly progress updates to clearly  articulate a strategy to achieve the sub-goal, and any barriers or  challenges along with associated actions and milestones, are better  positioned to demonstrate progress. As shown in figure 9, the People and  Culture CAP goal quarterly progress update demonstrated the link  between the milestones, and the result it intends to achieve, by identifying  specific milestones or major actions and performance measures for each  sub-goal or outcome. OPM staff for the People and Culture CAP goal told  us that aligning sub-goals and milestones allows them to see how all  milestones are contributing to the overall needs of the CAP goal. It also  allows them to review the information at the sub-goal level and roll the  information up to the CAP goal level when necessary.
Taken together, the sub-goals, milestones, and performance measures, if  available, should demonstrate how the goal team is measuring progress  related to the broader CAP goal. An effective tool to demonstrate that the  goal team’s activities are contributing to the goal’s desired long-term  outcome is to develop a model that describes the logical relationship  between an agency’s inputs, activities, outputs, and outcomes. By  making those linkages explicit, decision makers can have more focused  and meaningful discussions for how proposed strategies or milestones  are tied to desired results, and how to measure the success of strategy  execution and impact.
Identifying, agreeing with, and understanding the logical relationship  between the goal team’s activities and the desired outcome is an  important step in developing a sound and cohesive plan for  implementation. As we have previously reported, the relationship between  activities—such as milestones—and outcomes should be periodically  assessed through program evaluations or other methods to determine if  outcomes are being achieved as expected, and should be revised as  necessary. The quarterly progress reviews of CAP goals required by  GPRAMA is one vehicle for assessing implementation of the goals and  adjusting strategies as needed. As directed by OMB, any adjustments to  the goal strategy or next steps should be clearly reported in the  subsequent quarterly progress updates published on Performance.gov to  allow the public and decision makers to easily follow the implementation  strategy and goal progress.
section_title: Selected CAP Goal Teams Can Be More Transparent about the Status of Their Efforts to Develop Performance Measures
paragraphs: All of the goal teams we reviewed reported efforts to develop  performance measures as part of their strategy to improve performance  of these complex and crosscutting issues. Performance goals and  measures allow CAP goal teams and other stakeholders to track the  progress they are making toward their goals and provide critical  information on which to base decisions for improving programs or  activities. OMB and PIC staff overseeing the CAP goals told us they are  aware that improvement can be made in goal teams’ tracking and  reporting of progress using performance measures, and they hope to  continue to improve goal teams’ capacity to track performance. OMB and  the PIC directed CAP goal teams to select or develop performance  measures that are relevant, well defined, timely, reliable, and capable of  being influenced by the actions of contributing organizations.
We found that three of the seven selected goal teams reported specific  actions they have completed, or are completing, to develop performance  measures. The actions included identifying existing and needed data,  collecting new data, testing and validating the data for reliability, and  setting the baseline for measurement. For example:
Throughout the goal period, the STEM Education CAP goal team  consistently reported on the steps they were taking to develop  performance measures, and we were able to track their efforts from  one quarter to the next. For example, the goal team identified the  need to collect additional data to develop performance measures  through a long-term survey. In the fourth quarter of 2015, the goal  team reported they had completed some milestones—and others  were on track—related to conducting in-depth testing to add a new  item on undergraduate mathematics instruction for the High School  Longitudinal Survey (see figure 10). The goal team reported three  related milestones that, according to the goal team, should provide  survey results by December 2017. At that point the team expects to  be able to test and validate a new performance measure and  establish a baseline and targets for future measurement.
The Job-Creating Investment CAP goal team reported that they are  collecting baseline data and trends to ensure the methodology  supporting future performance measures is valid. The goal team  reported that it conducted a feasibility study to capture data on foreign  direct investment in the U.S. through a new evaluation system and  database. It also reported that it is working to collect data on client  satisfaction with the services it received. The CAP goal team plans to  use these sources of data to measure the impact of its work.
The Customer Service CAP goal team reported steps it plans to take  to develop a measure tracking the percentage of customer-facing  federal programs showing improvements in their customer feedback  data. For example, the goal team identified milestones for  implementation into fiscal year 2017 to identify the core customer- facing federal programs that have a plan to improve customer service  and collect customer service feedback data. The goal team reported  that these programs would conduct a self-assessment and develop  strategies to improve customer service followed by assessment of the  data to identify its baseline customer feedback data.
For the other four CAP goals we reviewed, the goal teams told us about  the actions they are taking to develop performance measures. However,  their quarterly reports on Performance.gov did not contain this  information. For example, the Lab-to-Market CAP goal leaders told us  that they are conducting public literature reviews and commissioning  studies that may help the goal team identify potential quantitative and  qualitative performance measures to assess the economic impact of  federal technology transfer activities. The goal team reported having  completed milestones related to this effort in 2014 and 2015, but has not  reported additional steps it has taken, or plans to take, to test or validate  the results of the studies. As a result, we could not determine if the goal  team is making progress towards establishing performance measures.
section_title: Conclusions
paragraphs: The CAP goals we reviewed are intended to drive progress in important  and complex areas, such as improving information technology and  customer service interactions across government, coordinating federal  STEM education efforts, and improving federal hiring practices. We found  that OMB’s updated guidance to CAP goal teams and the quarterly  reporting template have assisted goal teams in managing implementation  of the goals and in meeting the GPRAMA reporting requirements. Further,  OMB and the PIC’s efforts to build the capacity of the CAP goal teams to  implement the goals has resulted in increased leadership attention and  improved interagency collaboration for these goals.
We found that CAP goal teams are meeting a number of GPRAMA  reporting requirements, including identifying contributors and reporting  quarterly results and milestones. However, most of the CAP goal teams  we reviewed have not established quarterly targets for all performance  measures. In an effort to assist each team with addressing their reporting  gaps, OMB and the PIC are working to improve collection and reporting of  performance information. We found that during the first two years of goal  implementation, OMB and CAP goal teams made improvements to their  quarterly progress updates by adding new performance measures,  providing additional information on performance trends, and reporting on  new prospects for performance improvement, among other  improvements.
CAP goal teams told us that they are using milestones to track and report  progress quarterly. We generally found that the goal teams are aligning  their quarterly activities with longer-term strategies to achieve the desired  goal outcomes. Further, all of the selected CAP goal teams reported that  they are working to develop performance measures, and are at various  stages of the process. But, they are not consistently reporting on their  efforts to develop these measures. Given OMB’s, the PIC’s, and CAP  goal teams’ emphasis on developing measures that are relevant and well  defined, greater transparency is needed to track goal team’s efforts  quarterly. Because the CAP goal teams are working to implement policies  and activities that span multiple agencies, and in some cases  government-wide, it is important that the goal teams clearly communicate  the steps they are taking to develop performance measures to ensure  that the measures will be aligned with major activities and clearly  understood by contributors to the goals. With improved performance  information, the CAP goal teams will be better positioned to demonstrate  the progress that they are making, and will help ensure goal achievement  at the end of the 4-year goal period in 2018.
section_title: Recommendation for Executive Action
paragraphs: To improve the transparency of public reporting on CAP goal progress,  we recommend that the Director of OMB, working with the PIC, take the  following action:
Report on Performance.gov the actions that CAP goal teams are  taking, or plan to take, to develop performance measures and  quarterly targets.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report for review and comment to the Director  of OMB, the Director of the Office of Science and Technology Policy  (OSTP), and the Secretaries of the Departments of Commerce, Energy,  State, Transportation, and Veterans Affairs, and the Directors of the  National Science Foundation and the Office of Personnel Management,  and the Commissioner of the Social Security Administration.
On May 3, 2016, OMB staff provided us with oral comments on the report.  OMB staff generally agreed with the recommendation in the report, and  provided us with technical clarifications, which we have incorporated as  appropriate. Officials from the Departments of Energy and Commerce  provided technical clarifications, which we incorporated, as appropriate.
OSTP and the Departments of State, Transportation, Veterans Affairs, the  National Science Foundation, the Office of Personnel Management, and  the Social Security Administration had no comments on the report. The  written response from the Social Security Administration is reproduced in  appendix III.
We are sending copies of this report to the Director of OMB and the  heads of the agencies we reviewed as well as appropriate congressional  committees and other interested parties. In addition, this report is  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Selected Cross-Agency Priority (CAP) Goals and Related Sub-goals
paragraphs: Selected CAP Goal   Customer Service  Increase citizen satisfaction and promote  positive experiences with the federal  government by making it faster and easier for  individuals and businesses to complete  transactions and receive quality services.  Job-Creating Investment  Encourage foreign direct investment, spurring  job growth by improving federal investment  tools and resources, while also increasing  interagency coordination.  Lab-to-Market  Increase the economic impact of federally- funded research and development (R&D) by  accelerating and improving the transfer of  new technologies from the laboratory to the  commercial marketplace.  Open Data  Fuel entrepreneurship and innovation, and  improve government efficiency and  effectiveness by unlocking the value of  government data and adopting management  approaches that promote interoperability and  openness of these data.  People and Culture  Innovate by unlocking the full potential of the  workforce we have today and building the  workforce we need for tomorrow.
Sub-goals  1. Improve top customer service interactions  2.  Develop and implement standards, practices, and tools  3.  Feedback and transparency  4.  Focus on the frontline  1.  Promote and market the United States as the premier investment destination  2.  3.
Smarter IT Delivery  Eliminate barriers and create new incentives  to enable the federal government to procure,  build, and provide world-class, cost-effective  information technology (IT) delivery for its  citizens, and hold agencies accountable to  modern IT development and customer  service standards.
STEM Education  Improve Science, Technology, Engineering,  and Mathematics (STEM) Education by  implementing the Federal STEM Education 5- Year Strategic Plan, announced in May 2013.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: The GPRA Modernization Act of 2010 (GPRAMA) included a provision for  us to periodically report on the implementation of the act, and this report  is part of that series. The objectives of this report are to assess (1) the  extent to which lessons learned from implementing the interim cross- agency priority (CAP) goals were incorporated into the governance of the  current CAP goals; (2) the extent to which GPRAMA requirements for  reporting on CAP goal progress are included in the selected CAP goal  quarterly progress updates; and (3) the initial reported progress in  implementing the selected CAP goals. In September 2015, we reported  on lessons learned from the interim CAP goal period—which ended in  March 2014—and provided an assessment of CAP goal teams’ initial  reported progress implementing the current set of 2014-2018 CAP goals.
To conduct our assessment, we selected 7 of the 15 CAP goals for more  in-depth review. We randomly selected four of the seven CAP goals  including Open Data, STEM Education, Job-Creating Investment, and  Lab-to-Market. We also included three goals in our review because we  have completed and ongoing work related to the Customer Service,  People and Culture, and Smarter IT Delivery CAP goals. Our sample of  goals is nongeneralizable and is not representative of all CAP goals.
We interviewed Office of Management and Budget (OMB) and  Performance Improvement Council (PIC) staff responsible for the  management and implementation of the current CAP goals, as well as  agency officials and staff, including CAP goal leaders and members of the  seven CAP goal teams. We requested to meet with representatives from  the Presidential Personnel Office and National Economic Council who  have responsibility for leading the People and Culture and Job-Creating  Investment CAP goals. Staff from those offices directed us to obtain  feedback on these goals from responsible staff at OMB. We reviewed  OMB and PIC guidance, relevant documentation, our prior related work,  and quarterly progress updates published on Performance.gov from the  second quarter of fiscal year 2014 through the first quarter of fiscal year  2016, published in March 2016.
To determine the extent to which selected CAP goal quarterly progress  updates reflect GPRAMA requirements for assessing and reporting CAP  goal progress, we conducted a content analysis—using NVivo software  and Excel—of the quarterly updates from the second quarter of fiscal year  2015 through the first quarter of fiscal year 2016. Specifically, we  reviewed each quarterly progress update and coded the content into one  or more of the five major actions (as described below) to implement a  CAP goal, which we developed from an initial review of the selected  quarterly progress updates, and OMB and PIC guidance to CAP goal  teams.
Purpose - establishing the goal;
Governance - identification of leaders and contributors;
Planning - development of strategies and performance measures;
Implementation - execution of plans and use of performance  information; and
Results - reporting results on Performance.gov.
To improve the validity of results, each plan was coded independently by  two of our analysts, who then compared their analysis in three stages. In  cases where there was disagreement, the two coders met and either  came to agreement on a code, or a third coder made a judgment about  correct coding.
In the first stage of content analysis, the coders independently assigned  the content in each of the select CAP goal quarterly progress updates into  one or more of the five actions of goal implementation described above.  For the second stage of content analysis, the coders assigned the  relevant GPRAMA requirements to one or more of the five actions of goal  implementation. During the first two stages of content analysis, we  determined if an element or requirement was present in the plan (one or  more coded entries). For the third and final stage of content analysis, we  assessed the extent to which the progress update met the reporting  requirements.
To determine the extent to which the seven selected CAP goals’ quarterly  progress updates reflect GPRAMA requirements, we conducted a content  analysis of the four quarterly updates published from March 2015 through  December 2015. To fully meet each requirement, the quarterly progress  updates had to provide all information required by GPRAMA. We  analyzed 13 requirements – 6 that we determined were relevant at the  CAP goal level, and 7 that we determined were relevant at the sub-goal  level. For those requirements analyzed at the sub-goal level, we  determined a goal had partially met a requirement if the required  information was present in the quarterly progress update for at least one  of the goal’s sub-goals. For requirements that were not met, we  determined the required information was not present in the quarterly  progress update. We shared our analysis with each of the selected CAP  goal teams and interviewed agency staff to collect their feedback on our  analysis. We also collected and reviewed relevant documentation  provided by agencies, OMB, and the PIC.
One of the GPRAMA reporting requirements is for OMB, in coordination  with CAP goal teams, to identify and publish the key agencies,  organizations, program activities, regulations, tax expenditures, policies,  and other activities that contribute to each CAP goal on Performance.gov.  To determine the extent to which selected CAP goals met this  requirement, we reviewed the published lists of contributors in the  quarterly progress updates, and interviewed CAP goal teams and OMB  staff responsible for managing the CAP goals. We did not evaluate the  process that goal teams used to identify the contributors.
To assess the extent to which the selected CAP goal teams have made  progress during the first 2 years of implementation from March 2014  through December 2015, we reviewed the quarterly progress updates  published on Performance.gov and relevant agency documents,  interviewed CAP goal team staff to discuss challenges, efforts to mitigate  challenges, and key achievements. Because the scope of our review was  to examine the implementation of quarterly reporting requirements, we did  not evaluate whether these goals were appropriate indicators of  performance, were sufficiently ambitious, or met other dimensions of  quality.
We conducted our work from January 2015 to May 2016 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
section_title: Appendix III: Comments from the Social Security Administration
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: J. Christopher Mihm, (202) 512-6806 or mihmj@gao.gov.
section_title: Staff Acknowledgements
paragraphs: In addition to the contact named above, Sarah E. Veale, Assistant  Director, and Peter Beck, Analyst-in-Charge, supervised the development  of this report. Margaret Adams, Jehan Chase, Jillian Feirson, Jennifer  Felder, and Steven Putansu made significant contributions to all aspects  of this report. Lisette Baylor, Adam Cowles, Robert Gebhart, David  Hinchman, Donna Miller, Christopher Murray, Nyree Ryder Tee, and  MaryLynn Sergent provided additional assistance.