Managing for Results: Agencies Report Positive Effects of Data-Driven Reviews on Performance but Some Should Strengthen Practices
Released: 2015-07-07T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: How federal leaders manage the operations and performance of their agencies significantly affects their ability to achieve important outcomes critical to public health and safety. GAO's previous work has identified weaknesses in agencies' use of performance information that can hinder achievement of critical results.

This report is part of GAO's response to a statutory requirement to review GPRAMA implementation. It examines (1) the extent to which agencies are conducting data-driven performance reviews consistent with GPRAMA requirements, OMB guidance, and leading practices; and (2) how reviews have affected performance, collaboration, accountability, and efficiency in agencies, and how positive effects can be sustained.

GAO surveyed PIOs at 23 agencies, followed up to clarify responses, and interviewed officials involved in reviews at 5 agencies. These agencies were selected based on size and the extent to which leaders use reviews, as reported on a 2013 survey. GAO also reviewed OMB guidance and relevant documentation from agencies.

section_title: What GAO Found
paragraphs: The GPRA Modernization Act of 2010 (GPRAMA) requires that federal agencies review progress on agency priority goals (APG) at least once a quarter. GPRAMA requires that reviews be conducted by top agency leaders, involve APG goal leaders and other contributors, and be used to identify at-risk goals and strategies to improve performance. While GPRAMA requires that agencies conduct reviews, it also required the Office of Management and Budget (OMB) to prepare guidance on its implementation. Since 2011, OMB has provided guidance on how reviews should be conducted, specifying they should be held in person. Further, GAO previously identified nine leading practices for reviews.

Agencies Reported Review Practices Consistent with Requirements and Guidance. Of the 23 agencies GAO surveyed, most reported conducting data-driven reviews consistent with requirements, guidance, and leading practices. Specifically, most agencies reported:

conducting data-driven review meetings at least once a quarter, with several agencies holding them more frequently (20 agencies);

conducting Chief Operating Officer (COO)-led reviews, or reviews led jointly by the COO and Performance Improvement Officer (PIO) (19);

always or often involving PIOs (22) and APG goal leaders (21) in reviews;

always or often collecting and analyzing relevant data in advance of reviews, and incorporating these data into meeting materials (22);

always or often using review meetings to assess APG progress (20); and

always or often identifying follow-up actions to be taken after review meetings (18), an action that is positively correlated with the reported impact of reviews on agency performance improvement.

Agency Review Practices Inconsistent with Requirements and Guidance. Some agency practices were inconsistent with requirements or guidance. For instance, the Department of Homeland Security (DHS) reported that it does not hold in-person reviews, and the Departments of Agriculture (USDA) and Health and Human Services (HHS) reported that they do not hold regular, in-person reviews each quarter. The Department of State (State) reported that progress on each APG is only reviewed in an in-person review once a year, rather than each quarter, as required. The Department of Defense (DOD), USDA, and State also reported that their reviews are not led by their agency heads or COO. DOD also reported it rarely identifies follow-up actions to be taken after meetings.

Agencies Reported Positive Effects of Reviews . Most agencies reported their reviews have had positive effects on progress towards agency goals, collaboration between agency officials, the ability to hold officials accountable for progress, and efforts to improve the efficiency of operations. According to agency officials, reviews can bring together people, analytical insights, and resources to rigorously assess progress on goals or milestones, develop collaborative solutions to problems, enhance individual and collective accountability for performance, and review efforts to improve efficiency. Agencies reported that sustaining these effects requires ongoing leadership commitment, institutionalizing review processes, and demonstrating value to participants.

section_title: What GAO Recommends
paragraphs: To ensure that agency reviews are consistent with requirements, guidance, and leading practices, GAO is making recommendations to five agencies. DHS, HHS, and USDA concurred with the recommendations. DOD and State concurred with all but one recommendation—to ensure the COO leads the reviews—with which they partially concurred. GAO believes these recommendations are valid, as discussed in the report.

section_title: Letter
paragraphs: The effective operation of federal agencies has a significant effect on the  health, safety, and security of the American public, and how federal  leaders manage the operations and performance of their agencies  influences their ability to achieve important outcomes. Our previous work  has found that leadership’s active use of performance information to  guide decision making leads to better managed programs and improved  results. Our work has also found that federal agencies can use  performance information to identify and correct performance problems,  improve program implementation and organizational processes, and  make other important management and resource allocation decisions.  However, our past surveys of federal managers have identified continuing  weaknesses in the use of performance information by agencies that can  hinder their ability to achieve critical results.

To improve performance and results by increasing the use of  performance information by agency leaders and managers, Congress  included in the GPRA Modernization Act of 2010 (GPRAMA) a  requirement that agencies review progress on agency priority goals  (APG) at least once a quarter. This requirement was based on the “Stat”  model of frequent, in-person, leadership-driven reviews of performance— a model that has been widely adopted by state and local governments.

In its guidance to agencies on how to implement reviews, the Office of  Management and Budget (OMB) has emphasized that frequent, data- driven performance reviews provide a mechanism for agency leaders to:  (1) assess the organization’s performance; (2) bring together the people,  resources, and analysis needed to drive progress on agency priorities; (3)  diagnose performance problems and identify improvement opportunities  through the analysis of data; (4) identify lessons learned from past  experience; and (5) decide on next steps to increase performance and  productivity.  As OMB has also noted, these practices are designed to  shift the emphasis away from the passive collection and reporting of  performance information to a model where performance information is  actively used by agency officials to inform decision-making. The latter is  more likely to lead to performance improvements.

OMB Circular No. A-11, Preparation, Submission, and Execution of the Budget, pt. 6,  Section 270.3 (August 2014). improvements. This report builds on that earlier work, and is part of our  response to a statutory requirement that we evaluate how the  implementation of the GPRA Modernization Act of 2010 (GPRAMA) is  affecting performance management in federal agencies, and whether  performance management is being used by agencies to improve the  efficiency and effectiveness of agency programs.examines (1) the extent to which agencies are conducting data-driven  performance reviews in a manner consistent with GPRAMA requirements,  OMB guidance, and leading practices for reviews; and (2) how data- driven performance reviews have affected performance, collaboration,  accountability, and efficiency within agencies, and how positive effects  can be sustained.

To address both objectives, we conducted a survey of Performance  Improvement Officers (PIOs) at 23 executive agencies. We asked PIOs for information about the frequency of review meetings; leadership of, and  participation in, review meetings; preparation for, execution of, and follow- up on, review meetings; challenges; and perceived impacts on agency  performance, collaboration, efficiency, and accountability for results. We  received responses from all 23 agency PIOs.

To further address both objectives, we selected five case-study agencies  for a more in-depth assessment of their data-driven review processes. To  provide additional detail and illustrative examples to supplement  government-wide data collected through our survey, we used these more  in-depth reviews to gather information about agency practices and  officials’ perceptions of effects that review meetings have had. We  selected a sample of agencies that reflect a range of characteristics,  including agency size and the extent to which agency leadership uses  quarterly performance reviews to drive progress toward goals, as  reported by respondents to our 2013 Federal Managers Survey. We  selected the Departments of Commerce (Commerce), Health and Human  Services (HHS), and Transportation (DOT); the General Services  Administration (GSA); and Social Security Administration (SSA). At each  agency, we selected at least two agency priority goals (APG) to obtain the  perspective of APG leaders and their staff on the agency’s data-driven  review process.

To allow us to corroborate information collected through surveys,  interviews, and observations, and strengthen our confidence in the  reliability of the self-reported survey responses, we requested supporting  documents from 12 agencies, representing more than 50 percent of the  agencies surveyed, to verify survey responses related to review meeting  frequency, leadership, participation, content, and follow-up. The 12  agencies included the 5 agencies selected for more in-depth review,  several agencies whose survey responses required clarification, and  several additional agencies selected at random. Examples of documents  submitted by agencies included review meeting attendance or invitations,  agendas, presentation slides, and briefings and summary reports.  Through the survey, and subsequent follow-up, we learned that the  Department of Homeland Security (DHS) does not hold the OMB- required, in-person review meetings, and has not done so since  December 2013. For this reason, the summaries of survey responses in  this report exclude DHS, with the exception of table 2, which describes  the frequency of review meetings at each agency.

GPRAMA established the position of agency Chief Operating Officer (COO) and  required that the deputy head of the agency, or equivalent—such as a Deputy Secretary— serve in this role. The function of the COO is to: 1) provide overall organization  management to improve agency performance and achieve the mission and goals of the  agency through the use of strategic and performance planning, measurement, analysis,  regular assessment of progress, and use of performance information to improve the  results achieved; 2) advise and assist the head of agency in carrying out the performance  planning, reporting, and review requirements of GPRAMA; 3) oversee agency-specific  efforts to improve management functions within the agency and across government; and  4) coordinate and collaborate with relevant personnel within and external to the agency  who have a significant role in contributing to and achieving the mission and goals of the  agency. 31 U.S.C. § 1123. performance staff to periodically come together to discuss performance  review practices in their agencies.

We observed agency review meetings at HHS, GSA, and SSA. We also  observed one sub-agency-level review meeting at GSA. Observing review  meetings allowed us to gain firsthand knowledge of how the meetings are  conducted in these agencies. This provided context, increased our  familiarity with the process, and corroborated information gained through  other means. While we requested to observe a review meeting at both  Commerce and DOT, we were not allowed to do so. The agencies cited  concerns that our presence could inhibit open discussion.

Lastly, to address the first objective, we compared what we learned about  the review processes at all 23 agencies with requirements for review  meetings established in GPRAMA, as well as standards set forth in OMB  guidance and leading practices for data-driven reviews we previously   To address the second objective, we analyzed our survey  identified.data to determine how agencies characterized the effects of their review  meetings. We also used interviews with officials and documentation from  our selected agencies to identify illustrative examples of the effects  review meetings have had, and to identify actions they have taken to  sustain the positive effects of the reviews.

We conducted our work from July 2014 to July 2015 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusion  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
subsections: section_title: Background
paragraphs: Data-driven performance reviews are regularly scheduled, structured  meetings used by organizational leaders and managers to review and  analyze data on progress toward key performance goals and other  management-improvement priorities. They are generally used to target  areas where leaders want to achieve near-term performance  improvements, or to accelerate progress through focused senior  leadership attention.

Over the past several years, Congress and the executive branch have  taken steps to improve federal performance management by requiring  that agencies conduct regular data-driven review meetings. In 2010, OMB  released a memorandum establishing the expectation that federal  agencies would hold data-driven reviews at least once every quarter to  review progress on their priority goals and assure that follow-up steps  would be taken to achieve improved outcomes. The memorandum  specified that discussions during these meetings were to be guided by  analyses of performance data, to focus on progress toward desired  outcomes, to explore why variations between targets and actual outcomes occurred, and to prompt adjustments when needed.

Congress, through the passage of GPRAMA, made the expectation that  agencies would hold regular data-driven reviews a statutory requirement.  Specifically, GPRAMA requires that, not less than quarterly, the head of  each agency and COO, with the support of the PIO, should review  progress on agency priority goals (see text box).

GPRAMA Requirement for Quarterly Priority Progress Reviews   GPRAMA requires that, not less than quarterly, at all agencies required to develop agency  priority goals, the head of the agency and Chief Operating Officer, with the support of the  agency Performance Improvement Officer, shall:   For each agency priority goal, review with the appropriate goal leader the progress  achieved during the most recent quarter, overall trends, and the likelihood of meeting  the planned level of performance;

Coordinate with relevant personnel within and outside the agency that contribute to  the accomplishment of each agency priority goal;

Assess whether relevant organizations, program activities, regulations, policies, and  other activities are contributing as planned to the agency priority goals;

Categorize agency priority goals by risk of not achieving the planned level of  performance; and  For agency priority goals at greatest risk of not meeting the planned level of  performance, identify prospects and strategies for performance improvement,  including any needed changes to agency program activities, regulations, policies, or  other activities.

While GPRAMA established requirements for agencies to conduct the  reviews, GPRAMA also required that OMB prepare guidance on the  implementation of GPRAMA. In 2011, OMB released guidance for federal  agencies that reinforced the requirements in GPRAMA, specified that the  reviews should be held in person, and outlined the specific purposes of  the data-driven review meetings, the roles and responsibilities of agency  leaders involved in the review process, and how the reviews should be  conducted. In 2012, OMB released updated guidance for data-driven  reviews.throughout this report.

GAO-13-228. and federal level who shared their experiences and lessons learned.These practices, along with additional insights on why the application of  these practices is important, are noted and summarized throughout this  report.

Nine Leading Practices Identified by GAO That Can Be Used to Promote Successful  Data-Driven Performance Reviews

Reviews are conducted frequently and regularly.

Leaders use data-driven reviews as a leadership strategy to drive performance  improvement.

Key players attend reviews to facilitate problem solving.

Rigorous preparations enable meaningful performance discussions.

There is capacity to collect accurate, useful, and timely performance data.

Staff have skills to analyze and clearly communicate complex data for decision  making.

Reviews ensure alignment between goals, program activities, and resources.

Leaders hold managers accountable for diagnosing performance problems and  identifying strategies for improvement.

Participants engage in rigorous and sustained follow-up on issues identified during  reviews.

Taken together, the GPRAMA requirements, OMB guidance, and leading  practices identify the elements necessary to carry out effective data- driven reviews: (1) those that are used to engage agency leaders in the  rigorous assessment of agency performance; (2) support faster and better  informed responses to identified performance problems; (3) improve  communication and collaboration across an agency; and (4) enhance  individual and collective accountability for improving progress toward  agency goals.

section_title: Most Agencies Reported Conducting Data-Driven Reviews Consistent with Requirements, Guidance, and Leading Practices
subsections: section_title: Nearly All Agencies Reported Holding Regular Data-Driven Review Meetings at Least Quarterly
paragraphs: GPRAMA Requirement: Quarterly Reviews   GPRAMA requires that agency leaders conduct reviews on progress toward agency  priority goals (APGs) not less than quarterly.

OMB Guidance: Quarterly Reviews  OMB guidance directs agency leaders to run data-driven performance reviews on each of  their APGs at least quarterly.  This guidance also stresses that reviews should be  conducted  in person, as significant experience in federal agencies, states, localities, and  other countries demonstrates that in-person engagement of senior leaders greatly  accelerates learning and performance improvement.

Leading Practice for Data-Driven Reviews: Frequency and Regularity  Data-driven review meetings should be frequent and regularly scheduled.

Data-driven performance review meetings that are held frequently and  regularly help foster a culture of active and ongoing performance  management, problem solving, and continuous improvement. As OMB  has noted, the purpose of conducting performance reviews at least  quarterly is to ensure that agency leaders regularly review agency  performance on top priorities, along with the short- and long-term actions  agencies are taking to improve performance, and bring together the  people, resources, and analysis needed to drive progress on priority  goals.

Of the 23 CFO Act agencies surveyed, 20 agencies reported that they  hold data-driven review meetings at least quarterly, with some agencies  holding them more frequently. See table 2 for a summary of the frequency  with which each agency holds in-person review meetings to, among other  things, review progress on APGs. As shown in the table, the Department  of Homeland Security (DHS) does not hold the required in-person review  meetings.

The five case-study agencies we selected for more in-depth review – the  Departments of Commerce (Commerce), Health and Human Services  (HHS), and Transportation (DOT); General Services Administration  (GSA); and Social Security Administration (SSA) – all hold in-person  review meetings involving agency leaders and APG goal leaders at  different frequencies. This reflects differences in leadership preferences  and organizational structures and processes. See appendix II for more  detailed information on the approach used by each of the five selected  case-study agencies.
subsections: section_title: A Few Agencies Reported Conducting Less Frequent in- Person Reviews
paragraphs: OMB guidance is clear that reviews should be held in person to bring  together senior leaders and officials involved in all levels of program  delivery. This can help ensure coordination across agency silos and  enable rapid decision making. This guidance states that while written  communication may replace in-person review meetings in rare  circumstances, it should only be a stopgap measure to continue  performance reviews in a process that otherwise operates primarily in  person.

A few agencies, including the Department of Agriculture (USDA), DHS,  and HHS, reported that they do not hold in-person reviews of progress on  APGs at least quarterly as called for in GPRAMA and OMB guidance.  The Department of State (State) reported that agency officials participate  in one data-driven review meeting each quarter; however, each meeting  is used to review progress on only one APG.

Agriculture. According to USDA officials, the Deputy Secretary  meets weekly with officials and staff from the Office of Budget and  Program Analysis (OBPA), including the PIO, to discuss budget  and regulatory issues, which also provides opportunities to  discuss APG progress and performance-related issues. USDA  officials also told us that written updates on APGs and  performance data are provided quarterly, and that the Deputy  Secretary and PIO meet as necessary to review progress toward,  and discuss issues related to, specific APGs. However, these are  not regularly scheduled meetings. In addition, they told us that  staff from the OBPA have frequent conversations with program  officials as part of the review of regulatory documents, funding  availability notices, executive budget documents, and other  related documents so they have not had separate, regularly  scheduled meetings to discuss progress toward the APGs.  However, in subsequent discussions with USDA officials, they  informed us that they intend to begin holding regularly scheduled  quarterly meetings led by the COO and involving senior USDA  leadership, as directed by OMB in Circular A-11 guidance.

Homeland Security. DHS reported that in-person review  meetings ceased due to competing priorities and demands at the  end of 2013, when a change in leadership brought alternative  management priorities. Agency leaders continue to review written  performance updates quarterly. According to a DHS official, a  meeting involving the Deputy Secretary and APG goal leaders to  review goal results from fiscal year 2015 is being scheduled.

Health and Human Services. HHS leaders hold in-person review  meetings for each APG twice a year and review APG progress  two more times a year through reviews of written progress  updates. Officials from HHS said that, due to the longer-term  nature of the agency’s APGs, performance data that are tracked  for each goal show little meaningful change from quarter to  quarter, so agency officials have not considered meeting quarterly  to be an effective use of participants’ time. One HHS official said  that managers convene meetings with their program teams more  frequently to track progress on efforts contributing to each goal.

State. Each year, State holds one joint, in-person meeting to  review progress on the APG to support the implementation of low  emission development strategies, which it co-leads with the U.S.  Agency for International Development (USAID). State also holds  one meeting each year to review progress on the APG to improve  consular service delivery. State officials attend three other reviews  on APGs led by USAID held throughout the year. Therefore,  progress on each APG is only reviewed by officials from State in  an in-person review meeting once a year. A State official stated  that the performance measures used to track progress on the two  APGs for which the agency either leads or co-leads show little  meaningful change from quarter to quarter, and believes it would  not be beneficial for stakeholders to attend meetings more often  than annually. The official also said, however, that the agency’s  PIO reviews the data and written updates provided by APG goal  leaders each quarter, and updates the Deputy Secretary.

As GPRAMA requirements and OMB guidance are clear that in-person  review meetings should be held at least once a quarter, and that progress  on each APG should be reviewed each quarter in these meetings, the  approaches of these four agencies – DHS, HHS, State, and USDA – are  not consistent with requirements for the frequency and expected  characteristics of reviews. Furthermore, OMB guidance states that these  reviews should not be conducted through written documents, and that  agency leaders should use performance review meetings as an  opportunity to engage those involved in all levels of program delivery. The  lack of frequent, regular, in-person review meetings could result in missed  opportunities for leaders and key officials at these four agencies to have  regular, in-depth discussions of performance on top agency priorities.  Such meetings could also allow them to actively promote ongoing  coordination and accountability, address identified challenges or  problems in a timely manner, and encourage continuous improvement in  agency performance and operations.

As OMB guidance also clarifies, APGs are defined as a “near-term” result  or achievement that agency leaders want to accomplish within  approximately 24 months through focused leadership attention. While the  guidance states that APGs can advance progress toward longer-term,  outcome-focused strategic goals and objectives, the APGs are designed  to be near-term improvements in outcomes, customer service, or  efficiency. Even in those instances when new quantitative performance  data are not available for review in meetings, more frequent in-person  reviews still provide the opportunity to review goal leader progress in  completing shorter-term milestones or initiatives contributing to progress  on the goals, and promptly address any identified problems. In fact,  GPRAMA and OMB guidance both state that agencies should have  clearly-defined, quarterly milestones to track progress on their APGs.

section_title: Nearly All Agencies Reported Their Reviews Are Led by the Chief Operating Officer
paragraphs: GPRAMA Requirement: Leadership   GPRAMA requires that the agency head and Chief Operating Officer (COO) conduct  reviews with the support of the Performance Improvement Officer (PIO).

OMB Guidance: Leadership  OMB guidance directs that the agency head and/or COO must conduct reviews with the  support of the PIO and the PIO’s office.

Leading Practice for Data-Driven Reviews: Leadership  Agency leaders should be directly and visibly engaged in the reviews.

According to OMB, significant experience at federal agencies, states,  localities, and other countries demonstrates that in-person engagement of  senior leaders in review meetings greatly accelerates learning and  performance improvement. The personal engagement of agency leaders  in the review meetings also demonstrates their commitment to  improvement across the agency and, as mentioned above, facilitates  coordination across agency silos and rapid decision making. As OMB has  also noted, frequent, data-driven reviews also send a signal throughout  the organization that agency leaders are focused on effective and efficient  implementation to improve the delivery of results. GPRAMA recognized  that the direct involvement of leaders is a critical factor to drive  performance improvement within an agency. Thus, it requires that agency  heads and COOs conduct the reviews. OMB’s guidance for agencies on  how to conduct the reviews also emphasized the importance of  leadership involvement in data-driven performance reviews, directing the  agency head, COO, or both to conduct the review. As we have previously  reported, the commitment of agency leaders to make decisions and  manage programs on the basis of performance information, and inspire  others to embrace such a model – which review meetings can be used to  do – is critical to increase the use of performance information throughout  an agency.

We found through our survey that 19 of 22 agencies that held review  meetings reported that the meetings were led by their agency head or  COO, or jointly by the COO and PIO. See table 3 below for specific  agency responses. Furthermore, we found through our survey that review  meetings are used as a tool to enhance the engagement of top agency  leadership in an agency’s performance management process. In fact, all  22 agencies reported that their reviews have had a positive effect on the  engagement of top agency leadership in the agency’s performance  management process, with 13 reporting a large positive effect.

Although their exact roles varied, officials from our five selected case- study agencies reported that agency heads or COOs were actively  involved in their agency’s review processes, and led the meetings through  the following activities: focusing on agency priorities and directly  communicating their expectations, asking questions, reinforcing individual  and collective accountability, encouraging collaboration, offering  assistance with problem solving or identifying available resources, and  sharing perspectives from discussions with external stakeholders. For  example, during the review meeting we observed at GSA, the  Administrator and other leaders engaged in the discussion by challenging  assumptions about the status of goal progress, asking questions about  factors driving changes in specific performance measures, and  encouraging goal leaders to identify areas of risk and plans for  addressing challenges.

Some case-study agency officials we interviewed stated that a key role of  agency leaders in the meetings is to set a positive example by  demonstrating their commitment to, and involvement in, agency  performance management processes, and by communicating that  participation in reviews is a priority. For example, at SSA, the agency  head personally presided over bi-monthly review meetings and created a  new office (the Office of the Chief Strategic Officer) to support expanded  performance management and data analysis efforts. According to SSA  officials we spoke with, in one review meeting the agency head brought  focused attention from across the agency on a priority goal that was  showing insufficient progress. She convened a follow-up meeting  requesting that offices throughout SSA articulate how they would  contribute to progress on the goal. This proved to be a useful technique  for establishing a broader sense of accountability for contributions to the  goal and helped identify new strategies to improve progress.

Less than half of the agencies (8 of 22) reported in our survey that getting  or sustaining the participation of top agency leadership in the reviews was  a challenge. However, we heard of instances in which logistical  challenges can make it difficult for the agency head or COO to participate  in each review meeting. For example, a Department of Transportation  (DOT) official reported that leadership participation at the review  meetings, which are held separately with representatives of each of  DOT’s 10 operating administrations each quarter, was a moderate  challenge because key leaders may not be able to attend due to last- minute scheduling changes or conflicts. The agency has decided that in  these situations they will continue with scheduled meetings with other  leadership team members, such as the General Counsel of DOT, leading  the meeting. Officials at DOT said that ensuring the review meetings are  held regularly is important because it avoids logistical challenges  presented by rescheduling meetings, helps minimize preparation time by  ensuring staff do not have to recreate meeting materials, encourages  attendance, and leads to more productive discussions because staff are  assured they will have a regular opportunities to raise issues for high- level attention.
subsections: section_title: A Few Agencies Reported Their COOs Do Not Lead Reviews
paragraphs: USDA, the Department of Defense (DOD), and State reported in our  survey that the agency head or COO does not lead meetings that are  used to review progress on APGs, as specified by GPRAMA and OMB  guidance.

Agriculture. In its survey response and additional follow-up  communication, USDA reported that the meetings between the  Deputy Secretary and PIO that are held to discuss APG progress  are led by the PIO, who presents information to the Deputy  Secretary in these meetings.

Defense. DOD reported that its APGs are reviewed in meetings of  the Defense Business Council (DBC), which has responsibility for  the development and review of DOD’s performance goals. DBC  meetings, however, are led by the Deputy Chief Management  Officer, who is the PIO of DOD, rather than the Deputy Secretary  of Defense, who is the COO. According to meeting attendance  lists shared by DOD, neither the agency head nor Deputy  Secretary/COO lead or regularly attend these reviews.

State. State reported that its PIO leads the review meeting for the  one APG that State leads, and co-leads the review meeting with  USAID for the one APG that is shared by the two agencies. A  State official explained that the agency feels the PIO is  appropriately suited for the role as leader of the review meetings  as she also serves as the agency’s senior budget official. This  dual role, according to the official, allows her to integrate  performance with knowledge of agency resources. Neither the  agency head nor the Deputy Secretary/COO lead or regularly  attend these reviews.

These practices, however, are not consistent with OMB guidance, which  clearly states that agency heads or COOs should conduct in-person  meetings used to review progress on APGs. Leading practices  emphasize that having leaders actively engaged in the reviews helps  ensure that participants take the reviews seriously. As OMB has similarly  noted, the involvement of COOs is critical to bringing a broader set of  actors together to solve problems across the organization. Therefore,  because the agency head or COO does not lead review meetings at  these three agencies, the review process may be viewed as less of a  priority by agency officials. This could have a detrimental effect on  participation in reviews. It could also reduce opportunities for top agency  leaders to reinforce responsibility and accountability, and to personally  communicate their priorities and perspective to agency managers and  staff.

section_title: Nearly All Agencies Reported That Performance Improvement Officers, Goal Leaders, and Other Key Contributors Participate in Reviews
paragraphs: GPRAMA Requirement: Participation of Priority Goal Leaders and Other Relevant  Personnel  GPRAMA requires that agency leaders include agency priority goal (APG) leaders in their  reviews and coordinate with other relevant personnel within and outside the agency that  contribute to the accomplishment of each APG.

OMB Guidance: Participation of Priority Goal Leaders and Other Relevant  Personnel  OMB guidance reinforces this requirement by requiring that agency leaders include APG  goal leaders, or their designees, in the reviews, along with, as appropriate, relevant  personnel within and outside the agency who contribute to the accomplishment of each  APG.

Leading Practices for Data-Driven Reviews: Participation by Key Personnel   Reviews should include personnel with programmatic knowledge and responsibility for the  specific performance issues being discussed. In addition, the participation of officials with  functional management responsibilities, such as information technology, financial  management, and human capital, can facilitate problem solving by providing managers  from across the agency with a forum to communicate with each other.

When officials from various offices and levels of management participate  in review meetings, the meetings provide opportunities to have honest,  informed discussions about performance with all key players present, and  facilitate collaboration and group problem solving. Officials representing  their program or area of responsibility may also feel increased  accountability for results when forced to report on progress in front of  leadership and peers.

Survey responses show that participation of PIOs in review meetings is  strong, with 20 of 22 agencies reporting that PIOs always attend review  meetings, and 2 reporting that their PIOs often attend. See figure 1 for  reported frequency of participation in review meetings by agency  leadership and other key contributors. As the highest officials dedicated to  managing agency-wide performance management efforts, PIOs hold a  unique position within their agencies and are key participants in the  review meetings. PIOs and agency performance staff also engage in a  variety of activities that directly support successful review meetings.  Through discussions with agency officials and survey results, we found  that responsibilities of PIOs and performance staff may include  overseeing preparations for review meetings, including the collection and  analysis of data, creation of presentation materials, and convening  preparatory meetings; co-leading review meetings; and managing follow- up on action items identified in review meetings.

Participation by APG goal leaders in review meetings is also strong.  Twenty-one out of 22 agencies reported that their goal leaders always or  often participate in review meetings. Through our discussions with goal  leaders we learned that they also play a key role in the review meetings,  and present information on progress toward goals, respond to questions  from agency leaders, identify problems or challenges and propose  strategies to address them, and request support or assistance.

Most agencies also reported that other key officials with responsibility for  agency financial management, human capital, information technology,  and legal matters attend their review meetings. While there was variation  across agencies on the frequency with which these officials participate in  review meetings, 11 agencies reported that their Chief Financial Officers  (CFO), Chief Human Capital Officers (CHCO), Chief Information Officers  (CIO), Chief Acquisition Officers (CAO), and representatives from their  Office of General Counsel (OGC) always or often attend the reviews.  Officials from three of our five selected case-study agencies discussed  the benefits of including chief officers in their reviews. These benefits  include providing a cross-cutting agency perspective and specialized  expertise to inform decisions, and offering assistance, resources, and  problem solving support. For example, one DOT official described how  the discussions in review meetings often focus on regulations that are  under review by the Office of Information and Regulatory Affairs (OIRA) at  OMB, which, in some instances, reviews regulations before they can be  finalized. According to DOT officials, in the department’s review meetings,  officials discuss the progress and plans of rulemakings with the  Secretary’s office or OGC, such as facilitating early engagement with  OIRA to address analytical issues. According to DOT officials, this is  significant because the issuance of rules is an important tool that the  department uses to promote progress toward its APGs. For instance, in  its reporting on progress toward the APG to reduce the rate of roadway  fatalities, DOT identified a number of proposed and final rules designed to  reduce the risk of fatalities and serious injuries through enhancements to  the safety of vehicles and roadways.
subsections: section_title: Two Agencies Reported Limited Participation in Review Meetings
paragraphs: USDA and State provided responses indicating that participation in their  reviews is not fully consistent with requirements, guidance, and leading  practices.

Agriculture. USDA responded to our survey that APG goal  leaders participate in review meetings about half of the time.  Through follow-up communication, USDA officials clarified that  meetings between the Deputy Secretary and PIO in which APG  progress is reviewed generally do not involve goal leaders.  Officials also said, however, that when the Deputy Secretary had  specific questions on APG progress, the Office of Budget and  Program Analysis (OBPA) would schedule a follow-up meeting  attended by the Deputy Secretary, PIO, goal leaders, and,  occasionally, performance staff. USDA officials also reported that  their CFO, CHCO, CIO, and CAO are rarely involved in the  meetings, and that the General Counsel never attends. In  subsequent follow-up communication, USDA officials stated that if  additional information or action is needed from administrative,  program, or policy officials, then the PIO and OBPA staff will act  as a liaison, relaying questions and information between these  officials and the Deputy Secretary. USDA officials said that given  USDA’s large size and the complex and diverse nature of its  multiple missions, it is generally easier logistically to have the PIO  meet with the Deputy Secretary, rather than trying to schedule a  meeting involving additional senior officials.

State. State responded to our survey that the CFO, CHCO, CIO,  CAO, and General Counsel never attend review meetings. Upon  subsequent follow-up with State officials, they could not provide  an example of when these officials had been invited or attended  review meetings, but said that they plan to invite officials with  functional management responsibilities as appropriate in the  future.

Not involving APG goal leaders in regular reviews of goal progress is  inconsistent with GPRAMA requirements and OMB guidance; by not  doing so, USDA may be missing opportunities for direct communication  between agency leaders and relevant program staff about progress,  challenges, and strategies for improvement. In addition, by not regularly  including officials with functional management or legal expertise, as  leading practices suggest, USDA and State may also miss opportunities  to address performance issues in which human capital, information  technology, acquisitions, or legal expertise could play a significant role in  the development of solutions.

section_title: Agencies Reported Little External Involvement in Reviews
paragraphs: As we reported in our earlier evaluation of agency performance review  meetings, OMB guidance and leading practices indicate that including key  players from other agencies can lead to more effective collaboration and  goal achievement. Specifically, OMB guidance states that agencies  should include, as appropriate, relevant personnel from outside the  agency who contribute to the accomplishment of an APG or other priority.  When key players are excluded from performance reviews, agencies may  miss opportunities to have all the relevant parties apply their knowledge  of the issues and participate in developing solutions to performance  problems. Instead, agencies will need to rely on potentially duplicative  parallel coordination mechanisms, which could result in less than optimal  performance improvement strategies.

Only two agencies, State and USAID, reported that they always or often  include officials from outside the agency in their review meetings. These  are also the two agencies that hold joint sessions to review progress on  their shared APG. Most agencies, however, reported that external  contributors never participate in their reviews.

In February 2013, we recommended that OMB work with the PIC and  other relevant groups to identify and share promising practices to help  agencies extend their performance reviews to include, as relevant,  representatives from outside organizations that contribute to achieving  their agency performance goals. OMB generally concurred with the  recommendation, and in July 2014, staff from OMB and the PIC told us  that meetings of the PIC Internal Reviews Working Group have been  used to discuss the inclusion of representatives from external  organizations in agency reviews, as appropriate. In March 2015, OMB  staff said that while they have found that at times it is useful to engage  external stakeholders in improving program delivery, agencies view  reviews as internal agency management meetings. Thus, they believe it  would not always be appropriate to regularly include external  representatives. According to PIC staff, the PIC continues to work with  agencies to identify examples where agencies have included  representatives from outside organizations in quarterly reviews, and to  identify promising practices based on those experiences. As those  promising practices are identified, PIC staff plan to disseminate them  through the PIC Internal Reviews Working Group and other venues. We  will continue to monitor these efforts and periodically report on their  status.

section_title: All Agencies Reported Collecting and Analyzing Data for Reviews
paragraphs: GPRAMA Requirement: Review of Quarterly and Trend Data on Priority Goal  Progress  GPRAMA requires that participants review, for each APG, progress achieved during the  most recent quarter, overall trend data, and the likelihood of meeting the planned level of  performance.

OMB Guidance: Review of Quarterly and Trend Data on Priority Goal Progress   OMB guidance reinforces this requirement by directing participants to review progress  achieved during the most recent quarter, overall trend data, and the likelihood of meeting  the planned level of performance. It also says that, in the reviews, agency leaders should  hold goal leaders accountable for knowing the quality of their data, for having a plan to  improve it if necessary, and for filling critical evidence or other information gaps.

Leading Practice for Data-Driven Reviews: Collecting and Analyzing Performance  Data  Participants in a data-driven review meeting must have up-to-date, accurate data on  performance to have a meaningful discussion about progress toward goals and  milestones. The capacity to collect relevant and timely data and the ability to analyze it to  identify key trends, areas of strong or weak performance, and possible causal factors are  critical to successful reviews.

As we have previously reported, the capacity to collect and analyze  accurate and useful data is critical to successful data-driven reviews.  The collection and analysis of valid, up-to-date performance data in  advance of data-driven review meetings is necessary to ensure that the  most timely data and information are used to inform discussions in  meetings, and that key trends or areas of strong or weak performance  have been identified. The collection and analysis of up-to-date data for  review meetings is also necessary because GPRAMA and OMB guidance  require that reviews be used to review progress toward APGs.

All 22 agencies reported that they always or often collect data on APG  performance measures and milestones in advance of their review  meetings. Furthermore, all 22 agencies reported that they always or often  analyze these data to identify key performance trends or patterns and  areas of strong or weak performance. See figure 2 below for information  on the frequency with which agencies reported that they take specific  data collection and analysis actions prior to their review meetings.

All five of our selected case-study agencies established processes for  collecting and analyzing performance data in advance of their review  meetings. At each of the agencies, officials told us that those managing  the preparation for review meetings collect updated performance data  from goal leaders and their staffs. Some agencies used a standardized  template to collect and organize the performance data and other relevant  information about progress toward goals and milestones such as risks,  challenges, and future actions. GSA also used an online spreadsheet that  offices were required to regularly update with new information on  progress toward specific agency goals or milestones. See figure 3 for a  screenshot of this spreadsheet.

Collecting accurate and timely data is critical for successful performance  reviews, but our survey found that 19 of 22 agencies identified this as a  challenge. This finding is consistent with our previous survey of agency  PIOs, as well as past surveys conducted by the PIC, which found that the  primary challenges agencies faced when implementing reviews included  access to data and limitations in the capability of their data systems. It  appears, however, that the attention and scrutiny data receive through the  review process can help agencies identify and address problems or  limitations. In fact, 20 of 22 agencies reported that their reviews have had  a positive impact on the quality of the performance data used to track  progress and inform decision making within their agencies. According to  OMB staff, in February 2015, OMB and the PIC also formed a cross- agency working group on data quality comprised of agency and OMB  representatives. The stated objectives of the working group, which will  meet through August 2015, are to identify guidelines and practices that  would improve the reliability and quality of performance data and the  reporting process, and establish standards and consistency across the  federal government. We are assessing the quality of publicly reported  information on APGs in selected agencies and plan to discuss this cross- agency working group in more detail in an upcoming report in the summer  of 2015.

Officials from some of our five selected case-study agencies described  the actions they have taken to address challenges presented by lagging  or limited performance data. For example, to track progress on the  reduction of improper payments, the goal leader for SSA’s improper  payments APG previously received relevant data annually. To increase  the frequency with which new data are available, the PIO initiated a  conversation on increasing the frequency with which payment accuracy  data is received and the goal leader worked with the SSA Office of  Quality Review, which collects the data, to increase the frequency to  every 6 months. An SSA official said that having more current data has  given the agency a better indication, at an earlier stage, of its progress in  a given year and of any impacts its actions may be having. At HHS, a key  indicator tracked for the HHS early childhood education APG is the  number of states with Quality Rating and Improvement Systems that meet  seven benchmarks. The data for this indicator, however, are only  available annually. Because more frequently updated data are not  available, HHS officials asked the goal leader to disaggregate the data by  geographic region to allow for a more granular examination of conditions  and trends across regions. The Office of Child Care analyzed state  progress toward implementing Quality Rating and Improvement Systems  that met the seven benchmarks. As part of the analysis, the Office of  Child Care identified the most common gaps in the states, and created a  map that provided a visual representation of state progress toward the  goal.

Officials from two of our five selected case-study agencies also reported  specific challenges related to their capacity to perform data analysis to  inform performance management. As we have previously reported, this  helps ensure performance information is analyzed and communicated   For example, an SSA official  effectively, and used in a meaningful way.said that the agency had insufficient analytical capacity to perform deep,  detailed analysis of data on the use of SSA services and the relationships  between the use of these services and other factors. To address this  limitation, SSA created an Office of Performance Management and  Business Analytics to collaborate with other SSA offices to gather and  analyze agency data, and to perform complex data analyses. SSA also  facilitates initiatives like an internal training program where senior data  analysts train other staff. The agency is also seeking to hire an advanced  data scientist.

In April 2013, we reported on the importance of ensuring that agency  performance management staff have sufficient capacity to support  performance management in federal agencies, and recommended that  the Director of the Office of Personnel Management (OPM), in  coordination with the PIC and the Chief Learning Officer Council, work  with agencies to identify competency areas needing improvement within  agencies, and identify training that focuses on needed performance   OPM and OMB staff agreed with this  management competencies.recommendation. In July 2014, OPM told us that it had coordinated with  the PIC on this recommendation, and that the PIC would take  responsibility for the remaining actions needed to implement this  recommendation. In March 2015, OMB and PIC staff said that the PIC  has created a number of training programs designed to provide agency  officials with information about performance management, and  approaches for using performance management to improve agency  performance. The PIC has also created a public website,  LearnPerformance.gov, with informational resources on a range of topics,  including measurement, data and analysis, and reporting and  communicating performance information. We will continue to monitor  these efforts as training and other knowledge sharing efforts are  implemented and expanded.

section_title: All Agencies Reported Developing Materials and Many Reported Holding Preparatory Sessions for Reviews
paragraphs: GPRAMA Requirement: Identifying “At Risk” Goals   GPRAMA requires that agencies categorize agency priority goals (APGs) by risk of not  achieving the planned level of performance.

OMB Guidance: Identifying “At Risk” Goals  OMB guidance also directs agencies to identify APGs (or other priorities) at risk of not  achieving the planned level of performance and work with goal leaders to identify  strategies that support performance improvement. It also directs them to review variations  in performance trends across the organization and delivery partners, identify possible  reasons for the variance, and understand whether the variance points to promising  practices or problems needing greater attention.

Leading Practice for Data-Driven Reviews: Rigorous Preparation  Rigorous preparation is critical for effective performance reviews, as key participants must  be prepared to discuss issues related to their performance and progress toward goals.  After data have been collected and analyzed, they must be effectively communicated to  participants.

Following the completion of data collection and analysis, offices  responsible for supporting reviews will often compile summary materials  to help leaders and participants prepare for the reviews. These efforts to  ensure that participants are aware of the status of goals and milestones,  and key questions likely to be raised and discussed in the meetings, can  also be critical to the success of reviews. As we have also previously  reported, frequent and regular communication of performance information  is also critical to remind agency officials of their commitment to achieve  the agency’s goals, and to keep those goals in mind as they pursue their  day-to-day activities. It also helps ensure that leaders and managers have  opportunities to review information in time to take action to make  improvements.

All 22 agencies reported that they always or often develop presentation  slides or other meeting materials to communicate key data and analyses  to participants. Furthermore, all 22 agencies also reported that they  always or often distribute these materials to participants for review before  the meetings. See figure 4 below for information on how frequently  agencies report they take specific actions to prepare for review meetings.

All five of our selected case-study agencies developed presentation  slides, or other meeting materials, and distributed them to participants in  advance of their review meetings. In addition to presenting information on  progress toward agency goals and milestones, meeting materials may  also include discussions of key strategies and initiatives being employed  to influence progress, and any risks, challenges, or opportunities those  managing the goals are facing. In accordance with the GPRAMA and  OMB requirement that agencies categorize APGs by risk of not achieving  the planned level of performance, materials produced for meetings at all  five of our selected agencies included information or color-coded graphics  to indicate the likelihood a goal will be achieved and whether a goal is “off  track” or “at risk.” For two examples of materials prepared for review  meetings at SSA and HHS, see interactive figures 5 and 6.

Fourteen of 22 agencies reported that they always or often held a  preparatory session to review the agenda, data, and key discussion  points with participants before their review meetings. Officials from our  five selected case-study agencies described preparatory meetings that  officials from their agencies hold in advance of review meetings. Officials  from some of the five agencies also described how these preparatory  sessions can be valuable, as they allow agency leaders and goal leaders  to familiarize themselves with the data and discuss responses to potential  questions with knowledgeable staff.

General Services Administration. Two of the agency’s bureaus,  the Public Building Service and the Federal Acquisition Service,  hold regular meetings where managers from each service review  and discuss performance data presented later at the agency-level  performance review meetings. Officials see these meetings as not  only preparation for the agency-level review meetings, but as vital  to effectively managing the business of the services and making  progress toward identified goals.

Social Security Administration. To prepare for the quarterly  review meetings with the Acting Commissioner of SSA, the Chief  Strategic Officer/PIO meets with SSA’s deputy commissioners,  goal leaders, and appropriate staff to discuss progress toward  APGs, the status of efforts being employed to achieve them, and  the order in which goals should be discussed in the quarterly  review. This preparatory meeting is held 10 days before the  quarterly review meeting is scheduled to be held. Five days before  the quarterly review meeting, the Chief Strategic Officer/PIO  meets with the Acting Commissioner to prepare for the quarterly  meeting. At this meeting, they discuss goal progress and trends,  issues to be discussed during the review meeting, and potential  questions the Acting Commissioner could ask. Materials prepared  by the APG goal teams for the quarterly review are sent to the  Acting Commissioner 48 hours in advance of this preparatory  meeting.

Transportation. An official from the Federal Railroad  Administration (FRA) explained that, prior to FRA’s review  meetings with the Deputy Secretary of Transportation, staff hold  briefings for the FRA Administrator and each Associate  Administrator. This official said that these preparatory meetings for  FRA leadership are valuable because they allow FRA leadership  to ask questions of knowledgeable staff to better understand the  data and information they will ultimately present at the review  meeting with DOT leadership.

section_title: Most Agencies Reported Using Review Meetings to Review Progress on Priority Goals and Identify Strategies for Improvement
paragraphs: GPRAMA Requirement: Review of Priority Goal Progress and Identification of  Improvements  GPRAMA requires that agency leaders review progress on agency priority goals (APGs);  assess whether relevant organizations, programs, regulations, and policies are  contributing as planned; and identify strategies for performance improvement for those  goals at greatest risk of not achieving their planned levels of performance.

OMB Guidance: Review of Priority Goal Progress and Identification of  Improvements  OMB guidance reinforces this requirement by directing agency leaders to use in-person  review meetings to review progress on APGs; hold goal leaders accountable for knowing  whether their performance indicators are trending in the right direction and, if not, having a  plan to accelerate progress on the goal; identify APGs or other priorities at risk of not  achieving planned levels of performance; and work with goal leaders to identify strategies  that support improvement.

Leading Practice for Data-Driven Reviews: Accountability    Agency leaders should use review meetings to hold goal leaders and other responsible  managers accountable for knowing the progress being made in achieving goals and, if  progress is insufficient, understanding why and having a plan for improvement.

As mentioned throughout this report, a fundamental purpose of data- driven review meetings is to provide a mechanism for agency leaders to  assess an agency’s progress on key goals and milestones; analyze and  discuss data to identify goals at risk, performance problems, and  improvement opportunities; and ensure that goal contributors are held  accountable for their performance.

Through our survey, we found that most agencies reported they always or  often use their review meetings to assess progress and contributions, and  identify goals at risk. However, as shown in figure 7 below, there is some  variation reported across the 22 agencies on the frequency of specific  types of actions taken during review meetings.

Assessing progress on APGs. Reviewing progress on APGs on a  regular and ongoing basis is a key requirement of GPRAMA, and helps  ensure that agency leaders, goal leaders, and other contributors have  frequent opportunities to review recent progress and trends. Twenty of 22  agencies reported that their data-driven review meetings are always or  often used to review progress on APGs, including recent progress, overall  trends, and the status of related milestones.

Analyzing data to identify goals at risk and hold goal leaders  accountable. GPRAMA also requires that agencies identify and  categorize goals at risk of not achieving the planned level of performance.  Twenty-one of 22 agency PIOs reported that their review meetings are  always or often used to identify goals at risk, and to hold goal leaders  accountable for explaining why the goal is at risk, as well as strategies for  performance improvement.

Discussing contributions of program activities, policies, and  regulations and whether they should be changed to improve their  impact on priority goals. Assessing the contributions that organizations,  program activities, policies, and regulations are making toward the  achievement of goals is another critical part of efforts to use review  meetings to ensure accountability for the completion of commitments, and  to identify potential problems, effective practices, or strategies for  improvement. GPRAMA requires that agencies include these  assessments as part of their reviews. Twenty of 22 agencies reported  always or often discussing whether specific organizations or program  activities were contributing as planned to priority goals, and 18 of 22  reported always or often discussing the contributions of relevant policies  toward priority goals. In this way, data-driven review meetings can be  used to reinforce the alignment of higher-level agency goals with the  milestones and day-to-day activities of program officials contributing to  each goal. However, only 13 of 22 agencies reported always or often  discussing whether program activities, policies, and regulations should be  changed to improve their alignment with priority goals.

SSA officials described how they have used their review meetings to  discuss the contributions of programs, policies, and regulations, and  necessary changes. For example, SSA has an APG to expand the use of  video technology to hold benefit determination hearings. According to  SSA officials, initial discussions on this goal in quarterly review meetings  identified the challenge that those scheduled for video hearings were  opting out at the last minute, which led to unpredictable schedules and  down time for Administrative Law Judges. To address this issue and help  achieve the broader goal to expand the use of video hearings, the agency  determined that a regulatory change was needed to require claimants to  decline a video hearing within 30 days after the date the claimant  receives notice that the agency may schedule the claimant to appear at a  hearing by video teleconferencing. This change was designed to  decrease last-minute hearing cancellations and help them more efficiently  schedule video hearings. The milestones that were developed to track  progress on the development and implementation of the regulatory  change were regularly discussed in the quarterly meetings, which is one  of the factors that led to the agency working with OMB to expedite the  release of the regulation.

SSA believes that this June 2014 regulatory change will have long-term  benefits. However, SSA has acknowledged that in the short term they  may receive more opt-outs due to the 30-day notice requirement. For this  reason, they are tracking the opt-out rate for video hearings to measure  the impact of the new regulation, and are reviewing these data in their  quarterly meetings. In the quarterly review meeting we observed after the  regulation had been implemented, participants discussed several  potential consequences of the regulation, including the potential for an  increase in the opt-out rate, and possible strategies for addressing them,  such as additional regulatory or process changes.
subsections: section_title: Some Agencies Reported Taking Certain Actions during Reviews Less Frequently
paragraphs: Some agencies reported through our survey that they take certain actions  during review meetings about half of the time or less frequently. For  example, the Department of Labor (Labor) reported that it reviews APG  progress about half the time in review meetings. However, a Labor official  explained through follow-up communication that it responded this way  because each quarter it holds performance review meetings for each of  its 16 components and not all components have responsibility for one or  more of the APGs. Therefore, APG progress is discussed only during  review meetings for components that contribute to APGs. The agency  specified, however, that each quarter it reviews progress on each of its  APGs.

Three agencies—the Departments of Energy (Energy) and Health and  Human Services, and the National Science Foundation (NSF)—reported  that they rarely discuss whether program activities, policies, and  regulations should be changed to improve their alignment with priority  goals in review meetings. Two other agencies—the Department of  Defense (DOD) and the National Aeronautics and Space Administration  (NASA)—reported that they never hold these discussions. Through  follow-up, officials from Energy, HHS, NASA, and NSF clarified their  responses to this question and explained how their review meetings were  used to identify and discuss weaknesses or risks that could impact the  achievement of their goals, and discuss suggestions for improvement.

Energy. While Energy’s quarterly review meetings were used to  discuss APGs identified as “off track,” and review future plans and  milestones for these goals, an agency official said that other goal- specific meetings were held to drive action on at-risk goals. The  quarterly review meetings also served as an opportunity for senior  leaders not involved in the other meetings to discuss goal  progress and actions being taken to improve efforts in those areas  that are off track.

Health and Human Services. An HHS official stated that the  agency’s response was due mainly to the fact that review  meetings are generally not used to discuss regulatory changes,  with some exceptions, such as reviews held for the health  information technology APG. Instead, the official said that  discussions in HHS review meetings are focused primarily on  improving progress on APGs through better implementation and  execution of program activities and other management initiatives.  HHS leaders that attend the review meetings, however, may use  the information gained to inform decisions on longer-term policy or  regulatory changes.

NASA. A NASA official stated that the agency’s APGs are closely  aligned with specific agency programs and projects, and that  monthly data-driven performance review meetings are used to  discuss potential cost, schedule, technical, and programmatic  risks to meeting their milestones, as well as strategies for  improving performance. NASA does not, however, discuss  realigning or changing programs or policies to meet those  milestones. For example, in their quarterly reviews of the James  Webb Space Telescope program, participants have had  discussions of actions the program will undertake to meet its  milestones, but not of changing the program, or of reassigning this  work to a different agency program, as the program is the only  one with the capability to implement the work.

National Science Foundation. NSF officials stated that they have  no relevant regulations to discuss in the agency’s review  meetings, but NSF officials do discuss potential changes to  program activities and policies at review meetings as the need for  program or policy changes become apparent, which is generally  about half of the time. For example, NSF has an APG to improve  the nation’s capacity in data science by focusing NSF investments  in human capital, partnerships, and infrastructure that support  data science. Initial plans for this APG were set at the time the  goal was established and expressed as a series of quarterly  milestones. The timing of achievement of these milestones is  occasionally altered, and NSF review meetings have been used to  discuss these changes. In one such recent change, NSF officials  originally planned to support Big Data Regional Innovation Hubs in  fiscal year 2014, but decided to gather more community input to  increase the specificity and quality of its proposals. Request for Comments was published in the Federal Register,  with a public comment period ending November 1, 2014.According to an NSF official, the submissions to the request were  used to refine the solicitation for Big Data Regional Innovation  Hubs that was subsequently released in the second quarter of  fiscal year 2015. These timing changes were presented at each  review meeting and discussed as necessary.

Big Data Regional Innovation Hubs are designed to be consortiums of members from  academia, industry, and government that would foster collaboration amongst partners,  and focus on key Big Data challenges and opportunities in their regions of service. our survey that participants review progress on APGs only about half the  time, rarely identify goals at risk, and never discuss whether program  activities, policies, or regulations should be changed to improve their  alignment with priority goals. This is consistent with our own review of  documentation from DBC meetings, which indicated that a review of  APGs was not always included on the agenda. In those instances when  APG progress was reviewed, the information on APG progress included  in meeting materials was limited. For instance, materials prepared for  some meetings had only one slide with an aggregate count of how many  APGs were on or off track, and limited information on the status of  individual APGs. Without information on the status of individual APGs,  DOD’s review meetings are unlikely to foster meaningful discussions  about progress and trends. Further, if these review meetings are not  regularly used to assess progress on individual APGs, and to identify at- risk goals and potential improvements, it could mean missed  opportunities for DOD to address performance problems or accelerate  progress. DOD officials informed us, however, that over the next year,  they plan to revise their review process to ensure they conduct regular,  quarterly reviews of APG progress that involve discussions on progress  achieved in the most recent quarter, performance trends, and status of  related milestones; discussions of potential organization, program activity,  policy, or regulatory changes to improve alignment with, and impact on,  priority goals; and the identification of at-risk goals.

section_title: Most Agencies Reported Taking Steps to Follow up on Issues Identified in Reviews, but with Varying Frequencies
paragraphs: OMB Guidance: Follow-Up  OMB guidance directs agency leaders to agree on follow-up actions at each review  meeting and track timely follow-through.

Leading Practice for Data-Driven Reviews: Follow-Up  Rigorous and sustained follow-up on issues identified during meetings, including the  identification of the individual or office responsible for each follow-up action, is critical to  ensure the success of reviews as a performance improvement tool.

Identifying and agreeing upon actions that need to be taken following a  review meeting, and rigorously tracking the status of these actions to  completion, is a key element of OMB guidance as well as a leading  practice. Rigorous follow-up is also critical to the overall success of  reviews as a tool for addressing identified deficiencies and improving  performance.

According to our survey results, most agencies reported that they are  generally taking steps to identify and follow up on action items identified  in review meetings. However, this is an area where our survey indicated  there is less consistency in how frequently agencies are employing  specific practices. Figure 8 shows the frequency with which agencies  reported conducting specific follow-up actions.

The variation in how systematically agencies identify and follow-up on  action items from review meetings is also illustrated by the different  approaches that our five selected case-study agencies reported using to  identify and follow-up on action items, which are described in table 4.

The analysis of responses to our survey indicated that there is a  statistically significant, positive correlation between the frequency with  which an agency identifies and agrees on specific follow-up actions and  the perceived impact of review meetings on performance improvement.  Specifically, as shown in figure 9, all 13 agencies that reported that their  review meetings have had a major impact on performance improvement  also always or often identify and agree on follow-up actions during review  meetings. Agencies that reported their review meetings have had a minor  impact on performance improvement reported identifying and agreeing on  follow-up actions during review meetings less frequently. Similarly, our  analysis found that a statistically significant, positive correlation exists  between the frequency with which an agency uses its review meetings to  review the status of follow-up actions from the previous meeting, and the  perceived impact those reviews have on performance improvement.  These findings are consistent with surveys of agency PIOs administered  in the past by the PIC. These surveys found that agencies where reviews  have had a major impact on agency performance are more likely to  document specific action items with clear owners and due dates, and  review follow-up actions from previous meetings.

While OMB guidance and leading practices are clear that participants in  each review meeting should agree on follow-up actions and track follow- through, four agencies – DOD, Energy, NSF, and the Small Business  Administration (SBA) – reported through our survey that they identify and  agree on specific follow-up actions about half the time or less frequently.  Through follow-up with Energy, SBA, and NSF, officials from those  agencies further explained the actions they are taking, or have taken, to  identify, document, and track follow-up items, are consistent with OMB  guidance.

Energy. Energy reported through our survey that participants  identify and agree on specific follow-up actions in quarterly review  meetings about half of the time. According to an Energy official,  however, in instances where follow-up actions are identified, those  items are documented in a “Summary of Actions.” In addition to  using quarterly review meetings to identify follow-up actions, the  official stated that other topic-specific meetings are used to  identify and address follow-up items for specific APGs. For  example, the Summary of Actions from Energy’s August 2014  quarterly review meeting indicated that the Deputy Secretary  would hold a meeting with a specific agency official to review the  off-track status of a priority goal in more detail.

Small Business Administration. SBA reported through our  survey that participants would rarely identify and agree on specific  follow-up actions to be taken after meetings. During the course of  our review, however, SBA officials instituted changes to the  agency’s review processes as a result of new leadership, and  have given the SBA Office of Performance Management  responsibility for ensuring that all action items from their review  meetings, as well as “key takeaways” for discussion at the next  review, are recorded.

National Science Foundation. NSF reported through our survey  that participants rarely identify and agree on specific follow-up  actions. However, NSF officials stated that they chose this  response because their goals are based primarily on the  achievement of milestones and goal teams have already outlined  the specific actions they will be taking in goal documentation. The  status of actions to complete each of these milestones is then  reviewed in each review meeting. NSF officials also said that in  the event a follow-up action or course correction is identified in a  quarterly meeting, the status of these actions will be discussed in  bi-weekly meetings between the PIO and COO, who determine  whether the actions have been adequately addressed or whether  additional steps are required.

In contrast, DOD reported through our survey that participants in review  meetings rarely identify and agree on specific follow-up actions. After  subsequent follow-up with the agency, we found that DOD practices are  not consistent with OMB guidance or leading practices. Through our  review of documents from DOD review meetings, we also found there  was no information included in materials prepared before, or after, these  meetings to indicate that they are used to identify follow-up actions  related to APGs. In our follow-up communication with them, DOD officials  acknowledged the need to regularly identify follow-up actions, and  informed us that over the next year they plan to integrate the identification  of specific follow-up actions into their reviews.

Clearly identifying and documenting follow-up items, identifying the  individual or office responsible, and monitoring their status are important  to ensure that agreed upon actions are taken after DOD’s review  meetings. This is supported by the results of our analysis, which showed  that systematically identifying and following up on action items is  associated with review meetings having a greater impact as a  performance improvement tool. Furthermore, a failure to clearly identify  and document follow-up actions may lead to a situation at DOD in which  there is no commonly-held list of specific actions that will be taken after  review meetings, and a limited ability to hold accountable those  responsible for the completion of action items.

section_title: Most Agencies Reported Review Meetings Have Positively Affected Performance, Collaboration, Accountability, and Efficiency
paragraphs: The results of our survey on agency data-driven review practices indicate  that review meetings have had positive effects on progress toward  agency goals, collaboration between agency officials, the ability to hold  agency officials accountable for progress toward goals, and the ability to  identify opportunities to improve agency operations. COOs, PIOs, APG  goal leaders, and staff that we spoke with at the five selected agencies  reinforced these findings, and also shared examples that illustrate the  positive effects their data-driven review meetings are having in these  areas.
subsections: section_title: Nearly All Agencies Reported Review Meetings Have Positively Affected Progress toward Agency Goals
paragraphs: Nearly all agencies reported that their data-driven review meetings have  had a positive effect on progress toward the achievement of agency  goals, and on their ability to identify and mitigate risks to goal  achievement. As illustrated in figure 10, all 22 agencies reported that their  reviews have had a positive effect on progress toward their APGs, and 21  of 22 reported that their reviews have had a positive effect on their  agency’s ability to identify and mitigate risks to achieving priority goals.

In our discussions with officials from selected agencies, data-driven  review meetings were described as venues for agency leaders and  managers to assess progress toward key goals and milestones, the  status of ongoing initiatives and planned actions, potential solutions for  problems or challenges hindering progress, and additional support or  resources needed to improve performance.

Agency officials emphasized that discussions in their review meetings  tend to focus on those goals or issues most in need of attention, where  the achievement of a goal or milestone is at risk. In this way, reviews can  serve as early warning systems and facilitate focused discussions on  external, technical, or operational obstacles that may be hindering  progress, and the specific actions that should be taken to overcome them.

For example, SSA has an APG to increase the number of registrations for  its my Social Security portal by 15 percent per year in fiscal years 2014  and 2015. In 2014, however, through the review of data for SSA’s third  quarter review meeting, it became apparent to SSA leadership that the  agency was not on track to achieve its target for this goal. According to  officials, as part of the quarterly review process agency officials  completed a more thorough examination of reasons for this and found  that the agency would not be able to complete the development of  additional features, such as the ability to request a replacement Social  Security card, which were expected to drive higher volumes of traffic  to the portal. Understanding these limitations, SSA’s focus shifted to what  could be done by offices throughout the agency, using currently available  or attainable resources and technology, to support efforts to increase the  number of registrations. To achieve this, SSA leadership had different  offices within the agency, including Communications, Policy, and Budget,  specify the contributions they would make to help increase the number of  registrations. For example, the Office of Communications developed a  document outlining 26 activities the office was taking, or planned to take,  to promote my Social Security to potential users. Since then, the agency’s  quarterly review meetings have been used to review and reinforce the  commitments each office made.

In the quarterly review meeting that we observed, a representative of  SSA’s Communications office emphasized that supporting efforts to  increase my Social Security registrations is the office’s top priority, and  discussed an ongoing national marketing campaign, and marketing  activities targeted to advocates in the aging and disability communities  and third party tax preparers. While SSA was unable to meet the  registration goal for fiscal year 2014, according to SSA officials, these  efforts recently undertaken as a result of the review process have helped  generate an increase in registrations. Data from SSA’s fiscal year 2015  first quarter review show that there was a 46 percent increase in new  account registrations in October 2014 compared to the number of new  registrations in October 2013, and a 26 percent increase in December  2014 relative to December 2013.

Many agencies reported that they are also using their review meetings to  review progress on a broader suite of performance goals that go beyond  the requirement to review APGs. Nineteen of 22 agencies reported that  they always or often discuss progress on agency-wide goals or initiatives  beyond the APGs in their review meetings, while 20 of 22 agencies  reported that reviews have had a positive effect on their progress toward  the achievement of other performance goals. For example, according to  a GSA official, a long-standing challenge of the Public Buildings Service  (PBS) has been finalizing occupancy agreements in a timely fashion.2014, agency leadership made improving performance in this area a  specific goal for PBS, which was then often discussed during GSA’s  review meetings. According to GSA officials, due to the increased  attention on the status of goal progress and leadership commitment to  improving performance, the agency has surpassed its goals in this area.  According to GSA’s performance report, in fiscal year 2014, the agency  improved the on-time activation of occupancy agreements in owned  space to 98 percent and leased space to 90 percent, exceeding the  targets of 90 percent in owned space and 82 percent in leased space.  This is also an improvement from the on-time activation rates of 86   In  percent for owned space and 75 percent for leased space in fiscal year  2013.

section_title: Nearly All Agencies Reported Reviews Have Positively Affected Collaboration between Officials in Their Agencies
paragraphs: Twenty-one of 22 agencies reported that their data-driven reviews have  had a positive effect on collaboration between officials from different  offices or programs within the agency. Similarly, agency officials with  whom we spoke emphasized that review meetings provide opportunities  to bring together the people, analytical insights, and resources from  across an agency that are needed to improve progress on agency  priorities and to address any identified performance problems or  challenges. As we heard from agency officials, and summarized in figure  11, bringing leaders and officials from across an agency together  regularly to focus on shared goals and milestones can establish a shared  sense of purpose, encourage ongoing collaboration, and reduce  organizational silos. The review meetings also serve as action-forcing  events that provide an opportunity for officials from across an agency to  develop and implement collaborative solutions to identified problems.  These insights into the positive effects review meetings can have on  collaboration within agencies also reinforce their potential value as a tool  for promoting increased collaboration across agencies. As noted above,  this should encourage those who lead and manage agency reviews to  follow OMB guidance on the issue and be mindful of opportunities to  leverage reviews to involve relevant stakeholders from external agencies  or organizations.

Department of Health and Human Services (HHS) officials reported that  promoting collaboration between different offices that contribute to  individual APGs has been one of the most important effects of their  reviews. They also emphasized that reviews have been used to bring  APG contributors from across HHS together to discuss what can  collectively be done to support progress on the goals. For example, HHS  has an APG to increase the number of eligible providers who receive  incentive payments for the successful adoption or demonstration of  meaningful use of certified electronic health record (EHR) technology.  According to HHS officials, the goal requires a great deal of coordination  between the Office of the National Coordinator for Health IT (ONC) and  the Centers for Medicare & Medicaid Services (CMS). An HHS official  involved in these efforts explained that the two offices realize that, given  their shared ownership of the goal, they are expected to coordinate  effectively, and the reviews are used to reinforce this expectation and  ensure that ongoing coordination is occurring.

One way this has manifested itself is in the improved data sharing  arrangement that now exists between ONC and CMS. Under this  arrangement, CMS collects data related to the Medicare and Medicaid  EHR Incentive Programs, which provide financial incentives for the  “meaningful use” of certified EHR technology by health care providers.  According to ONC officials, the review process has helped encourage  more regular data sharing between CMS and ONC, with ONC receiving  monthly updates on EHR Incentive Program registration, attestation, and  payment data from CMS. The team supporting the priority goal is now  using these data to conduct ongoing evaluations of the characteristics of  providers at different stages in the program. According to an ONC official,  data on program participation are also shared by ONC and CMS during  monthly presentations to the Federal Advisory Committees on Health IT,  which illustrates how they are also using these data to facilitate  partnerships and public-facing discussions with other stakeholders.

Similarly, officials from GSA stated that they believe that the most  significant effect of the agency’s review meetings has been to enable  collaborative problem solving, where ideas for potential solutions can be  freely shared, and officials can request assistance from their colleagues  in offices throughout GSA. For example, as part of a 2013 reorganization,  the Office of Administrative Services (OAS) was given responsibility for  the management of GSA-occupied real estate. According to GSA officials,  through review meetings agency leaders identified that regional offices  were estimating project costs and footprints using different assumptions.  OAS was able to partner with PBS, which had experience working with  other agencies to estimate project costs and footprints, to leverage their  experience in the consideration of internal agency real estate needs. This  led to the creation of a new streamlined process for project plans, with the  head of OAS reviewing all plans to ensure they conform to consistent  standards. Agency leaders said this coordination likely would not have  happened if the two offices had remained siloed in their approaches, as  they were before they began coordinating in GSA’s regular review  meetings.

section_title: Nearly All Agencies Reported Reviews Have Positively Affected the Ability to Hold Officials Accountable for Progress
paragraphs: Having regular review meetings that require goal leaders and other  contributors to report out on their progress, and respond to direct  questions about the actions that they are taking, provide leaders with  important opportunities to clarify and reinforce responsibilities, motivate  actions necessary to complete milestones or improve performance, and  hold goal leaders and managers accountable. The involvement of agency  leaders in review meetings also helps ensure that the meetings are taken  seriously and viewed as a priority.

Twenty-one of 22 agencies reported that their data-driven reviews have  had a positive effect on their agency’s ability to hold goal leaders and  other officials accountable for progress towards goals and milestones.According to officials from selected agencies, the transparency of  performance information, and a review process that ensures it receives  appropriate scrutiny, produces an increased sense of accountability for  results.

Several agency officials emphasized that it is important for those leading  the meetings to establish a constructive, solution-oriented environment in  which officials can be open and honest about their progress, and any  problems and challenges they are facing. This outlook is also reinforced  by OMB guidance, which directs agency leaders to establish an  environment that promotes learning and openly sharing successes and  challenges. At the same time, while agency leaders must maintain an  environment in which participants feel comfortable raising problems and  challenges, officials we spoke to also emphasized that leaders must use  the meetings to hold goal leaders and managers accountable for having  well-thought-out strategies for how to overcome them or mitigate their  impact.

Leaders from Commerce said that they are using their regular review  meetings with bureau heads and goal leaders to support a cultural  change throughout the agency and to reinforce accountability for  performance at multiple levels of the organization. This change is one that  emphasizes regular and ongoing reviews of performance, accountability  for the completion of action items, more frequent and regular follow-up  through review meetings, and an increased urgency and pace of  implementation. For example, in 2014, Commerce established an APG  designed to increase the percentage of companies assisted by the Global  Markets (GM) program that achieve their export objectives.Commerce officials, the decision to focus on this new measure, which  places the focus on the satisfaction of clients, came out of input from GM  clients and staff, as well as discussions in departmental review meetings  about the need to clarify the objectives of the program and improve the  quality of assistance that trade and commercial specialists provide.

The key measure used to track progress on the priority goal was  designed to help drive changes in internal processes and behaviors by  focusing more clearly on activities and outcomes GM staff are able to  directly influence. Now, at the beginning of their interaction with a new  client, GM staff ask client businesses to define their needs and what they  want to achieve with the support of the program. This shift toward a more  consultative approach was designed to encourage staff to work with  clients to establish goals and expectations, design solutions responsive to  those goals, and identify potential challenges.

Commerce has also instituted new data collection procedures for the  goal. It now collects data on APG performance on a regular, weekly  basis, and in a way that will allow for comparisons across regions. The  GM Office of Strategic Planning sends out weekly updates to leaders and  managers with data on how each region, and GM as a whole, is  performing against weekly and annual targets for a number of key  metrics. These data on performance are also being reviewed by GM  leaders in monthly review meetings to see how regions are performing in  relation to one another and against established targets, and to identify  challenges that offices are experiencing and effective practices that could  be more widely shared.

According to Commerce officials, progress on the APG is discussed in  regular review meetings at the bureau and departmental levels. For  example, the Deputy Under Secretary for International Trade, who  oversees the daily operations of the International Trade Administration  (ITA), meets regularly with the APG goal leader to discuss performance  on the goal. Progress on the goal is also discussed in monthly meetings  of the ITA Management Council, which consists of the senior leaders of  each of ITA’s three business units. Finally, progress toward, and the  management of, the APG is also discussed in department-level review  meetings. Here, Commerce leaders provide APG goal representatives  with specific feedback and guidance, as well as information on other  agency-wide initiatives that could impact the ability of GM staff and  leadership to appropriately manage the program.

According to a Commerce official, the priority goal and associated  measures, data collection process, reviews at multiple levels, and a  system that holds staff accountable for identifying what clients want to  achieve and working to deliver on those expectations have provided an  increased sense of accountability and strong incentives for behavioral  change amongst staff within the program. Commerce reported that in the  first quarter of fiscal year 2015, 73 percent of GM clients reported that  they have achieved their export objectives, exceeding the current target  goal of 71 percent.

section_title: Most Agencies Report Reviews Have Positively Affected the Efficiency of Agency Operations
paragraphs: Seventeen of 22 agencies reported that their data-driven reviews have  had a positive effect on the efficiency of agency or program operations.Some of our selected agencies are using their review meetings as an  opportunity to review the status of management improvement initiatives  and to improve the efficiency of business operations.

For example, DOT’s review meetings have been used to uncover and  correct inefficiencies in its hiring process, which involves multiple offices  throughout DOT. According to a Federal Railroad Administration (FRA)  official we spoke with, staff were able to calculate how many days it took  to complete each step in its hiring process and identify which offices were  responsible for each step. The increased scrutiny this issue received  through DOT’s review meetings led to improvements in the average  number of days it now takes to hire a new employee. For example, for  FRA time-to-hire decreased from approximately 160 days in fiscal year  2012 to 77 days in the third quarter of fiscal year 2014.

Agency officials also emphasized that review meetings that bring together  leaders and officials from across an agency can increase the overall  efficiency of an agency’s decision making processes. These meetings  allow leaders and managers to discuss and respond to ideas, ask  questions, voice concerns, and immediately make decisions on how to  move forward. If these review meetings did not exist, officials explained  they would likely need to hold separate meetings, with questions and  answers moving up and down the agency’s lines of communication.  Having all key players present in one meeting makes this entire process  more efficient and allows for more timely action.

section_title: Sustaining Positive Effects of Reviews Requires Ongoing Leadership Commitment, Institutionalizing Review Processes, and Demonstrating Value to Participants
paragraphs: As reported through our survey and discussions with agency officials,  data-driven review meetings can improve agency performance and  results by increasing leadership oversight and management capacity to  use performance information, focusing attention on goals and priorities,  identifying areas where targeted improvements are needed, and  improving communication and collaboration across an agency.

Our survey data suggested, however, that sustaining a data-driven review  process over time and across leadership transitions can be a challenge  for agencies. Ten agencies reported that it has been a challenge to  continue holding reviews despite turnover of agency or priority goal  leadership, with 2 reporting it has been a great challenge, 2 a moderate  challenge, and 6 a small challenge. Through our discussions with  agency officials, we learned that sustaining review meetings and their  positive effects requires ongoing leadership commitment and  involvement, the institutionalization of review practices, and the  development of a broad base of support for the reviews through a shared  appreciation for the positive effects that review meetings produce. These  factors build on one another, as agency leaders, participants, and those  supporting the reviews engage in a cycle of ongoing actions, as shown in  figure 12.

First, agency officials emphasized that agency leaders’ commitment to  lead, support, and remain involved in the reviews is a key element that  must be in place for reviews and their positive effects to be sustained  over time. The experiences of our selected agencies show that active  involvement by agency leaders in review meetings is critical to establish  the importance of the meetings and the clear expectation that other  agency officials participate in them when called to do so. Furthermore,  through their ongoing involvement in the reviews and their communication  with other participants, leaders must reinforce that data-driven review  meetings remain a priority and are seen as a valuable tool to achieve key  agency objectives.

Second, agency officials indicated that reviews in which agency leaders  critically assess data on performance, and where follow-up actions are  identified and tracked, should be institutionalized and made a routine part  of the agency’s operations. Through GPRAMA, Congress took a critical  step to help ensure agencies do this by requiring that agency leaders   Agencies have also taken  conduct reviews frequently and regularly.several specific steps to institutionalize their review processes. For  instance, agency officials gave existing or newly established offices  responsibility for supporting the review process and ensuring that  meetings are carried out with regularity and consistency over time. Staff  in these offices generally manage data collection, meeting preparation,  and follow-up, offer training for staff, and provide analytical expertise that  helps support and inform the discussion in review meetings. Agencies  also established clear expectations and procedures for how reviews  would be carried out, including processes for preparation and follow-up.  Institutionalizing review processes in this way, and providing sufficient  institutional and staff support to manage them, can also facilitate the  maintenance of review processes across leadership changes, as the  process is made less dependent on the management style of a particular  leader.

As the cycle continues, officials also noted that agencies need to  continuously assess their review processes and address any identified  weaknesses by incorporating improvements that respond to the needs of  leaders and participants. While it is important to have a basic approach  that persists over time, the ongoing assessment and improvement of  review processes will help ensure reviews are adapted to meet the needs  of new leaders and participants and continue to be used in a way that  sustains their positive effects. For example, Commerce and SSA have  recently changed their review meetings and processes to strengthen the  focus on the agencies’ respective strategic goals. Officials with DOT and  GSA also indicated that they are considering and instituting changes to  their reviews. According to DOT officials, the agency formed a working  group that examined ways to more effectively structure DOT’s review  meetings. The group has also made recommendations to the Deputy  Secretary regarding the format of the meetings, participation, the  presentation of data, and the best ways to follow up on identified action  items. The recommendations are currently under review by DOT senior  leadership. At GSA, under a new Acting Administrator and Acting Deputy  Administrator, additional monthly and quarterly reviews of priority  initiatives and APGs are being instituted, while weekly review meetings  have been refocused on a subset of GSA’s key performance measures  and initiatives that are most ambitious or most in need of assistance and  focus.

Third, agency officials emphasized that it is critical for those leading,  managing, and participating in the reviews to assess, understand, and  communicate the results that review meetings produce to help develop a  broad base of support throughout an agency for sustaining review  processes over time. According to a number of agency officials we spoke  with, for agency leaders and managers—both political appointees and  career agency officials—to maintain their commitment to review  processes, the reviews must demonstrate that the benefits they provide  outweigh the costs in time and resources spent. If the reviews show  positive results, add value for participants, and have the support of senior  political and career leaders who are able to articulate their merits, this will  help sustain organization-wide commitment and increase the likelihood  that reviews will be continued following leadership transitions.

section_title: Conclusions
paragraphs: Most federal agencies reported conducting data-driven reviews frequently  and regularly, involving agency leaders and other key personnel, and  using the process to assess progress on agency goals and identify  strategies to address challenges or improve performance. These  practices are consistent with requirements, guidance, and leading  practices. Our findings also underscore the value of conducting frequent,  in-person, data-driven reviews as a leadership strategy and management  practice that can promote the use of performance information by agency  officials and produce improved results. Our survey results indicated that  agency data-driven review meetings enhanced their progress toward the  achievement of agency goals, the engagement of agency leaders in the  performance management process, the level of collaboration between  agency officials, the ability to hold agency officials accountable for  progress on goals and milestones, and the efficiency of agency  operations.

Through this work, however, we found that DHS was not holding in- person, data-driven reviews of its APGs, and that four other agencies  were conducting reviews in a manner that was not consistent with  requirements, guidance, or leading practices. We also found that data- driven review meetings should be held on a regular, frequent schedule,  actively involve senior agency leaders or other key officials, involve in- depth reviews of progress on agency goals, and be supported by rigorous  methods for identifying and tracking follow-up actions. Otherwise, there  could be missed opportunities for these agencies’ leaders to hold officials  accountable for progress toward identified goals and milestones, to take  timely and better informed action to address identified challenges, and to  encourage continuous improvements in agency performance and  operations.

section_title: Recommendations for Executive Action
paragraphs: To help ensure that agency review processes provide frequent, regular  opportunities to assess progress on agency priority goals (APG), and are  conducted in a manner consistent with GPRA Modernization Act of 2010  (GPRAMA) requirements, OMB guidance, and leading practices, we  recommend the following actions:

That the Secretary of Agriculture work with the COO and PIO to  modify the Department’s review processes to ensure that review  meetings: (1) are held at least quarterly; (2) are led by the agency  head or COO; (3) involve APG leaders; (4) and involve, as  appropriate, agency officials with functional management  responsibilities.

That the Secretary of Defense work with the COO and PIO to modify  the Department’s review processes to ensure that review meetings:  (1) are led by the agency head or COO; (2) are used to review  progress on all APGs at least once a quarter, discuss at-risk goals  and improvement strategies, and assess whether specific program  activities, policies, or other activities are contributing to goals as  planned; and (3) are used by participants to identify, agree upon,  document and track follow-up actions.

That the Secretary of Health and Human Services work with the COO  and PIO to modify the Department’s review process to ensure that  progress on each APG is reviewed in an in-person review meeting at  least quarterly.

That the Secretary of Homeland Security work with the COO and PIO  to reestablish regular, in-person, data-driven review meetings  conducted in a manner consistent with the requirements of GPRAMA,  OMB guidance, and leading practices outlined in this report.

That the Secretary of State work with the COO and PIO to modify the  Department’s review processes to ensure: (1) that progress on each  APG is reviewed in an in-person review meeting at least quarterly; (2)  and that the reviews are led by the agency head or COO; and (3)  involve, as appropriate, agency officials with functional management  responsibilities.

section_title: Agency Comments
paragraphs: We provided a draft of this report for review and comment to the  Secretaries of Agriculture, Commerce, Defense, Education, Energy,  Health and Human Services, Homeland Security, Housing and Urban  Development, Interior, Labor, State, Transportation, Treasury, and  Veterans Affairs; the Attorney General of the United States; the Directors  of the Office of Management and Budget, the Office of Personnel  Management, and the National Science Foundation (NSF); the  Administrators of the Environmental Protection Agency, National  Aeronautics and Space Administration (NASA), Small Business  Administration (SBA); the Acting Administrators of the General Services  Administration (GSA) and the U.S. Agency for International Development  (USAID); and the Acting Commissioner of Social Security.

We received written comments from the Departments of Defense (DOD),  Health and Human Services (HHS), Homeland Security (DHS), and State,  and the Social Security Administration (SSA). These responses are  reproduced in appendixes IV through VIII. The Department of Agriculture  (USDA) provided its response in an email transmitted on June 17, 2015.  DHS, HHS, and USDA concurred with our recommendations.

DOD and State concurred with all but one recommendation—to ensure  the COO leads the reviews—with which they partially concurred.

In its response, DOD concurred with our recommendations that  agency leaders modify the agency’s review process to ensure that  reviews are held to assess progress on APGs at least once a quarter,  to identify goals at risk, to assess contributions, and to identify and  track follow-up actions. DOD’s response said that the agency plans to  comply with the recommendations by November 30, 2015. DOD  partially concurred with our recommendation that the agency ensure  the reviews are led by at least the COO. DOD, in its response,  outlined a more specific role for its COO in future reviews. However,  we do not believe this role is sufficient to bring DOD’s practices in line  with the requirements of GPRAMA or with OMB guidance that the  agency head and/or the COO conduct the reviews.

In its response, State concurred with our recommendations that  reviews should be held on a quarterly basis and that the reviews  involve applicable functional management officials. State did not  explicitly agree or disagree with our recommendation that the reviews  be led by the agency head or COO. However, the agency’s response  indicated that the agency would continue to have the PIO lead their  review meetings. As outlined in the report, this is not consistent with  the requirements of GPRAMA or with OMB guidance that the agency  head and/or the COO conduct the reviews.

We believe that DOD and State have both interpreted the language of  OMB’s Circular A-11 in a way that provides them with the flexibility to  delegate responsibility for conducting data-driven performance reviews to  the PIO. As DOD notes, A-11 provides agencies with flexibility at key  points to design a performance management system that best meets the  agency’s needs. However, it is also important to emphasize that OMB’s  Circular A-11 guidance clearly and unambiguously states in six separate  sections that the COO is responsible for running agency reviews, and that  these reviews should be held quarterly. The guidance also specifies that  reviews must be held in person. As OMB has also noted, the personal  engagement of agency leaders demonstrates commitment to  improvement across the organization, ensures coordination across  agency silos, and enables rapid decision making. The personal  engagement of the COO in the data-driven reviews of progress on APGs  is also critical given that, under GPRAMA, APGs are to reflect the  agency’s highest priorities, and COOs are responsible for improving the  management and performance of the agency through the regular  assessment of progress and the use of performance information. For  these reasons, we believe that these recommendations to follow  requirements and guidance remain valid.

The following agencies provided technical comments that were  incorporated into the draft as appropriate: Department of Energy, NASA,  NSF, SBA, SSA, USAID, and USDA.

The following agencies had no comments on the draft report: The  Departments of Commerce, Labor, Education, Housing and Urban  Development, Interior, Justice, Treasury, and Veterans Affairs; the  Environmental Protection Agency; the General Services Administration;  the Office of Management and Budget; and the Office of Personnel  Management.

We are sending copies of this report to the Director of OMB as well as  appropriate congressional committees and other interested parties. The  report is also available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions concerning this report, please  contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our   Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix XI.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report is part of our response to a statutory requirement that we  evaluate how the implementation of the GPRA Modernization Act of 2010  (GPRAMA) is affecting performance management in federal agencies,  and whether performance management is being used by agencies to  improve the efficiency and effectiveness of agency programs.  Specifically, this report examines (1) the extent to which agencies are  conducting data-driven performance reviews in a manner consistent with  GPRAMA requirements, Office of Management and Budget (OMB)  guidance, and leading practices; and (2) how agency data-driven  performance reviews have affected performance, collaboration,  accountability, and efficiency within agencies, and how positive effects  can be sustained.

To address these objectives, we assessed review practices at the 23  executive agencies that fall under the purview of the sections of GPRAMA  that relate to agency performance reviews. GPRAMA states that the 24  agencies identified by the amended Chief Financial Officers Act of 1990  (CFO Act), or those agencies otherwise determined by OMB, are required  to develop agency priority goals (APG) and to review progress on these   Because OMB did not require  by conducting reviews at least quarterly. the Nuclear Regulatory Commission to develop APGs for 2014-2015, we  confined our study to the other 23 CFO Act agencies.

To address both objectives, we surveyed Performance Improvement  Officers (PIOs) at the 23 executive agencies. This method allowed us to collect government-wide information about the variety of practices being  used by agencies to review progress on goals. We asked PIOs for  information about the frequency of review meetings; leadership of, and  participation in, review meetings; preparation for, execution of, and follow- up on, review meetings; challenges; and perceived effects of review  meetings on agency performance, collaboration, efficiency, and  accountability for results. Although agencies may conduct a variety of  reviews as part of their performance management processes, we asked  respondents to consider only in-person, data-driven review meetings that  included discussion of APGs when answering our survey questions.  GPRAMA provisions apply directly to these types of reviews. We  administered the surveys between October and December 2014,  transmitting and receiving the surveys as attachments to e-mails. We  received responses from all 23 agency PIOs, signifying a 100 percent  response rate. To minimize errors related to difficulties interpreting  questions, we pretested the survey with three agency performance  management officials to ensure that our questions were clear, complete,  and unbiased, and that answering the survey did not place an undue  burden on respondents. One of our methodology specialists assisted in  developing our survey to ensure that survey questions captured the  intended information. We revised our survey questions based on  feedback from the pretesters and our methodologist. We verified our data  entry and analysis programs for accuracy.

To further address both objectives, we selected five case-study agencies  for a more in-depth assessment of their data-driven review processes. To  supplement government-wide data collected through our survey, we used  these more in-depth reviews to gather information about specific agency  practices and about agency officials’ perceptions of any impacts that  review meetings have had. This allowed us to collect additional detail and  illustrative examples. We selected a sample of agencies that reflected a  range of key characteristics, while excluding agencies from our selection  that had been used as case studies for related recent or ongoing work, to  avoid overburdening those agencies. The key characteristics we identified  were agency size, as indicated by number of civilian employees; the  extent to which agency leadership uses quarterly performance reviews to  drive progress toward goals, as reported by respondents to GAO’s 2013  Federal Managers Survey; and agency compliance with basic GPRAMA  requirements to hold quarterly in-person performance review meetings,  as identified by a GAO team conducting related work. Using these  criteria, we selected the Departments of Commerce (Commerce), Health  and Human Services (HHS), and Transportation (DOT); the General  Services Administration (GSA); and the Social Security Administration  (SSA). Our sample of agencies is non-generalizable, and should not be  considered representative of all agencies.

At each case-study agency selected for in-depth review, we also selected  APGs to obtain the perspective of APG leaders and their staff on the  agency’s data-driven review process. For those agencies with more than  two APG leaders, we developed a selection process to determine which  APG leaders we would request to interview. When possible, we excluded  APG leaders who had been selected for interviews as part of related  recent or ongoing work to avoid overburdening those officials. We  selected goal leaders to produce a sample that reflected varied agency  progress against APGs, where we excluded APGs for which progress  was unclear. In cases when we had to choose between two APGs with  the same type of progress, we prioritized the APG with a larger number of  indicators. In cases where we selected an official who leads two APGs,  we selected both APGs, resulting in three total APGs selected for the  agency.

To allow us to corroborate information collected through surveys,  interviews, and observations, and strengthen our confidence in the  reliability of the self-reported survey responses, we requested supporting  documents from 12 agencies, representing more than 50 percent of the  agencies surveyed, related to review meeting frequency, leadership,  participation, content, and follow-up. The 12 agencies included the 5  agencies selected for more in-depth review, several agencies whose  survey responses required clarification, and several additional agencies  at random. Examples of documents submitted by agencies included  review meeting attendance or invitations, agendas, presentation slides,  and briefings and summary reports. Based on our findings, we  determined that the survey data were sufficiently reliable for the purposes  of this report. In conducting the survey and subsequent follow-up, we  learned that the Department of Homeland Security (DHS) does not hold  in-person review meetings, and has not done so since December 2013.  For this reason, the summaries of survey responses in this report exclude  DHS.

We also addressed both objectives by interviewing agency officials who  play a central role in the review meetings, including the agency Chief  Operating Officer (COO), the PIO, and two APG leaders. These  interviews provided us with detailed information from individual officials’  perspectives and helped us to corroborate information collected through  other means. We used a consistent set of questions for each type of  official, which included the official’s objectives for review meetings; the  official’s role in preparing for, participating in, and conducting follow-up  after review meetings; and the official’s experience of any effects of  review meetings. We also interviewed staff from OMB and the  Performance Improvement Council (PIC) to obtain information on  previous surveys of agency PIOs administered by the PIC, their  perspective on the implementation and effectiveness of reviews, and to  learn about the role played by the PIC’s Internal Reviews Working Group,  which has served as a forum for agency performance staff to periodically  come together to discuss performance review practices in their agencies.

We also addressed both objectives by observing agency-level review  meetings at HHS, GSA, and SSA, as well as one sub-agency-level review  meeting at GSA. Observing review meetings allowed us to gain firsthand  knowledge of review meeting processes. This served to provide context,  increase our familiarity with the process, and corroborate information  gained through other means. While we requested to observe a review  meeting at both Commerce and DOT, we were not allowed to do so due  to agency concerns that our presence could inhibit open discussion.

To address the first objective, we compared what we learned about the  review processes at all 23 agencies with requirements for review  meetings established in GPRAMA, as well as standards set forth in  guidance in OMB’s Circular A-11 and leading practices for data-driven   To address the second objective, we  reviews we previously identified.analyzed our survey data to determine how agencies characterized the  effects of their review meetings. We also used interviews with officials  and documentation from our selected agencies to identify illustrative  examples of the effects review meetings have produced, and to identify  actions they have taken to sustain the benefits of the reviews.

Because the scope of our review was to examine the implementation of  data-driven review processes in agencies, we did not evaluate whether  APGs were appropriate indicators of performance, sufficiently ambitious,  or met other dimensions of quality. Although agency performance  information was reported in illustrative examples of data-driven review  meeting materials, as well as illustrative examples of the effects of review  meetings, we did not independently assess the accuracy of the agency  performance information cited in these examples.

We conducted our work from July 2014 to July 2015 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusion  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our finding and conclusions based on our  audit objectives.

section_title: Appendix II: Descriptions of Data-Driven Review Meetings at Selected Agencies
subsections: section_title: Agency General Services Administration (GSA)
paragraphs: Description  In 2014 and 2015, the Administrator of GSA convened weekly review meetings. At  least once a month, the Public Buildings Service and the Federal Acquisition  Service, which had responsibility for the agency’s two agency priority goals (APG),  presented on their key performance measures, including APGs. Other offices within  the agency also presented on their goals once a month. The Administrator also held  additional weekly meetings with the heads of PBS and FAS to discuss key  performance measures, including APGs, in greater detail.

section_title: Department of Commerce
paragraphs: In 2014, the Deputy Secretary of Commerce held biweekly review meetings, with  each meeting focusing on one to three of the agency’s 27 strategic objectives, and  including a review of progress on related APGs. In 2015, Commerce changed the  frequency of review meetings from biweekly to monthly. Beginning in 2015, the  Deputy Secretary held monthly meetings, with each meeting focusing on reviewing  progress on one of the agency’s five strategic goals, including related APGs. The  Deputy Secretary also held monthly check-in meetings with each of the five  strategic goal teams to discuss progress on the goals and related APGs.

section_title: Social Security Administration (SSA)
paragraphs: In 2014, the Acting Commissioner of SSA held quarterly review meetings that  covered all four of SSA’s APGs, as well as selected additional performance  measures. In 2015, in addition to these quarterly reviews, SSA began to convene  additional “theme-based” reviews to discuss cross-cutting issues, like human  resource management, which, according to SSA officials, are also intended to  include a discussion of relevant APGs. SSA plans to hold three of these “theme- based” reviews in 2015.

section_title: Department of Transportation (DOT)
paragraphs: In 2014 and 2015, the Deputy Secretary held four to five “management review  meetings” a year with representatives of each of DOT’s 10 operating  administrations. These meetings focused primarily on the status of each operating  administration’s pending and proposed regulations, but also included a discussion  of relevant performance measures, including APGs.

section_title: Department of Health and Human Services (HHS)
paragraphs: In 2014 and 2015, twice a year the Deputy Secretary and Performance  Improvement Officer held in-person review meetings to review progress on each of  the agency’s five APGs. According to HHS officials, agency leaders reviewed  written progress updates for each APG during quarters without in-person review  meetings.

section_title: Appendix III: Responses to Questions from GAO’s Survey on Agency Data-Driven Performance Review Meetings
paragraphs: To address our research questions related to agency data-driven review  practices and effects we distributed a survey to the performance  improvement officer (PIO) at each of the 23 agencies with agency priority  goals (APGs). We received responses from all 23 PIOs. Through our  survey and subsequent follow-up, however, we learned that the  Department of Homeland Security (DHS) is not currently holding in- person data-driven review meetings, so its responses are not included in  the aggregated results presented below. Therefore, unless otherwise  indicated, the number of responses to each question is 22. There were  nine questions in the survey, six of which contained multiple  subquestions. Tables 1 through 9 below show our survey questions and  aggregated responses. For more information about our methodology for  designing and administering the survey, see appendix I.

section_title: Appendix IV: Comments from the Department of Homeland Security

section_title: Appendix V: Comments from the Department of Defense

section_title: Appendix VI: Comments from the Department of Health and Human Services

section_title: Appendix VII: Comments from the Department of State

section_title: Appendix VIII: Comments from the Social Security Administration

section_title: Appendix IX: Full Text for Interactive Figure 5

section_title: Appendix X: Full Text for Interactive Figure 6

section_title: Appendix XI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Elizabeth Curda (Acting Director)  and Adam Miles supervised the development of this report. Linda Collins,  Shelby Kain, Kathleen Padulchick, Steven Putansu, and A.J. Stephens  made significant contributions to this report. Dierdre Duffy and Robert  Robinson also made key contributions.


Question-summary annotations: questions: question: What does GPRAMA mandate federal agencies do with regards to their respective goals?
answer: The GPRA Modernization Act of 2010 (GPRAMA) requires that federal agencies review progress on agency priority goals (APG) at least once a quarter.

question: What criteria does GPRAMA specify each agency's review should meet?
answer: GPRAMA requires that reviews be conducted by top agency leaders, involve APG goal leaders and other contributors, and be used to identify at-risk goals and strategies to improve performance.
child_questions: question: What else does GPRAMA mandate with regards to the OMB?
answer: While GPRAMA requires that agencies conduct reviews, it also required the Office of Management and Budget (OMB) to prepare guidance on its implementation.

question: What guidance has GAO and OMB provided to each agency?
answer: Since 2011, OMB has provided guidance on how reviews should be conducted, specifying they should be held in person. Further, GAO previously identified nine leading practices for reviews.

questions: question: How can some agencies be liable to inconsistencies in performance reviews?
answer: Agency Review Practices Inconsistent with Requirements and Guidance. Some agency practices were inconsistent with requirements or guidance. For instance, the Department of Homeland Security (DHS) reported that it does not hold in-person reviews, and the Departments of Agriculture (USDA) and Health and Human Services (HHS) reported that they do not hold regular, in-person reviews each quarter. The Department of State (State) reported that progress on each APG is only reviewed in an in-person review once a year, rather than each quarter, as required. The Department of Defense (DOD), USDA, and State also reported that their reviews are not led by their agency heads or COO.
child_questions: question: To what extent does DOD idenfity any follow-up actions to be taken?
answer: DOD also reported it rarely identifies follow-up actions to be taken after meetings.

questions: question: In what areas did these performance reviews yield positive effects?
answer: Agencies Reported Positive Effects of Reviews . Most agencies reported their reviews have had positive effects on progress towards agency goals, collaboration between agency officials, the ability to hold officials accountable for progress, and efforts to improve the efficiency of operations.
child_questions: question: What possible long-term effects did agency officials discover about performance reviews?
answer: According to agency officials, reviews can bring together people, analytical insights, and resources to rigorously assess progress on goals or milestones, develop collaborative solutions to problems, enhance individual and collective accountability for performance, and review efforts to improve efficiency. Agencies reported that sustaining these effects requires ongoing leadership commitment, institutionalizing review processes, and demonstrating value to participants.

questions: question: To what end is GAO making recommendations to 5 agencies?
answer: To ensure that agency reviews are consistent with requirements, guidance, and leading practices, GAO is making recommendations to five agencies.
child_questions: question: How did the agencies respond to the GAO recommendations?
answer: DHS, HHS, and USDA concurred with the recommendations. DOD and State concurred with all but one recommendation—to ensure the COO leads the reviews—with which they partially concurred.

question: How did GAO react to these responses?
answer: GAO believes these recommendations are valid, as discussed in the report.