Managing for Results: Agencies Report Positive Effects of Data-Driven Reviews on Performance but Some Should Strengthen Practices
Released: 2015-07-07T00:00:00

Letter

Background

Data-driven performance reviews are regularly scheduled, structured meetings used by organizational leaders and managers to review and analyze data on progress toward key performance goals and other management-improvement priorities. They are generally used to target areas where leaders want to achieve near-term performance improvements, or to accelerate progress through focused senior leadership attention.

Over the past several years, Congress and the executive branch have taken steps to improve federal performance management by requiring that agencies conduct regular data-driven review meetings. In 2010, OMB released a memorandum establishing the expectation that federal agencies would hold data-driven reviews at least once every quarter to review progress on their priority goals and assure that follow-up steps would be taken to achieve improved outcomes. The memorandum specified that discussions during these meetings were to be guided by analyses of performance data, to focus on progress toward desired outcomes, to explore why variations between targets and actual outcomes occurred, and to prompt adjustments when needed.

Congress, through the passage of GPRAMA, made the expectation that agencies would hold regular data-driven reviews a statutory requirement. Specifically, GPRAMA requires that, not less than quarterly, the head of each agency and COO, with the support of the PIO, should review progress on agency priority goals (see text box).

GPRAMA Requirement for Quarterly Priority Progress Reviews GPRAMA requires that, not less than quarterly, at all agencies required to develop agency priority goals, the head of the agency and Chief Operating Officer, with the support of the agency Performance Improvement Officer, shall: For each agency priority goal, review with the appropriate goal leader the progress achieved during the most recent quarter, overall trends, and the likelihood of meeting the planned level of performance;

Coordinate with relevant personnel within and outside the agency that contribute to the accomplishment of each agency priority goal;

Assess whether relevant organizations, program activities, regulations, policies, and other activities are contributing as planned to the agency priority goals;

Categorize agency priority goals by risk of not achieving the planned level of performance; and For agency priority goals at greatest risk of not meeting the planned level of performance, identify prospects and strategies for performance improvement, including any needed changes to agency program activities, regulations, policies, or other activities.

While GPRAMA established requirements for agencies to conduct the reviews, GPRAMA also required that OMB prepare guidance on the implementation of GPRAMA. In 2011, OMB released guidance for federal agencies that reinforced the requirements in GPRAMA, specified that the reviews should be held in person, and outlined the specific purposes of the data-driven review meetings, the roles and responsibilities of agency leaders involved in the review process, and how the reviews should be conducted. In 2012, OMB released updated guidance for data-driven reviews.throughout this report.

GAO-13-228. and federal level who shared their experiences and lessons learned.These practices, along with additional insights on why the application of these practices is important, are noted and summarized throughout this report.

Nine Leading Practices Identified by GAO That Can Be Used to Promote Successful Data-Driven Performance Reviews

Reviews are conducted frequently and regularly.

Leaders use data-driven reviews as a leadership strategy to drive performance improvement.

Key players attend reviews to facilitate problem solving.

Rigorous preparations enable meaningful performance discussions.

There is capacity to collect accurate, useful, and timely performance data.

Staff have skills to analyze and clearly communicate complex data for decision making.

Reviews ensure alignment between goals, program activities, and resources.

Leaders hold managers accountable for diagnosing performance problems and identifying strategies for improvement.

Participants engage in rigorous and sustained follow-up on issues identified during reviews.

Taken together, the GPRAMA requirements, OMB guidance, and leading practices identify the elements necessary to carry out effective data- driven reviews: (1) those that are used to engage agency leaders in the rigorous assessment of agency performance; (2) support faster and better informed responses to identified performance problems; (3) improve communication and collaboration across an agency; and (4) enhance individual and collective accountability for improving progress toward agency goals.

Most Agencies Reported Conducting Data-Driven Reviews Consistent with Requirements, Guidance, and Leading Practices

Nearly All Agencies Reported Holding Regular Data-Driven Review Meetings at Least Quarterly

GPRAMA Requirement: Quarterly Reviews GPRAMA requires that agency leaders conduct reviews on progress toward agency priority goals (APGs) not less than quarterly.

OMB Guidance: Quarterly Reviews OMB guidance directs agency leaders to run data-driven performance reviews on each of their APGs at least quarterly. This guidance also stresses that reviews should be conducted in person, as significant experience in federal agencies, states, localities, and other countries demonstrates that in-person engagement of senior leaders greatly accelerates learning and performance improvement.

Leading Practice for Data-Driven Reviews: Frequency and Regularity Data-driven review meetings should be frequent and regularly scheduled.

Data-driven performance review meetings that are held frequently and regularly help foster a culture of active and ongoing performance management, problem solving, and continuous improvement. As OMB has noted, the purpose of conducting performance reviews at least quarterly is to ensure that agency leaders regularly review agency performance on top priorities, along with the short- and long-term actions agencies are taking to improve performance, and bring together the people, resources, and analysis needed to drive progress on priority goals.

Of the 23 CFO Act agencies surveyed, 20 agencies reported that they hold data-driven review meetings at least quarterly, with some agencies holding them more frequently. See table 2 for a summary of the frequency with which each agency holds in-person review meetings to, among other things, review progress on APGs. As shown in the table, the Department of Homeland Security (DHS) does not hold the required in-person review meetings.

The five case-study agencies we selected for more in-depth review – the Departments of Commerce (Commerce), Health and Human Services (HHS), and Transportation (DOT); General Services Administration (GSA); and Social Security Administration (SSA) – all hold in-person review meetings involving agency leaders and APG goal leaders at different frequencies. This reflects differences in leadership preferences and organizational structures and processes. See appendix II for more detailed information on the approach used by each of the five selected case-study agencies.

A Few Agencies Reported Conducting Less Frequent in- Person Reviews

OMB guidance is clear that reviews should be held in person to bring together senior leaders and officials involved in all levels of program delivery. This can help ensure coordination across agency silos and enable rapid decision making. This guidance states that while written communication may replace in-person review meetings in rare circumstances, it should only be a stopgap measure to continue performance reviews in a process that otherwise operates primarily in person.

A few agencies, including the Department of Agriculture (USDA), DHS, and HHS, reported that they do not hold in-person reviews of progress on APGs at least quarterly as called for in GPRAMA and OMB guidance. The Department of State (State) reported that agency officials participate in one data-driven review meeting each quarter; however, each meeting is used to review progress on only one APG.

Agriculture. According to USDA officials, the Deputy Secretary meets weekly with officials and staff from the Office of Budget and Program Analysis (OBPA), including the PIO, to discuss budget and regulatory issues, which also provides opportunities to discuss APG progress and performance-related issues. USDA officials also told us that written updates on APGs and performance data are provided quarterly, and that the Deputy Secretary and PIO meet as necessary to review progress toward, and discuss issues related to, specific APGs. However, these are not regularly scheduled meetings. In addition, they told us that staff from the OBPA have frequent conversations with program officials as part of the review of regulatory documents, funding availability notices, executive budget documents, and other related documents so they have not had separate, regularly scheduled meetings to discuss progress toward the APGs. However, in subsequent discussions with USDA officials, they informed us that they intend to begin holding regularly scheduled quarterly meetings led by the COO and involving senior USDA leadership, as directed by OMB in Circular A-11 guidance.

Homeland Security. DHS reported that in-person review meetings ceased due to competing priorities and demands at the end of 2013, when a change in leadership brought alternative management priorities. Agency leaders continue to review written performance updates quarterly. According to a DHS official, a meeting involving the Deputy Secretary and APG goal leaders to review goal results from fiscal year 2015 is being scheduled.

Health and Human Services. HHS leaders hold in-person review meetings for each APG twice a year and review APG progress two more times a year through reviews of written progress updates. Officials from HHS said that, due to the longer-term nature of the agency’s APGs, performance data that are tracked for each goal show little meaningful change from quarter to quarter, so agency officials have not considered meeting quarterly to be an effective use of participants’ time. One HHS official said that managers convene meetings with their program teams more frequently to track progress on efforts contributing to each goal.

State. Each year, State holds one joint, in-person meeting to review progress on the APG to support the implementation of low emission development strategies, which it co-leads with the U.S. Agency for International Development (USAID). State also holds one meeting each year to review progress on the APG to improve consular service delivery. State officials attend three other reviews on APGs led by USAID held throughout the year. Therefore, progress on each APG is only reviewed by officials from State in an in-person review meeting once a year. A State official stated that the performance measures used to track progress on the two APGs for which the agency either leads or co-leads show little meaningful change from quarter to quarter, and believes it would not be beneficial for stakeholders to attend meetings more often than annually. The official also said, however, that the agency’s PIO reviews the data and written updates provided by APG goal leaders each quarter, and updates the Deputy Secretary.

As GPRAMA requirements and OMB guidance are clear that in-person review meetings should be held at least once a quarter, and that progress on each APG should be reviewed each quarter in these meetings, the approaches of these four agencies – DHS, HHS, State, and USDA – are not consistent with requirements for the frequency and expected characteristics of reviews. Furthermore, OMB guidance states that these reviews should not be conducted through written documents, and that agency leaders should use performance review meetings as an opportunity to engage those involved in all levels of program delivery. The lack of frequent, regular, in-person review meetings could result in missed opportunities for leaders and key officials at these four agencies to have regular, in-depth discussions of performance on top agency priorities. Such meetings could also allow them to actively promote ongoing coordination and accountability, address identified challenges or problems in a timely manner, and encourage continuous improvement in agency performance and operations.

As OMB guidance also clarifies, APGs are defined as a “near-term” result or achievement that agency leaders want to accomplish within approximately 24 months through focused leadership attention. While the guidance states that APGs can advance progress toward longer-term, outcome-focused strategic goals and objectives, the APGs are designed to be near-term improvements in outcomes, customer service, or efficiency. Even in those instances when new quantitative performance data are not available for review in meetings, more frequent in-person reviews still provide the opportunity to review goal leader progress in completing shorter-term milestones or initiatives contributing to progress on the goals, and promptly address any identified problems. In fact, GPRAMA and OMB guidance both state that agencies should have clearly-defined, quarterly milestones to track progress on their APGs.

Nearly All Agencies Reported Their Reviews Are Led by the Chief Operating Officer

GPRAMA Requirement: Leadership GPRAMA requires that the agency head and Chief Operating Officer (COO) conduct reviews with the support of the Performance Improvement Officer (PIO).

OMB Guidance: Leadership OMB guidance directs that the agency head and/or COO must conduct reviews with the support of the PIO and the PIO’s office.

Leading Practice for Data-Driven Reviews: Leadership Agency leaders should be directly and visibly engaged in the reviews.

According to OMB, significant experience at federal agencies, states, localities, and other countries demonstrates that in-person engagement of senior leaders in review meetings greatly accelerates learning and performance improvement. The personal engagement of agency leaders in the review meetings also demonstrates their commitment to improvement across the agency and, as mentioned above, facilitates coordination across agency silos and rapid decision making. As OMB has also noted, frequent, data-driven reviews also send a signal throughout the organization that agency leaders are focused on effective and efficient implementation to improve the delivery of results. GPRAMA recognized that the direct involvement of leaders is a critical factor to drive performance improvement within an agency. Thus, it requires that agency heads and COOs conduct the reviews. OMB’s guidance for agencies on how to conduct the reviews also emphasized the importance of leadership involvement in data-driven performance reviews, directing the agency head, COO, or both to conduct the review. As we have previously reported, the commitment of agency leaders to make decisions and manage programs on the basis of performance information, and inspire others to embrace such a model – which review meetings can be used to do – is critical to increase the use of performance information throughout an agency.

We found through our survey that 19 of 22 agencies that held review meetings reported that the meetings were led by their agency head or COO, or jointly by the COO and PIO. See table 3 below for specific agency responses. Furthermore, we found through our survey that review meetings are used as a tool to enhance the engagement of top agency leadership in an agency’s performance management process. In fact, all 22 agencies reported that their reviews have had a positive effect on the engagement of top agency leadership in the agency’s performance management process, with 13 reporting a large positive effect.

Although their exact roles varied, officials from our five selected case- study agencies reported that agency heads or COOs were actively involved in their agency’s review processes, and led the meetings through the following activities: focusing on agency priorities and directly communicating their expectations, asking questions, reinforcing individual and collective accountability, encouraging collaboration, offering assistance with problem solving or identifying available resources, and sharing perspectives from discussions with external stakeholders. For example, during the review meeting we observed at GSA, the Administrator and other leaders engaged in the discussion by challenging assumptions about the status of goal progress, asking questions about factors driving changes in specific performance measures, and encouraging goal leaders to identify areas of risk and plans for addressing challenges.

Some case-study agency officials we interviewed stated that a key role of agency leaders in the meetings is to set a positive example by demonstrating their commitment to, and involvement in, agency performance management processes, and by communicating that participation in reviews is a priority. For example, at SSA, the agency head personally presided over bi-monthly review meetings and created a new office (the Office of the Chief Strategic Officer) to support expanded performance management and data analysis efforts. According to SSA officials we spoke with, in one review meeting the agency head brought focused attention from across the agency on a priority goal that was showing insufficient progress. She convened a follow-up meeting requesting that offices throughout SSA articulate how they would contribute to progress on the goal. This proved to be a useful technique for establishing a broader sense of accountability for contributions to the goal and helped identify new strategies to improve progress.

Less than half of the agencies (8 of 22) reported in our survey that getting or sustaining the participation of top agency leadership in the reviews was a challenge. However, we heard of instances in which logistical challenges can make it difficult for the agency head or COO to participate in each review meeting. For example, a Department of Transportation (DOT) official reported that leadership participation at the review meetings, which are held separately with representatives of each of DOT’s 10 operating administrations each quarter, was a moderate challenge because key leaders may not be able to attend due to last- minute scheduling changes or conflicts. The agency has decided that in these situations they will continue with scheduled meetings with other leadership team members, such as the General Counsel of DOT, leading the meeting. Officials at DOT said that ensuring the review meetings are held regularly is important because it avoids logistical challenges presented by rescheduling meetings, helps minimize preparation time by ensuring staff do not have to recreate meeting materials, encourages attendance, and leads to more productive discussions because staff are assured they will have a regular opportunities to raise issues for high- level attention.

A Few Agencies Reported Their COOs Do Not Lead Reviews

USDA, the Department of Defense (DOD), and State reported in our survey that the agency head or COO does not lead meetings that are used to review progress on APGs, as specified by GPRAMA and OMB guidance.

Agriculture. In its survey response and additional follow-up communication, USDA reported that the meetings between the Deputy Secretary and PIO that are held to discuss APG progress are led by the PIO, who presents information to the Deputy Secretary in these meetings.

Defense. DOD reported that its APGs are reviewed in meetings of the Defense Business Council (DBC), which has responsibility for the development and review of DOD’s performance goals. DBC meetings, however, are led by the Deputy Chief Management Officer, who is the PIO of DOD, rather than the Deputy Secretary of Defense, who is the COO. According to meeting attendance lists shared by DOD, neither the agency head nor Deputy Secretary/COO lead or regularly attend these reviews.

State. State reported that its PIO leads the review meeting for the one APG that State leads, and co-leads the review meeting with USAID for the one APG that is shared by the two agencies. A State official explained that the agency feels the PIO is appropriately suited for the role as leader of the review meetings as she also serves as the agency’s senior budget official. This dual role, according to the official, allows her to integrate performance with knowledge of agency resources. Neither the agency head nor the Deputy Secretary/COO lead or regularly attend these reviews.

These practices, however, are not consistent with OMB guidance, which clearly states that agency heads or COOs should conduct in-person meetings used to review progress on APGs. Leading practices emphasize that having leaders actively engaged in the reviews helps ensure that participants take the reviews seriously. As OMB has similarly noted, the involvement of COOs is critical to bringing a broader set of actors together to solve problems across the organization. Therefore, because the agency head or COO does not lead review meetings at these three agencies, the review process may be viewed as less of a priority by agency officials. This could have a detrimental effect on participation in reviews. It could also reduce opportunities for top agency leaders to reinforce responsibility and accountability, and to personally communicate their priorities and perspective to agency managers and staff.

Nearly All Agencies Reported That Performance Improvement Officers, Goal Leaders, and Other Key Contributors Participate in Reviews

GPRAMA Requirement: Participation of Priority Goal Leaders and Other Relevant Personnel GPRAMA requires that agency leaders include agency priority goal (APG) leaders in their reviews and coordinate with other relevant personnel within and outside the agency that contribute to the accomplishment of each APG.

OMB Guidance: Participation of Priority Goal Leaders and Other Relevant Personnel OMB guidance reinforces this requirement by requiring that agency leaders include APG goal leaders, or their designees, in the reviews, along with, as appropriate, relevant personnel within and outside the agency who contribute to the accomplishment of each APG.

Leading Practices for Data-Driven Reviews: Participation by Key Personnel Reviews should include personnel with programmatic knowledge and responsibility for the specific performance issues being discussed. In addition, the participation of officials with functional management responsibilities, such as information technology, financial management, and human capital, can facilitate problem solving by providing managers from across the agency with a forum to communicate with each other.

When officials from various offices and levels of management participate in review meetings, the meetings provide opportunities to have honest, informed discussions about performance with all key players present, and facilitate collaboration and group problem solving. Officials representing their program or area of responsibility may also feel increased accountability for results when forced to report on progress in front of leadership and peers.

Survey responses show that participation of PIOs in review meetings is strong, with 20 of 22 agencies reporting that PIOs always attend review meetings, and 2 reporting that their PIOs often attend. See figure 1 for reported frequency of participation in review meetings by agency leadership and other key contributors. As the highest officials dedicated to managing agency-wide performance management efforts, PIOs hold a unique position within their agencies and are key participants in the review meetings. PIOs and agency performance staff also engage in a variety of activities that directly support successful review meetings. Through discussions with agency officials and survey results, we found that responsibilities of PIOs and performance staff may include overseeing preparations for review meetings, including the collection and analysis of data, creation of presentation materials, and convening preparatory meetings; co-leading review meetings; and managing follow- up on action items identified in review meetings.

Participation by APG goal leaders in review meetings is also strong. Twenty-one out of 22 agencies reported that their goal leaders always or often participate in review meetings. Through our discussions with goal leaders we learned that they also play a key role in the review meetings, and present information on progress toward goals, respond to questions from agency leaders, identify problems or challenges and propose strategies to address them, and request support or assistance.

Most agencies also reported that other key officials with responsibility for agency financial management, human capital, information technology, and legal matters attend their review meetings. While there was variation across agencies on the frequency with which these officials participate in review meetings, 11 agencies reported that their Chief Financial Officers (CFO), Chief Human Capital Officers (CHCO), Chief Information Officers (CIO), Chief Acquisition Officers (CAO), and representatives from their Office of General Counsel (OGC) always or often attend the reviews. Officials from three of our five selected case-study agencies discussed the benefits of including chief officers in their reviews. These benefits include providing a cross-cutting agency perspective and specialized expertise to inform decisions, and offering assistance, resources, and problem solving support. For example, one DOT official described how the discussions in review meetings often focus on regulations that are under review by the Office of Information and Regulatory Affairs (OIRA) at OMB, which, in some instances, reviews regulations before they can be finalized. According to DOT officials, in the department’s review meetings, officials discuss the progress and plans of rulemakings with the Secretary’s office or OGC, such as facilitating early engagement with OIRA to address analytical issues. According to DOT officials, this is significant because the issuance of rules is an important tool that the department uses to promote progress toward its APGs. For instance, in its reporting on progress toward the APG to reduce the rate of roadway fatalities, DOT identified a number of proposed and final rules designed to reduce the risk of fatalities and serious injuries through enhancements to the safety of vehicles and roadways.

Two Agencies Reported Limited Participation in Review Meetings

USDA and State provided responses indicating that participation in their reviews is not fully consistent with requirements, guidance, and leading practices.

Agriculture. USDA responded to our survey that APG goal leaders participate in review meetings about half of the time. Through follow-up communication, USDA officials clarified that meetings between the Deputy Secretary and PIO in which APG progress is reviewed generally do not involve goal leaders. Officials also said, however, that when the Deputy Secretary had specific questions on APG progress, the Office of Budget and Program Analysis (OBPA) would schedule a follow-up meeting attended by the Deputy Secretary, PIO, goal leaders, and, occasionally, performance staff. USDA officials also reported that their CFO, CHCO, CIO, and CAO are rarely involved in the meetings, and that the General Counsel never attends. In subsequent follow-up communication, USDA officials stated that if additional information or action is needed from administrative, program, or policy officials, then the PIO and OBPA staff will act as a liaison, relaying questions and information between these officials and the Deputy Secretary. USDA officials said that given USDA’s large size and the complex and diverse nature of its multiple missions, it is generally easier logistically to have the PIO meet with the Deputy Secretary, rather than trying to schedule a meeting involving additional senior officials.

State. State responded to our survey that the CFO, CHCO, CIO, CAO, and General Counsel never attend review meetings. Upon subsequent follow-up with State officials, they could not provide an example of when these officials had been invited or attended review meetings, but said that they plan to invite officials with functional management responsibilities as appropriate in the future.

Not involving APG goal leaders in regular reviews of goal progress is inconsistent with GPRAMA requirements and OMB guidance; by not doing so, USDA may be missing opportunities for direct communication between agency leaders and relevant program staff about progress, challenges, and strategies for improvement. In addition, by not regularly including officials with functional management or legal expertise, as leading practices suggest, USDA and State may also miss opportunities to address performance issues in which human capital, information technology, acquisitions, or legal expertise could play a significant role in the development of solutions.

Agencies Reported Little External Involvement in Reviews

As we reported in our earlier evaluation of agency performance review meetings, OMB guidance and leading practices indicate that including key players from other agencies can lead to more effective collaboration and goal achievement. Specifically, OMB guidance states that agencies should include, as appropriate, relevant personnel from outside the agency who contribute to the accomplishment of an APG or other priority. When key players are excluded from performance reviews, agencies may miss opportunities to have all the relevant parties apply their knowledge of the issues and participate in developing solutions to performance problems. Instead, agencies will need to rely on potentially duplicative parallel coordination mechanisms, which could result in less than optimal performance improvement strategies.

Only two agencies, State and USAID, reported that they always or often include officials from outside the agency in their review meetings. These are also the two agencies that hold joint sessions to review progress on their shared APG. Most agencies, however, reported that external contributors never participate in their reviews.

In February 2013, we recommended that OMB work with the PIC and other relevant groups to identify and share promising practices to help agencies extend their performance reviews to include, as relevant, representatives from outside organizations that contribute to achieving their agency performance goals. OMB generally concurred with the recommendation, and in July 2014, staff from OMB and the PIC told us that meetings of the PIC Internal Reviews Working Group have been used to discuss the inclusion of representatives from external organizations in agency reviews, as appropriate. In March 2015, OMB staff said that while they have found that at times it is useful to engage external stakeholders in improving program delivery, agencies view reviews as internal agency management meetings. Thus, they believe it would not always be appropriate to regularly include external representatives. According to PIC staff, the PIC continues to work with agencies to identify examples where agencies have included representatives from outside organizations in quarterly reviews, and to identify promising practices based on those experiences. As those promising practices are identified, PIC staff plan to disseminate them through the PIC Internal Reviews Working Group and other venues. We will continue to monitor these efforts and periodically report on their status.

All Agencies Reported Collecting and Analyzing Data for Reviews

GPRAMA Requirement: Review of Quarterly and Trend Data on Priority Goal Progress GPRAMA requires that participants review, for each APG, progress achieved during the most recent quarter, overall trend data, and the likelihood of meeting the planned level of performance.

OMB Guidance: Review of Quarterly and Trend Data on Priority Goal Progress OMB guidance reinforces this requirement by directing participants to review progress achieved during the most recent quarter, overall trend data, and the likelihood of meeting the planned level of performance. It also says that, in the reviews, agency leaders should hold goal leaders accountable for knowing the quality of their data, for having a plan to improve it if necessary, and for filling critical evidence or other information gaps.

Leading Practice for Data-Driven Reviews: Collecting and Analyzing Performance Data Participants in a data-driven review meeting must have up-to-date, accurate data on performance to have a meaningful discussion about progress toward goals and milestones. The capacity to collect relevant and timely data and the ability to analyze it to identify key trends, areas of strong or weak performance, and possible causal factors are critical to successful reviews.

As we have previously reported, the capacity to collect and analyze accurate and useful data is critical to successful data-driven reviews. The collection and analysis of valid, up-to-date performance data in advance of data-driven review meetings is necessary to ensure that the most timely data and information are used to inform discussions in meetings, and that key trends or areas of strong or weak performance have been identified. The collection and analysis of up-to-date data for review meetings is also necessary because GPRAMA and OMB guidance require that reviews be used to review progress toward APGs.

All 22 agencies reported that they always or often collect data on APG performance measures and milestones in advance of their review meetings. Furthermore, all 22 agencies reported that they always or often analyze these data to identify key performance trends or patterns and areas of strong or weak performance. See figure 2 below for information on the frequency with which agencies reported that they take specific data collection and analysis actions prior to their review meetings.

All five of our selected case-study agencies established processes for collecting and analyzing performance data in advance of their review meetings. At each of the agencies, officials told us that those managing the preparation for review meetings collect updated performance data from goal leaders and their staffs. Some agencies used a standardized template to collect and organize the performance data and other relevant information about progress toward goals and milestones such as risks, challenges, and future actions. GSA also used an online spreadsheet that offices were required to regularly update with new information on progress toward specific agency goals or milestones. See figure 3 for a screenshot of this spreadsheet.

Collecting accurate and timely data is critical for successful performance reviews, but our survey found that 19 of 22 agencies identified this as a challenge. This finding is consistent with our previous survey of agency PIOs, as well as past surveys conducted by the PIC, which found that the primary challenges agencies faced when implementing reviews included access to data and limitations in the capability of their data systems. It appears, however, that the attention and scrutiny data receive through the review process can help agencies identify and address problems or limitations. In fact, 20 of 22 agencies reported that their reviews have had a positive impact on the quality of the performance data used to track progress and inform decision making within their agencies. According to OMB staff, in February 2015, OMB and the PIC also formed a cross- agency working group on data quality comprised of agency and OMB representatives. The stated objectives of the working group, which will meet through August 2015, are to identify guidelines and practices that would improve the reliability and quality of performance data and the reporting process, and establish standards and consistency across the federal government. We are assessing the quality of publicly reported information on APGs in selected agencies and plan to discuss this cross- agency working group in more detail in an upcoming report in the summer of 2015.

Officials from some of our five selected case-study agencies described the actions they have taken to address challenges presented by lagging or limited performance data. For example, to track progress on the reduction of improper payments, the goal leader for SSA’s improper payments APG previously received relevant data annually. To increase the frequency with which new data are available, the PIO initiated a conversation on increasing the frequency with which payment accuracy data is received and the goal leader worked with the SSA Office of Quality Review, which collects the data, to increase the frequency to every 6 months. An SSA official said that having more current data has given the agency a better indication, at an earlier stage, of its progress in a given year and of any impacts its actions may be having. At HHS, a key indicator tracked for the HHS early childhood education APG is the number of states with Quality Rating and Improvement Systems that meet seven benchmarks. The data for this indicator, however, are only available annually. Because more frequently updated data are not available, HHS officials asked the goal leader to disaggregate the data by geographic region to allow for a more granular examination of conditions and trends across regions. The Office of Child Care analyzed state progress toward implementing Quality Rating and Improvement Systems that met the seven benchmarks. As part of the analysis, the Office of Child Care identified the most common gaps in the states, and created a map that provided a visual representation of state progress toward the goal.

Officials from two of our five selected case-study agencies also reported specific challenges related to their capacity to perform data analysis to inform performance management. As we have previously reported, this helps ensure performance information is analyzed and communicated For example, an SSA official effectively, and used in a meaningful way.said that the agency had insufficient analytical capacity to perform deep, detailed analysis of data on the use of SSA services and the relationships between the use of these services and other factors. To address this limitation, SSA created an Office of Performance Management and Business Analytics to collaborate with other SSA offices to gather and analyze agency data, and to perform complex data analyses. SSA also facilitates initiatives like an internal training program where senior data analysts train other staff. The agency is also seeking to hire an advanced data scientist.

In April 2013, we reported on the importance of ensuring that agency performance management staff have sufficient capacity to support performance management in federal agencies, and recommended that the Director of the Office of Personnel Management (OPM), in coordination with the PIC and the Chief Learning Officer Council, work with agencies to identify competency areas needing improvement within agencies, and identify training that focuses on needed performance OPM and OMB staff agreed with this management competencies.recommendation. In July 2014, OPM told us that it had coordinated with the PIC on this recommendation, and that the PIC would take responsibility for the remaining actions needed to implement this recommendation. In March 2015, OMB and PIC staff said that the PIC has created a number of training programs designed to provide agency officials with information about performance management, and approaches for using performance management to improve agency performance. The PIC has also created a public website, LearnPerformance.gov, with informational resources on a range of topics, including measurement, data and analysis, and reporting and communicating performance information. We will continue to monitor these efforts as training and other knowledge sharing efforts are implemented and expanded.

All Agencies Reported Developing Materials and Many Reported Holding Preparatory Sessions for Reviews

GPRAMA Requirement: Identifying “At Risk” Goals GPRAMA requires that agencies categorize agency priority goals (APGs) by risk of not achieving the planned level of performance.

OMB Guidance: Identifying “At Risk” Goals OMB guidance also directs agencies to identify APGs (or other priorities) at risk of not achieving the planned level of performance and work with goal leaders to identify strategies that support performance improvement. It also directs them to review variations in performance trends across the organization and delivery partners, identify possible reasons for the variance, and understand whether the variance points to promising practices or problems needing greater attention.

Leading Practice for Data-Driven Reviews: Rigorous Preparation Rigorous preparation is critical for effective performance reviews, as key participants must be prepared to discuss issues related to their performance and progress toward goals. After data have been collected and analyzed, they must be effectively communicated to participants.

Following the completion of data collection and analysis, offices responsible for supporting reviews will often compile summary materials to help leaders and participants prepare for the reviews. These efforts to ensure that participants are aware of the status of goals and milestones, and key questions likely to be raised and discussed in the meetings, can also be critical to the success of reviews. As we have also previously reported, frequent and regular communication of performance information is also critical to remind agency officials of their commitment to achieve the agency’s goals, and to keep those goals in mind as they pursue their day-to-day activities. It also helps ensure that leaders and managers have opportunities to review information in time to take action to make improvements.

All 22 agencies reported that they always or often develop presentation slides or other meeting materials to communicate key data and analyses to participants. Furthermore, all 22 agencies also reported that they always or often distribute these materials to participants for review before the meetings. See figure 4 below for information on how frequently agencies report they take specific actions to prepare for review meetings.

All five of our selected case-study agencies developed presentation slides, or other meeting materials, and distributed them to participants in advance of their review meetings. In addition to presenting information on progress toward agency goals and milestones, meeting materials may also include discussions of key strategies and initiatives being employed to influence progress, and any risks, challenges, or opportunities those managing the goals are facing. In accordance with the GPRAMA and OMB requirement that agencies categorize APGs by risk of not achieving the planned level of performance, materials produced for meetings at all five of our selected agencies included information or color-coded graphics to indicate the likelihood a goal will be achieved and whether a goal is “off track” or “at risk.” For two examples of materials prepared for review meetings at SSA and HHS, see interactive figures 5 and 6.

Fourteen of 22 agencies reported that they always or often held a preparatory session to review the agenda, data, and key discussion points with participants before their review meetings. Officials from our five selected case-study agencies described preparatory meetings that officials from their agencies hold in advance of review meetings. Officials from some of the five agencies also described how these preparatory sessions can be valuable, as they allow agency leaders and goal leaders to familiarize themselves with the data and discuss responses to potential questions with knowledgeable staff.

General Services Administration. Two of the agency’s bureaus, the Public Building Service and the Federal Acquisition Service, hold regular meetings where managers from each service review and discuss performance data presented later at the agency-level performance review meetings. Officials see these meetings as not only preparation for the agency-level review meetings, but as vital to effectively managing the business of the services and making progress toward identified goals.

Social Security Administration. To prepare for the quarterly review meetings with the Acting Commissioner of SSA, the Chief Strategic Officer/PIO meets with SSA’s deputy commissioners, goal leaders, and appropriate staff to discuss progress toward APGs, the status of efforts being employed to achieve them, and the order in which goals should be discussed in the quarterly review. This preparatory meeting is held 10 days before the quarterly review meeting is scheduled to be held. Five days before the quarterly review meeting, the Chief Strategic Officer/PIO meets with the Acting Commissioner to prepare for the quarterly meeting. At this meeting, they discuss goal progress and trends, issues to be discussed during the review meeting, and potential questions the Acting Commissioner could ask. Materials prepared by the APG goal teams for the quarterly review are sent to the Acting Commissioner 48 hours in advance of this preparatory meeting.

Transportation. An official from the Federal Railroad Administration (FRA) explained that, prior to FRA’s review meetings with the Deputy Secretary of Transportation, staff hold briefings for the FRA Administrator and each Associate Administrator. This official said that these preparatory meetings for FRA leadership are valuable because they allow FRA leadership to ask questions of knowledgeable staff to better understand the data and information they will ultimately present at the review meeting with DOT leadership.

Most Agencies Reported Using Review Meetings to Review Progress on Priority Goals and Identify Strategies for Improvement

GPRAMA Requirement: Review of Priority Goal Progress and Identification of Improvements GPRAMA requires that agency leaders review progress on agency priority goals (APGs); assess whether relevant organizations, programs, regulations, and policies are contributing as planned; and identify strategies for performance improvement for those goals at greatest risk of not achieving their planned levels of performance.

OMB Guidance: Review of Priority Goal Progress and Identification of Improvements OMB guidance reinforces this requirement by directing agency leaders to use in-person review meetings to review progress on APGs; hold goal leaders accountable for knowing whether their performance indicators are trending in the right direction and, if not, having a plan to accelerate progress on the goal; identify APGs or other priorities at risk of not achieving planned levels of performance; and work with goal leaders to identify strategies that support improvement.

Leading Practice for Data-Driven Reviews: Accountability Agency leaders should use review meetings to hold goal leaders and other responsible managers accountable for knowing the progress being made in achieving goals and, if progress is insufficient, understanding why and having a plan for improvement.

As mentioned throughout this report, a fundamental purpose of data- driven review meetings is to provide a mechanism for agency leaders to assess an agency’s progress on key goals and milestones; analyze and discuss data to identify goals at risk, performance problems, and improvement opportunities; and ensure that goal contributors are held accountable for their performance.

Through our survey, we found that most agencies reported they always or often use their review meetings to assess progress and contributions, and identify goals at risk. However, as shown in figure 7 below, there is some variation reported across the 22 agencies on the frequency of specific types of actions taken during review meetings.

Assessing progress on APGs. Reviewing progress on APGs on a regular and ongoing basis is a key requirement of GPRAMA, and helps ensure that agency leaders, goal leaders, and other contributors have frequent opportunities to review recent progress and trends. Twenty of 22 agencies reported that their data-driven review meetings are always or often used to review progress on APGs, including recent progress, overall trends, and the status of related milestones.

Analyzing data to identify goals at risk and hold goal leaders accountable. GPRAMA also requires that agencies identify and categorize goals at risk of not achieving the planned level of performance. Twenty-one of 22 agency PIOs reported that their review meetings are always or often used to identify goals at risk, and to hold goal leaders accountable for explaining why the goal is at risk, as well as strategies for performance improvement.

Discussing contributions of program activities, policies, and regulations and whether they should be changed to improve their impact on priority goals. Assessing the contributions that organizations, program activities, policies, and regulations are making toward the achievement of goals is another critical part of efforts to use review meetings to ensure accountability for the completion of commitments, and to identify potential problems, effective practices, or strategies for improvement. GPRAMA requires that agencies include these assessments as part of their reviews. Twenty of 22 agencies reported always or often discussing whether specific organizations or program activities were contributing as planned to priority goals, and 18 of 22 reported always or often discussing the contributions of relevant policies toward priority goals. In this way, data-driven review meetings can be used to reinforce the alignment of higher-level agency goals with the milestones and day-to-day activities of program officials contributing to each goal. However, only 13 of 22 agencies reported always or often discussing whether program activities, policies, and regulations should be changed to improve their alignment with priority goals.

SSA officials described how they have used their review meetings to discuss the contributions of programs, policies, and regulations, and necessary changes. For example, SSA has an APG to expand the use of video technology to hold benefit determination hearings. According to SSA officials, initial discussions on this goal in quarterly review meetings identified the challenge that those scheduled for video hearings were opting out at the last minute, which led to unpredictable schedules and down time for Administrative Law Judges. To address this issue and help achieve the broader goal to expand the use of video hearings, the agency determined that a regulatory change was needed to require claimants to decline a video hearing within 30 days after the date the claimant receives notice that the agency may schedule the claimant to appear at a hearing by video teleconferencing. This change was designed to decrease last-minute hearing cancellations and help them more efficiently schedule video hearings. The milestones that were developed to track progress on the development and implementation of the regulatory change were regularly discussed in the quarterly meetings, which is one of the factors that led to the agency working with OMB to expedite the release of the regulation.

SSA believes that this June 2014 regulatory change will have long-term benefits. However, SSA has acknowledged that in the short term they may receive more opt-outs due to the 30-day notice requirement. For this reason, they are tracking the opt-out rate for video hearings to measure the impact of the new regulation, and are reviewing these data in their quarterly meetings. In the quarterly review meeting we observed after the regulation had been implemented, participants discussed several potential consequences of the regulation, including the potential for an increase in the opt-out rate, and possible strategies for addressing them, such as additional regulatory or process changes.

Some Agencies Reported Taking Certain Actions during Reviews Less Frequently

Some agencies reported through our survey that they take certain actions during review meetings about half of the time or less frequently. For example, the Department of Labor (Labor) reported that it reviews APG progress about half the time in review meetings. However, a Labor official explained through follow-up communication that it responded this way because each quarter it holds performance review meetings for each of its 16 components and not all components have responsibility for one or more of the APGs. Therefore, APG progress is discussed only during review meetings for components that contribute to APGs. The agency specified, however, that each quarter it reviews progress on each of its APGs.

Three agencies—the Departments of Energy (Energy) and Health and Human Services, and the National Science Foundation (NSF)—reported that they rarely discuss whether program activities, policies, and regulations should be changed to improve their alignment with priority goals in review meetings. Two other agencies—the Department of Defense (DOD) and the National Aeronautics and Space Administration (NASA)—reported that they never hold these discussions. Through follow-up, officials from Energy, HHS, NASA, and NSF clarified their responses to this question and explained how their review meetings were used to identify and discuss weaknesses or risks that could impact the achievement of their goals, and discuss suggestions for improvement.

Energy. While Energy’s quarterly review meetings were used to discuss APGs identified as “off track,” and review future plans and milestones for these goals, an agency official said that other goal- specific meetings were held to drive action on at-risk goals. The quarterly review meetings also served as an opportunity for senior leaders not involved in the other meetings to discuss goal progress and actions being taken to improve efforts in those areas that are off track.

Health and Human Services. An HHS official stated that the agency’s response was due mainly to the fact that review meetings are generally not used to discuss regulatory changes, with some exceptions, such as reviews held for the health information technology APG. Instead, the official said that discussions in HHS review meetings are focused primarily on improving progress on APGs through better implementation and execution of program activities and other management initiatives. HHS leaders that attend the review meetings, however, may use the information gained to inform decisions on longer-term policy or regulatory changes.

NASA. A NASA official stated that the agency’s APGs are closely aligned with specific agency programs and projects, and that monthly data-driven performance review meetings are used to discuss potential cost, schedule, technical, and programmatic risks to meeting their milestones, as well as strategies for improving performance. NASA does not, however, discuss realigning or changing programs or policies to meet those milestones. For example, in their quarterly reviews of the James Webb Space Telescope program, participants have had discussions of actions the program will undertake to meet its milestones, but not of changing the program, or of reassigning this work to a different agency program, as the program is the only one with the capability to implement the work.

National Science Foundation. NSF officials stated that they have no relevant regulations to discuss in the agency’s review meetings, but NSF officials do discuss potential changes to program activities and policies at review meetings as the need for program or policy changes become apparent, which is generally about half of the time. For example, NSF has an APG to improve the nation’s capacity in data science by focusing NSF investments in human capital, partnerships, and infrastructure that support data science. Initial plans for this APG were set at the time the goal was established and expressed as a series of quarterly milestones. The timing of achievement of these milestones is occasionally altered, and NSF review meetings have been used to discuss these changes. In one such recent change, NSF officials originally planned to support Big Data Regional Innovation Hubs in fiscal year 2014, but decided to gather more community input to increase the specificity and quality of its proposals. Request for Comments was published in the Federal Register, with a public comment period ending November 1, 2014.According to an NSF official, the submissions to the request were used to refine the solicitation for Big Data Regional Innovation Hubs that was subsequently released in the second quarter of fiscal year 2015. These timing changes were presented at each review meeting and discussed as necessary.

Big Data Regional Innovation Hubs are designed to be consortiums of members from academia, industry, and government that would foster collaboration amongst partners, and focus on key Big Data challenges and opportunities in their regions of service. our survey that participants review progress on APGs only about half the time, rarely identify goals at risk, and never discuss whether program activities, policies, or regulations should be changed to improve their alignment with priority goals. This is consistent with our own review of documentation from DBC meetings, which indicated that a review of APGs was not always included on the agenda. In those instances when APG progress was reviewed, the information on APG progress included in meeting materials was limited. For instance, materials prepared for some meetings had only one slide with an aggregate count of how many APGs were on or off track, and limited information on the status of individual APGs. Without information on the status of individual APGs, DOD’s review meetings are unlikely to foster meaningful discussions about progress and trends. Further, if these review meetings are not regularly used to assess progress on individual APGs, and to identify at- risk goals and potential improvements, it could mean missed opportunities for DOD to address performance problems or accelerate progress. DOD officials informed us, however, that over the next year, they plan to revise their review process to ensure they conduct regular, quarterly reviews of APG progress that involve discussions on progress achieved in the most recent quarter, performance trends, and status of related milestones; discussions of potential organization, program activity, policy, or regulatory changes to improve alignment with, and impact on, priority goals; and the identification of at-risk goals.

Most Agencies Reported Taking Steps to Follow up on Issues Identified in Reviews, but with Varying Frequencies

OMB Guidance: Follow-Up OMB guidance directs agency leaders to agree on follow-up actions at each review meeting and track timely follow-through.

Leading Practice for Data-Driven Reviews: Follow-Up Rigorous and sustained follow-up on issues identified during meetings, including the identification of the individual or office responsible for each follow-up action, is critical to ensure the success of reviews as a performance improvement tool.

Identifying and agreeing upon actions that need to be taken following a review meeting, and rigorously tracking the status of these actions to completion, is a key element of OMB guidance as well as a leading practice. Rigorous follow-up is also critical to the overall success of reviews as a tool for addressing identified deficiencies and improving performance.

According to our survey results, most agencies reported that they are generally taking steps to identify and follow up on action items identified in review meetings. However, this is an area where our survey indicated there is less consistency in how frequently agencies are employing specific practices. Figure 8 shows the frequency with which agencies reported conducting specific follow-up actions.

The variation in how systematically agencies identify and follow-up on action items from review meetings is also illustrated by the different approaches that our five selected case-study agencies reported using to identify and follow-up on action items, which are described in table 4.

The analysis of responses to our survey indicated that there is a statistically significant, positive correlation between the frequency with which an agency identifies and agrees on specific follow-up actions and the perceived impact of review meetings on performance improvement. Specifically, as shown in figure 9, all 13 agencies that reported that their review meetings have had a major impact on performance improvement also always or often identify and agree on follow-up actions during review meetings. Agencies that reported their review meetings have had a minor impact on performance improvement reported identifying and agreeing on follow-up actions during review meetings less frequently. Similarly, our analysis found that a statistically significant, positive correlation exists between the frequency with which an agency uses its review meetings to review the status of follow-up actions from the previous meeting, and the perceived impact those reviews have on performance improvement. These findings are consistent with surveys of agency PIOs administered in the past by the PIC. These surveys found that agencies where reviews have had a major impact on agency performance are more likely to document specific action items with clear owners and due dates, and review follow-up actions from previous meetings.

While OMB guidance and leading practices are clear that participants in each review meeting should agree on follow-up actions and track follow- through, four agencies – DOD, Energy, NSF, and the Small Business Administration (SBA) – reported through our survey that they identify and agree on specific follow-up actions about half the time or less frequently. Through follow-up with Energy, SBA, and NSF, officials from those agencies further explained the actions they are taking, or have taken, to identify, document, and track follow-up items, are consistent with OMB guidance.

Energy. Energy reported through our survey that participants identify and agree on specific follow-up actions in quarterly review meetings about half of the time. According to an Energy official, however, in instances where follow-up actions are identified, those items are documented in a “Summary of Actions.” In addition to using quarterly review meetings to identify follow-up actions, the official stated that other topic-specific meetings are used to identify and address follow-up items for specific APGs. For example, the Summary of Actions from Energy’s August 2014 quarterly review meeting indicated that the Deputy Secretary would hold a meeting with a specific agency official to review the off-track status of a priority goal in more detail.

Small Business Administration. SBA reported through our survey that participants would rarely identify and agree on specific follow-up actions to be taken after meetings. During the course of our review, however, SBA officials instituted changes to the agency’s review processes as a result of new leadership, and have given the SBA Office of Performance Management responsibility for ensuring that all action items from their review meetings, as well as “key takeaways” for discussion at the next review, are recorded.

National Science Foundation. NSF reported through our survey that participants rarely identify and agree on specific follow-up actions. However, NSF officials stated that they chose this response because their goals are based primarily on the achievement of milestones and goal teams have already outlined the specific actions they will be taking in goal documentation. The status of actions to complete each of these milestones is then reviewed in each review meeting. NSF officials also said that in the event a follow-up action or course correction is identified in a quarterly meeting, the status of these actions will be discussed in bi-weekly meetings between the PIO and COO, who determine whether the actions have been adequately addressed or whether additional steps are required.

In contrast, DOD reported through our survey that participants in review meetings rarely identify and agree on specific follow-up actions. After subsequent follow-up with the agency, we found that DOD practices are not consistent with OMB guidance or leading practices. Through our review of documents from DOD review meetings, we also found there was no information included in materials prepared before, or after, these meetings to indicate that they are used to identify follow-up actions related to APGs. In our follow-up communication with them, DOD officials acknowledged the need to regularly identify follow-up actions, and informed us that over the next year they plan to integrate the identification of specific follow-up actions into their reviews.

Clearly identifying and documenting follow-up items, identifying the individual or office responsible, and monitoring their status are important to ensure that agreed upon actions are taken after DOD’s review meetings. This is supported by the results of our analysis, which showed that systematically identifying and following up on action items is associated with review meetings having a greater impact as a performance improvement tool. Furthermore, a failure to clearly identify and document follow-up actions may lead to a situation at DOD in which there is no commonly-held list of specific actions that will be taken after review meetings, and a limited ability to hold accountable those responsible for the completion of action items.

Most Agencies Reported Review Meetings Have Positively Affected Performance, Collaboration, Accountability, and Efficiency

The results of our survey on agency data-driven review practices indicate that review meetings have had positive effects on progress toward agency goals, collaboration between agency officials, the ability to hold agency officials accountable for progress toward goals, and the ability to identify opportunities to improve agency operations. COOs, PIOs, APG goal leaders, and staff that we spoke with at the five selected agencies reinforced these findings, and also shared examples that illustrate the positive effects their data-driven review meetings are having in these areas.

Nearly All Agencies Reported Review Meetings Have Positively Affected Progress toward Agency Goals

Nearly all agencies reported that their data-driven review meetings have had a positive effect on progress toward the achievement of agency goals, and on their ability to identify and mitigate risks to goal achievement. As illustrated in figure 10, all 22 agencies reported that their reviews have had a positive effect on progress toward their APGs, and 21 of 22 reported that their reviews have had a positive effect on their agency’s ability to identify and mitigate risks to achieving priority goals.

In our discussions with officials from selected agencies, data-driven review meetings were described as venues for agency leaders and managers to assess progress toward key goals and milestones, the status of ongoing initiatives and planned actions, potential solutions for problems or challenges hindering progress, and additional support or resources needed to improve performance.

Agency officials emphasized that discussions in their review meetings tend to focus on those goals or issues most in need of attention, where the achievement of a goal or milestone is at risk. In this way, reviews can serve as early warning systems and facilitate focused discussions on external, technical, or operational obstacles that may be hindering progress, and the specific actions that should be taken to overcome them.

For example, SSA has an APG to increase the number of registrations for its my Social Security portal by 15 percent per year in fiscal years 2014 and 2015. In 2014, however, through the review of data for SSA’s third quarter review meeting, it became apparent to SSA leadership that the agency was not on track to achieve its target for this goal. According to officials, as part of the quarterly review process agency officials completed a more thorough examination of reasons for this and found that the agency would not be able to complete the development of additional features, such as the ability to request a replacement Social Security card, which were expected to drive higher volumes of traffic to the portal. Understanding these limitations, SSA’s focus shifted to what could be done by offices throughout the agency, using currently available or attainable resources and technology, to support efforts to increase the number of registrations. To achieve this, SSA leadership had different offices within the agency, including Communications, Policy, and Budget, specify the contributions they would make to help increase the number of registrations. For example, the Office of Communications developed a document outlining 26 activities the office was taking, or planned to take, to promote my Social Security to potential users. Since then, the agency’s quarterly review meetings have been used to review and reinforce the commitments each office made.

In the quarterly review meeting that we observed, a representative of SSA’s Communications office emphasized that supporting efforts to increase my Social Security registrations is the office’s top priority, and discussed an ongoing national marketing campaign, and marketing activities targeted to advocates in the aging and disability communities and third party tax preparers. While SSA was unable to meet the registration goal for fiscal year 2014, according to SSA officials, these efforts recently undertaken as a result of the review process have helped generate an increase in registrations. Data from SSA’s fiscal year 2015 first quarter review show that there was a 46 percent increase in new account registrations in October 2014 compared to the number of new registrations in October 2013, and a 26 percent increase in December 2014 relative to December 2013.

Many agencies reported that they are also using their review meetings to review progress on a broader suite of performance goals that go beyond the requirement to review APGs. Nineteen of 22 agencies reported that they always or often discuss progress on agency-wide goals or initiatives beyond the APGs in their review meetings, while 20 of 22 agencies reported that reviews have had a positive effect on their progress toward the achievement of other performance goals. For example, according to a GSA official, a long-standing challenge of the Public Buildings Service (PBS) has been finalizing occupancy agreements in a timely fashion.2014, agency leadership made improving performance in this area a specific goal for PBS, which was then often discussed during GSA’s review meetings. According to GSA officials, due to the increased attention on the status of goal progress and leadership commitment to improving performance, the agency has surpassed its goals in this area. According to GSA’s performance report, in fiscal year 2014, the agency improved the on-time activation of occupancy agreements in owned space to 98 percent and leased space to 90 percent, exceeding the targets of 90 percent in owned space and 82 percent in leased space. This is also an improvement from the on-time activation rates of 86 In percent for owned space and 75 percent for leased space in fiscal year 2013.

Nearly All Agencies Reported Reviews Have Positively Affected Collaboration between Officials in Their Agencies

Twenty-one of 22 agencies reported that their data-driven reviews have had a positive effect on collaboration between officials from different offices or programs within the agency. Similarly, agency officials with whom we spoke emphasized that review meetings provide opportunities to bring together the people, analytical insights, and resources from across an agency that are needed to improve progress on agency priorities and to address any identified performance problems or challenges. As we heard from agency officials, and summarized in figure 11, bringing leaders and officials from across an agency together regularly to focus on shared goals and milestones can establish a shared sense of purpose, encourage ongoing collaboration, and reduce organizational silos. The review meetings also serve as action-forcing events that provide an opportunity for officials from across an agency to develop and implement collaborative solutions to identified problems. These insights into the positive effects review meetings can have on collaboration within agencies also reinforce their potential value as a tool for promoting increased collaboration across agencies. As noted above, this should encourage those who lead and manage agency reviews to follow OMB guidance on the issue and be mindful of opportunities to leverage reviews to involve relevant stakeholders from external agencies or organizations.

Department of Health and Human Services (HHS) officials reported that promoting collaboration between different offices that contribute to individual APGs has been one of the most important effects of their reviews. They also emphasized that reviews have been used to bring APG contributors from across HHS together to discuss what can collectively be done to support progress on the goals. For example, HHS has an APG to increase the number of eligible providers who receive incentive payments for the successful adoption or demonstration of meaningful use of certified electronic health record (EHR) technology. According to HHS officials, the goal requires a great deal of coordination between the Office of the National Coordinator for Health IT (ONC) and the Centers for Medicare & Medicaid Services (CMS). An HHS official involved in these efforts explained that the two offices realize that, given their shared ownership of the goal, they are expected to coordinate effectively, and the reviews are used to reinforce this expectation and ensure that ongoing coordination is occurring.

One way this has manifested itself is in the improved data sharing arrangement that now exists between ONC and CMS. Under this arrangement, CMS collects data related to the Medicare and Medicaid EHR Incentive Programs, which provide financial incentives for the “meaningful use” of certified EHR technology by health care providers. According to ONC officials, the review process has helped encourage more regular data sharing between CMS and ONC, with ONC receiving monthly updates on EHR Incentive Program registration, attestation, and payment data from CMS. The team supporting the priority goal is now using these data to conduct ongoing evaluations of the characteristics of providers at different stages in the program. According to an ONC official, data on program participation are also shared by ONC and CMS during monthly presentations to the Federal Advisory Committees on Health IT, which illustrates how they are also using these data to facilitate partnerships and public-facing discussions with other stakeholders.

Similarly, officials from GSA stated that they believe that the most significant effect of the agency’s review meetings has been to enable collaborative problem solving, where ideas for potential solutions can be freely shared, and officials can request assistance from their colleagues in offices throughout GSA. For example, as part of a 2013 reorganization, the Office of Administrative Services (OAS) was given responsibility for the management of GSA-occupied real estate. According to GSA officials, through review meetings agency leaders identified that regional offices were estimating project costs and footprints using different assumptions. OAS was able to partner with PBS, which had experience working with other agencies to estimate project costs and footprints, to leverage their experience in the consideration of internal agency real estate needs. This led to the creation of a new streamlined process for project plans, with the head of OAS reviewing all plans to ensure they conform to consistent standards. Agency leaders said this coordination likely would not have happened if the two offices had remained siloed in their approaches, as they were before they began coordinating in GSA’s regular review meetings.

Nearly All Agencies Reported Reviews Have Positively Affected the Ability to Hold Officials Accountable for Progress

Having regular review meetings that require goal leaders and other contributors to report out on their progress, and respond to direct questions about the actions that they are taking, provide leaders with important opportunities to clarify and reinforce responsibilities, motivate actions necessary to complete milestones or improve performance, and hold goal leaders and managers accountable. The involvement of agency leaders in review meetings also helps ensure that the meetings are taken seriously and viewed as a priority.

Twenty-one of 22 agencies reported that their data-driven reviews have had a positive effect on their agency’s ability to hold goal leaders and other officials accountable for progress towards goals and milestones.According to officials from selected agencies, the transparency of performance information, and a review process that ensures it receives appropriate scrutiny, produces an increased sense of accountability for results.

Several agency officials emphasized that it is important for those leading the meetings to establish a constructive, solution-oriented environment in which officials can be open and honest about their progress, and any problems and challenges they are facing. This outlook is also reinforced by OMB guidance, which directs agency leaders to establish an environment that promotes learning and openly sharing successes and challenges. At the same time, while agency leaders must maintain an environment in which participants feel comfortable raising problems and challenges, officials we spoke to also emphasized that leaders must use the meetings to hold goal leaders and managers accountable for having well-thought-out strategies for how to overcome them or mitigate their impact.

Leaders from Commerce said that they are using their regular review meetings with bureau heads and goal leaders to support a cultural change throughout the agency and to reinforce accountability for performance at multiple levels of the organization. This change is one that emphasizes regular and ongoing reviews of performance, accountability for the completion of action items, more frequent and regular follow-up through review meetings, and an increased urgency and pace of implementation. For example, in 2014, Commerce established an APG designed to increase the percentage of companies assisted by the Global Markets (GM) program that achieve their export objectives.Commerce officials, the decision to focus on this new measure, which places the focus on the satisfaction of clients, came out of input from GM clients and staff, as well as discussions in departmental review meetings about the need to clarify the objectives of the program and improve the quality of assistance that trade and commercial specialists provide.

The key measure used to track progress on the priority goal was designed to help drive changes in internal processes and behaviors by focusing more clearly on activities and outcomes GM staff are able to directly influence. Now, at the beginning of their interaction with a new client, GM staff ask client businesses to define their needs and what they want to achieve with the support of the program. This shift toward a more consultative approach was designed to encourage staff to work with clients to establish goals and expectations, design solutions responsive to those goals, and identify potential challenges.

Commerce has also instituted new data collection procedures for the goal. It now collects data on APG performance on a regular, weekly basis, and in a way that will allow for comparisons across regions. The GM Office of Strategic Planning sends out weekly updates to leaders and managers with data on how each region, and GM as a whole, is performing against weekly and annual targets for a number of key metrics. These data on performance are also being reviewed by GM leaders in monthly review meetings to see how regions are performing in relation to one another and against established targets, and to identify challenges that offices are experiencing and effective practices that could be more widely shared.

According to Commerce officials, progress on the APG is discussed in regular review meetings at the bureau and departmental levels. For example, the Deputy Under Secretary for International Trade, who oversees the daily operations of the International Trade Administration (ITA), meets regularly with the APG goal leader to discuss performance on the goal. Progress on the goal is also discussed in monthly meetings of the ITA Management Council, which consists of the senior leaders of each of ITA’s three business units. Finally, progress toward, and the management of, the APG is also discussed in department-level review meetings. Here, Commerce leaders provide APG goal representatives with specific feedback and guidance, as well as information on other agency-wide initiatives that could impact the ability of GM staff and leadership to appropriately manage the program.

According to a Commerce official, the priority goal and associated measures, data collection process, reviews at multiple levels, and a system that holds staff accountable for identifying what clients want to achieve and working to deliver on those expectations have provided an increased sense of accountability and strong incentives for behavioral change amongst staff within the program. Commerce reported that in the first quarter of fiscal year 2015, 73 percent of GM clients reported that they have achieved their export objectives, exceeding the current target goal of 71 percent.

Most Agencies Report Reviews Have Positively Affected the Efficiency of Agency Operations

Seventeen of 22 agencies reported that their data-driven reviews have had a positive effect on the efficiency of agency or program operations.Some of our selected agencies are using their review meetings as an opportunity to review the status of management improvement initiatives and to improve the efficiency of business operations.

For example, DOT’s review meetings have been used to uncover and correct inefficiencies in its hiring process, which involves multiple offices throughout DOT. According to a Federal Railroad Administration (FRA) official we spoke with, staff were able to calculate how many days it took to complete each step in its hiring process and identify which offices were responsible for each step. The increased scrutiny this issue received through DOT’s review meetings led to improvements in the average number of days it now takes to hire a new employee. For example, for FRA time-to-hire decreased from approximately 160 days in fiscal year 2012 to 77 days in the third quarter of fiscal year 2014.

Agency officials also emphasized that review meetings that bring together leaders and officials from across an agency can increase the overall efficiency of an agency’s decision making processes. These meetings allow leaders and managers to discuss and respond to ideas, ask questions, voice concerns, and immediately make decisions on how to move forward. If these review meetings did not exist, officials explained they would likely need to hold separate meetings, with questions and answers moving up and down the agency’s lines of communication. Having all key players present in one meeting makes this entire process more efficient and allows for more timely action.

Sustaining Positive Effects of Reviews Requires Ongoing Leadership Commitment, Institutionalizing Review Processes, and Demonstrating Value to Participants

As reported through our survey and discussions with agency officials, data-driven review meetings can improve agency performance and results by increasing leadership oversight and management capacity to use performance information, focusing attention on goals and priorities, identifying areas where targeted improvements are needed, and improving communication and collaboration across an agency.

Our survey data suggested, however, that sustaining a data-driven review process over time and across leadership transitions can be a challenge for agencies. Ten agencies reported that it has been a challenge to continue holding reviews despite turnover of agency or priority goal leadership, with 2 reporting it has been a great challenge, 2 a moderate challenge, and 6 a small challenge. Through our discussions with agency officials, we learned that sustaining review meetings and their positive effects requires ongoing leadership commitment and involvement, the institutionalization of review practices, and the development of a broad base of support for the reviews through a shared appreciation for the positive effects that review meetings produce. These factors build on one another, as agency leaders, participants, and those supporting the reviews engage in a cycle of ongoing actions, as shown in figure 12.

First, agency officials emphasized that agency leaders’ commitment to lead, support, and remain involved in the reviews is a key element that must be in place for reviews and their positive effects to be sustained over time. The experiences of our selected agencies show that active involvement by agency leaders in review meetings is critical to establish the importance of the meetings and the clear expectation that other agency officials participate in them when called to do so. Furthermore, through their ongoing involvement in the reviews and their communication with other participants, leaders must reinforce that data-driven review meetings remain a priority and are seen as a valuable tool to achieve key agency objectives.

Second, agency officials indicated that reviews in which agency leaders critically assess data on performance, and where follow-up actions are identified and tracked, should be institutionalized and made a routine part of the agency’s operations. Through GPRAMA, Congress took a critical step to help ensure agencies do this by requiring that agency leaders Agencies have also taken conduct reviews frequently and regularly.several specific steps to institutionalize their review processes. For instance, agency officials gave existing or newly established offices responsibility for supporting the review process and ensuring that meetings are carried out with regularity and consistency over time. Staff in these offices generally manage data collection, meeting preparation, and follow-up, offer training for staff, and provide analytical expertise that helps support and inform the discussion in review meetings. Agencies also established clear expectations and procedures for how reviews would be carried out, including processes for preparation and follow-up. Institutionalizing review processes in this way, and providing sufficient institutional and staff support to manage them, can also facilitate the maintenance of review processes across leadership changes, as the process is made less dependent on the management style of a particular leader.

As the cycle continues, officials also noted that agencies need to continuously assess their review processes and address any identified weaknesses by incorporating improvements that respond to the needs of leaders and participants. While it is important to have a basic approach that persists over time, the ongoing assessment and improvement of review processes will help ensure reviews are adapted to meet the needs of new leaders and participants and continue to be used in a way that sustains their positive effects. For example, Commerce and SSA have recently changed their review meetings and processes to strengthen the focus on the agencies’ respective strategic goals. Officials with DOT and GSA also indicated that they are considering and instituting changes to their reviews. According to DOT officials, the agency formed a working group that examined ways to more effectively structure DOT’s review meetings. The group has also made recommendations to the Deputy Secretary regarding the format of the meetings, participation, the presentation of data, and the best ways to follow up on identified action items. The recommendations are currently under review by DOT senior leadership. At GSA, under a new Acting Administrator and Acting Deputy Administrator, additional monthly and quarterly reviews of priority initiatives and APGs are being instituted, while weekly review meetings have been refocused on a subset of GSA’s key performance measures and initiatives that are most ambitious or most in need of assistance and focus.

Third, agency officials emphasized that it is critical for those leading, managing, and participating in the reviews to assess, understand, and communicate the results that review meetings produce to help develop a broad base of support throughout an agency for sustaining review processes over time. According to a number of agency officials we spoke with, for agency leaders and managers—both political appointees and career agency officials—to maintain their commitment to review processes, the reviews must demonstrate that the benefits they provide outweigh the costs in time and resources spent. If the reviews show positive results, add value for participants, and have the support of senior political and career leaders who are able to articulate their merits, this will help sustain organization-wide commitment and increase the likelihood that reviews will be continued following leadership transitions.

Conclusions

Most federal agencies reported conducting data-driven reviews frequently and regularly, involving agency leaders and other key personnel, and using the process to assess progress on agency goals and identify strategies to address challenges or improve performance. These practices are consistent with requirements, guidance, and leading practices. Our findings also underscore the value of conducting frequent, in-person, data-driven reviews as a leadership strategy and management practice that can promote the use of performance information by agency officials and produce improved results. Our survey results indicated that agency data-driven review meetings enhanced their progress toward the achievement of agency goals, the engagement of agency leaders in the performance management process, the level of collaboration between agency officials, the ability to hold agency officials accountable for progress on goals and milestones, and the efficiency of agency operations.

Through this work, however, we found that DHS was not holding in- person, data-driven reviews of its APGs, and that four other agencies were conducting reviews in a manner that was not consistent with requirements, guidance, or leading practices. We also found that data- driven review meetings should be held on a regular, frequent schedule, actively involve senior agency leaders or other key officials, involve in- depth reviews of progress on agency goals, and be supported by rigorous methods for identifying and tracking follow-up actions. Otherwise, there could be missed opportunities for these agencies’ leaders to hold officials accountable for progress toward identified goals and milestones, to take timely and better informed action to address identified challenges, and to encourage continuous improvements in agency performance and operations.

Recommendations for Executive Action

To help ensure that agency review processes provide frequent, regular opportunities to assess progress on agency priority goals (APG), and are conducted in a manner consistent with GPRA Modernization Act of 2010 (GPRAMA) requirements, OMB guidance, and leading practices, we recommend the following actions:

That the Secretary of Agriculture work with the COO and PIO to modify the Department’s review processes to ensure that review meetings: (1) are held at least quarterly; (2) are led by the agency head or COO; (3) involve APG leaders; (4) and involve, as appropriate, agency officials with functional management responsibilities.

That the Secretary of Defense work with the COO and PIO to modify the Department’s review processes to ensure that review meetings: (1) are led by the agency head or COO; (2) are used to review progress on all APGs at least once a quarter, discuss at-risk goals and improvement strategies, and assess whether specific program activities, policies, or other activities are contributing to goals as planned; and (3) are used by participants to identify, agree upon, document and track follow-up actions.

That the Secretary of Health and Human Services work with the COO and PIO to modify the Department’s review process to ensure that progress on each APG is reviewed in an in-person review meeting at least quarterly.

That the Secretary of Homeland Security work with the COO and PIO to reestablish regular, in-person, data-driven review meetings conducted in a manner consistent with the requirements of GPRAMA, OMB guidance, and leading practices outlined in this report.

That the Secretary of State work with the COO and PIO to modify the Department’s review processes to ensure: (1) that progress on each APG is reviewed in an in-person review meeting at least quarterly; (2) and that the reviews are led by the agency head or COO; and (3) involve, as appropriate, agency officials with functional management responsibilities.

Agency Comments

We provided a draft of this report for review and comment to the Secretaries of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, Interior, Labor, State, Transportation, Treasury, and Veterans Affairs; the Attorney General of the United States; the Directors of the Office of Management and Budget, the Office of Personnel Management, and the National Science Foundation (NSF); the Administrators of the Environmental Protection Agency, National Aeronautics and Space Administration (NASA), Small Business Administration (SBA); the Acting Administrators of the General Services Administration (GSA) and the U.S. Agency for International Development (USAID); and the Acting Commissioner of Social Security.

We received written comments from the Departments of Defense (DOD), Health and Human Services (HHS), Homeland Security (DHS), and State, and the Social Security Administration (SSA). These responses are reproduced in appendixes IV through VIII. The Department of Agriculture (USDA) provided its response in an email transmitted on June 17, 2015. DHS, HHS, and USDA concurred with our recommendations.

DOD and State concurred with all but one recommendation—to ensure the COO leads the reviews—with which they partially concurred.

In its response, DOD concurred with our recommendations that agency leaders modify the agency’s review process to ensure that reviews are held to assess progress on APGs at least once a quarter, to identify goals at risk, to assess contributions, and to identify and track follow-up actions. DOD’s response said that the agency plans to comply with the recommendations by November 30, 2015. DOD partially concurred with our recommendation that the agency ensure the reviews are led by at least the COO. DOD, in its response, outlined a more specific role for its COO in future reviews. However, we do not believe this role is sufficient to bring DOD’s practices in line with the requirements of GPRAMA or with OMB guidance that the agency head and/or the COO conduct the reviews.

In its response, State concurred with our recommendations that reviews should be held on a quarterly basis and that the reviews involve applicable functional management officials. State did not explicitly agree or disagree with our recommendation that the reviews be led by the agency head or COO. However, the agency’s response indicated that the agency would continue to have the PIO lead their review meetings. As outlined in the report, this is not consistent with the requirements of GPRAMA or with OMB guidance that the agency head and/or the COO conduct the reviews.

We believe that DOD and State have both interpreted the language of OMB’s Circular A-11 in a way that provides them with the flexibility to delegate responsibility for conducting data-driven performance reviews to the PIO. As DOD notes, A-11 provides agencies with flexibility at key points to design a performance management system that best meets the agency’s needs. However, it is also important to emphasize that OMB’s Circular A-11 guidance clearly and unambiguously states in six separate sections that the COO is responsible for running agency reviews, and that these reviews should be held quarterly. The guidance also specifies that reviews must be held in person. As OMB has also noted, the personal engagement of agency leaders demonstrates commitment to improvement across the organization, ensures coordination across agency silos, and enables rapid decision making. The personal engagement of the COO in the data-driven reviews of progress on APGs is also critical given that, under GPRAMA, APGs are to reflect the agency’s highest priorities, and COOs are responsible for improving the management and performance of the agency through the regular assessment of progress and the use of performance information. For these reasons, we believe that these recommendations to follow requirements and guidance remain valid.

The following agencies provided technical comments that were incorporated into the draft as appropriate: Department of Energy, NASA, NSF, SBA, SSA, USAID, and USDA.

The following agencies had no comments on the draft report: The Departments of Commerce, Labor, Education, Housing and Urban Development, Interior, Justice, Treasury, and Veterans Affairs; the Environmental Protection Agency; the General Services Administration; the Office of Management and Budget; and the Office of Personnel Management.

We are sending copies of this report to the Director of OMB as well as appropriate congressional committees and other interested parties. The report is also available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix XI.

Appendix I: Objectives, Scope, and Methodology

This report is part of our response to a statutory requirement that we evaluate how the implementation of the GPRA Modernization Act of 2010 (GPRAMA) is affecting performance management in federal agencies, and whether performance management is being used by agencies to improve the efficiency and effectiveness of agency programs. Specifically, this report examines (1) the extent to which agencies are conducting data-driven performance reviews in a manner consistent with GPRAMA requirements, Office of Management and Budget (OMB) guidance, and leading practices; and (2) how agency data-driven performance reviews have affected performance, collaboration, accountability, and efficiency within agencies, and how positive effects can be sustained.

To address these objectives, we assessed review practices at the 23 executive agencies that fall under the purview of the sections of GPRAMA that relate to agency performance reviews. GPRAMA states that the 24 agencies identified by the amended Chief Financial Officers Act of 1990 (CFO Act), or those agencies otherwise determined by OMB, are required to develop agency priority goals (APG) and to review progress on these Because OMB did not require by conducting reviews at least quarterly. the Nuclear Regulatory Commission to develop APGs for 2014-2015, we confined our study to the other 23 CFO Act agencies.

To address both objectives, we surveyed Performance Improvement Officers (PIOs) at the 23 executive agencies. This method allowed us to collect government-wide information about the variety of practices being used by agencies to review progress on goals. We asked PIOs for information about the frequency of review meetings; leadership of, and participation in, review meetings; preparation for, execution of, and follow- up on, review meetings; challenges; and perceived effects of review meetings on agency performance, collaboration, efficiency, and accountability for results. Although agencies may conduct a variety of reviews as part of their performance management processes, we asked respondents to consider only in-person, data-driven review meetings that included discussion of APGs when answering our survey questions. GPRAMA provisions apply directly to these types of reviews. We administered the surveys between October and December 2014, transmitting and receiving the surveys as attachments to e-mails. We received responses from all 23 agency PIOs, signifying a 100 percent response rate. To minimize errors related to difficulties interpreting questions, we pretested the survey with three agency performance management officials to ensure that our questions were clear, complete, and unbiased, and that answering the survey did not place an undue burden on respondents. One of our methodology specialists assisted in developing our survey to ensure that survey questions captured the intended information. We revised our survey questions based on feedback from the pretesters and our methodologist. We verified our data entry and analysis programs for accuracy.

To further address both objectives, we selected five case-study agencies for a more in-depth assessment of their data-driven review processes. To supplement government-wide data collected through our survey, we used these more in-depth reviews to gather information about specific agency practices and about agency officials’ perceptions of any impacts that review meetings have had. This allowed us to collect additional detail and illustrative examples. We selected a sample of agencies that reflected a range of key characteristics, while excluding agencies from our selection that had been used as case studies for related recent or ongoing work, to avoid overburdening those agencies. The key characteristics we identified were agency size, as indicated by number of civilian employees; the extent to which agency leadership uses quarterly performance reviews to drive progress toward goals, as reported by respondents to GAO’s 2013 Federal Managers Survey; and agency compliance with basic GPRAMA requirements to hold quarterly in-person performance review meetings, as identified by a GAO team conducting related work. Using these criteria, we selected the Departments of Commerce (Commerce), Health and Human Services (HHS), and Transportation (DOT); the General Services Administration (GSA); and the Social Security Administration (SSA). Our sample of agencies is non-generalizable, and should not be considered representative of all agencies.

At each case-study agency selected for in-depth review, we also selected APGs to obtain the perspective of APG leaders and their staff on the agency’s data-driven review process. For those agencies with more than two APG leaders, we developed a selection process to determine which APG leaders we would request to interview. When possible, we excluded APG leaders who had been selected for interviews as part of related recent or ongoing work to avoid overburdening those officials. We selected goal leaders to produce a sample that reflected varied agency progress against APGs, where we excluded APGs for which progress was unclear. In cases when we had to choose between two APGs with the same type of progress, we prioritized the APG with a larger number of indicators. In cases where we selected an official who leads two APGs, we selected both APGs, resulting in three total APGs selected for the agency.

To allow us to corroborate information collected through surveys, interviews, and observations, and strengthen our confidence in the reliability of the self-reported survey responses, we requested supporting documents from 12 agencies, representing more than 50 percent of the agencies surveyed, related to review meeting frequency, leadership, participation, content, and follow-up. The 12 agencies included the 5 agencies selected for more in-depth review, several agencies whose survey responses required clarification, and several additional agencies at random. Examples of documents submitted by agencies included review meeting attendance or invitations, agendas, presentation slides, and briefings and summary reports. Based on our findings, we determined that the survey data were sufficiently reliable for the purposes of this report. In conducting the survey and subsequent follow-up, we learned that the Department of Homeland Security (DHS) does not hold in-person review meetings, and has not done so since December 2013. For this reason, the summaries of survey responses in this report exclude DHS.

We also addressed both objectives by interviewing agency officials who play a central role in the review meetings, including the agency Chief Operating Officer (COO), the PIO, and two APG leaders. These interviews provided us with detailed information from individual officials’ perspectives and helped us to corroborate information collected through other means. We used a consistent set of questions for each type of official, which included the official’s objectives for review meetings; the official’s role in preparing for, participating in, and conducting follow-up after review meetings; and the official’s experience of any effects of review meetings. We also interviewed staff from OMB and the Performance Improvement Council (PIC) to obtain information on previous surveys of agency PIOs administered by the PIC, their perspective on the implementation and effectiveness of reviews, and to learn about the role played by the PIC’s Internal Reviews Working Group, which has served as a forum for agency performance staff to periodically come together to discuss performance review practices in their agencies.

We also addressed both objectives by observing agency-level review meetings at HHS, GSA, and SSA, as well as one sub-agency-level review meeting at GSA. Observing review meetings allowed us to gain firsthand knowledge of review meeting processes. This served to provide context, increase our familiarity with the process, and corroborate information gained through other means. While we requested to observe a review meeting at both Commerce and DOT, we were not allowed to do so due to agency concerns that our presence could inhibit open discussion.

To address the first objective, we compared what we learned about the review processes at all 23 agencies with requirements for review meetings established in GPRAMA, as well as standards set forth in guidance in OMB’s Circular A-11 and leading practices for data-driven To address the second objective, we reviews we previously identified.analyzed our survey data to determine how agencies characterized the effects of their review meetings. We also used interviews with officials and documentation from our selected agencies to identify illustrative examples of the effects review meetings have produced, and to identify actions they have taken to sustain the benefits of the reviews.

Because the scope of our review was to examine the implementation of data-driven review processes in agencies, we did not evaluate whether APGs were appropriate indicators of performance, sufficiently ambitious, or met other dimensions of quality. Although agency performance information was reported in illustrative examples of data-driven review meeting materials, as well as illustrative examples of the effects of review meetings, we did not independently assess the accuracy of the agency performance information cited in these examples.

We conducted our work from July 2014 to July 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusion based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our finding and conclusions based on our audit objectives.

Appendix II: Descriptions of Data-Driven Review Meetings at Selected Agencies

Agency General Services Administration (GSA)

Description In 2014 and 2015, the Administrator of GSA convened weekly review meetings. At least once a month, the Public Buildings Service and the Federal Acquisition Service, which had responsibility for the agency’s two agency priority goals (APG), presented on their key performance measures, including APGs. Other offices within the agency also presented on their goals once a month. The Administrator also held additional weekly meetings with the heads of PBS and FAS to discuss key performance measures, including APGs, in greater detail.

Department of Commerce

In 2014, the Deputy Secretary of Commerce held biweekly review meetings, with each meeting focusing on one to three of the agency’s 27 strategic objectives, and including a review of progress on related APGs. In 2015, Commerce changed the frequency of review meetings from biweekly to monthly. Beginning in 2015, the Deputy Secretary held monthly meetings, with each meeting focusing on reviewing progress on one of the agency’s five strategic goals, including related APGs. The Deputy Secretary also held monthly check-in meetings with each of the five strategic goal teams to discuss progress on the goals and related APGs.

Social Security Administration (SSA)

In 2014, the Acting Commissioner of SSA held quarterly review meetings that covered all four of SSA’s APGs, as well as selected additional performance measures. In 2015, in addition to these quarterly reviews, SSA began to convene additional “theme-based” reviews to discuss cross-cutting issues, like human resource management, which, according to SSA officials, are also intended to include a discussion of relevant APGs. SSA plans to hold three of these “theme- based” reviews in 2015.

Department of Transportation (DOT)

In 2014 and 2015, the Deputy Secretary held four to five “management review meetings” a year with representatives of each of DOT’s 10 operating administrations. These meetings focused primarily on the status of each operating administration’s pending and proposed regulations, but also included a discussion of relevant performance measures, including APGs.

Department of Health and Human Services (HHS)

In 2014 and 2015, twice a year the Deputy Secretary and Performance Improvement Officer held in-person review meetings to review progress on each of the agency’s five APGs. According to HHS officials, agency leaders reviewed written progress updates for each APG during quarters without in-person review meetings.

Appendix III: Responses to Questions from GAO’s Survey on Agency Data-Driven Performance Review Meetings

To address our research questions related to agency data-driven review practices and effects we distributed a survey to the performance improvement officer (PIO) at each of the 23 agencies with agency priority goals (APGs). We received responses from all 23 PIOs. Through our survey and subsequent follow-up, however, we learned that the Department of Homeland Security (DHS) is not currently holding in- person data-driven review meetings, so its responses are not included in the aggregated results presented below. Therefore, unless otherwise indicated, the number of responses to each question is 22. There were nine questions in the survey, six of which contained multiple subquestions. Tables 1 through 9 below show our survey questions and aggregated responses. For more information about our methodology for designing and administering the survey, see appendix I.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: Comments from the Department of Defense

Appendix VI: Comments from the Department of Health and Human Services

Appendix VII: Comments from the Department of State

Appendix VIII: Comments from the Social Security Administration

Appendix IX: Full Text for Interactive Figure 5

Appendix X: Full Text for Interactive Figure 6

Appendix XI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Elizabeth Curda (Acting Director) and Adam Miles supervised the development of this report. Linda Collins, Shelby Kain, Kathleen Padulchick, Steven Putansu, and A.J. Stephens made significant contributions to this report. Dierdre Duffy and Robert Robinson also made key contributions.