Managing for Results: Data-Driven Performance Reviews Show Promise But Agencies Should Explore How to Involve Other Relevant Agencies
Published: 2013-02-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Given the federal government's central role in addressing many of the American public's most pressing concerns, it is critical that government performance is managed effectively. GAO's previous work has shown that many federal agencies have struggled to adopt effective performance management practices. Congress took steps to improve federal performance management with the passage of the Government Performance and Results Act Modernization Act of 2010 (GPRAMA), which included a provision for agency leaders to conduct quarterly, data-driven performance reviews. As part of GAO's mandate to review GPRAMA implementation, this report (1) identifies practices that can promote successful data-driven performance reviews at the federal level and examines how they are being implemented at selected agencies and across the government, and (2) examines the impact of quarterly datadriven performance reviews on selected agencies' progress toward high priority and other performance goals. To address these objectives, GAO reviewed academic and policy literature; information from practitioners at the local, state, and federal level; and Office of Management and Budget (OMB) guidance. GAO surveyed performance improvement officers at 24 federal agencies and examined review implementation at three agencies--DOE, SBA, and Treasury.
section_title: What GAO Found
paragraphs: GAO identified nine leading practices to promote successful data-driven performance reviews--referred to as quarterly performance reviews--at the federal level.
Agency leaders use data-driven reviews as a leadership strategy to drive performance improvement.
Key players attend reviews to facilitate problem solving.
Reviews ensure alignment between agency goals, program activities, and resources.
Agency leaders hold managers accountable for diagnosing performance problems and identifying strategies for improvement.
Agency has capacity to collect accurate, useful, and timely performance data.
Agency staff have skills to analyze and clearly communicate complex data for decision making.
Rigorous preparations enable meaningful performance discussions.
Reviews are conducted on a frequent and regularly scheduled basis.
Participants engage in rigorous and sustained follow-up on issues identified during reviews.
Most officials GAO interviewed at the Department of Energy (DOE), Small Business Administration (SBA), and Department of the Treasury (Treasury) attributed improvements in performance and decision making to the reviews. DOE, SBA, and Treasury officials said their reviews allowed different functional management groups and program areas within their agencies to collaborate and identify strategies which led to performance improvements. GAO's survey of performance improvement officers indicated that there was little to no involvement in the reviews from other agencies that could help achieve agency goals. This was also true at DOE, SBA, and Treasury, where officials expressed concerns about including outsiders in their reviews and described other means of coordinating with them. However, OMB guidance--along with a leading practice GAO identified--indicates that including key players from other agencies can lead to more effective collaboration and goal achievement.
section_title: What GAO Recommends
paragraphs: GAO recommends that OMB identify and share practices to use the reviews for interagency collaboration, when relevant, to achieve agency goals. OMB staff generally agreed with the recommendation.

section_title: Letter
paragraphs: How federal agencies manage toward better performance has a  significant effect on many of the American public’s most pressing  concerns—ranging from unemployment to food safety to national security.  However, our previous work has shown that many federal agencies have  struggled to adopt effective performance management practices.  Specifically, our work over the past 15 years has indicated that using  performance data for decision making can lead to better results, but in  several surveys we have done, less than half of federal managers  reported using performance data for decision making to a great or very  great extent. In looking for solutions to this long-standing problem,  Congress identified an effective management tool that has been widely  adopted by local and state governments—data-driven performance  review meetings, often referred to as “Stat” meetings. Congress took  steps to improve federal performance management with the passage of  the Government Performance and Results Act Modernization Act of 2010  (GPRAMA), which included a specific provision for quarterly performance  reviews, modeled after those at the local and state level.
As part of our mandate to review the implementation of GPRAMA, this  report (1) identifies practices that can promote successful data-driven  performance reviews at the federal level and examines how these  reviews are being implemented at selected agencies and across the  government, and (2) examines the impact of quarterly data-driven  performance reviews on selected agencies’ progress toward high priority  and other performance goals. This report is the second in a series that  examines how agencies are implementing various GPRAMA  requirements.
To address the first objective, we identified practices that can promote  successful data-driven reviews at the federal level by conducting a review  of relevant academic and policy literature, including our previous reports.  We refined these practices with additional information obtained from  practitioners at the local, state, and federal level who shared their  experiences and lessons learned. We also compared these practices  with recent GPRAMA related guidance in the Office of Management and  Budget’s (OMB) Circular No. A-11 and found them broadly consistent.  We observed two data-driven review meetings at the Department of the  Treasury (Treasury), which was one of the agencies selected to address  our reporting objectives. We also examined how these reviews are being  implemented at agencies across the government by conducting a survey  of performance improvement officers (PIO) in the 24 agencies covered by  the Chief Financial Officers Act of 1990 and subject to GPRAMA’s  requirements. We received responses from all 24 PIOs—a 100 percent  response rate.
To address both objectives, we selected three agencies to examine  implementation of GPRAMA-mandated quarterly performance reviews in  greater depth—Department of Energy (DOE), Small Business  Administration (SBA), and Treasury. We selected these three agencies  because they have been performing data-driven reviews for at least one  year and, together, use a mix of government tools—such as direct  service, regulations, grants, loans, and tax expenditures—to achieve their  performance goals, among other reasons. At each selected agency, we  focused on two agency priority goals (APG) to examine how quarterly  performance reviews affected the agency components responsible for  achieving performance outcomes. Because the scope of our review was  to examine data-driven performance reviews as a leadership strategy, we  did not evaluate whether these goals were appropriate indicators of  agency performance, sufficiently ambitious, or met other dimensions of  quality.
We also reviewed memorandums, internal briefings, and other materials  agencies used to prepare for the reviews, as well as documents used  during the reviews and follow-up materials. We conducted interviews with  officials at OMB, the Performance Improvement Council (PIC), and  officials involved in each agency’s performance review process. We  asked to observe at least one review meeting at each agency. Treasury  allowed us to observe two review meetings—one focused on the Bureau  of the Fiscal Service and one on the Internal Revenue Service (IRS).  DOE and SBA did not allow us to observe their meetings, citing concerns  that our presence could inhibit open discussion. During the interviews, we  asked officials to identify any challenges to effective implementation they  faced as the process evolved or any lessons they learned. We also asked  officials to identify examples of any impacts on performance that they  attributed to the reviews.
We conducted our work from April 2012 to February 2013 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
subsections: section_title: Background
paragraphs: The model for data-driven performance reviews was established in the  early 1990s by New York City Police Department (NYPD) leadership as a  strategy to reduce crime. Dubbed “CompStat,” the NYPD’s weekly  reviews grew from the premise that use of crime data could enable  leadership to make better-informed, more effective decisions. CompStat  followed four key tenets:  1.  Accurate and timely intelligence.  2.  Effective tactics.  3.  Rapid deployment.  4.  Relentless follow-up and assessment.
As the law enforcement community observed NYPD’s improved  accountability and bottom-line results in crime reduction, many police  departments nationwide began to replicate CompStat in their own  organizations. The CompStat model was subsequently adopted by many  cities, municipalities, and some states—most notably Washington and  Maryland—as a general performance management tool. Many of these  efforts were patterned on the same four tenets as CompStat.
Nearly two decades later, the Obama administration began to encourage  the use of data-driven review meetings as a performance management  tool through several memorandums issued by OMB in 2010 and 2011,  and a June 2011 executive order. During this timeframe, GPRAMA  introduced the concept of data-driven performance reviews at the federal  level with a provision that federal agencies conduct quarterly performance  reviews on progress toward their APGs. Specifically, agencies are  required to assess how relevant programs and activities contribute to  achieving APGs; categorize goals by their risk of not being achieved; and  for those at risk, identify strategies to improve performance. GPRAMA  also specified that the reviews must occur on at least a quarterly basis  and involve key leadership and other relevant parties both within and  outside the agency. GPRAMA required agencies to begin conducting  quarterly performance reviews by June 2011. Some agencies began  conducting data-driven reviews earlier, in response to the executive  order, OMB guidance, or other performance management efforts.
Several efforts aid agencies in implementing the GPRAMA-required  quarterly performance reviews. The PIC established a working group on  internal agency performance reviews. The working group meets monthly  to share leading practices and discuss strategies for improving  performance. Participation in the working group is voluntary and  according to OMB, PIOs or designees from 21 agencies across the  federal government are currently represented. In addition, OMB  supported performance review implementation by issuing new guidance  in OMB Circular A-11 during August 2012.
DOE, SBA, and Treasury each had experience with data-driven  performance reviews before the June 2011 GPRAMA implementation  deadline. Officials at both SBA and Treasury said that when new  leadership came in, they brought interest in data-driven decision making,  along with key staff that were experienced at data-driven review or data  analysis. DOE officials said that their experience with the American  Recovery and Reinvestment Act of 2009 (Recovery Act) work led them to  begin data-driven performance reviews along with the GPRAMA  requirements.     DOE built on its Recovery Act-related performance reviews with the  establishment of its quarterly performance reviews, called business  quarterly reviews, in 2011. According to DOE officials, the Deputy  Secretary leads the review meetings with participation by under  secretaries, and each review covers the department’s 8 priority goals,  15 mission-related key goals, and 8 management and operations key  goals. The Associate Deputy Secretary, who is responsible for  agency-wide management and operations goals, also participates in  the reviews. The review meeting follows a structured format, with the  first half focusing on performance goals and the second half focusing  on fiscal issues, such as budget execution.   SBA began conducting reviews, called quarterly performance reviews,  during the third quarter of 2009. SBA started by holding separate  meetings by program area but changed the review format in early  2010 to include all program offices at each review meeting. According  to SBA officials, the Deputy Administrator leads SBA’s quarterly  performance reviews with support from the chief operating officer  (COO) and the PIO. The PIO also assembles the data and  disseminates follow-up action items. Review meetings also include  senior officials from functional management areas (e.g., information  technology, human capital, procurement, etc.). In addition to  discussing agency priority goals, SBA reviews cover the rest of the  agency’s performance goals and objectives, as time permits.    Treasury started conducting department-level performance reviews,  called Stats, in 2010. Treasury’s reviews, led by the Deputy Secretary,  are performed on an agency component-by-component basis.  Reviews also involve senior officials responsible for functional  management areas such as information technology, financial  management, and procurement, as well as policy officials. The Stat  process is managed by Treasury’s Office of the Deputy Assistant  Secretary for Management and Budget, which reports to the PIO.  The Stats cover Treasury’s two priority goals along with a range of  other department programmatic and operational goals and priority  projects.
section_title: Practices that Promote Successful Data-Driven Performance Reviews at the Federal Level
subsections: section_title: Agency Leaders Use Data- Driven Reviews as a Leadership Strategy to Drive Performance Improvement
paragraphs: Our analysis confirmed that to be successful, data-driven reviews should  be used as a leadership strategy to drive performance improvement.  Agency leadership must be directly and visibly engaged in the review  process and invest the time necessary to understand and interpret the  data being discussed during the meetings. Moreover, GPRAMA requires  an agency head and deputy head to conduct quarterly priority progress  reviews which fosters ownership and helps ensure that participants take  the reviews seriously and that decisions and commitments can be made.  OMB Circular No. A-11 also emphasizes the importance of leadership  involvement in quarterly performance reviews, allowing the COO, the  agency head, or both to conduct the review.
Our survey of PIOs indicated that agency leadership, with the exception  of agency heads, actively participated in these reviews. This was  consistent with what we found at DOE, SBA, and Treasury. The sidebar  at left shows the survey results on the participation of key positions.
At Treasury, where we observed two Stat meetings, the Deputy Secretary  used the Stat meetings to challenge participants to stretch toward  ambitious performance goals and to provide possible solutions for any  issues that were discussed. By holding separate reviews for all of  Treasury’s bureaus and key offices, the Deputy Secretary committed a  significant amount of time to these reviews. Other Treasury officials—at  both the department headquarters and bureau levels—were appreciative  of the amount of time the Deputy Secretary devoted to the reviews. The  Deputy Secretary’s high-level position allowed him to speak with clear  authority in the department and garner the attention of component agency  leaders on performance issues. For example, during one review meeting  that we observed, the Deputy Secretary challenged Treasury’s bureau  leaders to develop new strategies for the department to collect delinquent  debt payments owed to the federal government.
We did not observe quarterly performance meetings at DOE or SBA.  However, officials we interviewed said that top leadership was actively  driving performance discussions. At DOE, the Deputy Secretary led the  quarterly review sessions and guided performance discussions with the  department’s under secretaries on progress made toward achieving their  respective performance goals. DOE’s Deputy Secretary noted that it is  important that under secretaries take ownership of the performance  review process to ensure the reviews are useful. At SBA, the Deputy  Administrator led the reviews with support from the COO and PIO.  According to officials, SBA’s leadership asks probing questions of  program heads concerning SBA’s performance goals.
Officials from DOE, SBA, and Treasury expressed concern about  maintaining the continuity of leadership engagement in performance  review meetings and pointed to ways to help ensure that the review  process continues with transitions to new agency leadership. For  example, officials at DOE pointed out that they have a responsibility to  remind new leadership of the legislative requirement for performance  reviews as well as OMB’s guidance on these reviews. In addition, SBA  officials noted that having a mix of career and political leadership involved  in the reviews can facilitate continuity of the review practices since career  officials generally span successive administrations. At Treasury,  performance budgeting staff developed written standard operating  procedures as a way to document the review process and pave the way  for new leadership engagement at the department.
To encourage new leadership to take ownership of the performance  reviews, several agency officials noted the importance of designing the  reviews to fit the leadership style and preferences of the incoming leader.  For example, one official said that it is critical for the incoming leader of  the review meetings to provide input into the presentation format and  structure of the review based upon personal preferences. At Treasury,  headquarters staff responsible for managing the performance reviews  worked closely with the Deputy Secretary to develop a review template  and meeting format that met his needs.
section_title: Key Players Attend Reviews to Facilitate Problem Solving
paragraphs: Our analysis indicated that performance review meeting participants  should include high-level leaders and managers with an agency-wide  perspective as well as those with programmatic knowledge and  responsibility for the specific performance issues likely to be raised. In  addition, participants should typically include those with agency-wide  functional management responsibilities, such as information technology,  budget, and human capital. This enables the reviews to facilitate problem  solving by breaking down information silos and providing managers from  across the agency and other contributing organizations with a forum to  communicate with each other and identify improvement strategies and  agree upon specific next steps. At the city and state level, data-driven  performance reviews typically include senior management from multiple  agencies. In addition to the benefit of in-person meeting attendance, there  is also value in having key players participate in other parts of the review  process, such as the data review and analysis leading up to the meetings  and the follow-up actions that arise from the meetings. Consistent with  this practice, OMB’s guidance directs agencies to include, as appropriate,  relevant personnel from within and outside the agency in the review  meetings.
According to our survey, most PIOs—21 of 24—reported that their  reviews included the participants needed to facilitate problem solving and  identify performance improvements the majority of the time. Further, 19  PIOs reported that goal leaders had large involvement in their agency’s  performance reviews, and 15 PIOs reported that internal contributors to  agency goals and functional management chiefs, such as the CFO, had  large involvement in their agency’s reviews.
At DOE, SBA, and Treasury, officials said they found that including senior  management responsible for specific mission program areas—as well as  those with functional management responsibilities in areas such as  budget, information technology, human capital, procurement, and legal  counsel—enhanced performance improvement efforts at review  meetings. For example, both DOE and Treasury officials said discussions  with budget officials concerning performance goals provided an  opportunity for performance issues to be discussed in the context of  budget issues where relevant.
However, differences in the scope of the reviews influenced which key  players attended the review meetings. For instance, while Treasury’s  reviews focused primarily on a specific component organization, such as  a bureau or policy office, representatives from multiple components  attended the review meetings when the achievement of a performance  goal crossed component lines, or when the components had other  commonalities. One Treasury official noted that the Bureau of Engraving  and Printing and the U.S. Mint were asked to sit in on each others’  sessions, as they both work under the Treasurer and have similar  operations.
Treasury officials indicated that one of the reasons why they chose to  focus their performance reviews on individual component agencies—such  as IRS and the Bureau of the Fiscal Service—was that their performance  goals are generally aligned by component rather than shared by  components. In addition, according to a senior Treasury official, focusing  reviews on components allows time for more in-depth reviews. Treasury’s  reviews include officials with department-level responsibilities in functional  management areas such as budget, procurement, and legal counsel who  can contribute to problem solving or who may be called on to take follow- up actions. In addition, Treasury officials noted that component-by- component reviews enable Treasury not only to have attendance from the  bureau head, but also from more members of the bureau’s leadership  team, such as the bureau CFO and others.
Officials said that if reviews were held with all of Treasury’s components  at one time it would be impractical to gather all key members of bureau  leadership because of the limited time which would be available for  discussion on any particular topic. Treasury officials said that their  practice of using the same information template for the Stat review with  each component enables the reviews to cover similar issues across  components and identify actions that should be addressed collaboratively,  even though all components do not participate in the review meetings at  the same time.
Conversely, at DOE and SBA, each review meeting included leadership  participation from across all agency mission areas. However, DOE and  SBA were similar to Treasury in that they included officials with agency- wide functional management responsibilities in the reviews. Officials  noted that DOE’s reviews provided an opportunity for senior leadership to  discuss the context around their performance goals and improve results  with help from other areas of the department. In addition, the presence of  senior officials across DOE’s programs provided an opportunity to share  leading practices in various areas across program lines. At SBA, officials  noted that many senior career officials attending these reviews had  experience managing multiple programs and their broader experiences  helped them understand and identify relevant performance metrics across  program areas. In addition, officials mentioned that, in contrast to a large  department with many discrete mission areas, SBA’s programs are  focused on achieving a relatively narrow mission of supporting small  businesses, which creates more opportunities for cross-program  collaboration.
According to our survey results, PIOs did not see getting the right  personnel included in the meetings as a challenge: Only 2 of 24 PIOs  reported a challenge in including those managers or staff needed to  facilitate problem solving and identify improvement opportunities.  However, officials from DOE, SBA, and Treasury did note challenges in  getting the right mix and number of participants to most effectively  facilitate performance improvement efforts at the meetings. See figure 1.
According to survey results, 16 of 24 PIOs indicated that there was little to  no involvement in the reviews from external officials who contribute to the  achievement of agency goals. This is consistent with what we found at  DOE, SBA, and Treasury, as stakeholders from outside the agencies did  not participate in their performance review meetings.
At Treasury, officials said they addressed performance issues requiring  external collaboration by conducting follow-up meetings with relevant  external participants. These meetings were scheduled as part of  Treasury’s practice of following up on issues raised during the review  sessions. For instance, during a Treasury review meeting we observed,  the Deputy Secretary directed staff to arrange a meeting with OMB to  help address an obstacle to achieving a performance goal.
Likewise, DOE and SBA officials said they undertook collaborative  actions as a result of discussions in their quarterly performance reviews,  but neither planned to invite external representatives to their meetings.  For example, SBA’s reviews led to multiple discussions with other federal  agencies on whether some of their government contracts that were going  to large businesses could go to small businesses instead.
Officials we interviewed cited several concerns that may explain why, at  present, agencies are generally not including external participants—from  other federal agencies or other relevant organizations—in their reviews.  First, officials did not include external stakeholders because they wanted  to keep discussions focused on internal problem-solving and were  concerned that including external parties might inhibit open discussions  on performance issues. Second, reviews at DOE, SBA, and Treasury  mainly focused on goals achieved through internal contributors at each  agency or office and officials noted that it would not currently be an  efficient use of time to include external parties, even though external  issues are discussed. One official said that the logistics of including high- ranking agency managers from other agencies could make it difficult to  schedule review meetings on a timely basis. Another official who had  experience managing data-driven performance reviews at different levels  of government noted that city- and state-level reviews tend to be run by a  mayor or governor with direct authority over the various agencies that  participate in the reviews. This official pointed out that circumstances are  different at the federal level, where an agency head could invite but not  require outside participation and would not have control over the  information shared or whether follow-up action was carried out.
However, we have previously reported that agencies can collaborate  more effectively across organizational lines when presented with a clear  and compelling rationale to do so and when agency leaders demonstrate  their commitment to working collaboratively. Agency goals that require  the efforts of more than one agency could serve as such a compelling  rationale—even in the absence of direct authority requiring such  collaboration. Moreover, our prior work has shown that agencies which  participated in various planning and decision-making forums together— such as interagency councils or planning bodies—reported that such  interactions contributed to achieving their goals. Specifically, agencies  reported that such participation opened lines of communication, fostered  trust, and helped build relationships, which can in turn lead to more  effective collaboration across agency lines. Despite the concerns that  DOE, SBA, and Treasury raised about including external participants in  their reviews, our survey results indicate that some agencies are doing  so: 4 of 24 PIOs reported moderate to large involvement of external  officials who contribute to the achievement of agency goals. In addition,  OMB officials provided an example of two agencies which have been  successfully making use of quarterly performance reviews to collaborate  on their APGs. These officials told us that the Departments of Housing  and Urban Development and Veterans Affairs—which both contribute to  efforts to reduce veterans’ homelessness—had conducted several Stat  meetings jointly. According to OMB officials, program staff members from  both agencies regularly participate in HUD Stat meetings, where they  jointly analyze performance data to understand trends, identify best  practices, and prioritize the actions needed to achieve veteran  homelessness goals. Officials reported that these collaborative meetings  have contributed to better outcomes. Moreover, officials from both OMB  and the PIC indicated that agencies have increasingly been observing  others’ review meetings as a means of learning about different practices  with no apparent harm to the effectiveness of the meetings. This suggests  that the challenges, if any, to outside participation can be overcome.
While there are many approaches to managing performance to achieve  goals that rely on multiple agencies, few are likely to provide the benefit of  bringing together the leadership and all the key players to solve problems  and motivate performance improvement. Moreover, when key players are  excluded from quarterly performance reviews, agencies may be missing  opportunities to have all the relevant parties participate in developing  solutions to performance problems. Instead, agencies will need to rely on  potentially duplicative parallel coordination mechanisms, which could  result in less than optimal performance improvement strategies.
section_title: Reviews Ensure Alignment between Agency Goals, Program Activities, and Resources
paragraphs: Our analysis showed that quarterly performance reviews should be used  to align an organization’s resources, programs, and activities to ensure  they are contributing to the achievement of agency goals. To help ensure  that reviews are focusing on the appropriate interim goals and measures,  agencies can develop models that describe the logical relationship  between an agency’s inputs, activities, outputs, and outcomes. These  logical relationships, sometimes called logic models, should be  periodically assessed to determine if outcomes are being achieved as  expected, and should be revised if necessary.
Our survey results indicated that PIOs do not find goal alignment to be a  challenging aspect of implementing quarterly performance reviews, with  13 of 24 PIOs reporting that ensuring alignment between performance  reviews and strategic goals and performance objectives was easy. Only  one PIO reported this as being a challenge. Moreover, 22 PIOs reported  that the majority of their reviews are aligned with strategic goals and  performance objectives.
Consistent with our survey results, we found that DOE, SBA, and  Treasury had selected performance metrics, initiatives, and other areas of  focus in their reviews that were linked to the accomplishment of APGs  and other goals, such as strategic plan objectives and key operational  goals. Officials from both DOE and Treasury described the processes  they undertook to choose useful performance information to frame the  performance review discussion. For example, DOE narrowed its list of  more than 190 performance measures to provide leadership with a  focused view of the department’s key goals, while also providing sufficient  depth of information to be meaningful. Treasury officials responsible for  designing the department’s quarterly performance reviews described  extensive interactions with the Deputy Secretary and the bureaus to  identify performance measures that could be used to promote discussion  of performance issues and opportunities for improvement.
In addition, the three agencies described efforts to use logic models,  project milestones, and other approaches to identify early information on  how they were progressing toward long-term outcome goals, which  officials said could be challenging to monitor. For example, DOE’s Office  of Energy Efficiency and Renewable Energy created a template for  program managers to develop logic models linking program inputs and  outputs to longer-term performance objectives, such as achieving clean  generation of 80 percent of the nation’s electricity by 2035. DOE identified  outputs or intermediate outcomes that contribute to the clean generation  of energy, such as reducing the cost of solar energy, which the  department measures to indicate progress toward its 2035 goal.  According to officials, logic models helped program staff communicate a  coherent story about how the program’s key activities contribute to its  goals.
At Treasury, performance budgeting staff said they included the status of  priority projects—such as IRS plans to develop a streamlined, user- friendly website—in its quarterly performance review information (see  figure 2). Officials explained that these priority projects were designated  as such because Treasury sees them as the critical path to achieving  agency priority goals and other key longer-term outcomes, which could  not always be tracked on a quarterly basis. For example, the Deputy  Secretary wanted to monitor the status of IRS’s website update because  it is seen to be a lever toward the agency’s priority goal to improve the  voluntary tax compliance rate, which can only be measured with data that  lags by approximately five years.
Our analysis showed that because data-driven reviews are to foster  improved performance, the focus of accountability should be on the  responsible manager’s role in addressing problems and bringing about  positive change. Agency leaders should hold goal leaders and other  responsible managers accountable for knowing the progress being made  in achieving goals and, if progress is insufficient, understanding why and  having a plan for improvement. If data is insufficient for gauging progress,  managers should be held accountable for improving the quality of the  data so that it is sufficient for decision making. Managers should also be  held accountable for identifying and replicating effective practices to  improve performance. In addition, the goals addressed in the reviews  should be aligned with managers’ and staff’s individual performance goals  to create a line of sight that reinforces the connection between strategic  goals and day-to-day activities of managers and staff.
GPRAMA introduced specific roles and responsibilities for agency heads,  COOs, PIOs, and goal leaders in conducting quarterly performance  reviews. For each APG, agency heads and COOs, with support from the  PIO, must:   review with the appropriate goal leader the progress achieved during  the most recent quarter, overall trend data, and the likelihood of  meeting the planned level of performance;   assess whether relevant organizations, program activities,  regulations, policies, and other activities are contributing as planned  to agency priority goals; categorize agency priority goals by risk of not achieving the planned  level of performance; and identify prospects and strategies for performance improvement,  including any needed changes to agency program activities,  regulations, policies, or other activities for agency priority goals at  greatest risk of not meeting the planned level of performance.
Across the government, a majority of PIOs reported that their agency’s  reviews met these GPRAMA requirements, as indicated in figure 3.
Our survey results indicated that most PIOs—21 of 24—reported using  the reviews to identify actionable opportunities for performance  improvement at least half the time.
Consistent with the survey results, DOE, SBA, and Treasury reported that  top agency leadership held officials accountable for identifying  performance problems and opportunities for improvement. For example,  Treasury officials said—and the sessions we observed confirmed—that  their Stat meetings focus on performance and are a vehicle for the  Deputy Secretary to challenge bureau heads to ensure their bureaus  continually improve. For example, the Deputy Secretary reported that  many of the bureaus had not reviewed their management metrics, such  as internal controls, diversity issues and employee survey scores, for  some time and the Stat reviews are an opportunity to engage the bureaus  on these issues. At the Stat reviews we attended, we observed the  Deputy Secretary discussing such management metrics. For example, he  asked bureau leadership to explain why their scores on a government- wide employee satisfaction survey had dipped during the past year,  questioned a decline in survey response rates, and discussed specific  next steps that he could take to support plans for improvement.
At DOE, officials said that their quarterly performance reviews focused on  areas where they were not on track to meet performance goals and ways  to address this. One official at DOE noted that the Deputy Secretary  expected the under secretaries to be accountable for all of their goals at  the meeting. At SBA, officials said the focus of the meetings is to fix  problems and that it is important to be prepared because the  Administrator asks probing questions and managers are called out when  progress is not being made and asked to explain what is being done to  resolve the issues.
Each agency also reported taking steps to ensure that managers’  individual performance objectives are aligned with priority and other  agency performance goals. For example, SBA’s performance agreements  incorporate performance objectives which cascade from the agency’s  priority goals and other performance goals and are aligned with its  strategic plan. Samples of performance agreements we reviewed  identified “performance elements linked to organizational goals.” For  example, a district manager had a performance objective to hold outreach  events to connect small businesses to contracting opportunities. This  objective clearly links to SBA’s APG to increase small business  participation in government contracting.
section_title: Agency Has Capacity to Collect Accurate, Useful, and Timely Performance Data
paragraphs: Our analysis indicated that the capacity to collect and analyze accurate,  useful, and timely data is critical to successful data-driven reviews.  Agencies should track both outputs and outcomes. Agencies should also  look for opportunities to leverage data produced by other agency  components or outside entities. In addition, having the capacity to  disaggregate data according to demographic, geographic, or other  relevant characteristics can aid in highlighting significant variation, which  can help meeting participants to pinpoint problems and identify solutions.  Agencies also need to plan for the time and resources required to  generate and communicate performance data in a timely manner. Easy  access to relevant databases and systems-generated analysis, such as  providing analysts with the ability to develop performance reports without  relying on information technology staff, can streamline the data collection  and analysis processes.
While having accurate, timely, and useful data available is critical to  successful performance reviews, 16 of 24 PIOs reported that this was a  challenge—more than any other practice we asked about. However, all  16 of those PIOs also responded that accurate, timely, and useful data is  available for their agency’s reviews about half the time or more, which  may indicate that some agencies have found ways to address this  challenge.
Our review of DOE, SBA, and Treasury illustrates how agencies can  overcome some challenges to data availability. Our analysis of quarterly  performance review documents indicated that each agency was  producing data-rich analyses that identified trends and potential  performance issues. However, agency officials described initial  challenges in these areas and said that improving their capacities for data  collection and analysis took time. For example, SBA’s Office of  Government Contracting and Business Development collects data on the  percentage of all federal agency contracts being awarded to small  businesses. The office is dependent on a General Services  Administration database, the Federal Procurement Database System- Next Generation (FPDS-NG), for its information. SBA officials were  concerned about the quality of the data since each federal agency enters  its own information. To address this concern, SBA officials said they  provided agencies with individualized reports of potential anomalies in  their small business contracting data. This process allowed agencies to  verify and correct if necessary the anomalies before SBA published the  annual Small Business Procurement Scorecard report. For example, if  an agency listed a contract in FPDS-NG as a small business set-aside at  the same time that the agency listed the contract as an open procurement  competition, this would be flagged. SBA would then notify the responsible  agency and give it an opportunity to correct the anomaly in FPDS-NG. In  addition, SBA noted instances where performance data lag behind the  performance review cycle. For example, the Department of Defense holds  its procurement data back from FPDS-NG for one quarter for national  security purposes. SBA officials said that instead of waiting for the next  quarter, they obtain preliminary information from the Department of  Defense.
section_title: Agency Staff Have Skills to Analyze and Clearly Communicate Complex Data for Decision Making
paragraphs: Our analysis indicated that agencies need staff with the skills to assess  performance data for coverage and quality and to identify key trends,  areas of strong or weak performance, and possible causal factors. In  addition, those messages need to be effectively communicated to  management and staff that will play a role in identifying and solving  performance problems and making related decisions. Analysts and  managers should carefully consider the type and amount of information  that will be useful for performance reviews, as well as how to present the  information to audiences with varying levels of technical or quantitative  skills. Providing the right amount of easy-to-understand performance  information can promote effective decision making during the quarterly  performance reviews. For example, focusing the presentation on the  message the data tell about performance, using well-designed graphics,  and grounding the data in relevant context are effective communication  techniques.
Our PIO survey results included information on 15 specific competencies  associated with performance improvement responsibilities. For each of  these competencies, the majority of PIOs reported the competencies  were present among performance improvement staff to a large extent.  However, survey results were less positive about 2 competencies  specifically related to analytic abilities. Of the 24 PIOs, 9 reported that  performance measurement competencies and 10 reported that  organizational performance analysis competencies were present among  their performance improvement staff to a small or moderate extent.
PIOs at DOE, SBA, and Treasury each described the teams they had  assembled to support their performance improvement efforts. For  example, SBA’s Deputy PIO had performance analysts to support the  quarterly performance reviews and many other performance management  activities, such as the production of a weekly dashboard of key  performance metrics. However, SBA officials acknowledged that some  staff were less comfortable working with data and they perceived this as a  skills gap that needed to be addressed. These officials said they are  addressing this through a combination of training and hiring. For example,  as part of its leadership training, SBA began developing courses related  to “decision support;” officials said the courses were designed to lead to  competencies in spreadsheet development and analysis, presentation  delivery, development of decision support datasets, and other analytic  and presentation skills. Participants began training in late summer of  2012 with courses titled Principles of Analytics and Analytic Boot Camp.
Having staff with abilities to communicate analyses effectively is an  important factor in successful performance reviews, and most PIOs—22  of 24—reported that data and relevant analyses are presented effectively  to participants in their agency’s reviews about half the time or more.  However, 11 of those 22 PIOs also reported that effective presentation of  data and relevant analysis was challenging—the second largest  challenge cited by the PIOs among the practices we asked about.
Consistent with our survey results, officials we interviewed at SBA and  Treasury described the challenges they faced in developing skills sets  that bridge the gap between data analysis and effective communication.  At SBA, the Office of Performance Management developed internal  training to help SBA managers improve their ability to communicate the  message that the data suggest. One official recounted initial struggles to  interpret data and then effectively communicate the key points relevant to  performance improvement to those who were not analysts. He noted that  the SBA Administrator told senior management that a “data dump” was  not helpful, which helped them to realize what was needed. Training was  developed to move managers and staff beyond basic analytic skills, with  a focus on structuring presentations effectively, using data to drive  management decisions, and in general, “telling your story so you’re  drawing out insights, rather than just summarizing facts.” See figure 4.
At Treasury, performance budgeting staff developed a PowerPoint  presentation template that was distributed to each bureau to complete in  advance of the Stat meetings. The template provided a uniform data  collection tool that incorporated data presentation design principles, to  guide the bureaus in effectively communicating their message to the  Deputy Secretary. For example, templates for line charts prompted  bureaus to indicate whether the desired trend line direction was up or  down, since this is not always immediately apparent to high-ranking  reviewers who may not have the depth of background into the particular  program or operation. According to a Treasury official, designing an  effective presentation is as important as doing relevant, high-quality data  analysis. One official pointed out, “If no one reads or understands the  analysis, it doesn’t matter how good it was.”
SBA and Treasury officials responsible for managing the reviews also  described challenges in balancing presentation uniformity with the need  to provide context that varies. These officials noted that consistency was  key to making performance information quick and easy to absorb,  especially for leadership that has limited time to review such information.  However, SBA noted that the down side of consistency is people’s  tendencies to tune out information that appears to be repetitive. SBA’s  PIO and COO said they have to continually look for ways to keep the  performance review meetings engaging to participants and that “meeting  fatigue” can be a problem. Further, several bureau officials at Treasury  we interviewed said that while they understood the need for uniformity,  the templates did not always provide them with enough flexibility to  provide sufficient context for their performance information. While bureau  officials we interviewed said that the process and template had improved  over time, some felt that in the early days of the Stat reviews, they were  so limited in their ability to “tell their story” to the Deputy Secretary that  they did not think he was getting an accurate understanding of the issues.  However, our review of multiple Stat documents indicated that there was  a specific page in the template left open for issues the bureaus wanted to  raise, and further, that some bureaus appended information to the  template to provide additional context.
section_title: Rigorous Preparations Enable Meaningful Performance Discussions
paragraphs: Our analysis found that sufficient preparation for the performance review  meeting is critical for a successful review. Key participants must be  prepared to discuss agenda items related to their performance measures  and progress toward goals as well as any other issues to be addressed.  The time allotted to prepare for reviews also provides a prompt for  participants to continuously update their performance data, assess  progress toward their performance goals, and develop a response to any  performance issues identified. Also, data to be presented during the  reviews must be fully vetted prior to the meeting so that participants can  focus discussions on data trends and analysis rather than on whether the  data itself is correct.
According to our survey results, 22 of 24 PIOs reported that review  participants are adequately prepared for performance reviews more than  half the time, and 20 reported that, overall, it is not challenging for  participants to be prepared for the reviews. Nevertheless, several agency  officials from DOE, SBA, and Treasury said that there was a significant  time investment in preparing information and coordinating among  managers and analysts across headquarters and components or offices.  Officials at Treasury noted that the process of preparing for the reviews  forced the department and its component agencies to closely examine  performance data and make sure they could explain it to the Deputy  Secretary, and said this process was a valuable part of the performance  review. As one bureau-level official explained, nobody wants to go before  the Deputy Secretary with data that indicates a performance problem,  unless they are able to explain the issue and show that they have already  thought of strategies for improvement. As a result, preparing for the  reviews sometimes prompted participants to conduct additional analysis  and have advance discussions on how to address performance problems.
At DOE, SBA, and Treasury, we found several practices in place to  prepare participants for review meetings. Officials at each agency  stressed the importance of meeting preparation to ensure that the review  sessions were productive. For example, Treasury employed a rigorous  pre-meeting process which started with the performance budgeting staff  developing a PowerPoint template, in consultation with the Deputy  Secretary, specifying the performance information to be provided by each  component agency for its Stat session. Treasury officials said that  performance budgeting staff then met with component staff to discuss the  new template and any changes. The templates we reviewed were  organized into several categories, such as priority projects, management  metrics, and other issues, and were distributed in advance of the Stat  meeting and used as the meeting agenda. Treasury officials said there  were typically several rounds of revisions to the PowerPoint template  prior to the review session, with management and analysts at the  component level coordinating with their counterparts at the department.  Officials said that one of the goals of developing the template was to  ensure that all participants are fully prepared and able to engage in  meaningful discussions about performance. In particular, one official  explained that a guiding principle is that none of the participants should  ever be surprised by any of the topics to be discussed. In advance of  each review session, Treasury’s Deputy Secretary reviews the completed  document along with an explanatory briefing memo prepared by  performance budgeting staff, which provides relevant context for any  issues, suggests lines of questioning, and highlights particular decisions  to be made. Performance budgeting staff also review the completed Stat  template with component staff to discuss the contents and ensure that the  component is aware of issues likely to be brought to the attention of the  Deputy Secretary. Treasury’s senior officials emphasized the importance  of ensuring that data discussed at the meeting were sufficiently vetted  during meeting preparation so as not to spend time during the sessions  determining whether the information is correct. Several officials pointed to  instances in early review meetings that used valuable meeting time on  data issues because department-level and component staff disagreed on  baseline data used during the reviews.
DOE’s pre-meeting practices included briefings with the Deputy Secretary  and under secretaries and development of a Business Quarterly Review  binder that included descriptions of the program offices’ performance and  budget information, a list of attendees, and background notes, as well as  a list of actions from the previous performance review.
section_title: Reviews Are Conducted on a Frequent and Regularly Scheduled Basis
paragraphs: Our analysis showed that in-person meetings which are both frequent and  regularly scheduled are a defining characteristic of data-driven reviews.  Regularly scheduled meetings foster a culture of performance  management and continuous improvement. The frequency of the meeting  schedule should depend on the urgency of the problems to be fixed,  frequency with which the data are collected, and speed with which  agency action can have an impact on these data. GPRAMA requires that  starting no later than June 2011, agencies must conduct meetings at least  quarterly, but agencies may meet more frequently if it meets their needs.
According to our survey, all 24 of the PIOs reported that their agencies  were conducting performance reviews at least quarterly, with 7 of those  24 reporting conducting reviews more frequently. At DOE, SBA, and  Treasury, we found each agency had frequent, regularly scheduled  performance review meetings. Treasury’s Deputy Secretary met  separately with each of the bureaus two or three times a year. Generally,  one or two of the meetings with each Treasury bureau focused on  performance, and the other meetings focused on budget. Treasury also  holds Stat meetings on certain department-wide goals related to human  capital, procurement, and strategic sourcing. As a result, Treasury  conducted more than three dozen performance review meetings  throughout the year, with more than 100 reviews conducted as of  November 2012. DOE and SBA both met quarterly with representatives  from across their entire departments.
Our survey results indicated that while holding performance reviews as  scheduled is generally occurring, it may present challenges at some  agencies. Of 24 PIOs, 20 reported that their agency held the performance  reviews as scheduled more than half the time, with 2 of these PIOs  reporting scheduling as a challenge. In addition, the 4 PIOs who reported  that this practice was occurring about half of the time or less also reported  that it was a challenge.
Experiences at Treasury illustrated how it can be challenging to schedule  performance review meetings with high-ranking officials. At one bureau,  officials said that last-minute cancellations due to the Deputy Secretary’s  schedule caused inefficiencies since extensive meeting preparations,  including performance data analyses, had to be redone each time.  Officials from this bureau also commented that the performance review  meetings sometimes coincided with times during which the bureau had  heavy workloads. The officials noted that scheduling all of the meetings at  the beginning of the year would be helpful so that they could plan around  them. According to Treasury officials, currently sessions are scheduled 4  to 8 weeks in advance.
SBA found that it was best to schedule a standing date for the  performance review meetings, which are held on the third Tuesday after  the end of every quarter, rather than try to coordinate with numerous  senior officials’ schedules every time. By selecting this meeting time, SBA  was also able to leverage an existing weekly operations meeting, rather  than scheduling a separate meeting for the quarterly performance review.
section_title: Participants Engage in Rigorous and Sustained Follow-up on Issues Identified During Reviews
paragraphs: Rigorous and sustained follow-up on issues identified during the meetings  is also critical to ensure the success of the reviews as a performance  improvement tool. Important follow-up activities include identifying the  individual or office responsible for each action item as well as who will be  monitoring the follow-up. Follow-up actions should be included as agenda  items for subsequent reviews to hold responsible officials accountable for  addressing the issues raised and communicating what was done.
According to our survey results, follow-up activities are generally  occurring even though some PIOs reported that this practice was  challenging. In particular, 21 of 24 PIOs reported that follow-up activities  occurred at their agencies more than half the time. Although 7 PIOs  reported that follow-up activities were challenging, 5 of those 7 PIOs also  reported that this practice was occurring more than half the time, which  may indicate that some agencies have found ways to overcome these  challenges.
DOE, SBA, and Treasury each had a different approach to ensure that  follow-up items were carried out as agreed to at the review meetings and  to ensure responsible parties were held accountable. At DOE, post- meeting activities included asking program offices questions that were  generated by their review and assigning an analyst from headquarters  performance staff to work with mission program staff to prepare answers  to these follow-up questions and have them ready for the next  performance review meeting. According to an SBA official, follow-up  action items were typically discussed at weekly operations meetings,  which helped officials to integrate their action plans into these other  performance discussions. One example of an SBA follow-up item was to  develop new strategies related to increasing federal contracts with small  businesses. Officials said they ranked agencies by the total dollar value of  contracts they issued, and they targeted procurement representatives at  the top seven purchasing agencies for their outreach efforts.
At Treasury, its performance budgeting staff generated follow-up  memorandums immediately after the review meetings, naming action  items, responsible parties, and due dates. The status of follow-up items  from the previous review meetings was also incorporated into the  materials for the next review meeting so the Deputy Secretary could see if  there was any lagging action. In addition, performance budgeting staff  tracked overall performance on post-review follow-up to ensure that this  part of the process was being managed effectively (see figure 5).
Several officials from Treasury and SBA noted the importance of  receiving feedback from meeting participants to help improve the review  process. For example, Treasury conducted a formal feedback meeting  with bureau heads after the first round of review meetings and learned  that components wanted more data from the department’s performance  staff to help substantiate their program performance. More recently,  Treasury developed a survey to obtain formal feedback from participants  and other management and staff that contribute to the review process.  The survey included questions on the amount of time invested in  preparation and follow-up and asked respondents to rate their satisfaction  with various aspects of the review process, including the template design  and the guidance provided by performance budget staff. Respondents  were also asked to rate the importance of the various aspects of the  review to their bureau or office, among other questions. According to  Treasury officials who manage the Stat reviews, they have made specific  improvements based on the survey results, such as sending out  templates earlier to give components more time to prepare.
SBA also solicited feedback from meeting participants and made some  changes as a result. For example, an SBA official said that the agency  used to schedule its quarterly performance review meeting after all the  data were available, which was 45 days after the end of each quarter.  However, the official said that participants wanted the meeting to be held  when the majority of the data was available—only one of its offices’ data  lagged—to allow them to take any needed action on a timely basis. As a  result, the officials said SBA started scheduling the quarterly review  meeting on the third Tuesday after the end of each quarter and  conducting a separate meeting with the one SBA office that receives its  data after that date.
section_title: DOE, SBA, and Treasury Officials Attributed Improvements in Performance and Decision Making to Quarterly Performance Reviews, although Some Reported Minimal Impact
subsections: section_title: Reviews Fostered Intra- agency Collaboration and Problem Solving to Improve Performance
paragraphs: DOE, SBA, and Treasury officials said their quarterly performance  reviews allowed different functional management groups and program  areas within each agency to share information and ideas for performance  improvement. Officials said these reviews helped them to solve problems  that were impeding progress toward performance goals or to develop new  performance improvement strategies. In some cases, officials were able  to point to specific performance improvements that they attributed to the  reviews.
DOE officials said that quarterly reviews helped them take a more critical  look at subordinate activities contributing to the achievement of their  APGs. For instance, they examined barriers to achieving their  weatherization goals and found they could produce better results by  targeting not only individual homes but also multiuse buildings for  retrofitting. Therefore, they created a program to leverage resources from  nonprofit and private sector organizations focused on large scale retrofit  projects for buildings. In another example, DOE identified barriers to  achieving its solar energy cost reduction APG related to slow local  permitting processes for solar installations along with other local-level  activities that contributed toward the goal. To address these barriers,  DOE funded a new program called the Rooftop Solar Challenge in which  teams develop actions plans to standardize permit processes, update  planning and zoning codes, improve standards for connecting solar power  to the electric grid, and increase access to financing. Officials said they  expected these new approaches to improve DOE’s performance in  reducing solar energy costs.
According to officials, performance reviews at DOE also facilitated  information sharing across the agency that led to better results. For  example, officials said discussions at DOE’s quarterly performance  reviews led offices to share effective procurement practices. In this case,  offices have been sharing best practices related to strategic sourcing to  identify areas of cost reduction. For example, a recent reorganization  between Environmental Management and National Nuclear Security  Administration presented an opportunity for Environmental Management  to leverage existing capabilities in strategic sourcing. The performance  reviews have allowed for additional DOE-wide discussions on strategic  sourcing lessons-learned and partnerships to potentially achieve greater  efficiencies and cost savings.
Another example provided by SBA officials illustrated how using the  reviews to increase visibility of the small business contracting goal at  higher management levels led to the adoption of new performance  improvement strategies. After discussing contracting goal data at a  quarterly review, officials said the Administrator and Deputy Administrator  decided to call department secretaries at those federal agencies with the  most potential for awarding small business contracts to emphasize the  importance of the goal. Officials said other strategies, such as providing  more training for procurement center representatives, who, among other  things, assist small businesses in obtaining federal contracts, came out of  discussion at the reviews. Officials said they anticipated that the new  strategies that came out of the reviews would lead to better performance  toward the small business contracting goal.
SBA officials also provided an example that illustrated how the reviews  facilitated intra-agency collaboration that improved SBA’s bottom line.  Officials said that during a quarterly performance review meeting, the  head of the Office of Capital Access described anticipated staffing  shortfalls and the head of SBA’s Office of Disaster Assistance noted that  he expected to have staff members available during the slower season in  disaster assistance work, which is cyclical in nature. SBA’s Administrator  instructed the two offices to work together, and as a result, officials said  they were able to reduce the Office of Capital Access’s labor costs by 20  to 30 percent compared to the cost of paying employees overtime or  hiring temporary contractor labor.
section_title: Leadership Communication of Priorities at Reviews Motivated Performance Improvements
paragraphs: DOE, SBA, and Treasury officials said the quarterly performance reviews  provided a venue for top leadership to directly communicate their  priorities and the priorities of the administration, which led to performance  improvements in these areas. For example, at Treasury, nearly all of the  bureau-level officials we interviewed said the Stat meetings were valuable  because they allowed for a firsthand understanding of the Deputy  Secretary’s priorities for their bureaus. Bureau officials said that the  Deputy Secretary used these meetings to challenge their performance  targets and approaches to addressing performance problems and to  identify new opportunities for improvement. For example, the Bureau of  the Public Debt Commissioner said that upon hearing about the Deputy  Secretary’s interest in one of his ideas for improving how the agency  communicates its bond pricing approach to customers, he moved ahead  with developing an actionable strategy which he believes will ultimately  lead to better customer service. Treasury officials said the Stat review  process improved decision making by creating an environment where  meaningful discussions on improving performance were held, citing  performance improvements at the U.S. Mint as an example (see figure 6).
Citing another example, Treasury officials said that the Deputy Secretary  had made contracting with small business a department-wide priority.  Although increasing small business participation in government  contracting was a SBA priority goal, Treasury, like all federal agencies  subject to the Small Business Act, had its own target to meet to contribute  to the goal. Officials attributed increases in the department’s percentage  of contracts with small businesses—Treasury was the only agency to  achieve all of its fiscal year 2011 SBA small business prime contracting  goals—to the Deputy Secretary’s “relentless attention” at the Stat  meetings. At every Stat session with every bureau, the Deputy Secretary  reviewed the individual bureau’s performance against the small business  target goals. Officials said that Treasury’s chief procurement officer was  present at every Stat meeting to facilitate goal achievement.
The Financial Management Service (FMS) Commissioner cited another  example of how the Stat meetings provided a venue for the Deputy  Secretary to communicate his priorities, which led to better performance.  FMS has been pursuing several priority projects to modernize its  payments, collections, and central accounting systems that serve federal  agencies across the government. For example, one project is to replace a  paper process that many agencies use to accept vendor invoices with a  central invoicing system. FMS’ analysts estimated that a central invoicing  system could save the federal government $400 to $500 million annually,  as well as provide vendors with online access to the status of their  payments. FMS piloted the new system with the Bureau of Engraving and  Printing. The new system enabled the Bureau of Engraving and Printing  to pay their vendors more quickly than before, which resulted in the  vendors having to provide “prompt pay” discounts. The decision was  made to move forward government-wide, but to start with Treasury’s own  bureaus. Following a Stat meeting in which the FMS Commissioner cited  delays in adoption by other Treasury bureaus, the Deputy Secretary  made it clear that implementing the new system was a priority. The FMS  official said that, while he believed the adoption of the new system would  have happened eventually even without the Stat meetings, he credited  the Stat meetings with speeding up the process by three to four years.
A small number of Treasury bureau officials we interviewed had mixed  views on whether the Stat meetings had actually improved performance,  with some pointing out that certain performance improvements may have  occurred without the Stat meetings. For example, Treasury leadership  said they used the Stat sessions to closely monitor the consolidation of  Bureau of the Public Debt and FMS into a single bureau, the Bureau of  the Fiscal Service. Some officials thought they would have achieved the  various consolidation milestones and goals without the Stat reviews and  one said that the reviews added another layer of reporting that was time  consuming. However, Treasury department officials noted that  component management may not be aware of how the Deputy Secretary  uses the reviews to better understand performance issues, including, in  some cases, to ensure that the department is providing necessary  support to improve performance.
section_title: Conclusions
paragraphs: Our review of DOE, SBA, and Treasury—as well as our survey of 24  PIOs—indicated that data-driven quarterly performance reviews hold  promise as an effective management tool at the federal level. However,  unlike city- and state-level data-driven reviews, which typically include  representatives from multiple agencies, officials at DOE, SBA, and  Treasury viewed their quarterly performance review meetings as an  internal management tool and therefore did not open the reviews to  outside participation. Officials said they relied on other means of  collaborating with outside agencies and other partners that contribute to  achieving cross-cutting goals. Furthermore, the majority of PIOs we  surveyed indicated that there was little to no involvement in the reviews  from external officials who could contribute to achieving agency goals.  Successful data-driven performance reviews, which require extensive  preparation and significant leadership time, do not come without a cost,  so it is critical that agencies implement their reviews in a way that  maximizes their effectiveness. As the implementation of the various  GPRAMA provisions continues, agencies may need to reevaluate the  most effective way to engage outside contributors in the quarterly  performance review process for APGs and other performance goals that  depend on other organizations to achieve desired outcomes. While there  are many approaches to managing performance toward such goals,  agency quarterly performance reviews could provide opportunities to  bring together the leadership and all the key players needed to improve  cross-agency and internal agency performance.
section_title: Recommendation for Executive Action
paragraphs: To better leverage agency quarterly performance reviews as a  mechanism to manage performance toward agency priority and other  agency-level performance goals, we are recommending that the Director  of the Office of Management and Budget—working with the Performance  Improvement Council and other relevant groups—identify and share  promising practices to help agencies extend their quarterly performance  reviews to include, as relevant, representatives from outside  organizations that contribute to achieving their agency performance  goals.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOE, OMB, SBA, and Treasury for  review and comment. Each agency provided technical comments which  we incorporated as appropriate. OMB staff generally concurred with the  recommendation in our report. DOE and Treasury provided written  comments, which are reproduced in appendixes IV and V, agreeing with  our conclusions.
We are sending copies of this report to the appropriate congressional  committees and the Secretaries of Energy and Treasury, the  Administrator of SBA, and the Director of OMB. In addition, the report is  available at no charge on GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix VII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: As part of our mandate to review the implementation of the Government  Performance and Results Act Modernization Act of 2010 (GPRAMA), our  reporting objectives were to (1) identify practices that can promote  successful data-driven reviews at the federal level and examine how  these reviews are being implemented at selected agencies and across  the government, and (2) examine the impact of quarterly data-driven  performance reviews on selected agencies’ progress toward high priority  and other performance goals. This report is the second in a series that  examines how agencies are implementing various GPRAMA  requirements.
To identify practices that can promote successful data-driven reviews at  the federal level, we conducted a review of relevant academic and policy  literature, including our previous reports. Based on our literature review,  we developed a broad set of practices associated with the major  contributors to success for data-driven reviews. We refined these  practices with additional information obtained from practitioners at the  local, state, and federal level who shared their experiences and lessons  learned. As part of this engagement, we also compared these practices  with recent GPRAMA-related guidance in the Office of Management and  Budget’s (OMB) Circular No. A-11 and found them broadly consistent.
We observed two data-driven review meetings at the Department of the  Treasury (Treasury), which was one of the agencies selected to address  our reporting objectives.
To examine how these reviews are being implemented at agencies  across the government, we surveyed the performance improvement  officer (PIO) at each of the 24 federal agencies subject to the Chief  Financial Officers (CFO) Act. We surveyed these agencies because  GPRAMA directs us to focus on them in our reporting on the act.  Additionally, several provisions of the act apply specifically to these  agencies, including that the Performance Improvement Council (PIC)  include them as members. We received responses from 24 PIOs—a 100  percent response rate.
The web-based survey was administered from October 18, 2012, to  December 14, 2012. Respondents were sent an e-mail invitation to  complete the survey on a GAO web server using a unique username and  password. During the data collection period, we sent reminder e-mails  and made phone calls to nonresponding agencies. Because this was not  a sample survey, it has no sampling errors. The practical difficulties of  conducting any survey may also introduce nonsampling errors, such as  difficulties interpreting a particular question, which can introduce  unwanted variability into the survey results. We took steps to minimize  nonsampling errors by pretesting the questionnaire in person with PIOs  and Deputy PIOs at three different agencies. We conducted pretests to  make sure that the questions were clear and unbiased, the data and  information were readily obtainable, and the questionnaire did not place  an undue burden on respondents. We made appropriate revisions to the  content and format of the questionnaire after the pretests and  independent review. All data analysis programs used to generate survey  results were independently verified for accuracy. Additionally, in reviewing  the answers from agencies, we confirmed that PIOs had correctly  bypassed inapplicable questions (skip patterns). Based on our findings,  we determined that the survey data are sufficiently reliable for the  purposes of this engagement.
To evaluate how effectively selected agencies are applying the practices  of data-driven reviews in their GPRAMA-mandated quarterly performance  reviews, and the effectiveness of these reviews toward achieving agency  priority and other performance goals, we chose three agencies— Department of Energy (DOE), Small Business Administration (SBA), and  Department of the Treasury (Treasury).
Because one of our goals was to describe challenges and lessons  learned that will be useful to as many federal agencies as possible—as  well as to the administration and to Congress—we selected agencies for  case study that could provide a new illustration of challenges or lessons  learned, that is, agencies that were not the recent or current subject of  GAO or other public administration or public policy case studies. We also  looked for agencies that could provide broadly applicable case  illustrations, based on the scope of their mission, organizational  complexity, and the mix of government tools—such as direct service,  regulations, grants, loans, and tax expenditures—used to achieve their  performance goals.
In addition, we excluded agencies that were undergoing significant  management changes—such as leadership turnover or consolidation  review—that could prevent us from gaining a clear picture of performance  management or could interfere with our ability to make practicable  recommendations. We also excluded agencies that had less than one  year of experience in conducting data-driven reviews at the time that we  started our field work.
At each selected agency, we focused on two agency priority goals (APG)  to examine how quarterly performance reviews affected the agency  components responsible for achieving performance outcomes. Because  the scope of our review was to examine data-driven performance reviews  as a leadership strategy, we did not evaluate whether these goals were  appropriate indicators of agency performance, sufficiently ambitious, or  met other dimensions of quality. The following are the complete sets of  the agencies’ 2013 APGs. An asterisk indicates the ones we focused on  in our review:    Save low-income families money and energy through weatherization    Reduce the department’s Cold War legacy environmental footprint.    Reduce the cost of batteries for electric drive vehicles to help increase  the market for plug-in-hybrids and all electric vehicles and thereby  reduce petroleum use and greenhouse gas emissions.   Make solar energy as cheap as traditional sources of electricity.*    Reduce consumer energy use and costs for household appliances.    Prioritization of scientific facilities to ensure optimal benefit from  federal investments.   Make significant progress toward securing the most vulnerable  nuclear materials worldwide within four years.   Maintain the U.S. nuclear weapons stockpile and dismantle excess  nuclear weapons to meet national nuclear security requirements as  assigned by the President through the Nuclear Posture Review.    Process business loans as efficiently as possible.*  Increase small business participation in government contracting.*   Process disaster assistance applications efficiently.     Expand access to long term capital.
Increase electronic transactions with the public to improve service,  prevent fraud, and reduce costs.* Increase voluntary tax compliance.* To address both of our objectives, we reviewed memorandums, internal  briefings, and other materials agencies used to prepare for the reviews,  as well as documents used during the reviews and follow-up materials.  We conducted interviews with officials at OMB, the Performance  Improvement Council, and senior-level officials involved in each agency’s  performance review process to gain various perspectives on these  reviews.    DOE: We met with the Deputy Secretary, who also serves as DOE’s  chief operating officer (COO), and other senior-level officials including  but not limited to the PIO, the Deputy PIO, and budget officials. In  addition, we met with two APG goal leaders: the Deputy Assistant  Secretary for Energy Efficiency and the Deputy Assistant Secretary  for Renewable Energy.   SBA: We met with senior-level officials, including but not limited to the  SBA’s PIO, Deputy PIO, and COO. In addition, we met with SBA’s  Associate Administrator/Office of Capital Access, who serves as the  goal leader for one of the AGPs we reviewed.   Treasury: We met with the Deputy Secretary, who also serves as  Treasury’s COO, and other senior-level officials at the department  level, including the PIO, who also serves as the CFO, the Deputy PIO,  and budget officials. We also met with senior-level officials at  Treasury’s Bureau of the Fiscal Service, Bureau of the Public Debt,  Financial Management Service, and the Internal Revenue Service.
In addition, we met with the Assistant Fiscal Secretary who serves as  the goal leader for one of the APGs we reviewed.
We asked to observe at least one review meeting at each agency.  Treasury allowed us to observe two of its quarterly performance review  meetings—one focused on the Bureau of the Fiscal Service and one on  the Internal Revenue Service. DOE and SBA did not allow us to observe  their meetings, citing concerns that our presence could inhibit open  discussion. During the interviews, we asked officials to identify any  challenges to effective implementation they faced as the process evolved  or any lessons they learned. We also asked officials to identify examples  of any impacts on performance that they attributed to the reviews.
We conducted our work from April 2012 to February 2013 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
section_title: Appendix II: Responses to Selected Questions from GAO’s Survey on Agency Performance Management Positions and Responsibilities
paragraphs: To address several research objectives related to implementation of the  Government Performance and Results Act Modernization Act of 2010  (GPRAMA) we distributed a web-based survey to the performance  improvement officer (PIO) at each of the 24 agencies subject to the Chief  Financial Officers Act. We received responses from all 24 PIOs. There  were 45 survey questions; tables 1 through 9 below show questions and  responses for the nine questions that were directly applicable to the  research objectives in this report. We will publish the full survey results in  a report in the spring of 2013 on the implementation of key management  positions under GPRAMA. For more information about our methodology  for designing and distributing the survey, see appendix I.
section_title: Appendix III: Federal Agencies Subject to the Chief Financial Officers Act
section_title: Appendix IV: Comments from the Department of the Treasury
section_title: Appendix V: Comments from the Department of Energy
section_title: Appendix VI: Full Text for Interactive Figure 1 on Finding the Right Meeting Size and Composition for Effective Performance Reviews
paragraphs: This appendix includes the print version of the text contained in  interactive figure 1.
Several officials noted challenges in achieving the proper balance  between having a small number of meeting participants, which can allow  more air time for each participant, compared to a larger group which  might not be a productive use of time for those attending.
Picture: Treasury Deputy Secretary and COO Neal Wolin (top) leads the  department’s quarterly performance reviews, with support from Assistant  Secretary for Management Nani Coloretti, who also serves as Treasury’s  PIO.
In addition, balancing depth of knowledge and numbers of staff attending  these reviews is another consideration for finding the right meeting size.  One official pointed out that having a relatively small group can help  participants feel more comfortable in revealing performance problems but  that there is a risk of leaving out key players that need to be part of a  performance solution. Officials from each agency described different  approaches to finding the right meeting size and composition for effective  performance reviews, with several officials acknowledging that it takes  time to get it right.
Table 10 contains the rollover information from interactive figure 1 about  how the Department of Energy (DOE), the Small Business Administration  (SBA), and the Department of the Treasury (Treasury) conduct their  quarterly performance review meetings.
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: Elizabeth Curda and Laura Miller Craig managed this assignment. Darnita  Akers, Virginia Chanley, Don Kiggins, and Bob Yetvin made contributions  to all aspects of the report. Kim Gianopoulos, Kay Kuhlman, Judith  Kordahl, Jill Lacey, Sarah M. McGrath, Tim Minelli, Kathleen Padulchick,  William B. Shear, Albert Sim, and Meredith Trauner also provided  assistance. In addition, Sabrina Streagle provided legal support and  Donna Miller developed the report’s graphics.
section_title: Selected Bibliography
paragraphs: Behn, Robert. “Collaborating for Performance: Or Can There Exist Such a  Thing as CollaborationStat?” International Public Management Journal,  vol. 13, no. 4 (2010): 429-470.
Behn, Robert. “The Core Drivers Of CitiStat: It’s Not Just About the  Meetings and the Maps,” International Public Management Journal, vol. 8,  no. 3 (2005): 295-319.
Behn, Robert. “The Varieties of CitiStat,” Public Administration Review,  vol. 66, no. 3 (2006): 332-340.
Behn, Robert. “What All Mayors Would Like to Know About Baltimore’s  CitiStat Performance Strategy,” IBM Center for the Business of  Government Managing for Performance and Results Series (Washington,  D.C.: 2007).
Campbell, Mary. “Bringing Performance Excellence to the Public Sector:  Washington State’s Odyssey,” ASQ World Conference on Quality and  Improvement Proceedings, vol. 59 (2005): 279-288.
Esty, Daniel, and Reece Rushing. “The Promise of Data-Driven  Policymaking,” Issues in Science and Technology, vol. 23, no. 4 (Summer  2007): 67-72.
Fillichio, Carl. “Getting Ahead of the Curve: Baltimore and CitiStat,” Public  Manager, vol. 34, no. 2 (Summer 2005): 51-53.
Godown, Jeff. “The CompStat Process: Four Principles for Managing  Crime Reduction,” The Police Chief, vol. LXXVI, no. 8 (August 2009),  accessed on September 5, 2012,  http://www.policechiefmagazine.org/magazine/index.cfm?fuseaction=displ ay&article_id=1859&issue_id=82009.
Griffith, John, and Gadi Dechter. “Performance Reviews That Work: Four  Case Studies of Successful Performance Review Systems in the Federal  Government,” Center for American Progress (Washington, D.C.: February  2011).
Hatry, Harry and Elizabeth Davies, “A Guide to Data-Driven Performance  Reviews,” IBM Center for the Business of Government Improving  Performance Series (Washington, D.C.: 2011).
Henderson, Lenneal. “The Baltimore CitiStat Program: Performance and  Accountability,” IBM Endowment for the Business of Government  Managing For Results Series (Arlington, VA: May 2003).
Kettl, Donald. “China Looks West for Performance Management,”  Governing, August 2011, http://www.governing.com/columns/potomac- chronicle/china-looks-west-performance-management.html.
Kingsley, Christopher. “Smart Cities: PerformanceSTAT at 15,” University  of Pennsylvania Fels Institute of Government (Philadelphia, PA; October  2010).
Malinowski, Chris, and Sasha Page. “Top 10 Performance Measurement  Dos and Don’ts,” Government Finance Review, vol. 20, no. 5 (October  2004): 28+.
Moynihan, Donald, and Stéphane Lavertu. “Do Performance Reforms  Change How Federal Mangers Manage?” The Brookings Institution  Issues in Governance Studies Series, 52 (Washington, D.C.: October  2012).
O’Connell, Paul, and Daniel Forrester. “Think Differently: Update Your  Stats to Unlock Outcomes,” The Public Manager, vol. 38, no. 4 (Winter  2009-2010): 5-9.
O’Connell, Paul. “Using Performance Data for Accountability: The New  York City Police Department’s CompStat Model of Police Management,”  The PricewaterhouseCoopers Endowment for the Business of  Government Managing for Results Series (Arlington, VA: 2001).
Partnership for Public Service. “From Data to Decisions: The Power of  Analytics,” IBM Center for the Business of Government (Washington,  D.C.: November 2011).
Perez, Teresita and Reece Rushing. “The CitiStat Model: How Data- Driven Government Can Increase Efficiency and Effectiveness,” Center  for American Progress (Washington, D.C.: April 2007).
Queensland Audit Office. Better Practice Guide Performance Reviews.  Brisbane, Australia: 2010.
Sanger, Mary Bryna. “From Measurement to Management: Breaking  through the Barriers to State and Local Performance,” Public  Administration Review, S1 (December 2008): S70-85.
Scofield, Jennifer. “Asking Why Cuyahoga CountyStat,” Government  Finance Review, vol. 27, no. 6 (December 2011): 40-43.
Serpas, Ronal, and Matthew Morley. “The Next Step in Accountability- Driven Leadership: “CompStating” the CompStat data,” The Police Chief,  vol. LXXV, no. 5 (May 2008), accessed on September 5, 2012,  http://www.policechiefmagazine.org/magazine/index.cfm?fuseaction=displ ay_arch&article_id=1501&issue_id=52008.
Shane, Jon. “Compstat Process,” FBI Law Enforcement Bulletin, vol. 73,  no. 4 (April 2004): 12-21.
Sharp, Cathy, Jocelyn Jones, and Alison Smith. “What Do We Measure  and Why? An Evaluation of the Citistat Model of Performance  Management and its Applicability to the Scottish Public Sector,” Scottish  Executive Social Research (Edinburgh, UK: 2006).
Solan, David. “The EPAStat Quarterly Report and Lessons Learned,”  Public Performance & Management Review, vol. 33, no. 2 (December  2009): 222-240.
Steinberg, Harold. “Using Performance Information to Drive Performance  Improvement,” Association of Government Accountants Corporate  Partner Advisory GroupResearch Series: Report No. 29 (Alexandria, VA:  2011).
Useem, Greg. “Moving from Reporting Performance Information to Using  It,” Government Finance Review, vol. 25, no. 2 (April 2009): 47-50.
Weitzman, Beth, Diana Silver, and Caitlyn Brazill. “Efforts to Improve  Public Policy and Programs through Data Practice: Experiences in 15  Distressed American Cities,” Public Administration Review, vol. 66, no. 3  (May/June 2006): 386-399.
section_title: Related GAO Products
paragraphs: Managing for Results: Key Considerations for Implementing Interagency  Collaborative Mechanisms. GAO-12-1022. Washington, D.C.: September  27, 2012  Strategic Sourcing: Improved and Expanded Use Could Save Billions in  Annual Procurement Costs. GAO-12-919. Washington, D.C.: September  20, 2012  Solar Energy: Federal Initiatives Overlap but Take Measures to Avoid  Duplication. GAO-12-843. Washington, D.C.: August 30, 2012.
Entrepreneurial Assistance: Opportunities Exist to Improve Programs’  Collaboration, Data-Tracking, and Performance Management.  GAO-12-819. Washington, D.C.: August 23, 2012.
Managing for Results: GAO’s Work Related to the Interim Crosscutting  Priority Goals under the GPRA Modernization Act. GAO-12-620R.  Washington, D.C.: May 31, 2012.  2012 Annual Report: Opportunities to Reduce Duplication, Overlap and  Fragmentation, Achieve Savings, and Enhance Revenue.  GAO-12-342SP. Washington D.C: February 28, 2012.
Department of Energy: Additional Opportunities Exist to Streamline  Support Functions at NNSA and Office of Science Sites. GAO-12-255.  Washington, D.C.: January 31, 2012.
Small Business Administration: Progress Continues in Addressing  Reforms to the Disaster Loan Program. GAO-12-253T. Washington, D.C.:  November 30, 2011.
U.S. Coins: Replacing the $1 Note with a $1 Coin Would Provide a  Financial Benefit to the Government. GAO-11-281. Washington, D.C.:  March 4, 2011.
Government Performance: Strategies for Building a Results-Oriented and  Collaborative Culture in the Federal Government. GAO-09-1011T.  Washington, D.C.: September 24, 2009.
Government Performance: Lessons Learned for the Next Administration  on Using Performance Information to Improve Results. GAO-08-1026T.  Washington, D.C.: July 24, 2008.
Results Oriented Cultures: Creating a Clear Linkage between Individual  Performance and Organizational Success. GAO-03-488. Washington,  D.C.: March 14, 2003.