Managing for Results: Data-Driven Performance Reviews Show Promise But Agencies Should Explore How to Involve Other Relevant Agencies
Released: 2013-02-27T00:00:00

Letter

Background

The model for data-driven performance reviews was established in the early 1990s by New York City Police Department (NYPD) leadership as a strategy to reduce crime. Dubbed “CompStat,” the NYPD’s weekly reviews grew from the premise that use of crime data could enable leadership to make better-informed, more effective decisions. CompStat followed four key tenets: 1. Accurate and timely intelligence. 2. Effective tactics. 3. Rapid deployment. 4. Relentless follow-up and assessment.

As the law enforcement community observed NYPD’s improved accountability and bottom-line results in crime reduction, many police departments nationwide began to replicate CompStat in their own organizations. The CompStat model was subsequently adopted by many cities, municipalities, and some states—most notably Washington and Maryland—as a general performance management tool. Many of these efforts were patterned on the same four tenets as CompStat.

Nearly two decades later, the Obama administration began to encourage the use of data-driven review meetings as a performance management tool through several memorandums issued by OMB in 2010 and 2011, and a June 2011 executive order. During this timeframe, GPRAMA introduced the concept of data-driven performance reviews at the federal level with a provision that federal agencies conduct quarterly performance reviews on progress toward their APGs. Specifically, agencies are required to assess how relevant programs and activities contribute to achieving APGs; categorize goals by their risk of not being achieved; and for those at risk, identify strategies to improve performance. GPRAMA also specified that the reviews must occur on at least a quarterly basis and involve key leadership and other relevant parties both within and outside the agency. GPRAMA required agencies to begin conducting quarterly performance reviews by June 2011. Some agencies began conducting data-driven reviews earlier, in response to the executive order, OMB guidance, or other performance management efforts.

Several efforts aid agencies in implementing the GPRAMA-required quarterly performance reviews. The PIC established a working group on internal agency performance reviews. The working group meets monthly to share leading practices and discuss strategies for improving performance. Participation in the working group is voluntary and according to OMB, PIOs or designees from 21 agencies across the federal government are currently represented. In addition, OMB supported performance review implementation by issuing new guidance in OMB Circular A-11 during August 2012.

DOE, SBA, and Treasury each had experience with data-driven performance reviews before the June 2011 GPRAMA implementation deadline. Officials at both SBA and Treasury said that when new leadership came in, they brought interest in data-driven decision making, along with key staff that were experienced at data-driven review or data analysis. DOE officials said that their experience with the American Recovery and Reinvestment Act of 2009 (Recovery Act) work led them to begin data-driven performance reviews along with the GPRAMA requirements.  DOE built on its Recovery Act-related performance reviews with the establishment of its quarterly performance reviews, called business quarterly reviews, in 2011. According to DOE officials, the Deputy Secretary leads the review meetings with participation by under secretaries, and each review covers the department’s 8 priority goals, 15 mission-related key goals, and 8 management and operations key goals. The Associate Deputy Secretary, who is responsible for agency-wide management and operations goals, also participates in the reviews. The review meeting follows a structured format, with the first half focusing on performance goals and the second half focusing on fiscal issues, such as budget execution.  SBA began conducting reviews, called quarterly performance reviews, during the third quarter of 2009. SBA started by holding separate meetings by program area but changed the review format in early 2010 to include all program offices at each review meeting. According to SBA officials, the Deputy Administrator leads SBA’s quarterly performance reviews with support from the chief operating officer (COO) and the PIO. The PIO also assembles the data and disseminates follow-up action items. Review meetings also include senior officials from functional management areas (e.g., information technology, human capital, procurement, etc.). In addition to discussing agency priority goals, SBA reviews cover the rest of the agency’s performance goals and objectives, as time permits.  Treasury started conducting department-level performance reviews, called Stats, in 2010. Treasury’s reviews, led by the Deputy Secretary, are performed on an agency component-by-component basis. Reviews also involve senior officials responsible for functional management areas such as information technology, financial management, and procurement, as well as policy officials. The Stat process is managed by Treasury’s Office of the Deputy Assistant Secretary for Management and Budget, which reports to the PIO. The Stats cover Treasury’s two priority goals along with a range of other department programmatic and operational goals and priority projects.

Practices that Promote Successful Data-Driven Performance Reviews at the Federal Level

Agency Leaders Use Data- Driven Reviews as a Leadership Strategy to Drive Performance Improvement

Our analysis confirmed that to be successful, data-driven reviews should be used as a leadership strategy to drive performance improvement. Agency leadership must be directly and visibly engaged in the review process and invest the time necessary to understand and interpret the data being discussed during the meetings. Moreover, GPRAMA requires an agency head and deputy head to conduct quarterly priority progress reviews which fosters ownership and helps ensure that participants take the reviews seriously and that decisions and commitments can be made. OMB Circular No. A-11 also emphasizes the importance of leadership involvement in quarterly performance reviews, allowing the COO, the agency head, or both to conduct the review.

Our survey of PIOs indicated that agency leadership, with the exception of agency heads, actively participated in these reviews. This was consistent with what we found at DOE, SBA, and Treasury. The sidebar at left shows the survey results on the participation of key positions.

At Treasury, where we observed two Stat meetings, the Deputy Secretary used the Stat meetings to challenge participants to stretch toward ambitious performance goals and to provide possible solutions for any issues that were discussed. By holding separate reviews for all of Treasury’s bureaus and key offices, the Deputy Secretary committed a significant amount of time to these reviews. Other Treasury officials—at both the department headquarters and bureau levels—were appreciative of the amount of time the Deputy Secretary devoted to the reviews. The Deputy Secretary’s high-level position allowed him to speak with clear authority in the department and garner the attention of component agency leaders on performance issues. For example, during one review meeting that we observed, the Deputy Secretary challenged Treasury’s bureau leaders to develop new strategies for the department to collect delinquent debt payments owed to the federal government.

We did not observe quarterly performance meetings at DOE or SBA. However, officials we interviewed said that top leadership was actively driving performance discussions. At DOE, the Deputy Secretary led the quarterly review sessions and guided performance discussions with the department’s under secretaries on progress made toward achieving their respective performance goals. DOE’s Deputy Secretary noted that it is important that under secretaries take ownership of the performance review process to ensure the reviews are useful. At SBA, the Deputy Administrator led the reviews with support from the COO and PIO. According to officials, SBA’s leadership asks probing questions of program heads concerning SBA’s performance goals.

Officials from DOE, SBA, and Treasury expressed concern about maintaining the continuity of leadership engagement in performance review meetings and pointed to ways to help ensure that the review process continues with transitions to new agency leadership. For example, officials at DOE pointed out that they have a responsibility to remind new leadership of the legislative requirement for performance reviews as well as OMB’s guidance on these reviews. In addition, SBA officials noted that having a mix of career and political leadership involved in the reviews can facilitate continuity of the review practices since career officials generally span successive administrations. At Treasury, performance budgeting staff developed written standard operating procedures as a way to document the review process and pave the way for new leadership engagement at the department.

To encourage new leadership to take ownership of the performance reviews, several agency officials noted the importance of designing the reviews to fit the leadership style and preferences of the incoming leader. For example, one official said that it is critical for the incoming leader of the review meetings to provide input into the presentation format and structure of the review based upon personal preferences. At Treasury, headquarters staff responsible for managing the performance reviews worked closely with the Deputy Secretary to develop a review template and meeting format that met his needs.

Key Players Attend Reviews to Facilitate Problem Solving

Our analysis indicated that performance review meeting participants should include high-level leaders and managers with an agency-wide perspective as well as those with programmatic knowledge and responsibility for the specific performance issues likely to be raised. In addition, participants should typically include those with agency-wide functional management responsibilities, such as information technology, budget, and human capital. This enables the reviews to facilitate problem solving by breaking down information silos and providing managers from across the agency and other contributing organizations with a forum to communicate with each other and identify improvement strategies and agree upon specific next steps. At the city and state level, data-driven performance reviews typically include senior management from multiple agencies. In addition to the benefit of in-person meeting attendance, there is also value in having key players participate in other parts of the review process, such as the data review and analysis leading up to the meetings and the follow-up actions that arise from the meetings. Consistent with this practice, OMB’s guidance directs agencies to include, as appropriate, relevant personnel from within and outside the agency in the review meetings.

According to our survey, most PIOs—21 of 24—reported that their reviews included the participants needed to facilitate problem solving and identify performance improvements the majority of the time. Further, 19 PIOs reported that goal leaders had large involvement in their agency’s performance reviews, and 15 PIOs reported that internal contributors to agency goals and functional management chiefs, such as the CFO, had large involvement in their agency’s reviews.

At DOE, SBA, and Treasury, officials said they found that including senior management responsible for specific mission program areas—as well as those with functional management responsibilities in areas such as budget, information technology, human capital, procurement, and legal counsel—enhanced performance improvement efforts at review meetings. For example, both DOE and Treasury officials said discussions with budget officials concerning performance goals provided an opportunity for performance issues to be discussed in the context of budget issues where relevant.

However, differences in the scope of the reviews influenced which key players attended the review meetings. For instance, while Treasury’s reviews focused primarily on a specific component organization, such as a bureau or policy office, representatives from multiple components attended the review meetings when the achievement of a performance goal crossed component lines, or when the components had other commonalities. One Treasury official noted that the Bureau of Engraving and Printing and the U.S. Mint were asked to sit in on each others’ sessions, as they both work under the Treasurer and have similar operations.

Treasury officials indicated that one of the reasons why they chose to focus their performance reviews on individual component agencies—such as IRS and the Bureau of the Fiscal Service—was that their performance goals are generally aligned by component rather than shared by components. In addition, according to a senior Treasury official, focusing reviews on components allows time for more in-depth reviews. Treasury’s reviews include officials with department-level responsibilities in functional management areas such as budget, procurement, and legal counsel who can contribute to problem solving or who may be called on to take follow- up actions. In addition, Treasury officials noted that component-by- component reviews enable Treasury not only to have attendance from the bureau head, but also from more members of the bureau’s leadership team, such as the bureau CFO and others.

Officials said that if reviews were held with all of Treasury’s components at one time it would be impractical to gather all key members of bureau leadership because of the limited time which would be available for discussion on any particular topic. Treasury officials said that their practice of using the same information template for the Stat review with each component enables the reviews to cover similar issues across components and identify actions that should be addressed collaboratively, even though all components do not participate in the review meetings at the same time.

Conversely, at DOE and SBA, each review meeting included leadership participation from across all agency mission areas. However, DOE and SBA were similar to Treasury in that they included officials with agency- wide functional management responsibilities in the reviews. Officials noted that DOE’s reviews provided an opportunity for senior leadership to discuss the context around their performance goals and improve results with help from other areas of the department. In addition, the presence of senior officials across DOE’s programs provided an opportunity to share leading practices in various areas across program lines. At SBA, officials noted that many senior career officials attending these reviews had experience managing multiple programs and their broader experiences helped them understand and identify relevant performance metrics across program areas. In addition, officials mentioned that, in contrast to a large department with many discrete mission areas, SBA’s programs are focused on achieving a relatively narrow mission of supporting small businesses, which creates more opportunities for cross-program collaboration.

According to our survey results, PIOs did not see getting the right personnel included in the meetings as a challenge: Only 2 of 24 PIOs reported a challenge in including those managers or staff needed to facilitate problem solving and identify improvement opportunities. However, officials from DOE, SBA, and Treasury did note challenges in getting the right mix and number of participants to most effectively facilitate performance improvement efforts at the meetings. See figure 1.

According to survey results, 16 of 24 PIOs indicated that there was little to no involvement in the reviews from external officials who contribute to the achievement of agency goals. This is consistent with what we found at DOE, SBA, and Treasury, as stakeholders from outside the agencies did not participate in their performance review meetings.

At Treasury, officials said they addressed performance issues requiring external collaboration by conducting follow-up meetings with relevant external participants. These meetings were scheduled as part of Treasury’s practice of following up on issues raised during the review sessions. For instance, during a Treasury review meeting we observed, the Deputy Secretary directed staff to arrange a meeting with OMB to help address an obstacle to achieving a performance goal.

Likewise, DOE and SBA officials said they undertook collaborative actions as a result of discussions in their quarterly performance reviews, but neither planned to invite external representatives to their meetings. For example, SBA’s reviews led to multiple discussions with other federal agencies on whether some of their government contracts that were going to large businesses could go to small businesses instead.

Officials we interviewed cited several concerns that may explain why, at present, agencies are generally not including external participants—from other federal agencies or other relevant organizations—in their reviews. First, officials did not include external stakeholders because they wanted to keep discussions focused on internal problem-solving and were concerned that including external parties might inhibit open discussions on performance issues. Second, reviews at DOE, SBA, and Treasury mainly focused on goals achieved through internal contributors at each agency or office and officials noted that it would not currently be an efficient use of time to include external parties, even though external issues are discussed. One official said that the logistics of including high- ranking agency managers from other agencies could make it difficult to schedule review meetings on a timely basis. Another official who had experience managing data-driven performance reviews at different levels of government noted that city- and state-level reviews tend to be run by a mayor or governor with direct authority over the various agencies that participate in the reviews. This official pointed out that circumstances are different at the federal level, where an agency head could invite but not require outside participation and would not have control over the information shared or whether follow-up action was carried out.

However, we have previously reported that agencies can collaborate more effectively across organizational lines when presented with a clear and compelling rationale to do so and when agency leaders demonstrate their commitment to working collaboratively. Agency goals that require the efforts of more than one agency could serve as such a compelling rationale—even in the absence of direct authority requiring such collaboration. Moreover, our prior work has shown that agencies which participated in various planning and decision-making forums together— such as interagency councils or planning bodies—reported that such interactions contributed to achieving their goals. Specifically, agencies reported that such participation opened lines of communication, fostered trust, and helped build relationships, which can in turn lead to more effective collaboration across agency lines. Despite the concerns that DOE, SBA, and Treasury raised about including external participants in their reviews, our survey results indicate that some agencies are doing so: 4 of 24 PIOs reported moderate to large involvement of external officials who contribute to the achievement of agency goals. In addition, OMB officials provided an example of two agencies which have been successfully making use of quarterly performance reviews to collaborate on their APGs. These officials told us that the Departments of Housing and Urban Development and Veterans Affairs—which both contribute to efforts to reduce veterans’ homelessness—had conducted several Stat meetings jointly. According to OMB officials, program staff members from both agencies regularly participate in HUD Stat meetings, where they jointly analyze performance data to understand trends, identify best practices, and prioritize the actions needed to achieve veteran homelessness goals. Officials reported that these collaborative meetings have contributed to better outcomes. Moreover, officials from both OMB and the PIC indicated that agencies have increasingly been observing others’ review meetings as a means of learning about different practices with no apparent harm to the effectiveness of the meetings. This suggests that the challenges, if any, to outside participation can be overcome.

While there are many approaches to managing performance to achieve goals that rely on multiple agencies, few are likely to provide the benefit of bringing together the leadership and all the key players to solve problems and motivate performance improvement. Moreover, when key players are excluded from quarterly performance reviews, agencies may be missing opportunities to have all the relevant parties participate in developing solutions to performance problems. Instead, agencies will need to rely on potentially duplicative parallel coordination mechanisms, which could result in less than optimal performance improvement strategies.

Reviews Ensure Alignment between Agency Goals, Program Activities, and Resources

Our analysis showed that quarterly performance reviews should be used to align an organization’s resources, programs, and activities to ensure they are contributing to the achievement of agency goals. To help ensure that reviews are focusing on the appropriate interim goals and measures, agencies can develop models that describe the logical relationship between an agency’s inputs, activities, outputs, and outcomes. These logical relationships, sometimes called logic models, should be periodically assessed to determine if outcomes are being achieved as expected, and should be revised if necessary.

Our survey results indicated that PIOs do not find goal alignment to be a challenging aspect of implementing quarterly performance reviews, with 13 of 24 PIOs reporting that ensuring alignment between performance reviews and strategic goals and performance objectives was easy. Only one PIO reported this as being a challenge. Moreover, 22 PIOs reported that the majority of their reviews are aligned with strategic goals and performance objectives.

Consistent with our survey results, we found that DOE, SBA, and Treasury had selected performance metrics, initiatives, and other areas of focus in their reviews that were linked to the accomplishment of APGs and other goals, such as strategic plan objectives and key operational goals. Officials from both DOE and Treasury described the processes they undertook to choose useful performance information to frame the performance review discussion. For example, DOE narrowed its list of more than 190 performance measures to provide leadership with a focused view of the department’s key goals, while also providing sufficient depth of information to be meaningful. Treasury officials responsible for designing the department’s quarterly performance reviews described extensive interactions with the Deputy Secretary and the bureaus to identify performance measures that could be used to promote discussion of performance issues and opportunities for improvement.

In addition, the three agencies described efforts to use logic models, project milestones, and other approaches to identify early information on how they were progressing toward long-term outcome goals, which officials said could be challenging to monitor. For example, DOE’s Office of Energy Efficiency and Renewable Energy created a template for program managers to develop logic models linking program inputs and outputs to longer-term performance objectives, such as achieving clean generation of 80 percent of the nation’s electricity by 2035. DOE identified outputs or intermediate outcomes that contribute to the clean generation of energy, such as reducing the cost of solar energy, which the department measures to indicate progress toward its 2035 goal. According to officials, logic models helped program staff communicate a coherent story about how the program’s key activities contribute to its goals.

At Treasury, performance budgeting staff said they included the status of priority projects—such as IRS plans to develop a streamlined, user- friendly website—in its quarterly performance review information (see figure 2). Officials explained that these priority projects were designated as such because Treasury sees them as the critical path to achieving agency priority goals and other key longer-term outcomes, which could not always be tracked on a quarterly basis. For example, the Deputy Secretary wanted to monitor the status of IRS’s website update because it is seen to be a lever toward the agency’s priority goal to improve the voluntary tax compliance rate, which can only be measured with data that lags by approximately five years.

Our analysis showed that because data-driven reviews are to foster improved performance, the focus of accountability should be on the responsible manager’s role in addressing problems and bringing about positive change. Agency leaders should hold goal leaders and other responsible managers accountable for knowing the progress being made in achieving goals and, if progress is insufficient, understanding why and having a plan for improvement. If data is insufficient for gauging progress, managers should be held accountable for improving the quality of the data so that it is sufficient for decision making. Managers should also be held accountable for identifying and replicating effective practices to improve performance. In addition, the goals addressed in the reviews should be aligned with managers’ and staff’s individual performance goals to create a line of sight that reinforces the connection between strategic goals and day-to-day activities of managers and staff.

GPRAMA introduced specific roles and responsibilities for agency heads, COOs, PIOs, and goal leaders in conducting quarterly performance reviews. For each APG, agency heads and COOs, with support from the PIO, must: review with the appropriate goal leader the progress achieved during the most recent quarter, overall trend data, and the likelihood of meeting the planned level of performance;  assess whether relevant organizations, program activities, regulations, policies, and other activities are contributing as planned to agency priority goals; categorize agency priority goals by risk of not achieving the planned level of performance; and identify prospects and strategies for performance improvement, including any needed changes to agency program activities, regulations, policies, or other activities for agency priority goals at greatest risk of not meeting the planned level of performance.

Across the government, a majority of PIOs reported that their agency’s reviews met these GPRAMA requirements, as indicated in figure 3.

Our survey results indicated that most PIOs—21 of 24—reported using the reviews to identify actionable opportunities for performance improvement at least half the time.

Consistent with the survey results, DOE, SBA, and Treasury reported that top agency leadership held officials accountable for identifying performance problems and opportunities for improvement. For example, Treasury officials said—and the sessions we observed confirmed—that their Stat meetings focus on performance and are a vehicle for the Deputy Secretary to challenge bureau heads to ensure their bureaus continually improve. For example, the Deputy Secretary reported that many of the bureaus had not reviewed their management metrics, such as internal controls, diversity issues and employee survey scores, for some time and the Stat reviews are an opportunity to engage the bureaus on these issues. At the Stat reviews we attended, we observed the Deputy Secretary discussing such management metrics. For example, he asked bureau leadership to explain why their scores on a government- wide employee satisfaction survey had dipped during the past year, questioned a decline in survey response rates, and discussed specific next steps that he could take to support plans for improvement.

At DOE, officials said that their quarterly performance reviews focused on areas where they were not on track to meet performance goals and ways to address this. One official at DOE noted that the Deputy Secretary expected the under secretaries to be accountable for all of their goals at the meeting. At SBA, officials said the focus of the meetings is to fix problems and that it is important to be prepared because the Administrator asks probing questions and managers are called out when progress is not being made and asked to explain what is being done to resolve the issues.

Each agency also reported taking steps to ensure that managers’ individual performance objectives are aligned with priority and other agency performance goals. For example, SBA’s performance agreements incorporate performance objectives which cascade from the agency’s priority goals and other performance goals and are aligned with its strategic plan. Samples of performance agreements we reviewed identified “performance elements linked to organizational goals.” For example, a district manager had a performance objective to hold outreach events to connect small businesses to contracting opportunities. This objective clearly links to SBA’s APG to increase small business participation in government contracting.

Agency Has Capacity to Collect Accurate, Useful, and Timely Performance Data

Our analysis indicated that the capacity to collect and analyze accurate, useful, and timely data is critical to successful data-driven reviews. Agencies should track both outputs and outcomes. Agencies should also look for opportunities to leverage data produced by other agency components or outside entities. In addition, having the capacity to disaggregate data according to demographic, geographic, or other relevant characteristics can aid in highlighting significant variation, which can help meeting participants to pinpoint problems and identify solutions. Agencies also need to plan for the time and resources required to generate and communicate performance data in a timely manner. Easy access to relevant databases and systems-generated analysis, such as providing analysts with the ability to develop performance reports without relying on information technology staff, can streamline the data collection and analysis processes.

While having accurate, timely, and useful data available is critical to successful performance reviews, 16 of 24 PIOs reported that this was a challenge—more than any other practice we asked about. However, all 16 of those PIOs also responded that accurate, timely, and useful data is available for their agency’s reviews about half the time or more, which may indicate that some agencies have found ways to address this challenge.

Our review of DOE, SBA, and Treasury illustrates how agencies can overcome some challenges to data availability. Our analysis of quarterly performance review documents indicated that each agency was producing data-rich analyses that identified trends and potential performance issues. However, agency officials described initial challenges in these areas and said that improving their capacities for data collection and analysis took time. For example, SBA’s Office of Government Contracting and Business Development collects data on the percentage of all federal agency contracts being awarded to small businesses. The office is dependent on a General Services Administration database, the Federal Procurement Database System- Next Generation (FPDS-NG), for its information. SBA officials were concerned about the quality of the data since each federal agency enters its own information. To address this concern, SBA officials said they provided agencies with individualized reports of potential anomalies in their small business contracting data. This process allowed agencies to verify and correct if necessary the anomalies before SBA published the annual Small Business Procurement Scorecard report. For example, if an agency listed a contract in FPDS-NG as a small business set-aside at the same time that the agency listed the contract as an open procurement competition, this would be flagged. SBA would then notify the responsible agency and give it an opportunity to correct the anomaly in FPDS-NG. In addition, SBA noted instances where performance data lag behind the performance review cycle. For example, the Department of Defense holds its procurement data back from FPDS-NG for one quarter for national security purposes. SBA officials said that instead of waiting for the next quarter, they obtain preliminary information from the Department of Defense.

Agency Staff Have Skills to Analyze and Clearly Communicate Complex Data for Decision Making

Our analysis indicated that agencies need staff with the skills to assess performance data for coverage and quality and to identify key trends, areas of strong or weak performance, and possible causal factors. In addition, those messages need to be effectively communicated to management and staff that will play a role in identifying and solving performance problems and making related decisions. Analysts and managers should carefully consider the type and amount of information that will be useful for performance reviews, as well as how to present the information to audiences with varying levels of technical or quantitative skills. Providing the right amount of easy-to-understand performance information can promote effective decision making during the quarterly performance reviews. For example, focusing the presentation on the message the data tell about performance, using well-designed graphics, and grounding the data in relevant context are effective communication techniques.

Our PIO survey results included information on 15 specific competencies associated with performance improvement responsibilities. For each of these competencies, the majority of PIOs reported the competencies were present among performance improvement staff to a large extent. However, survey results were less positive about 2 competencies specifically related to analytic abilities. Of the 24 PIOs, 9 reported that performance measurement competencies and 10 reported that organizational performance analysis competencies were present among their performance improvement staff to a small or moderate extent.

PIOs at DOE, SBA, and Treasury each described the teams they had assembled to support their performance improvement efforts. For example, SBA’s Deputy PIO had performance analysts to support the quarterly performance reviews and many other performance management activities, such as the production of a weekly dashboard of key performance metrics. However, SBA officials acknowledged that some staff were less comfortable working with data and they perceived this as a skills gap that needed to be addressed. These officials said they are addressing this through a combination of training and hiring. For example, as part of its leadership training, SBA began developing courses related to “decision support;” officials said the courses were designed to lead to competencies in spreadsheet development and analysis, presentation delivery, development of decision support datasets, and other analytic and presentation skills. Participants began training in late summer of 2012 with courses titled Principles of Analytics and Analytic Boot Camp.

Having staff with abilities to communicate analyses effectively is an important factor in successful performance reviews, and most PIOs—22 of 24—reported that data and relevant analyses are presented effectively to participants in their agency’s reviews about half the time or more. However, 11 of those 22 PIOs also reported that effective presentation of data and relevant analysis was challenging—the second largest challenge cited by the PIOs among the practices we asked about.

Consistent with our survey results, officials we interviewed at SBA and Treasury described the challenges they faced in developing skills sets that bridge the gap between data analysis and effective communication. At SBA, the Office of Performance Management developed internal training to help SBA managers improve their ability to communicate the message that the data suggest. One official recounted initial struggles to interpret data and then effectively communicate the key points relevant to performance improvement to those who were not analysts. He noted that the SBA Administrator told senior management that a “data dump” was not helpful, which helped them to realize what was needed. Training was developed to move managers and staff beyond basic analytic skills, with a focus on structuring presentations effectively, using data to drive management decisions, and in general, “telling your story so you’re drawing out insights, rather than just summarizing facts.” See figure 4.

At Treasury, performance budgeting staff developed a PowerPoint presentation template that was distributed to each bureau to complete in advance of the Stat meetings. The template provided a uniform data collection tool that incorporated data presentation design principles, to guide the bureaus in effectively communicating their message to the Deputy Secretary. For example, templates for line charts prompted bureaus to indicate whether the desired trend line direction was up or down, since this is not always immediately apparent to high-ranking reviewers who may not have the depth of background into the particular program or operation. According to a Treasury official, designing an effective presentation is as important as doing relevant, high-quality data analysis. One official pointed out, “If no one reads or understands the analysis, it doesn’t matter how good it was.”

SBA and Treasury officials responsible for managing the reviews also described challenges in balancing presentation uniformity with the need to provide context that varies. These officials noted that consistency was key to making performance information quick and easy to absorb, especially for leadership that has limited time to review such information. However, SBA noted that the down side of consistency is people’s tendencies to tune out information that appears to be repetitive. SBA’s PIO and COO said they have to continually look for ways to keep the performance review meetings engaging to participants and that “meeting fatigue” can be a problem. Further, several bureau officials at Treasury we interviewed said that while they understood the need for uniformity, the templates did not always provide them with enough flexibility to provide sufficient context for their performance information. While bureau officials we interviewed said that the process and template had improved over time, some felt that in the early days of the Stat reviews, they were so limited in their ability to “tell their story” to the Deputy Secretary that they did not think he was getting an accurate understanding of the issues. However, our review of multiple Stat documents indicated that there was a specific page in the template left open for issues the bureaus wanted to raise, and further, that some bureaus appended information to the template to provide additional context.

Rigorous Preparations Enable Meaningful Performance Discussions

Our analysis found that sufficient preparation for the performance review meeting is critical for a successful review. Key participants must be prepared to discuss agenda items related to their performance measures and progress toward goals as well as any other issues to be addressed. The time allotted to prepare for reviews also provides a prompt for participants to continuously update their performance data, assess progress toward their performance goals, and develop a response to any performance issues identified. Also, data to be presented during the reviews must be fully vetted prior to the meeting so that participants can focus discussions on data trends and analysis rather than on whether the data itself is correct.

According to our survey results, 22 of 24 PIOs reported that review participants are adequately prepared for performance reviews more than half the time, and 20 reported that, overall, it is not challenging for participants to be prepared for the reviews. Nevertheless, several agency officials from DOE, SBA, and Treasury said that there was a significant time investment in preparing information and coordinating among managers and analysts across headquarters and components or offices. Officials at Treasury noted that the process of preparing for the reviews forced the department and its component agencies to closely examine performance data and make sure they could explain it to the Deputy Secretary, and said this process was a valuable part of the performance review. As one bureau-level official explained, nobody wants to go before the Deputy Secretary with data that indicates a performance problem, unless they are able to explain the issue and show that they have already thought of strategies for improvement. As a result, preparing for the reviews sometimes prompted participants to conduct additional analysis and have advance discussions on how to address performance problems.

At DOE, SBA, and Treasury, we found several practices in place to prepare participants for review meetings. Officials at each agency stressed the importance of meeting preparation to ensure that the review sessions were productive. For example, Treasury employed a rigorous pre-meeting process which started with the performance budgeting staff developing a PowerPoint template, in consultation with the Deputy Secretary, specifying the performance information to be provided by each component agency for its Stat session. Treasury officials said that performance budgeting staff then met with component staff to discuss the new template and any changes. The templates we reviewed were organized into several categories, such as priority projects, management metrics, and other issues, and were distributed in advance of the Stat meeting and used as the meeting agenda. Treasury officials said there were typically several rounds of revisions to the PowerPoint template prior to the review session, with management and analysts at the component level coordinating with their counterparts at the department. Officials said that one of the goals of developing the template was to ensure that all participants are fully prepared and able to engage in meaningful discussions about performance. In particular, one official explained that a guiding principle is that none of the participants should ever be surprised by any of the topics to be discussed. In advance of each review session, Treasury’s Deputy Secretary reviews the completed document along with an explanatory briefing memo prepared by performance budgeting staff, which provides relevant context for any issues, suggests lines of questioning, and highlights particular decisions to be made. Performance budgeting staff also review the completed Stat template with component staff to discuss the contents and ensure that the component is aware of issues likely to be brought to the attention of the Deputy Secretary. Treasury’s senior officials emphasized the importance of ensuring that data discussed at the meeting were sufficiently vetted during meeting preparation so as not to spend time during the sessions determining whether the information is correct. Several officials pointed to instances in early review meetings that used valuable meeting time on data issues because department-level and component staff disagreed on baseline data used during the reviews.

DOE’s pre-meeting practices included briefings with the Deputy Secretary and under secretaries and development of a Business Quarterly Review binder that included descriptions of the program offices’ performance and budget information, a list of attendees, and background notes, as well as a list of actions from the previous performance review.

Reviews Are Conducted on a Frequent and Regularly Scheduled Basis

Our analysis showed that in-person meetings which are both frequent and regularly scheduled are a defining characteristic of data-driven reviews. Regularly scheduled meetings foster a culture of performance management and continuous improvement. The frequency of the meeting schedule should depend on the urgency of the problems to be fixed, frequency with which the data are collected, and speed with which agency action can have an impact on these data. GPRAMA requires that starting no later than June 2011, agencies must conduct meetings at least quarterly, but agencies may meet more frequently if it meets their needs.

According to our survey, all 24 of the PIOs reported that their agencies were conducting performance reviews at least quarterly, with 7 of those 24 reporting conducting reviews more frequently. At DOE, SBA, and Treasury, we found each agency had frequent, regularly scheduled performance review meetings. Treasury’s Deputy Secretary met separately with each of the bureaus two or three times a year. Generally, one or two of the meetings with each Treasury bureau focused on performance, and the other meetings focused on budget. Treasury also holds Stat meetings on certain department-wide goals related to human capital, procurement, and strategic sourcing. As a result, Treasury conducted more than three dozen performance review meetings throughout the year, with more than 100 reviews conducted as of November 2012. DOE and SBA both met quarterly with representatives from across their entire departments.

Our survey results indicated that while holding performance reviews as scheduled is generally occurring, it may present challenges at some agencies. Of 24 PIOs, 20 reported that their agency held the performance reviews as scheduled more than half the time, with 2 of these PIOs reporting scheduling as a challenge. In addition, the 4 PIOs who reported that this practice was occurring about half of the time or less also reported that it was a challenge.

Experiences at Treasury illustrated how it can be challenging to schedule performance review meetings with high-ranking officials. At one bureau, officials said that last-minute cancellations due to the Deputy Secretary’s schedule caused inefficiencies since extensive meeting preparations, including performance data analyses, had to be redone each time. Officials from this bureau also commented that the performance review meetings sometimes coincided with times during which the bureau had heavy workloads. The officials noted that scheduling all of the meetings at the beginning of the year would be helpful so that they could plan around them. According to Treasury officials, currently sessions are scheduled 4 to 8 weeks in advance.

SBA found that it was best to schedule a standing date for the performance review meetings, which are held on the third Tuesday after the end of every quarter, rather than try to coordinate with numerous senior officials’ schedules every time. By selecting this meeting time, SBA was also able to leverage an existing weekly operations meeting, rather than scheduling a separate meeting for the quarterly performance review.

Participants Engage in Rigorous and Sustained Follow-up on Issues Identified During Reviews

Rigorous and sustained follow-up on issues identified during the meetings is also critical to ensure the success of the reviews as a performance improvement tool. Important follow-up activities include identifying the individual or office responsible for each action item as well as who will be monitoring the follow-up. Follow-up actions should be included as agenda items for subsequent reviews to hold responsible officials accountable for addressing the issues raised and communicating what was done.

According to our survey results, follow-up activities are generally occurring even though some PIOs reported that this practice was challenging. In particular, 21 of 24 PIOs reported that follow-up activities occurred at their agencies more than half the time. Although 7 PIOs reported that follow-up activities were challenging, 5 of those 7 PIOs also reported that this practice was occurring more than half the time, which may indicate that some agencies have found ways to overcome these challenges.

DOE, SBA, and Treasury each had a different approach to ensure that follow-up items were carried out as agreed to at the review meetings and to ensure responsible parties were held accountable. At DOE, post- meeting activities included asking program offices questions that were generated by their review and assigning an analyst from headquarters performance staff to work with mission program staff to prepare answers to these follow-up questions and have them ready for the next performance review meeting. According to an SBA official, follow-up action items were typically discussed at weekly operations meetings, which helped officials to integrate their action plans into these other performance discussions. One example of an SBA follow-up item was to develop new strategies related to increasing federal contracts with small businesses. Officials said they ranked agencies by the total dollar value of contracts they issued, and they targeted procurement representatives at the top seven purchasing agencies for their outreach efforts.

At Treasury, its performance budgeting staff generated follow-up memorandums immediately after the review meetings, naming action items, responsible parties, and due dates. The status of follow-up items from the previous review meetings was also incorporated into the materials for the next review meeting so the Deputy Secretary could see if there was any lagging action. In addition, performance budgeting staff tracked overall performance on post-review follow-up to ensure that this part of the process was being managed effectively (see figure 5).

Several officials from Treasury and SBA noted the importance of receiving feedback from meeting participants to help improve the review process. For example, Treasury conducted a formal feedback meeting with bureau heads after the first round of review meetings and learned that components wanted more data from the department’s performance staff to help substantiate their program performance. More recently, Treasury developed a survey to obtain formal feedback from participants and other management and staff that contribute to the review process. The survey included questions on the amount of time invested in preparation and follow-up and asked respondents to rate their satisfaction with various aspects of the review process, including the template design and the guidance provided by performance budget staff. Respondents were also asked to rate the importance of the various aspects of the review to their bureau or office, among other questions. According to Treasury officials who manage the Stat reviews, they have made specific improvements based on the survey results, such as sending out templates earlier to give components more time to prepare.

SBA also solicited feedback from meeting participants and made some changes as a result. For example, an SBA official said that the agency used to schedule its quarterly performance review meeting after all the data were available, which was 45 days after the end of each quarter. However, the official said that participants wanted the meeting to be held when the majority of the data was available—only one of its offices’ data lagged—to allow them to take any needed action on a timely basis. As a result, the officials said SBA started scheduling the quarterly review meeting on the third Tuesday after the end of each quarter and conducting a separate meeting with the one SBA office that receives its data after that date.

DOE, SBA, and Treasury Officials Attributed Improvements in Performance and Decision Making to Quarterly Performance Reviews, although Some Reported Minimal Impact

Reviews Fostered Intra- agency Collaboration and Problem Solving to Improve Performance

DOE, SBA, and Treasury officials said their quarterly performance reviews allowed different functional management groups and program areas within each agency to share information and ideas for performance improvement. Officials said these reviews helped them to solve problems that were impeding progress toward performance goals or to develop new performance improvement strategies. In some cases, officials were able to point to specific performance improvements that they attributed to the reviews.

DOE officials said that quarterly reviews helped them take a more critical look at subordinate activities contributing to the achievement of their APGs. For instance, they examined barriers to achieving their weatherization goals and found they could produce better results by targeting not only individual homes but also multiuse buildings for retrofitting. Therefore, they created a program to leverage resources from nonprofit and private sector organizations focused on large scale retrofit projects for buildings. In another example, DOE identified barriers to achieving its solar energy cost reduction APG related to slow local permitting processes for solar installations along with other local-level activities that contributed toward the goal. To address these barriers, DOE funded a new program called the Rooftop Solar Challenge in which teams develop actions plans to standardize permit processes, update planning and zoning codes, improve standards for connecting solar power to the electric grid, and increase access to financing. Officials said they expected these new approaches to improve DOE’s performance in reducing solar energy costs.

According to officials, performance reviews at DOE also facilitated information sharing across the agency that led to better results. For example, officials said discussions at DOE’s quarterly performance reviews led offices to share effective procurement practices. In this case, offices have been sharing best practices related to strategic sourcing to identify areas of cost reduction. For example, a recent reorganization between Environmental Management and National Nuclear Security Administration presented an opportunity for Environmental Management to leverage existing capabilities in strategic sourcing. The performance reviews have allowed for additional DOE-wide discussions on strategic sourcing lessons-learned and partnerships to potentially achieve greater efficiencies and cost savings.

Another example provided by SBA officials illustrated how using the reviews to increase visibility of the small business contracting goal at higher management levels led to the adoption of new performance improvement strategies. After discussing contracting goal data at a quarterly review, officials said the Administrator and Deputy Administrator decided to call department secretaries at those federal agencies with the most potential for awarding small business contracts to emphasize the importance of the goal. Officials said other strategies, such as providing more training for procurement center representatives, who, among other things, assist small businesses in obtaining federal contracts, came out of discussion at the reviews. Officials said they anticipated that the new strategies that came out of the reviews would lead to better performance toward the small business contracting goal.

SBA officials also provided an example that illustrated how the reviews facilitated intra-agency collaboration that improved SBA’s bottom line. Officials said that during a quarterly performance review meeting, the head of the Office of Capital Access described anticipated staffing shortfalls and the head of SBA’s Office of Disaster Assistance noted that he expected to have staff members available during the slower season in disaster assistance work, which is cyclical in nature. SBA’s Administrator instructed the two offices to work together, and as a result, officials said they were able to reduce the Office of Capital Access’s labor costs by 20 to 30 percent compared to the cost of paying employees overtime or hiring temporary contractor labor.

Leadership Communication of Priorities at Reviews Motivated Performance Improvements

DOE, SBA, and Treasury officials said the quarterly performance reviews provided a venue for top leadership to directly communicate their priorities and the priorities of the administration, which led to performance improvements in these areas. For example, at Treasury, nearly all of the bureau-level officials we interviewed said the Stat meetings were valuable because they allowed for a firsthand understanding of the Deputy Secretary’s priorities for their bureaus. Bureau officials said that the Deputy Secretary used these meetings to challenge their performance targets and approaches to addressing performance problems and to identify new opportunities for improvement. For example, the Bureau of the Public Debt Commissioner said that upon hearing about the Deputy Secretary’s interest in one of his ideas for improving how the agency communicates its bond pricing approach to customers, he moved ahead with developing an actionable strategy which he believes will ultimately lead to better customer service. Treasury officials said the Stat review process improved decision making by creating an environment where meaningful discussions on improving performance were held, citing performance improvements at the U.S. Mint as an example (see figure 6).

Citing another example, Treasury officials said that the Deputy Secretary had made contracting with small business a department-wide priority. Although increasing small business participation in government contracting was a SBA priority goal, Treasury, like all federal agencies subject to the Small Business Act, had its own target to meet to contribute to the goal. Officials attributed increases in the department’s percentage of contracts with small businesses—Treasury was the only agency to achieve all of its fiscal year 2011 SBA small business prime contracting goals—to the Deputy Secretary’s “relentless attention” at the Stat meetings. At every Stat session with every bureau, the Deputy Secretary reviewed the individual bureau’s performance against the small business target goals. Officials said that Treasury’s chief procurement officer was present at every Stat meeting to facilitate goal achievement.

The Financial Management Service (FMS) Commissioner cited another example of how the Stat meetings provided a venue for the Deputy Secretary to communicate his priorities, which led to better performance. FMS has been pursuing several priority projects to modernize its payments, collections, and central accounting systems that serve federal agencies across the government. For example, one project is to replace a paper process that many agencies use to accept vendor invoices with a central invoicing system. FMS’ analysts estimated that a central invoicing system could save the federal government $400 to $500 million annually, as well as provide vendors with online access to the status of their payments. FMS piloted the new system with the Bureau of Engraving and Printing. The new system enabled the Bureau of Engraving and Printing to pay their vendors more quickly than before, which resulted in the vendors having to provide “prompt pay” discounts. The decision was made to move forward government-wide, but to start with Treasury’s own bureaus. Following a Stat meeting in which the FMS Commissioner cited delays in adoption by other Treasury bureaus, the Deputy Secretary made it clear that implementing the new system was a priority. The FMS official said that, while he believed the adoption of the new system would have happened eventually even without the Stat meetings, he credited the Stat meetings with speeding up the process by three to four years.

A small number of Treasury bureau officials we interviewed had mixed views on whether the Stat meetings had actually improved performance, with some pointing out that certain performance improvements may have occurred without the Stat meetings. For example, Treasury leadership said they used the Stat sessions to closely monitor the consolidation of Bureau of the Public Debt and FMS into a single bureau, the Bureau of the Fiscal Service. Some officials thought they would have achieved the various consolidation milestones and goals without the Stat reviews and one said that the reviews added another layer of reporting that was time consuming. However, Treasury department officials noted that component management may not be aware of how the Deputy Secretary uses the reviews to better understand performance issues, including, in some cases, to ensure that the department is providing necessary support to improve performance.

Conclusions

Our review of DOE, SBA, and Treasury—as well as our survey of 24 PIOs—indicated that data-driven quarterly performance reviews hold promise as an effective management tool at the federal level. However, unlike city- and state-level data-driven reviews, which typically include representatives from multiple agencies, officials at DOE, SBA, and Treasury viewed their quarterly performance review meetings as an internal management tool and therefore did not open the reviews to outside participation. Officials said they relied on other means of collaborating with outside agencies and other partners that contribute to achieving cross-cutting goals. Furthermore, the majority of PIOs we surveyed indicated that there was little to no involvement in the reviews from external officials who could contribute to achieving agency goals. Successful data-driven performance reviews, which require extensive preparation and significant leadership time, do not come without a cost, so it is critical that agencies implement their reviews in a way that maximizes their effectiveness. As the implementation of the various GPRAMA provisions continues, agencies may need to reevaluate the most effective way to engage outside contributors in the quarterly performance review process for APGs and other performance goals that depend on other organizations to achieve desired outcomes. While there are many approaches to managing performance toward such goals, agency quarterly performance reviews could provide opportunities to bring together the leadership and all the key players needed to improve cross-agency and internal agency performance.

Recommendation for Executive Action

To better leverage agency quarterly performance reviews as a mechanism to manage performance toward agency priority and other agency-level performance goals, we are recommending that the Director of the Office of Management and Budget—working with the Performance Improvement Council and other relevant groups—identify and share promising practices to help agencies extend their quarterly performance reviews to include, as relevant, representatives from outside organizations that contribute to achieving their agency performance goals.

Agency Comments and Our Evaluation

We provided a draft of this report to DOE, OMB, SBA, and Treasury for review and comment. Each agency provided technical comments which we incorporated as appropriate. OMB staff generally concurred with the recommendation in our report. DOE and Treasury provided written comments, which are reproduced in appendixes IV and V, agreeing with our conclusions.

We are sending copies of this report to the appropriate congressional committees and the Secretaries of Energy and Treasury, the Administrator of SBA, and the Director of OMB. In addition, the report is available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII.

Appendix I: Objectives, Scope, and Methodology

As part of our mandate to review the implementation of the Government Performance and Results Act Modernization Act of 2010 (GPRAMA), our reporting objectives were to (1) identify practices that can promote successful data-driven reviews at the federal level and examine how these reviews are being implemented at selected agencies and across the government, and (2) examine the impact of quarterly data-driven performance reviews on selected agencies’ progress toward high priority and other performance goals. This report is the second in a series that examines how agencies are implementing various GPRAMA requirements.

To identify practices that can promote successful data-driven reviews at the federal level, we conducted a review of relevant academic and policy literature, including our previous reports. Based on our literature review, we developed a broad set of practices associated with the major contributors to success for data-driven reviews. We refined these practices with additional information obtained from practitioners at the local, state, and federal level who shared their experiences and lessons learned. As part of this engagement, we also compared these practices with recent GPRAMA-related guidance in the Office of Management and Budget’s (OMB) Circular No. A-11 and found them broadly consistent.

We observed two data-driven review meetings at the Department of the Treasury (Treasury), which was one of the agencies selected to address our reporting objectives.

To examine how these reviews are being implemented at agencies across the government, we surveyed the performance improvement officer (PIO) at each of the 24 federal agencies subject to the Chief Financial Officers (CFO) Act. We surveyed these agencies because GPRAMA directs us to focus on them in our reporting on the act. Additionally, several provisions of the act apply specifically to these agencies, including that the Performance Improvement Council (PIC) include them as members. We received responses from 24 PIOs—a 100 percent response rate.

The web-based survey was administered from October 18, 2012, to December 14, 2012. Respondents were sent an e-mail invitation to complete the survey on a GAO web server using a unique username and password. During the data collection period, we sent reminder e-mails and made phone calls to nonresponding agencies. Because this was not a sample survey, it has no sampling errors. The practical difficulties of conducting any survey may also introduce nonsampling errors, such as difficulties interpreting a particular question, which can introduce unwanted variability into the survey results. We took steps to minimize nonsampling errors by pretesting the questionnaire in person with PIOs and Deputy PIOs at three different agencies. We conducted pretests to make sure that the questions were clear and unbiased, the data and information were readily obtainable, and the questionnaire did not place an undue burden on respondents. We made appropriate revisions to the content and format of the questionnaire after the pretests and independent review. All data analysis programs used to generate survey results were independently verified for accuracy. Additionally, in reviewing the answers from agencies, we confirmed that PIOs had correctly bypassed inapplicable questions (skip patterns). Based on our findings, we determined that the survey data are sufficiently reliable for the purposes of this engagement.

To evaluate how effectively selected agencies are applying the practices of data-driven reviews in their GPRAMA-mandated quarterly performance reviews, and the effectiveness of these reviews toward achieving agency priority and other performance goals, we chose three agencies— Department of Energy (DOE), Small Business Administration (SBA), and Department of the Treasury (Treasury).

Because one of our goals was to describe challenges and lessons learned that will be useful to as many federal agencies as possible—as well as to the administration and to Congress—we selected agencies for case study that could provide a new illustration of challenges or lessons learned, that is, agencies that were not the recent or current subject of GAO or other public administration or public policy case studies. We also looked for agencies that could provide broadly applicable case illustrations, based on the scope of their mission, organizational complexity, and the mix of government tools—such as direct service, regulations, grants, loans, and tax expenditures—used to achieve their performance goals.

In addition, we excluded agencies that were undergoing significant management changes—such as leadership turnover or consolidation review—that could prevent us from gaining a clear picture of performance management or could interfere with our ability to make practicable recommendations. We also excluded agencies that had less than one year of experience in conducting data-driven reviews at the time that we started our field work.

At each selected agency, we focused on two agency priority goals (APG) to examine how quarterly performance reviews affected the agency components responsible for achieving performance outcomes. Because the scope of our review was to examine data-driven performance reviews as a leadership strategy, we did not evaluate whether these goals were appropriate indicators of agency performance, sufficiently ambitious, or met other dimensions of quality. The following are the complete sets of the agencies’ 2013 APGs. An asterisk indicates the ones we focused on in our review:  Save low-income families money and energy through weatherization  Reduce the department’s Cold War legacy environmental footprint.  Reduce the cost of batteries for electric drive vehicles to help increase the market for plug-in-hybrids and all electric vehicles and thereby reduce petroleum use and greenhouse gas emissions.  Make solar energy as cheap as traditional sources of electricity.*  Reduce consumer energy use and costs for household appliances.  Prioritization of scientific facilities to ensure optimal benefit from federal investments.  Make significant progress toward securing the most vulnerable nuclear materials worldwide within four years.  Maintain the U.S. nuclear weapons stockpile and dismantle excess nuclear weapons to meet national nuclear security requirements as assigned by the President through the Nuclear Posture Review.  Process business loans as efficiently as possible.* Increase small business participation in government contracting.*  Process disaster assistance applications efficiently.  Expand access to long term capital.

Increase electronic transactions with the public to improve service, prevent fraud, and reduce costs.* Increase voluntary tax compliance.* To address both of our objectives, we reviewed memorandums, internal briefings, and other materials agencies used to prepare for the reviews, as well as documents used during the reviews and follow-up materials. We conducted interviews with officials at OMB, the Performance Improvement Council, and senior-level officials involved in each agency’s performance review process to gain various perspectives on these reviews.  DOE: We met with the Deputy Secretary, who also serves as DOE’s chief operating officer (COO), and other senior-level officials including but not limited to the PIO, the Deputy PIO, and budget officials. In addition, we met with two APG goal leaders: the Deputy Assistant Secretary for Energy Efficiency and the Deputy Assistant Secretary for Renewable Energy.  SBA: We met with senior-level officials, including but not limited to the SBA’s PIO, Deputy PIO, and COO. In addition, we met with SBA’s Associate Administrator/Office of Capital Access, who serves as the goal leader for one of the AGPs we reviewed.  Treasury: We met with the Deputy Secretary, who also serves as Treasury’s COO, and other senior-level officials at the department level, including the PIO, who also serves as the CFO, the Deputy PIO, and budget officials. We also met with senior-level officials at Treasury’s Bureau of the Fiscal Service, Bureau of the Public Debt, Financial Management Service, and the Internal Revenue Service.

In addition, we met with the Assistant Fiscal Secretary who serves as the goal leader for one of the APGs we reviewed.

We asked to observe at least one review meeting at each agency. Treasury allowed us to observe two of its quarterly performance review meetings—one focused on the Bureau of the Fiscal Service and one on the Internal Revenue Service. DOE and SBA did not allow us to observe their meetings, citing concerns that our presence could inhibit open discussion. During the interviews, we asked officials to identify any challenges to effective implementation they faced as the process evolved or any lessons they learned. We also asked officials to identify examples of any impacts on performance that they attributed to the reviews.

We conducted our work from April 2012 to February 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Responses to Selected Questions from GAO’s Survey on Agency Performance Management Positions and Responsibilities

To address several research objectives related to implementation of the Government Performance and Results Act Modernization Act of 2010 (GPRAMA) we distributed a web-based survey to the performance improvement officer (PIO) at each of the 24 agencies subject to the Chief Financial Officers Act. We received responses from all 24 PIOs. There were 45 survey questions; tables 1 through 9 below show questions and responses for the nine questions that were directly applicable to the research objectives in this report. We will publish the full survey results in a report in the spring of 2013 on the implementation of key management positions under GPRAMA. For more information about our methodology for designing and distributing the survey, see appendix I.

Appendix III: Federal Agencies Subject to the Chief Financial Officers Act

Appendix IV: Comments from the Department of the Treasury

Appendix V: Comments from the Department of Energy

Appendix VI: Full Text for Interactive Figure 1 on Finding the Right Meeting Size and Composition for Effective Performance Reviews

This appendix includes the print version of the text contained in interactive figure 1.

Several officials noted challenges in achieving the proper balance between having a small number of meeting participants, which can allow more air time for each participant, compared to a larger group which might not be a productive use of time for those attending.

Picture: Treasury Deputy Secretary and COO Neal Wolin (top) leads the department’s quarterly performance reviews, with support from Assistant Secretary for Management Nani Coloretti, who also serves as Treasury’s PIO.

In addition, balancing depth of knowledge and numbers of staff attending these reviews is another consideration for finding the right meeting size. One official pointed out that having a relatively small group can help participants feel more comfortable in revealing performance problems but that there is a risk of leaving out key players that need to be part of a performance solution. Officials from each agency described different approaches to finding the right meeting size and composition for effective performance reviews, with several officials acknowledging that it takes time to get it right.

Table 10 contains the rollover information from interactive figure 1 about how the Department of Energy (DOE), the Small Business Administration (SBA), and the Department of the Treasury (Treasury) conduct their quarterly performance review meetings.

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Elizabeth Curda and Laura Miller Craig managed this assignment. Darnita Akers, Virginia Chanley, Don Kiggins, and Bob Yetvin made contributions to all aspects of the report. Kim Gianopoulos, Kay Kuhlman, Judith Kordahl, Jill Lacey, Sarah M. McGrath, Tim Minelli, Kathleen Padulchick, William B. Shear, Albert Sim, and Meredith Trauner also provided assistance. In addition, Sabrina Streagle provided legal support and Donna Miller developed the report’s graphics.

Selected Bibliography

Behn, Robert. “Collaborating for Performance: Or Can There Exist Such a Thing as CollaborationStat?” International Public Management Journal, vol. 13, no. 4 (2010): 429-470.

Behn, Robert. “The Core Drivers Of CitiStat: It’s Not Just About the Meetings and the Maps,” International Public Management Journal, vol. 8, no. 3 (2005): 295-319.

Behn, Robert. “The Varieties of CitiStat,” Public Administration Review, vol. 66, no. 3 (2006): 332-340.

Behn, Robert. “What All Mayors Would Like to Know About Baltimore’s CitiStat Performance Strategy,” IBM Center for the Business of Government Managing for Performance and Results Series (Washington, D.C.: 2007).

Campbell, Mary. “Bringing Performance Excellence to the Public Sector: Washington State’s Odyssey,” ASQ World Conference on Quality and Improvement Proceedings, vol. 59 (2005): 279-288.

Esty, Daniel, and Reece Rushing. “The Promise of Data-Driven Policymaking,” Issues in Science and Technology, vol. 23, no. 4 (Summer 2007): 67-72.

Fillichio, Carl. “Getting Ahead of the Curve: Baltimore and CitiStat,” Public Manager, vol. 34, no. 2 (Summer 2005): 51-53.

Godown, Jeff. “The CompStat Process: Four Principles for Managing Crime Reduction,” The Police Chief, vol. LXXVI, no. 8 (August 2009), accessed on September 5, 2012, http://www.policechiefmagazine.org/magazine/index.cfm?fuseaction=displ ay&article_id=1859&issue_id=82009.

Griffith, John, and Gadi Dechter. “Performance Reviews That Work: Four Case Studies of Successful Performance Review Systems in the Federal Government,” Center for American Progress (Washington, D.C.: February 2011).

Hatry, Harry and Elizabeth Davies, “A Guide to Data-Driven Performance Reviews,” IBM Center for the Business of Government Improving Performance Series (Washington, D.C.: 2011).

Henderson, Lenneal. “The Baltimore CitiStat Program: Performance and Accountability,” IBM Endowment for the Business of Government Managing For Results Series (Arlington, VA: May 2003).

Kettl, Donald. “China Looks West for Performance Management,” Governing, August 2011, http://www.governing.com/columns/potomac- chronicle/china-looks-west-performance-management.html.

Kingsley, Christopher. “Smart Cities: PerformanceSTAT at 15,” University of Pennsylvania Fels Institute of Government (Philadelphia, PA; October 2010).

Malinowski, Chris, and Sasha Page. “Top 10 Performance Measurement Dos and Don’ts,” Government Finance Review, vol. 20, no. 5 (October 2004): 28+.

Moynihan, Donald, and Stéphane Lavertu. “Do Performance Reforms Change How Federal Mangers Manage?” The Brookings Institution Issues in Governance Studies Series, 52 (Washington, D.C.: October 2012).

O’Connell, Paul, and Daniel Forrester. “Think Differently: Update Your Stats to Unlock Outcomes,” The Public Manager, vol. 38, no. 4 (Winter 2009-2010): 5-9.

O’Connell, Paul. “Using Performance Data for Accountability: The New York City Police Department’s CompStat Model of Police Management,” The PricewaterhouseCoopers Endowment for the Business of Government Managing for Results Series (Arlington, VA: 2001).

Partnership for Public Service. “From Data to Decisions: The Power of Analytics,” IBM Center for the Business of Government (Washington, D.C.: November 2011).

Perez, Teresita and Reece Rushing. “The CitiStat Model: How Data- Driven Government Can Increase Efficiency and Effectiveness,” Center for American Progress (Washington, D.C.: April 2007).

Queensland Audit Office. Better Practice Guide Performance Reviews. Brisbane, Australia: 2010.

Sanger, Mary Bryna. “From Measurement to Management: Breaking through the Barriers to State and Local Performance,” Public Administration Review, S1 (December 2008): S70-85.

Scofield, Jennifer. “Asking Why Cuyahoga CountyStat,” Government Finance Review, vol. 27, no. 6 (December 2011): 40-43.

Serpas, Ronal, and Matthew Morley. “The Next Step in Accountability- Driven Leadership: “CompStating” the CompStat data,” The Police Chief, vol. LXXV, no. 5 (May 2008), accessed on September 5, 2012, http://www.policechiefmagazine.org/magazine/index.cfm?fuseaction=displ ay_arch&article_id=1501&issue_id=52008.

Shane, Jon. “Compstat Process,” FBI Law Enforcement Bulletin, vol. 73, no. 4 (April 2004): 12-21.

Sharp, Cathy, Jocelyn Jones, and Alison Smith. “What Do We Measure and Why? An Evaluation of the Citistat Model of Performance Management and its Applicability to the Scottish Public Sector,” Scottish Executive Social Research (Edinburgh, UK: 2006).

Solan, David. “The EPAStat Quarterly Report and Lessons Learned,” Public Performance & Management Review, vol. 33, no. 2 (December 2009): 222-240.

Steinberg, Harold. “Using Performance Information to Drive Performance Improvement,” Association of Government Accountants Corporate Partner Advisory GroupResearch Series: Report No. 29 (Alexandria, VA: 2011).

Useem, Greg. “Moving from Reporting Performance Information to Using It,” Government Finance Review, vol. 25, no. 2 (April 2009): 47-50.

Weitzman, Beth, Diana Silver, and Caitlyn Brazill. “Efforts to Improve Public Policy and Programs through Data Practice: Experiences in 15 Distressed American Cities,” Public Administration Review, vol. 66, no. 3 (May/June 2006): 386-399.

Related GAO Products

Managing for Results: Key Considerations for Implementing Interagency Collaborative Mechanisms. GAO-12-1022. Washington, D.C.: September 27, 2012 Strategic Sourcing: Improved and Expanded Use Could Save Billions in Annual Procurement Costs. GAO-12-919. Washington, D.C.: September 20, 2012 Solar Energy: Federal Initiatives Overlap but Take Measures to Avoid Duplication. GAO-12-843. Washington, D.C.: August 30, 2012.

Entrepreneurial Assistance: Opportunities Exist to Improve Programs’ Collaboration, Data-Tracking, and Performance Management. GAO-12-819. Washington, D.C.: August 23, 2012.

Managing for Results: GAO’s Work Related to the Interim Crosscutting Priority Goals under the GPRA Modernization Act. GAO-12-620R. Washington, D.C.: May 31, 2012. 2012 Annual Report: Opportunities to Reduce Duplication, Overlap and Fragmentation, Achieve Savings, and Enhance Revenue. GAO-12-342SP. Washington D.C: February 28, 2012.

Department of Energy: Additional Opportunities Exist to Streamline Support Functions at NNSA and Office of Science Sites. GAO-12-255. Washington, D.C.: January 31, 2012.

Small Business Administration: Progress Continues in Addressing Reforms to the Disaster Loan Program. GAO-12-253T. Washington, D.C.: November 30, 2011.

U.S. Coins: Replacing the $1 Note with a $1 Coin Would Provide a Financial Benefit to the Government. GAO-11-281. Washington, D.C.: March 4, 2011.

Government Performance: Strategies for Building a Results-Oriented and Collaborative Culture in the Federal Government. GAO-09-1011T. Washington, D.C.: September 24, 2009.

Government Performance: Lessons Learned for the Next Administration on Using Performance Information to Improve Results. GAO-08-1026T. Washington, D.C.: July 24, 2008.

Results Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success. GAO-03-488. Washington, D.C.: March 14, 2003.