Managing for Results: Practices for Effective Agency Strategic Reviews
Released: 2015-07-29T00:00:00

Letter

Background

GPRAMA requires OMB to annually determine whether agencies have met the performance goals and objectives outlined in their performance plans and submit a report on unmet goals. In implementing this provision, OMB’s guidance directs agencies to continue reporting on unmet performance goals in their annual performance reports, as has been required since fiscal year 1999. In addition, OMB’s guidance directs agencies to conduct leadership-driven, annual reviews of progress towards each strategic objective—the outcome or impact the agency is intending to achieve—established in the agency’s strategic plan. Figure 1, an illustrative example from OMB’s guidance, shows how strategic objectives relate to other goals within an agency’s performance management structure.

Agencies began conducting these reviews in fiscal year 2014. The results from their first round of reviews were published in their annual performance reports in February 2015, as well as on Performance.gov, the central governmentwide performance reporting website implemented by OMB to meet GPRAMA requirements. OMB’s guidance directs agencies to provide a progress update for each strategic objective, including a brief summary of what progress was made and an explanation of the achievements made or challenges that have impeded progress.

As part of their reporting, agencies were to identify a portion of their objectives as (1) having demonstrated noteworthy progress and (2) focus areas for improvement.

According to OMB’s guidance, the results of these reviews should (1) inform long-term strategy; (2) inform annual planning and budget formulation; (3) facilitate identification and adoption of opportunities for improvement, including risk management; (4) identify areas where additional program evaluation, other studies, or analyses of performance data are needed to determine effectiveness or set priorities; (5) identify where additional skills or other capacity are needed; (6) improve decision- making response time; (7) strengthen collaboration on crosscutting issues; and (8) improve transparency.

The PIC also provided support to agencies as they began planning for and implementing their strategic reviews. According to OMB and PIC staff, through the PIC’s Internal Reviews Working Group, agency officials shared information about their planned strategic review processes as well as lessons learned from the initial round of reviews. The PIC also hosted several summits focused on strategic reviews and published a guide in August 2014 on leading effective strategic reviews, based on agencies’ initial experience.

Moving forward, OMB staff told us that they expect agencies’ strategic review processes will mature over time, and as such expect the results of those reviews to mature over time as well. According to OMB staff, they used the information conveyed in figure 2 to communicate to agencies that they likely would not be able to fulfill all requirements in OMB’s guidance in initial implementation, but instead should develop a maturity model to ensure they continue to strengthen the reviews over time.

Practices for Facilitating Effective Agency Strategic Reviews

Establish a Process for Conducting the Agency’s Strategic Reviews

Sufficient planning and preparation is important to ensure that the agency’s strategic review process is successful. Our February 2013 report on data-driven performance reviews found this was critical to a successful review. Planning enhances the quality, credibility, and usefulness of the review, and helps ensure that participants’ time and resources are used effectively. Establishing common purposes for strategic review meetings can build trust and encourage active participation by participants. In addition, developing common terminology, policies, and procedures, and clarifying roles and responsibilities helps facilitate collaboration for productive meetings. Participants need to be prepared to review progress towards their strategic objectives and determine any subsequent actions.

Key features for planning the strategic review include:

Leadership commitment and involvement. Agency leadership should be directly and visibly engaged in the review process and invest the time necessary to understand and interpret the evidence being presented. This involvement fosters ownership among those involved in the review and helps ensure that participants take the reviews seriously and can make decisions and commitments with the knowledge and backing of leadership.

Communication of expectations and time frames. Guidance and agendas provided in advance of review meetings can establish a common understanding of the purpose of the review, the process to be used, and time frames for completing the review. In addition, standardized templates used to collect and share key information are helpful to facilitate strategic review discussions and help to ensure consistency across reviews.

Accountability for results. The focus of accountability should be on the responsible objective leader’s role in credibly assessing progress in achieving a strategic objective using evidence. Agency leaders should hold objective leaders and other responsible managers accountable for knowing the progress being made in achieving outcomes and, if progress is insufficient, understanding why and having a plan for improvement. If evidence is insufficient for assessing progress, managers should be held accountable for improving the availability and quality of the evidence so that it can be used effectively for decision making. Managers should also be held accountable for identifying and replicating effective practices to improve performance.

In addition, OMB’s guidance strongly encourages agencies to leverage existing decision-making processes to conduct strategic reviews. According to the guidance, in most cases, the strategic reviews should be integrated into existing agency management processes to raise key decisions, issues, and analysis to agency leadership. OMB’s guidance also provides agencies flexibility in developing their processes, stating that agencies should use a tailored approach that is appropriate for the nature of the agency’s programs, operations, and strategic objectives and evidence available.

NASA Involved Senior Leaders across the Agency to Help Ensure Commitment to Strategic Review Process and Accountability for Results

In developing the agency’s strategic review process in late 2013, NASA’s Performance Improvement Officer (PIO) at the time and her staff sought input on the process from NASA senior leaders. This group included the leaders for each of NASA’s strategic objectives who typically represent the most senior official with direct oversight of the programs and activities supporting each objective, such as division directors and deputy associate administrators, among other senior positions. According to PIO staff, all of NASA’s guiding principles for the strategic review process were informed by senior leadership, such as using existing management processes and structures, promoting transparency, and making the process intuitive and easy to understand. NASA PIO staff told us that this helped create buy-in and understanding for the strategic review process.

Each strategic objective leader, along with deputy objective leaders and relevant NASA staff, was involved in conducting individual assessments of each objective and provided a suggested rating. For example, the Director of the Heliophysics Division was the strategic objective leader for the strategic review of the objective “Understand the Sun and its interactions with the Earth and the solar system, including space weather.” NASA’s PIO and her staff then led crosscutting reviews of these individual assessments to identify themes and provide an independent rating recommendation. Following the crosscutting review, NASA’s Chief Operating Officer (COO) determined final ratings during a briefing attended by the PIO and each of the strategic objective leaders. At that meeting, a member of NASA’s PIO staff summarized review findings and results to the COO. The COO then asked each strategic objective leader clarifying questions and sought suggestions that would lead to performance improvements before settling on the final rating. According to NASA PIO staff, this approach of having all strategic objective leaders (and relevant program staff) attend the entire briefing encouraged transparency, and the personal involvement of the COO encouraged accountability for results and performance improvements.

DHS Communicated Key Aspects of Its Strategic Review Process at a Kickoff Meeting Attended by Process Participants

DHS’s Office of the Chief Financial Officer/Office of Program Analysis and Evaluation (CFO/PA&E) leads departmental implementation of performance management activities, including strategic reviews. In addition, each component agency has a designated PIO and performance staff who coordinate efforts in their component agency as part of department-wide performance management activities. For example, for U.S. Citizenship and Immigration Services (USCIS), this role is performed by the Office of the USCIS CFO. In early January 2014, CFO/PA&E met with component PIOs to provide a basis for understanding and participating in the department’s first strategic reviews. Recognizing the important role that they played in the initial reviews, CFO/PA&E revised its orientation process for the 2015 strategic reviews to include a separate briefing for assessment leads—the senior executives who lead teams reviewing progress towards each strategic goal.

The DHS briefing slides informed participants about the related GPRAMA requirement and OMB’s guidance, as well as the purpose and expected benefits of the department’s strategic reviews, such as informing the next DHS strategic plan, strengthening collaboration, and informing program and budget reviews. The briefing provided an overview of the department’s strategic review process, describing a structured methodology for conducting the reviews and samples of four standard deliverables (templates) to be used to collect information from each assessment team. It also identified the roles and responsibilities for various participants in the process, including assessment leads and teams conducting the review of each goal, the component and DHS PIOs, and CFO/PA&E staff. The briefing also provided a timeline for implementing the department’s strategic reviews, with specific dates for key activities to be completed.

USDA Used an Existing Performance Review Process and Template to Facilitate Strategic Reviews

Among other responsibilities, USDA’s Office of Budget Policy and Analysis (OBPA) oversees implementation of the department’s performance management activities. According to the Associate Director of OBPA, who also serves as USDA’s PIO, his office and relevant component agencies provide regular performance updates to the Secretary on key initiatives, such as the Blueprint for Stronger Service, an effort launched in 2012 to enhance administrative services and management operations. For these updates, which primarily occur monthly, depending on the initiative, USDA uses a standard template, known as a “quad chart,” to collect and present information to the Secretary for decision making. Because of the Secretary’s familiarity with the quad chart format, the department adapted the chart for use in its strategic review process, known as the strategic objective annual review (SOAR) (see figure 3).

As illustrated in figure 3, the SOAR quad chart includes the following information: the relevant agency or office within USDA responsible for the objective and the officials leading the efforts, known as objective owners and lieutenants; the strategic objective and the strategic goal it supports; a summary of progress towards the objective and related achievements; key performance indicators along with actual performance results compared to targets; a discussion of challenges that could affect program outcomes; and a description of next steps, crosscutting analysis, or evaluations to improve objective performance.

Objective owners and lieutenants are responsible for populating the information in the quad charts. Subsequently, OBPA reviews the quad charts before they go to the Secretary to ensure consistency in information reported and progress assessments, identify any needed changes, and determine if the information provided could impact other initiatives across the department. According to OBPA officials, the quad charts provide USDA leadership with succinct and sufficient information to make decisions to improve performance, such as approving new or modifying existing strategies, or adjusting time frames.

Clarify and Clearly Define Measurable Outcomes for Each Strategic Objective to Be Reviewed

A strategic review starts with framing the outcome or impact the agency seeks to achieve. According to OMB guidance, strategic objectives should be relatively simple statements that break down the broader, mission-oriented strategic goals to a level that reflects the impact or outcome the agency is trying to achieve through its programs. Objectives should be framed so they can serve as standards against which an assessment can reasonably be performed to determine the effectiveness of the agency’s implementation of its programs, as well as progress toward the ultimate outcome.

In some cases, defining and measuring the outcome related to a strategic objective may be relatively straightforward. For example, increasing employment rates for participants who completed a training program is an outcome defined in a way that can be measured. However, where agencies are focused on more long-term or complex outcomes, determining if the agency is making progress each year can be more challenging. In these instances, the agency may need to break the strategic objectives into pieces that can be more easily be measured or assessed.

NASA Clarifies Its Long-term Scientific Research Outcomes through 10-Year Success Statements, Multiyear Performance Goals, Measures, and Milestones

As part of its performance framework, NASA has associated time frames with its goals, as illustrated in figure 4.

For the agency’s planning process for its 2014 strategic plan and annual strategic review, strategic objective leaders developed success statements that covered up to a 10-year time frame for each of their objectives. According to PIO staff, for the success statements, objective leaders and staff were asked to characterize or define the outcomes of success in implementing their objectives in the next 10 years by answering questions such as, “What will the agency have completed, obtained, contributed, advanced?” NASA officials told us that because it can be difficult to measure progress towards long-term, scientific discovery-oriented outcomes, they also rely on underlying multiyear performance goals, annual performance indicators, and milestones to better plan for and understand near-term progress towards those objectives. Table 1 illustrates how NASA clarified long-term and near- term progress for its objective to Understand the Sun.

To frame DHS strategic goal 3.1, “Strengthen and Effectively Administer the Immigration System,” in more concrete terms, the lead agency, USCIS, focused on three sub-goals. Table 2 identifies the sub-goals and describes them.

In addition to the sub-goals, USCIS also developed performance measures (known as strategic measures) as part of its ongoing performance monitoring efforts for this goal. For example, one measure is the average processing cycle time (in months) for naturalization applications. Taken together, the sub-goals and performance measures show how DHS has identified measureable pieces of its efforts related to the larger goal.

Identify the Strategies and Other Factors That Influence the Outcomes and Determine Which Are Most Important

It is critical to identify, at a conceptual level, the various strategies and factors that can help or hinder achievement of the strategic objective. The federal government uses numerous activities and policy implementation tools, such as loans, grants, contracts, social and economic regulations, insurance, and tax expenditures, among others (hereafter strategies) to help address public problems. However, since 2011, our annual series of reports examining federal programs has found that agencies often employed overlapping or fragmented program strategies that were poorly coordinated. In addition, because the federal government rarely works in isolation, the efforts of other levels of governments (local, state, and international) and sectors (private and nonprofit) frequently contribute to the achievement of an outcome as well. Beyond these strategies and efforts, factors both within and beyond the control of any particular agency—generally referred to as internal and external factors—may influence an outcome. Internally, these factors could include an agency’s culture, management practices, and business processes. External factors may include the economy, demographic trends, technological advances, and the natural environment.

The strategic review for each objective should take into account the comprehensive set of federal strategies, nonfederal efforts, and factors within and outside an agency’s control related to the outcome. The more complex the outcome, the more likely it is to be influenced by multiple strategies, nonfederal efforts, and factors. Although these influences may have been previously identified through an agency’s strategic planning process or similar vehicle, they should be revisited as part of the strategic review to determine if anything has changed. OMB’s guidance directs agencies to identify in their strategic plans the various organizations and policy tools, both within and external to the agency, that contribute to their strategic objectives. However, our work reviewing GPRAMA implementation has found weaknesses in agencies’ abilities to identify contributors to their goals. For example, in our April 2013 report on agency priority goals (APG), we found that agencies had not always identified external organizations and policy tools that contributed to their goals, although required by GPRAMA and OMB’s guidance. We recommended that OMB ensure agencies adhere to its guidance by providing complete information about the contributors to their APGs. OMB staff agreed with this recommendation. According to information provided by OMB staff in April 2015, agencies were asked to identify organizations, program activities, regulations, policies, tax expenditures, and other activities contributing to their 2014-2015 APGs, first as part of the September 2014 update to Performance.gov, with opportunities for revisions in subsequent quarterly updates. Our analysis found that agencies have made progress in identifying external organizations and programs for their APGs, but they did not present this information consistently on Performance.gov. Although each APG webpage has a location where agencies are to identify contributing programs, agencies did not always identify external organizations and programs there. Instead, they identified these external contributors elsewhere, such as APG overview or strategy sections, which could limit the ability of users to easily locate this information. We will continue to monitor progress on implementation of this recommendation.

Using existing knowledge, expertise, and evidence, those involved in the review should identify the strategies, nonfederal efforts, and factors that are likely to have the strongest influence on the outcome. This information will help to establish priorities for the scope of the review. There are a number of methods that can be used to map or model the causal relationships among the inputs, processes, and outputs produced by various strategies and the forces that influence achievement of outcomes, such as results mapping and logic modeling. These methods can help to clarify the issues that must be addressed conceptually to create change or achieve the intended outcome. By identifying and examining the various influences on the strategic objective or expected outcome during the strategic review, an agency can better understand how the existing set of program outputs and activities are contributing to the achievement of outcomes and whether gaps exist or changes are needed in light of all the other factors that are influencing outcomes.

Recognizing that some of these influences may present risks or challenges to achieving expected outcomes, OMB’s 2014 update to its guidance (covering agency’s strategic reviews in 2015) states that while agencies cannot mitigate all risks related to achieving strategic objectives and performance goals, they should identify, measure, and assess To that challenges related to mission delivery, to the extent possible. end, the guidance encourages agencies to institute an Enterprise Risk Management (ERM) approach, and leverage such efforts when conducting strategic reviews. The guidance defines ERM as an effective agency-wide approach for addressing the full spectrum of the organization’s risks by understanding their combined impact as an interrelated portfolio, rather than addressing risks within silos. The guidance further states that with an ERM approach, agencies can be better positioned to quickly gauge which risks are directly aligned to strategic objectives, and which have the highest probability of impacting the agency’s mission.opportunities and challenges are routinely identified, analyzed, and Such an approach can help ensure that addressed, as appropriate, enhancing the agency’s capacity to more efficiently and effectively determine priorities and allocate resources.

For DHS’s Lawful Trade and Travel Goal, Officials from Four Component Agencies Identified Contributing Programs and Ranked Them by Degree of Influence

DHS’s review of its strategic goal 2.2 “Safeguard and Expedite Lawful Trade and Travel” involved four component agencies: Customs and Border Protection (CBP, the designated lead agency for the review), the Transportation Security Administration (TSA), Immigration and Customs Enforcement (ICE), and the U.S. Coast Guard (Coast Guard). According to CBP officials, each of these component agencies plays a role in implementing strategies supporting this goal. According to DHS’s Strategic Plan for fiscal years 2014-2018, the strategies for this goal are to (1) safeguard key nodes, conveyances, and pathways; (2) manage the risk of people and goods in transit; and (3) maximize compliance with U.S. trade laws and promote U.S. economic security and com- petitiveness.

The goal leader—CBP’s Executive Director for Planning, Program Analysis, and Evaluation, within the Office of Field Operations, who also led the assessment team—asked participating officials from the four contributing agencies to identify which of their programs and activities contributed to the achievement of the goal, and then subsequently to rank them by level of influence. Table 3 provides illustrative examples of programs and activities that support this goal from each of the four contributing component agencies.

According to CBP officials who coordinated the review, participating officials determined that a few of the programs and activities they initially identified as contributing to the goal had relatively minor influence towards the outcome. In these instances, the programs and activities primarily supported another DHS goal. DHS officials decided to include only those programs that primarily supported the goal under review. For example, CBP officials determined that CBP’s Container Security Initiative, which works with foreign governments to examine potentially high-risk cargo prior to departure from the foreign port of origin, may have had influence on safeguarding trade and travel, but more directly supported another DHS goal, “Secure U.S. Air, Land, and Sea Borders and Approaches.”

Food and Nutrition Service Officials Use a Logic Model to Identify Programs and Factors That Influence USDA’s Efforts to Improve Access to Nutritious Foods

USDA’s Food and Nutrition Service (FNS) seeks to increase food security and reduce hunger by providing children and low-income people access to food, a healthful diet, and nutrition education in a way that supports American agriculture and inspires public confidence. FNS uses a logic model (figure 5) to understand how its programs and other factors influence outcomes related to USDA’s objective to “improve access to nutritious food.” FNS first developed the logic model in the early 2000s as part of an effort to better integrate performance measurement into its operations. FNS officials told us that the concepts included in the logic model are often used when the agency is making decisions about performance measurement and evaluation because it shows the connections among program inputs, outputs, and overall outcomes. By making those linkages explicit, decision makers can have more focused and meaningful discussions for how proposed strategies are tied to desired results and how to measure the success of strategy execution and impact, according to FNS officials. As part of the strategic review process, FNS used its logic model to reaffirm the connections between program outputs and related outcomes.

As illustrated in figure 5, the logic model shows how the output of FNS’s programs (left column) contribute to relevant near-term and long-term outcomes (the three columns to the right). The model covers five contributing FNS programs: Child and Adult Care Food Program (CACFP), Fresh Fruit and Vegetable Program (FFVP), National School Lunch Program (NSLP), Supplemental Nutrition Assistance Program (SNAP), and Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). At each level, the logic model identifies related performance measures as well as external factors that could influence progress, such as FNS and state implementing agency resource levels, competing priorities and policies, and food price and availability.

Identify Key Stakeholders to Participate in the Review

Because the achievement of outcomes may be complex and involve a variety of contributors from within an agency, or include other federal agencies, levels of government, and sectors, it is critical to consider which key stakeholders should be involved in a strategic review. Each of these stakeholders provides a unique perspective on their contribution or view of progress of the outcome under review.

OMB’s guidance and our past work reinforce the importance of including key stakeholders in the review. OMB’s guidance states that the analysis of each objective should be conducted at the objective lead level, with support from relevant bureaus and programs, and that the COO and PIO office should be involved in analysis and decision making across all objectives. Our prior report on effective practices for data-driven performance reviews also indicated that performance review participants should include high-level leaders and managers with an agency-wide perspective, as well as those with programmatic knowledge and responsibility for the specific performance issues likely to be raised. Each of the six agencies covered by our work for this report developed strategic review processes that involved relevant internal stakeholders, from contributing program officials to the agency head or COO.

Agencies should also consider including the perspectives of relevant third-party policy experts, academics, professional associations, end users/clients or advocacy groups that represent them in the review process. When outcomes are complex and involve multiple organizations, it is also important to establish how existing collaboration mechanisms can facilitate joint data collection, analysis, and reporting, or if new networks should be established. In some cases, there may be an existing interagency group, such as a task force, that has been formed to achieve an outcome. Our prior work has shown that agencies that participated in various planning and decision-making forums together—such as interagency councils or planning bodies—reported that such interactions contributed to achieving their goals. Specifically, agencies reported that such participation opened lines of communication, fostered trust, and helped build relationships, which can in turn lead to more effective collaboration across agency lines.

In spite of the compelling rationale for all parties contributing to an outcome to collaborate, our past work on GPRAMA implementation has found that agencies generally have not included external stakeholders when reviewing progress on an outcome. In our report on implementation of data-driven reviews in February 2013, we concluded that as the implementation of various GPRAMA provisions continues, agencies may need to reevaluate the most effective way to engage outside stakeholders in the performance review processes for APGs and other performance goals that depend on other organizations to achieve desired outcomes.

We recommended that OMB work with the PIC and other relevant groups to identify and share promising practices to help agencies extend their quarterly performance reviews to include, as relevant, representatives from outside organizations that contribute to achieving their agency performance goals. OMB staff generally agreed with this recommendation. As of April 2015, OMB staff told us that agencies continue to find that most APG reviews are appropriately focused on internal agency management, rather than involving external stakeholders. Therefore, OMB and the PIC have focused recent efforts on developing and sharing promising practices related to conducting reviews internal to the agencies or on improving evidence/measurement. We will continue to monitor progress on implementation of this recommendation.

HUD Considered Input from Contributors Internal and External to the Department Regarding Efforts to End Homelessness

GAO-13-228.

Office of the Secretary, who serves as the objective leader. The review involved officials from the Office of Special Needs Assistance Programs within the Office of Community Planning and Development, which administers the department’s homelessness programs, such as the Continuum of Care Program, which funds local networks of organizations to quickly rehouse individuals and minimize the trauma and dislocation caused to individuals, families, and communities by homelessness. In addition, the review included participants from other HUD offices, such as the Office of Public and Indian Housing and the Office of Multifamily Housing, whose programs can assist in ending homelessness. For example, the Office of Public and Indian Housing’s Housing Choice Voucher Program provides rental subsidies for low-income families, which may include families experiencing homelessness. This crosscutting and inclusive approach reinforced one of HUD’s strategies supporting this objective—to fully engage and leverage mainstream housing assistance to build capacity among public housing agencies and multifamily owners to admit homeless households into their units.

Although no one outside of HUD directly participated in the review, HUD officials stated that they leveraged their existing relationship with officials at the U.S. Interagency Council on Homelessness (Interagency Council) and the 18 other federal agencies that comprise it to better understand how other federal programs are contributing to progress in ending This included attending and homelessness for the target populations.participating in various meetings, including quarterly meetings with the Interagency Council principals, staff-level coordinating meetings, and targeted working groups, such as bimonthly meetings of the chronic and family homelessness working group. In addition, as part of this objective, HUD and the Department of Veterans Affairs (VA) share an APG to end veterans homelessness. HUD officials described regular coordination between the two agencies, in conjunction with Interagency Council officials, to monitor progress towards the goal.

DHS Included Officials from Various Contributing Offices to Provide Differing Perspectives on Its Goal to Strengthen and Effectively Administer the Immigration System

In its strategic review of DHS strategic goal 3.1, “Strengthen and Effectively Administer the Immigration System,” USCIS involved two organizations that understand and promote the appropriate level of attention to the rights and views of USCIS customers—DHS’s Office for Civil Rights and Civil Liberties (CRCL) and the Office of the Citizenship and Immigration Services Ombudsman (CISOMB). While both offices are within DHS, organizationally they are located outside of USCIS. CRCL supports the department’s mission to secure the nation while preserving individual liberty, fairness, and equality under the law. CISOMB, which was created by Congress in 2002, assists industry and other employers with the services and benefits provided by USCIS. CISOMB maintains neutrality and identifies issues where trends or policy could be corrected with USCIS by making formal recommendations and providing an annual report to Congress.

According to USCIS officials, strategic review participants from CISOMB and CRCL were able to offer perspectives that reflected the views of those who receive services and benefits provided by USCIS. The presence of a CRCL representative helped to ensure that concerns related to civil rights and civil liberties were given proper consideration when discussing the administration of citizenship and immigration benefits, according to those involved in the review. For example, the CRCL representative shared that while reaching certain output or outcome goals is important, it is also critical to clearly communicate the various means through which USCIS customers can contest, appeal, or seek reconsideration of certain adverse determinations involving DHS employees or programs, or to correct outdated or otherwise incorrect information that could impact determinations. According to USCIS officials, the CRCL representative’s comment led them to evaluate, during the strategic review, whether the agency was clearly communicating the various avenues for customers to seek redress. They subsequently determined that it was.

Overall, the USCIS officials involved in the strategic review told us that the presence of CISOMB representatives helped ensure the review accurately portrayed the views and experiences of customers and employers that interacted with and received benefits from USCIS. A representative from CISOMB told us that because of their institutional knowledge regarding the impact of USCIS activities, CISOMB officials involved in the strategic review were able to ask informed questions about the evidence presented during the strategic review. In one instance, CISOMB representatives encouraged USCIS participants to broaden their assessment beyond quantitative output data to identify the impact of the agency’s public engagement efforts. USCIS officials said this was valuable input from the CISOMB representatives, and refocused the review to also look at the quality and end results of USCIS’s services to its customers.

EPA’s Office of Solid Waste and Emergency Response Collaborated with the Agency for Toxic Substances and Disease Registry to Improve Measures of Superfund Program’s Health Impacts

Officials in EPA’s Office of Solid Waste and Emergency Response (OSWER) told us that, concurrent with the strategic review of an objective related to the cleanup and reuse of contaminated sites, they launched a working group with the Agency for Toxic Substances and Disease Registry (ATSDR) located within the Center for Disease Control and Prevention at the Department of Health and Human Services. This group was created to collaborate in better understanding methodology to assess human health at Superfund sites. OSWER is responsible for providing policy, guidance, and direction for EPA’s emergency response and waste programs, including the Superfund program. The Superfund program responds to abandoned and active hazardous waste sites and accidental chemical releases. ATSDR is responsible for performing specific functions concerning the effect on public health of hazardous substances in the environment, such as public health assessments of waste sites and health consultations concerning specific hazardous substances.

The objectives of the working group were to develop measures to estimate the number of people exposed to or potentially exposed to contaminants at Superfund sites, as well as the number of people who are now protected as a result of actions taken by OSWER and ATSDR. From this collaboration, the working group made recommendations to improve the methodology for determining measures to assess health impacts and OSWER’s clean-up efforts at its clean-up sites. While the collaborative effort was not completed in time to be incorporated into the fiscal year 2014 strategic review findings, OSWER officials told us the project had stronger internal support because the type of evidence the working group was seeking to develop could help with reviewing progress on the strategic objective. Going forward, OSWER officials told us that the results of the EPA/ATSDR working group could help subsequent strategic reviews by producing better evidence of the Superfund program’s effectiveness in achieving the “Promote Sustainable and Livable Communities” objective.

Identify and Assess Evidence Related to Strategic Objective Achievement

Given the long-term and complex nature of many outcomes, the strategic review should be informed by a variety of evidence regarding the implementation of strategies and their effectiveness in achieving the outcome. OMB’s guidance states that the strategic review process should consider multiple perspectives and sources of evidence to understand the progress made on each strategic objective. This should include progress made by the agency towards the performance goals and measures related to the strategic objective as well as program evaluations, research studies, data, and policy analysis relevant to the objective or its related programs.evidence, studies conducted by external entities, such as academics, think tanks, nonprofits, associations, and oversight entities (such as ourselves or Inspectors General), may prove useful to the review.

While performance measurement and program evaluations can serve as key evidence for assessing progress, our past work has identified issues with agencies’ capacities to develop and use these types of evidence in decision making.

Performance measurement is the ongoing monitoring and reporting of program accomplishments, particularly progress toward preestablished goals. Because of its ongoing nature, performance measurement can serve as an early warning system to management and as a vehicle for improving accountability to the public. Although our work on federal performance measurement during the past 2 decades has found an increase in the reported presence of different types of performance measures across the government, it has not resulted in similar increases in the reported use of performance information in decision making. Moreover, in June 2013, we found that agencies continue to face common, long-standing difficulties in measuring the performance of various types of federal programs and activities—contracts, direct services, grants, regulations, research and development, and tax expenditures. We recommended that the Director of OMB work with the PIC to develop a detailed approach to examine these difficulties across agencies, including identifying and sharing any promising practices from agencies that have overcome difficulties in measuring the performance of these program types. OMB staff agreed with this recommendation. As of April 2015, OMB and the PIC have taken some initial steps to address this recommendation in a few areas, such as acquisition management (contracts). In addition, according to information provided by OMB staff, the PIC formed a working group on performance measurement that, in part, is focusing on how to develop appropriate performance measures. However, OMB has not yet developed a comprehensive and detailed approach to address these issues as envisioned in our report. We will continue to monitor progress on implementation of this recommendation.

Program evaluations are individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working. A program evaluation’s typically more in-depth examination of program performance and context allows for an overall assessment of whether the program works and identification of adjustments that may improve its results. However, as reported in June 2013 based on results from a governmentwide survey, we found that most federal managers lacked recent evaluations of their programs.reported that an evaluation had been completed within the past 5 years of any program, operation, or project in which they were involved. Another 40 percent of managers reported that they did not know if an evaluation had been completed. However, 80 percent of managers who did have evaluations reported that those evaluations contributed to a moderate or greater extent to improving program management or performance, and to assessing program effectiveness or value.

Our past work has found that the capacity to collect and analyze useful evidence is critical to successful reviews. To be useful to various decision makers, evidence must be accessible, accurate, complete, credible, consistent, relevant, timely, and valid. In addition, having the capacity to disaggregate data according to demographic, geographic, or other relevant characteristics can aid in highlighting significant variation, which can help meeting participants to pinpoint problems and identify solutions. Agencies also need to plan for the time and resources required to generate and communicate performance data and other evidence in a timely manner. Easy access to relevant databases and systems- generated analysis, such as providing analysts with the ability to develop performance reports without relying on information technology staff, can streamline the data collection and analysis processes.

HUD’s Office of Policy Development and Research Supported Strategic Reviews by Identifying Relevant Research and Evaluation

HUD’s Office of Policy Development and Research (PD&R) is responsible for maintaining current information on housing needs, market conditions, and existing programs, as well as conducting research on priority housing and community development issues. According to HUD performance staff, PD&R supported the review of each strategic objective by providing a template containing the most relevant research and evaluations related to each objective. This included both HUD-funded and external evidence. In addition, HUD’s performance staff asked objective leaders to supplement the evidence provided by PD&R with any additional evidence they thought would inform the review. Figure 6 provides examples of the research and evaluations that informed the department’s review of its objective to end homelessness for veterans, people experiencing chronic homelessness, families, youth, and children.

HUD performance staff told us that when a strategy has a clear outcome measure tied to departmental funding and support, identifying or developing relevant research was a lower priority because the existing measures provided an understanding of progress towards a goal or objective. For example, they told us that HUD has outcome information for its rental housing programs, in terms of individuals who are subsequently housed. HUD performance staff told us they were more concerned about developing new performance measures and identifying relevant research to inform policy changes where existing strategies lacked clear measures. For instance, HUD does not have broader outcome information on how all of its rental housing programs help individuals become more self-sufficient in terms of obtaining further education or employment.

USCIS Officials Compiled and Categorized Key Performance Information and Evidence to Inform Its Review

Prior to kicking off the review for DHS’s goal to strengthen and administer the immigration system, USCIS performance staff compiled relevant evidence—including agency performance data, program evaluations, and relevant reports by us and the DHS Inspector General—into a database to allow strategic review team members to focus on analyzing the evidence and determining progress in achieving the goal. In compiling the database, performance staff summarized key findings from the evidence and provided potential users with the source of the evidence so they could obtain additional context, if necessary. Further, they categorized the evidence to allow for easy sorting by users. For example, the evidence could be sorted by the sub-goal to which it was related; DHS’s four assessment areas it supported; whether it represented an accomplishment, planned activity, challenge/recommendation, a study, or other information; and key contributing organization within USCIS. Figure 7 provides an excerpt from this database, illustrating how agency performance data on processing applications was categorized.

For its objective to promote sustainable and livable communities, OSWER officials developed what they called a “ladder of evidence”—a framework and inventory of relevant performance information, scientific studies, academic research, and program evaluations, which they then assessed and categorized by strength. Officials said the different levels (types) of evidence allowed them to better assess and communicate the results of OSWER’s programs.

The first level of evidence provides descriptive data, covering information about what OSWER does, whom it serves and why, and performance trends over time. For example, one performance measure at this level is the number of Superfund sites with human exposure to contamination under control.

The second level of evidence identifies a relationship between OSWER’s activities and its outcomes. It provides evidence about the effectiveness of program implementation which can help identify promising practices or problematic areas for further study.

The third level of evidence establishes a causal link between OSWER’s programs and the impact they are having on human health and environmental outcomes.

Figure 8 provides additional information about each level of evidence along with illustrative examples.

Assess Effectiveness in Achieving Strategic Objectives and Identify Actions Needed to Improve Implementation and Impact

Using relevant evidence, strategic review participants should assess whether strategies are being implemented as planned and whether they are having the desired effect, as well as whether other factors are influencing results. The review may highlight areas where action is needed to improve or enhance implementation and impact. The following questions, based broadly on practices from OMB’s guidance and our past work on performance management, could help participants focus and facilitate this assessment and determine any needed actions.

If progress is lagging, why and what actions (strategy changes, revised management practices, legislative or budgetary proposals, etc.) could lead to better results?

Are there any potential gaps in strategy? Conversely, is there any unnecessary overlap and duplication? Addressing such issues could lead to improvements in effectiveness and efficiency.

Where progress is sufficient or exceeding expectations, are there strategies or practices that could be replicated and/or scaled to further enhance effectiveness?

Have there been recent changes in the agency’s operating environment that need to be addressed?

Are there strengths/opportunities on which to capitalize?

Are there weaknesses/threats that need to be overcome?

If the review identified evidence gaps, what steps will the agency take to develop sufficient evidence?

In addition, OMB’s guidance suggests additional actions that agencies should consider, which could lead to enhanced performance. These include benchmarking information from others trying to accomplish the same or similar objectives or using the same or similar key process, and identifying lessons learned from past efforts to continuously improve service delivery and resolve management challenges.

Education Modified an Existing Program and Created a New One to Help Address Challenges Related to Turning around Schools and Closing Achievement Gaps

Education officials told us that, in addition to the Department’s policy development and spending plan review, they used the strategic review process to assess how recent changes to its School Improvement Grant (SIG) program contributed to progress in one of the department’s strategic objectives. Education’s SIG program is designed to fund significant reforms in low-performing schools in support of the department’s objective to “accelerate achievement by supporting states and districts in turning around low-performing schools and closing achievement gaps, and developing models of next-generation high schools.” According to Education’s Fiscal Year 2014 Annual Performance Report and Fiscal Year 2016 Performance Plan, turning around the lowest-performing schools takes several years to show progress and success. Education reported that since 2009, more than 1,700 schools have received up to $2 million for 3 years through the SIG program to implement intervention models intended to turn around the lowest- performing schools. While nearly two-thirds of the schools have made progress, the remaining schools have either not shown progress or had decreased performance.

Through their ongoing SIG program monitoring, Education officials told us they learned about two challenges grantees reported facing that could be hindering progress and developed new strategies intended to address them. First, officials at state and local educational agencies expressed concerns to Education officials about sustaining turnaround efforts, since they are long term in nature and SIG program funds were only available for 3 years.approaches to better support sustainability. Using waiver authority, the department gave grantees flexibility to extend their use of existing funding into a fourth year. In addition, beginning with its fiscal year 2014 appropriations, Education obtained additional authority for state educational authorities to make school improvement grants for up to 5 Education officials told us they took two different years. Education officials told us grantees also expressed concerns about a lack of principals with knowledge about or experience in turning around schools. Recognizing the importance of sustained leadership commitment, the department launched a new grant program in 2014, the Turnaround School Leaders program. This program provides funding for 3 years to local educational agencies to help ensure that leaders at schools eligible for or receiving SIG program funds possess the specialized skills needed to drive successful efforts to turn those schools around.

DHS Is Developing New Performance Measures to Address Gaps in Data Identified during Strategic Review

Although CBP, TSA, ICE, and Coast Guard officials determined during their review that sufficient progress was being made on DHS’s Goal 2.2 to Safeguard and Expedite Lawful Trade and Travel, they also identified gaps to address in performance monitoring. Officials told us that they realized that while they tracked a number of performance measures related to aspects of trade, they had none regarding the travel portion of the objective (see table 3 below). Further, while most of the existing measures addressed enforcement and security, they noted that they had few measures that addressed the facilitation aspects of their mission— reducing barriers to the efficient flow of trade and travel.

According to one of the CBP officials who coordinated the review for this goal from the Office of Planning, Program Analysis, and Evaluation (PPAE), CBP is responsible for most of the activities that would be covered by the gaps in performance information. He told us that CBP has been working to address these weaknesses since they were identified last year. For example, he shared that PPAE has been working with the Trusted Traveler Division within the Office of Field Operations (OFO) to develop travel-specific performance measures. These measures would address the land border and air travel modes, the principal avenues by which most international travelers enter the country. One or more of the travel measures developed is to address the facilitation aspect of CBP’s mission, as expressed in Goal 2.2. In addition, he told us that PPAE is working with the Cargo and Conveyance Security Directorate within OFO to develop a trade facilitation measure. CBP expects to complete the formulation of these measures during calendar year 2015, and plans to subsequently submit them to DHS as formal performance measures to begin reporting in the second quarter of fiscal year 2016.

Develop a Process to Monitor Progress on Needed Actions

Much like we found for data-driven reviews, thorough and sustained follow-up on issues identified during strategic reviews is critical to the success of the reviews as a performance improvement tool. To ensure that actions identified as a result of the strategic review are carried out in the period between reviews, the agency should have a process to track these actions and communicate the progress made towards them. Such a process should identify, among other things for each action item, the responsible party, target completion dates, and significant milestones. In addition, agency leadership should hold responsible officials accountable for taking the agreed upon actions and communicating what has been done routinely. For example, agencies could use their existing quarterly performance review processes to monitor progress on strategic review action items, in line with the emphasis in OMB’s guidance for using existing agency management processes for strategic reviews.

OMB’s guidance further reinforces this practice by stating that agencies must incorporate actions to maintain or improve progress toward each objective, along with related implementation activities, into their next annual performance plan or other operating plans. For the fiscal year 2016 annual performance plan, this is to include, at a minimum, the agency’s summary of plans to improve or maintain performance, key milestones planned for the next year with completion dates, and efforts to close evidence gaps, as appropriate.

HUD Is Using Its Existing Performance Review Process to Track Progress on Action Items Stemming from Strategic Reviews

HUD’s performance reviews for its agency priority goals, known as HUDStat meetings, occur frequently and regularly (quarterly). To conduct its strategic reviews, HUD broadened the focus of its HUDStat meetings in one quarter to review progress toward its strategic objectives. For both sets of meetings, HUD’s performance staff have developed a process for identifying and tracking action items stemming from the reviews.

According to HUD performance staff, action items can be identified in a number of ways, including by the Secretary or PIO during reviews of materials prior to the HUDStat meeting, by meeting participants during the HUDStat session, or in a postmeeting session among the Secretary, PIO, and objective leads. HUD’s performance staff then compile and share a list of action items by objective or goal to all participants via e- mail within a day of the HUDStat meeting to ensure agreement. These are then added to a central tracking database for all action items. For each action item, the tracking database identifies the responsible party, a target completion date, any interim dates (milestones), and a status update. For example, following the 2014 strategic review for HUD’s objective to “end homelessness for veterans, people experiencing chronic homelessness, families, youth, and children,” one action item identified during the review was to establish targets for homeless family admissions to public housing, tenant-based vouchers, and project-based vouchers. It identifies the Office of Public and Indian Housing and the Office of Multifamily Housing as the responsible parties. According to HUD officials, as of April 2015, the Office of Public and Indian Housing is working to understand the capacity of local partners and will subsequently set targets. The Office of Multifamily Housing began collecting homeless admissions data in late 2014 and requiring it in February 2015. However, it is at least a year off from establishing and validating a baseline, and subsequently setting a target. HUD performance staff told us they will use the department’s 2015 strategic reviews to reinforce accountability for setting these targets.

HUD’s performance staff told us they work with responsible parties to update the status of each action item and provide a report to the Deputy Secretary regularly. According to HUD performance staff, following the 2014 strategic review, these updates occurred either biweekly or monthly, and for the 2015 strategic review they will occur biweekly.

USDA Instituted Quarterly Updates to Track Progress on Action Items Stemming from Strategic Reviews

USDA uses quarterly updates to the SOAR quad charts to keep the Secretary and other senior leaders informed of ongoing progress towards the objectives, as well as any related challenges. This includes providing updated information on the status of actions that were identified in prior quarters. For example, the Food Safety and Inspection Service (FSIS), which is responsible for ensuring that the nation’s commercial supply of meat, poultry, and egg products is safe, wholesome, and correctly labeled and packaged, is the lead agency for USDA’s strategic objective to “protect public health to ensure food is safe.” As part of the initial SOAR quad chart, from the second quarter of 2014, one of the next steps FSIS identified for this objective was to ensure continued progress in controlling Salmonella by developing new performance standards targeting chicken parts and ground poultry, and improving the agency’s verification sampling plans. According to USDA, Salmonella is the leading known cause of bacterial foodborne illness and death in the country, causing an estimated 1.3 million illnesses, and between 400 and 500 deaths annually. As part of its SOAR quad chart update for the fourth quarter of 2014, FSIS noted that it had developed a workplan for the Federal Register to announce and seek public comment on draft performance standards for Salmonella in chicken parts and ground chicken as part of the progress update. However, FSIS also noted in the significant challenges section that the draft rule was deemed “significant” by OMB,and FSIS was also responding to internal comments prior to moving forward with publication.

Agency Comments

We provided a draft of the report to the Director of the Office of Management and Budget, the Secretary of the Department of Agriculture, the Secretary of the Department of Education, the Secretary of the Department of Homeland Security, the Secretary of the Department of Housing and Urban Development, the Administrator of the National Aeronautics and Space Administration, and the Administrator of the Environmental Protection Agency for comment.

OMB staff and officials from the six agencies generally agreed with the findings presented in this report. In addition, DHS, Education, EPA, HUD, NASA, and OMB provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to interested congressional committees, the Director of the Office of Management and Budget, the Secretary of the Department of Agriculture, the Secretary of the Department of Education, the Secretary of the Department of Homeland Security, the Secretary of the Department of Housing and Urban Development, the Administrator of the National Aeronautics and Space Administration, the Administrator of the Environmental Protection Agency, and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix II.

Appendix I: Objective, Scope, and Methodology

We are required to review implementation of the GPRA Modernization Act of 2010 (GPRAMA) at several critical junctures. This report is part of our response to that mandate. Our specific objective for this report was to identify and illustrate, through case agency examples, practices that facilitate effective strategic reviews by federal agencies.

To identify practices, we analyzed and synthesized information gathered from a literature review we conducted, which covered public administration and public policy journals, business administration journals, our body of work on performance management and program evaluation, and other sources on policies and practices that can facilitate or challenge the effectiveness of strategic reviews as a decision-making tool. We also conducted interviews with performance management and evaluation experts representing different levels of government (local, state, federal), sectors (e.g. public, non-profit, foundations), and nations, who had experience with implementing elements of strategic reviews or academic and/or consultative expertise in this area.and interviews experts based on the results of our literature review (i.e., the authors of relevant articles or books included in our review). Based on suggestions from those individuals, we expanded our list of experts and conducted a second round of interviews.

Using the information we obtained from our literature review and expert interviews, we developed a broad set of practices for conducting effective strategic reviews. We refined the practices through our audit work at selected agencies (see next paragraphs). We also compared our practices with legal requirements in GPRAMA, guidance from the Office of Management and Budget (OMB), and a guide for conducting strategic reviews developed by the Performance Improvement Council (PIC), and found them to be broadly consistent.

To help illustrate and refine our draft practices, we selected a non- generalizeable sample of agencies based on several criteria and analyses. We limited the initial population for selection to the 24 agencies covered by the Chief Financial Officers Act of 1990 (CFO Act), as amended, because GPRAMA directs us to periodically evaluate how implementation of the act is affecting performance management at those agencies. We further refined the list to exclude two agencies, the Departments of Defense (DOD) and Veterans Affairs (VA), from selection. We excluded DOD because the department had not published strategic objectives related to its 2014 strategic goals at the time of our selection process. We excluded VA because of ongoing corrective actions it was taking to address significant shortcomings in the accuracy and reliability of certain performance information.

Because agencies conducted their initial strategic reviews in 2014 as we were selecting our sample, we could not use information about agencies’ strategic review processes to inform selection. As a proxy, we used relevant agency-level results on selected items from our 2013 survey of federal managers on performance and management issues to approximate if agencies had robust review processes and selected agencies with varying levels of robustness. These survey items covered the extent to which agency leadership was committed and involved in performance management activities, as well as the use of performance information.

We also considered the extent to which agency strategic review processes had a greater chance of addressing areas of fragmentation, overlap, and duplication, and high-risk issues identified in our past work. We have previously reported that effective implementation of strategic reviews could help identify opportunities to reduce, eliminate, or better manage instances of fragmentation, overlap, and duplication because, as part of the reviews, agencies are to identify the various organizations, programs, regulations, tax expenditures, policies, and other activities that contribute to each objective both within and outside the agency.addition, because agencies are to identify goals and strategies to resolve major management challenges they face, strategic reviews could also identify opportunities to better address issues on our High Risk List.

We also took into consideration agency size, based on the number full- time equivalent employees, given the potential for variation in review practices due to organizational size and capacity. Based on the criteria and analyses outlined above, we selected the Departments of Agriculture, Education, Homeland Security, and Housing and Urban Development, and the Environmental Protection Agency and National Aeronautics and Space Administration. These selections were also in line with suggestions we independently obtained from staff in OMB’s Office of Performance and Personnel Management who had reviewed each of the agencies’ plans for conducting their strategic reviews as well as the results of those reviews.

To identify illustrative examples for each of our practices from the six selected agencies and to further refine our practices, we reviewed documentation about agencies’ strategic review processes and results, including guidance, meeting agendas, relevant evidence used to inform the review, and internal and published summaries of the results. We also conducted interviews with officials involved in conducting strategic reviews at the six selected agencies—which included agency Performance Improvement Officers and their staff, strategic objective leaders, and strategic review participants—and staff from OMB and the PIC.

We conducted this performance audit from August 2013 to July 2015 in accordance with generally accepted government auditing standards.Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the above contact, Elizabeth Curda (Acting Director) and Benjamin T. Licht (Assistant Director) supervised this review and the development of the resulting report. Crystal Bernard, Virginia Chanley, Jehan Chase, Carole J. Cimitile, Emily Gruenwald, and Katherine Wulff made significant contributions to this report. Robert Robinson developed the graphics for this report. Sandra Beattie, Ellen Grady, Adam Miles, Jason Vassilicos, and Dan Webb verified the information contained in this report.