Managing for Results: Practices for Effective Agency Strategic Reviews
Published: 2015-07-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The GPRA Modernization Act of 2010 (GPRAMA) provides important tools that can help inform federal decision making. In implementing GPRAMA, the Office of Management and Budget (OMB) established a strategic review process in which agencies, beginning in 2014, were to annually assess their progress in achieving each strategic objective—the outcome the agency is intending to achieve—in their strategic plans.
GPRAMA requires GAO to periodically review its implementation. This report identifies and illustrates practices that facilitate effective strategic reviews.
To identify such practices, GAO analyzed and synthesized information from a variety of sources, including GPRAMA's requirements; OMB guidance; a review of relevant literature; and interviews with experts in performance management and evaluation and OMB staff. To refine and illustrate the practices, GAO reviewed strategic review documentation and interviewed relevant officials from six selected agencies: USDA, Education, DHS, HUD, EPA, and NASA. GAO selected these agencies based on several factors. This included the extent to which agency strategic review processes had a greater chance of addressing areas identified in GAO's work on fragmentation, overlap, and duplication or high-risk issues, and agency results on selected items in GAO's 2013 survey of federal managers on performance and management issues.
In commenting on a draft of this report, OMB and the six selected agencies generally agreed with the findings.
section_title: What GAO Found
paragraphs: GAO identified seven practices federal agencies can employ to facilitate effective strategic reviews and illustrated aspects of those practices through examples from the strategic review processes conducted at the Departments of Agriculture (USDA), Education (Education), Homeland Security (DHS), and Housing and Urban Development (HUD), and the Environmental Protection Agency (EPA), and the National Aeronautics and Space Administration (NASA).
1. Establish a process for conducting strategic reviews. NASA developed a strategic review process that involved senior leaders in individual assessments and a rating of each strategic objective, a crosscutting review to identify themes and provide independent rating recommendations, and a briefing to the Chief Operating Officer to determine final ratings.
2. Clarify and clearly define measurable outcomes for each strategic objective. NASA officials defined what would constitute success in 10 years for each strategic objective and used underlying performance goals, indicators, and milestones to better plan for and understand near-term progress towards their long-term scientific outcomes.
3. Review the strategies and other factors that influence the outcomes and determine which are most important. USDA's Food and Nutrition Service developed a model showing how the output of its programs contribute to relevant near-term and long-term outcomes related to the department's objective to improve access to nutritious foods. The model also identifies external factors that could influence progress, such as food prices.
4. Identify and include key stakeholders in the review. Contributors from various agencies, levels of government, and sectors may be involved in achieving an outcome. While the six agencies involved internal stakeholders in their strategic reviews, GAO did not find instances of external stakeholder involvement. In some cases, agencies took steps to incorporate external perspectives, such as HUD leveraging its existing relationship with officials at the U.S. Interagency Council on Homelessness to better understand how other federal programs are contributing to progress towards its objective to end homelessness for target populations.
5. Identify and assess evidence related to strategic objective achievement. For EPA's objective to promote sustainable and livable communities, officials developed a framework and inventory of relevant performance information, scientific studies, academic research, and program evaluations, which they then assessed and categorized by strength.
6. Assess effectiveness in achieving strategic objectives and identify actions needed to improve implementation and impact. For DHS's goal to safeguard and expedite lawful trade and travel, officials determined that sufficient progress was being made, but identified gaps in monitoring efforts, such as a lack of performance measures related to travel. DHS officials are taking steps to develop measures to address the gaps.
7. Develop a process to monitor progress on needed actions. HUD broadened its existing process for tracking progress on actions items identified at its quarterly performance reviews to also cover those from strategic reviews. HUD staff update the status of each action item regularly—planned to be biweekly following the 2015 strategic reviews.

section_title: Letter
paragraphs: The federal government is one of the world’s largest and most diverse  entities, with about $3.5 trillion in outlays in fiscal year 2013, funding an  extensive array of programs and operations. It faces a number of  significant fiscal, management, and performance challenges in  responding to the diverse and increasingly complex issues it seeks to  address. Addressing these challenges will require actions on multiple  fronts. For example, program structures that are outmoded, fragmented,  overlapping, or duplicative and not up to the challenges of the times must  be reformed or restructured. In addition, weaknesses in management  capacity, both government-wide and in individual agencies, undermine  efficient and effective government. Moving forward, federal decision  makers will be confronted with making tough choices in setting priorities  as well as reforming programs and management practices to better link  resources to results.
In that regard, the performance planning and reporting framework  originally put into place by the Government Performance and Results Act  of 1993 (GPRA), and significantly enhanced by the GPRA Modernization  Act of 2010 (GPRAMA), provides important tools that can help inform  congressional and executive branch decision making to address  challenges the federal government faces. The Office of Management and  Budget’s (OMB) 2012 guidance implementing GPRAMA established a  strategic review process in which agencies, beginning in 2014, were to  conduct leadership-driven, annual reviews of their progress towards  achieving each strategic objective—the outcome or impact the agency is  intending to achieve through its various programs and initiatives— established in their strategic plans.
Effective implementation of strategic reviews could help identify  opportunities to reduce, eliminate, or better manage instances of  fragmentation, overlap, and duplication because agencies are to identify  the various organizations, program activities, regulations, tax  expenditures, policies, and other activities that contribute to each  objective, both within and outside the agency. Where progress in  achieving an objective is lagging, the reviews are intended to identify  strategies for improvement, such as strengthening collaboration to better  address crosscutting challenges, or using evidence to identify and  implement more effective program designs. If successfully implemented  in a way that is open, inclusive, and transparent—to Congress, delivery  partners, and a full range of stakeholders—this approach could help  decision makers assess the relative contributions of various programs to  a given objective. Successful strategic reviews could also help decision  makers identify and assess the interplay of public policy tools that are  being used to ensure that those tools are effective and mutually  reinforcing, and results are being efficiently achieved.
We are required to review implementation of GPRAMA at several critical  junctures.specific objective for this report was to identify and illustrate, through case  agency examples, practices that facilitate effective strategic reviews by  federal agencies. To identify and illustrate the practices, we analyzed and  synthesized information gathered from   This report is part of our response to that mandate. Our  related legal requirements in GPRAMA and OMB guidance for  implementing those requirements; a literature review we conducted, which covered public administration  and public policy journals, business administration journals, our body  of work on performance management and program evaluation, and  other sources on policies and practices that can facilitate or challenge  the effectiveness of strategic reviews as a decision-making tool; a guide for conducting strategic reviews developed by the  Performance Improvement Council (PIC); documentation from six selected agencies’ strategic review processes  and results, including guidance, meeting agendas, relevant evidence  used to inform the review, and internal and published summaries of  the results;  interviews we conducted with more than 30 performance  management and evaluation experts representing different levels of  government, sectors (e.g. public, non-profit, foundations), and nations,  who had experience with implementing elements of strategic reviews  or academic or consultative expertise in this area;interviews we conducted with officials involved in conducting strategic  reviews at six selected agencies and staff from OMB and the PIC.
We selected six agencies to illustrate the practices we developed—the  Departments of Agriculture (USDA), Education (Education), Homeland  Security (DHS), and Housing and Urban Development (HUD), and the  Environmental Protection Agency (EPA), and the National Aeronautics  and Space Administration (NASA). We selected these agencies based on  various criteria. This included the extent to which agency strategic review  processes had a greater chance of addressing areas of fragmentation,  overlap, and duplication, and high-risk issues identified in our past  work. We also considered agency results on selected leadership  involvement and performance information use items in our 2013 survey of  federal managers on performance and management issues. Our  selection was also informed by agency size, based on the number of full- time equivalent employees, and suggestions about agencies with robust  review processes from OMB staff with government-wide perspective on  agency strategic reviews. See appendix I for additional information about  the objective, scope, and methodology for this report.
We conducted this performance audit from August 2013 to July 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: GPRAMA requires OMB to annually determine whether agencies have  met the performance goals and objectives outlined in their performance  plans and submit a report on unmet goals. In implementing this  provision, OMB’s guidance directs agencies to continue reporting on  unmet performance goals in their annual performance reports, as has  been required since fiscal year 1999. In addition, OMB’s guidance  directs agencies to conduct leadership-driven, annual reviews of progress  towards each strategic objective—the outcome or impact the agency is  intending to achieve—established in the agency’s strategic plan. Figure  1, an illustrative example from OMB’s guidance, shows how strategic  objectives relate to other goals within an agency’s performance  management structure.
Agencies began conducting these reviews in fiscal year 2014. The results  from their first round of reviews were published in their annual  performance reports in February 2015, as well as on Performance.gov,  the central governmentwide performance reporting website implemented  by OMB to meet GPRAMA requirements. OMB’s guidance directs  agencies to provide a progress update for each strategic objective,  including a brief summary of what progress was made and an explanation  of the achievements made or challenges that have impeded progress.
As part of their reporting, agencies were to identify a portion of their  objectives as (1) having demonstrated noteworthy progress and (2) focus  areas for improvement.
According to OMB’s guidance, the results of these reviews should (1)  inform long-term strategy; (2) inform annual planning and budget  formulation; (3) facilitate identification and adoption of opportunities for  improvement, including risk management; (4) identify areas where  additional program evaluation, other studies, or analyses of performance  data are needed to determine effectiveness or set priorities; (5) identify  where additional skills or other capacity are needed; (6) improve decision- making response time; (7) strengthen collaboration on crosscutting  issues; and (8) improve transparency.
The PIC also provided support to agencies as they began planning for  and implementing their strategic reviews. According to OMB and PIC  staff, through the PIC’s Internal Reviews Working Group, agency officials  shared information about their planned strategic review processes as well  as lessons learned from the initial round of reviews. The PIC also hosted  several summits focused on strategic reviews and published a guide in  August 2014 on leading effective strategic reviews, based on agencies’  initial experience.
Moving forward, OMB staff told us that they expect agencies’ strategic  review processes will mature over time, and as such expect the results of  those reviews to mature over time as well. According to OMB staff, they  used the information conveyed in figure 2 to communicate to agencies  that they likely would not be able to fulfill all requirements in OMB’s  guidance in initial implementation, but instead should develop a maturity  model to ensure they continue to strengthen the reviews over time.
section_title: Practices for Facilitating Effective Agency Strategic Reviews
subsections: section_title: Establish a Process for Conducting the Agency’s Strategic Reviews
paragraphs: Sufficient planning and preparation is important to ensure that the  agency’s strategic review process is successful. Our February 2013  report on data-driven performance reviews found this was critical to a  successful review. Planning enhances the quality, credibility, and  usefulness of the review, and helps ensure that participants’ time and  resources are used effectively. Establishing common purposes for  strategic review meetings can build trust and encourage active  participation by participants. In addition, developing common terminology,  policies, and procedures, and clarifying roles and responsibilities helps  facilitate collaboration for productive meetings. Participants need to be  prepared to review progress towards their strategic objectives and  determine any subsequent actions.
Key features for planning the strategic review include:
Leadership commitment and involvement. Agency leadership  should be directly and visibly engaged in the review process and  invest the time necessary to understand and interpret the evidence  being presented. This involvement fosters ownership among those  involved in the review and helps ensure that participants take the  reviews seriously and can make decisions and commitments with the  knowledge and backing of leadership.
Communication of expectations and time frames. Guidance and  agendas provided in advance of review meetings can establish a  common understanding of the purpose of the review, the process to  be used, and time frames for completing the review. In addition,  standardized templates used to collect and share key information are  helpful to facilitate strategic review discussions and help to ensure  consistency across reviews.
Accountability for results. The focus of accountability should be on  the responsible objective leader’s role in credibly assessing progress  in achieving a strategic objective using evidence. Agency leaders  should hold objective leaders and other responsible managers  accountable for knowing the progress being made in achieving  outcomes and, if progress is insufficient, understanding why and  having a plan for improvement. If evidence is insufficient for assessing  progress, managers should be held accountable for improving the  availability and quality of the evidence so that it can be used  effectively for decision making. Managers should also be held  accountable for identifying and replicating effective practices to  improve performance.
In addition, OMB’s guidance strongly encourages agencies to leverage  existing decision-making processes to conduct strategic reviews.  According to the guidance, in most cases, the strategic reviews should be  integrated into existing agency management processes to raise key  decisions, issues, and analysis to agency leadership. OMB’s guidance  also provides agencies flexibility in developing their processes, stating  that agencies should use a tailored approach that is appropriate for the  nature of the agency’s programs, operations, and strategic objectives and  evidence available.
subsections: section_title: NASA Involved Senior Leaders across the Agency to Help Ensure Commitment to Strategic Review Process and Accountability for Results
paragraphs: In developing the agency’s strategic review process in late 2013, NASA’s  Performance Improvement Officer (PIO) at the time and her staff sought  input on the process from NASA senior leaders. This group included the  leaders for each of NASA’s strategic objectives who typically represent  the most senior official with direct oversight of the programs and activities  supporting each objective, such as division directors and deputy  associate administrators, among other senior positions. According to PIO  staff, all of NASA’s guiding principles for the strategic review process  were informed by senior leadership, such as using existing management  processes and structures, promoting transparency, and making the  process intuitive and easy to understand. NASA PIO staff told us that this  helped create buy-in and understanding for the strategic review process.
Each strategic objective leader, along with deputy objective leaders and  relevant NASA staff, was involved in conducting individual assessments  of each objective and provided a suggested rating. For example, the  Director of the Heliophysics Division was the strategic objective leader for  the strategic review of the objective “Understand the Sun and its  interactions with the Earth and the solar system, including space  weather.” NASA’s PIO and her staff then led crosscutting reviews of these  individual assessments to identify themes and provide an independent  rating recommendation. Following the crosscutting review, NASA’s Chief  Operating Officer (COO) determined final ratings during a briefing  attended by the PIO and each of the strategic objective leaders. At that  meeting, a member of NASA’s PIO staff summarized review findings and  results to the COO. The COO then asked each strategic objective leader  clarifying questions and sought suggestions that would lead to  performance improvements before settling on the final rating. According  to NASA PIO staff, this approach of having all strategic objective leaders  (and relevant program staff) attend the entire briefing encouraged  transparency, and the personal involvement of the COO encouraged  accountability for results and performance improvements.
section_title: DHS Communicated Key Aspects of Its Strategic Review Process at a Kickoff Meeting Attended by Process Participants
paragraphs: DHS’s Office of the Chief Financial Officer/Office of Program Analysis and  Evaluation (CFO/PA&E) leads departmental implementation of  performance management activities, including strategic reviews. In  addition, each component agency has a designated PIO and performance  staff who coordinate efforts in their component agency as part of  department-wide performance management activities. For example, for  U.S. Citizenship and Immigration Services (USCIS), this role is performed  by the Office of the USCIS CFO. In early January 2014, CFO/PA&E met  with component PIOs to provide a basis for understanding and  participating in the department’s first strategic reviews. Recognizing the  important role that they played in the initial reviews, CFO/PA&E revised  its orientation process for the 2015 strategic reviews to include a separate  briefing for assessment leads—the senior executives who lead teams  reviewing progress towards each strategic goal.
The DHS briefing slides informed participants about the related GPRAMA  requirement and OMB’s guidance, as well as the purpose and expected  benefits of the department’s strategic reviews, such as informing the next  DHS strategic plan, strengthening collaboration, and informing program  and budget reviews. The briefing provided an overview of the  department’s strategic review process, describing a structured  methodology for conducting the reviews and samples of four standard  deliverables (templates) to be used to collect information from each  assessment team. It also identified the roles and responsibilities for  various participants in the process, including assessment leads and  teams conducting the review of each goal, the component and DHS PIOs,  and CFO/PA&E staff. The briefing also provided a timeline for  implementing the department’s strategic reviews, with specific dates for  key activities to be completed.
section_title: USDA Used an Existing Performance Review Process and Template to Facilitate Strategic Reviews
paragraphs: Among other responsibilities, USDA’s Office of Budget Policy and  Analysis (OBPA) oversees implementation of the department’s  performance management activities. According to the Associate Director  of OBPA, who also serves as USDA’s PIO, his office and relevant  component agencies provide regular performance updates to the  Secretary on key initiatives, such as the Blueprint for Stronger Service, an  effort launched in 2012 to enhance administrative services and  management operations. For these updates, which primarily occur  monthly, depending on the initiative, USDA uses a standard template,  known as a “quad chart,” to collect and present information to the  Secretary for decision making. Because of the Secretary’s familiarity with  the quad chart format, the department adapted the chart for use in its  strategic review process, known as the strategic objective annual review  (SOAR) (see figure 3).
As illustrated in figure 3, the SOAR quad chart includes the following  information:  the relevant agency or office within USDA responsible for the  objective and the officials leading the efforts, known as objective  owners and lieutenants;  the strategic objective and the strategic goal it supports; a summary of progress towards the objective and related  achievements;  key performance indicators along with actual performance results  compared to targets; a discussion of challenges that could affect program outcomes; and a description of next steps, crosscutting analysis, or evaluations to  improve objective performance.
Objective owners and lieutenants are responsible for populating the  information in the quad charts. Subsequently, OBPA reviews the quad  charts before they go to the Secretary to ensure consistency in  information reported and progress assessments, identify any needed  changes, and determine if the information provided could impact other  initiatives across the department. According to OBPA officials, the quad  charts provide USDA leadership with succinct and sufficient information to  make decisions to improve performance, such as approving new or  modifying existing strategies, or adjusting time frames.
section_title: Clarify and Clearly Define Measurable Outcomes for Each Strategic Objective to Be Reviewed
paragraphs: A strategic review starts with framing the outcome or impact the agency  seeks to achieve. According to OMB guidance, strategic objectives  should be relatively simple statements that break down the broader,  mission-oriented strategic goals to a level that reflects the impact or  outcome the agency is trying to achieve through its programs. Objectives  should be framed so they can serve as standards against which an  assessment can reasonably be performed to determine the effectiveness  of the agency’s implementation of its programs, as well as progress  toward the ultimate outcome.
In some cases, defining and measuring the outcome related to a strategic  objective may be relatively straightforward. For example, increasing  employment rates for participants who completed a training program is an  outcome defined in a way that can be measured. However, where  agencies are focused on more long-term or complex outcomes,  determining if the agency is making progress each year can be more  challenging. In these instances, the agency may need to break the  strategic objectives into pieces that can be more easily be measured or  assessed.
subsections: section_title: NASA Clarifies Its Long-term Scientific Research Outcomes through 10-Year Success Statements, Multiyear Performance Goals, Measures, and Milestones
paragraphs: As part of its performance framework, NASA has associated time frames  with its goals, as illustrated in figure 4.
For the agency’s planning process for its 2014 strategic plan and annual  strategic review, strategic objective leaders developed success  statements that covered up to a 10-year time frame for each of their  objectives. According to PIO staff, for the success statements, objective  leaders and staff were asked to characterize or define the outcomes of  success in implementing their objectives in the next 10 years by  answering questions such as, “What will the agency have completed,  obtained, contributed, advanced?” NASA officials told us that because it  can be difficult to measure progress towards long-term, scientific  discovery-oriented outcomes, they also rely on underlying multiyear  performance goals, annual performance indicators, and milestones to  better plan for and understand near-term progress towards those  objectives. Table 1 illustrates how NASA clarified long-term and near- term progress for its objective to Understand the Sun.
To frame DHS strategic goal 3.1, “Strengthen and Effectively Administer  the Immigration System,” in more concrete terms, the lead agency,  USCIS, focused on three sub-goals. Table 2 identifies the sub-goals and  describes them.
In addition to the sub-goals, USCIS also developed performance  measures (known as strategic measures) as part of its ongoing  performance monitoring efforts for this goal. For example, one measure is  the average processing cycle time (in months) for naturalization  applications. Taken together, the sub-goals and performance measures  show how DHS has identified measureable pieces of its efforts related to  the larger goal.
section_title: Identify the Strategies and Other Factors That Influence the Outcomes and Determine Which Are Most Important
paragraphs: It is critical to identify, at a conceptual level, the various strategies and  factors that can help or hinder achievement of the strategic objective. The  federal government uses numerous activities and policy implementation  tools, such as loans, grants, contracts, social and economic regulations,  insurance, and tax expenditures, among others (hereafter strategies) to  help address public problems. However, since 2011, our annual series  of reports examining federal programs has found that agencies often  employed overlapping or fragmented program strategies that were poorly  coordinated. In addition, because the federal government rarely works  in isolation, the efforts of other levels of governments (local, state, and  international) and sectors (private and nonprofit) frequently contribute to  the achievement of an outcome as well. Beyond these strategies and  efforts, factors both within and beyond the control of any particular  agency—generally referred to as internal and external factors—may  influence an outcome. Internally, these factors could include an agency’s  culture, management practices, and business processes. External factors  may include the economy, demographic trends, technological advances,  and the natural environment.
The strategic review for each objective should take into account the  comprehensive set of federal strategies, nonfederal efforts, and factors  within and outside an agency’s control related to the outcome. The more  complex the outcome, the more likely it is to be influenced by multiple  strategies, nonfederal efforts, and factors. Although these influences may  have been previously identified through an agency’s strategic planning  process or similar vehicle, they should be revisited as part of the strategic  review to determine if anything has changed. OMB’s guidance directs  agencies to identify in their strategic plans the various organizations and  policy tools, both within and external to the agency, that contribute to their  strategic objectives. However, our work reviewing GPRAMA  implementation has found weaknesses in agencies’ abilities to identify  contributors to their goals. For example, in our April 2013 report on  agency priority goals (APG), we found that agencies had not always  identified external organizations and policy tools that contributed to their  goals, although required by GPRAMA and OMB’s guidance. We  recommended that OMB ensure agencies adhere to its guidance by  providing complete information about the contributors to their APGs. OMB  staff agreed with this recommendation. According to information provided  by OMB staff in April 2015, agencies were asked to identify organizations,  program activities, regulations, policies, tax expenditures, and other  activities contributing to their 2014-2015 APGs, first as part of the  September 2014 update to Performance.gov, with opportunities for  revisions in subsequent quarterly updates. Our analysis found that  agencies have made progress in identifying external organizations and  programs for their APGs, but they did not present this information  consistently on Performance.gov. Although each APG webpage has a  location where agencies are to identify contributing programs, agencies  did not always identify external organizations and programs there.  Instead, they identified these external contributors elsewhere, such as  APG overview or strategy sections, which could limit the ability of users to  easily locate this information. We will continue to monitor progress on  implementation of this recommendation.
Using existing knowledge, expertise, and evidence, those involved in the  review should identify the strategies, nonfederal efforts, and factors that  are likely to have the strongest influence on the outcome. This information  will help to establish priorities for the scope of the review. There are a  number of methods that can be used to map or model the causal  relationships among the inputs, processes, and outputs produced by  various strategies and the forces that influence achievement of outcomes,  such as results mapping and logic modeling. These methods can help  to clarify the issues that must be addressed conceptually to create  change or achieve the intended outcome. By identifying and examining  the various influences on the strategic objective or expected outcome  during the strategic review, an agency can better understand how the  existing set of program outputs and activities are contributing to the  achievement of outcomes and whether gaps exist or changes are needed  in light of all the other factors that are influencing outcomes.
Recognizing that some of these influences may present risks or  challenges to achieving expected outcomes, OMB’s 2014 update to its  guidance (covering agency’s strategic reviews in 2015) states that while  agencies cannot mitigate all risks related to achieving strategic objectives  and performance goals, they should identify, measure, and assess   To that  challenges related to mission delivery, to the extent possible. end, the guidance encourages agencies to institute an Enterprise Risk  Management (ERM) approach, and leverage such efforts when  conducting strategic reviews. The guidance defines ERM as an effective  agency-wide approach for addressing the full spectrum of the  organization’s risks by understanding their combined impact as an  interrelated portfolio, rather than addressing risks within silos. The  guidance further states that with an ERM approach, agencies can be  better positioned to quickly gauge which risks are directly aligned to  strategic objectives, and which have the highest probability of impacting  the agency’s mission.opportunities and challenges are routinely identified, analyzed, and   Such an approach can help ensure that  addressed, as appropriate, enhancing the agency’s capacity to more  efficiently and effectively determine priorities and allocate resources.
subsections: section_title: For DHS’s Lawful Trade and Travel Goal, Officials from Four Component Agencies Identified Contributing Programs and Ranked Them by Degree of Influence
paragraphs: DHS’s review of its strategic goal 2.2 “Safeguard and Expedite Lawful  Trade and Travel” involved four component agencies: Customs and  Border Protection (CBP, the designated lead agency for the review), the  Transportation Security Administration (TSA), Immigration and Customs  Enforcement (ICE), and the U.S. Coast Guard (Coast Guard). According  to CBP officials, each of these component agencies plays a role in  implementing strategies supporting this goal. According to DHS’s  Strategic Plan for fiscal years 2014-2018, the strategies for this goal are  to (1) safeguard key nodes, conveyances, and pathways; (2) manage the  risk of people and goods in transit; and (3) maximize compliance with  U.S. trade laws and promote U.S. economic security and com- petitiveness.
The goal leader—CBP’s Executive Director for Planning, Program  Analysis, and Evaluation, within the Office of Field Operations, who also  led the assessment team—asked participating officials from the four  contributing agencies to identify which of their programs and activities  contributed to the achievement of the goal, and then subsequently to rank  them by level of influence. Table 3 provides illustrative examples of  programs and activities that support this goal from each of the four  contributing component agencies.
According to CBP officials who coordinated the review, participating  officials determined that a few of the programs and activities they initially  identified as contributing to the goal had relatively minor influence  towards the outcome. In these instances, the programs and activities  primarily supported another DHS goal. DHS officials decided to include  only those programs that primarily supported the goal under review. For  example, CBP officials determined that CBP’s Container Security  Initiative, which works with foreign governments to examine potentially  high-risk cargo prior to departure from the foreign port of origin, may have  had influence on safeguarding trade and travel, but more directly  supported another DHS goal, “Secure U.S. Air, Land, and Sea Borders  and Approaches.”
section_title: Food and Nutrition Service Officials Use a Logic Model to Identify Programs and Factors That Influence USDA’s Efforts to Improve Access to Nutritious Foods
paragraphs: USDA’s Food and Nutrition Service (FNS) seeks to increase food security  and reduce hunger by providing children and low-income people access  to food, a healthful diet, and nutrition education in a way that supports  American agriculture and inspires public confidence. FNS uses a logic  model (figure 5) to understand how its programs and other factors  influence outcomes related to USDA’s objective to “improve access to  nutritious food.” FNS first developed the logic model in the early 2000s as  part of an effort to better integrate performance measurement into its  operations. FNS officials told us that the concepts included in the logic  model are often used when the agency is making decisions about  performance measurement and evaluation because it shows the  connections among program inputs, outputs, and overall outcomes. By  making those linkages explicit, decision makers can have more focused  and meaningful discussions for how proposed strategies are tied to  desired results and how to measure the success of strategy execution  and impact, according to FNS officials. As part of the strategic review  process, FNS used its logic model to reaffirm the connections between  program outputs and related outcomes.
As illustrated in figure 5, the logic model shows how the output of FNS’s  programs (left column) contribute to relevant near-term and long-term  outcomes (the three columns to the right). The model covers five  contributing FNS programs: Child and Adult Care Food Program  (CACFP), Fresh Fruit and Vegetable Program (FFVP), National School  Lunch Program (NSLP), Supplemental Nutrition Assistance Program  (SNAP), and Special Supplemental Nutrition Program for Women, Infants,  and Children (WIC). At each level, the logic model identifies related  performance measures as well as external factors that could influence  progress, such as FNS and state implementing agency resource levels,  competing priorities and policies, and food price and availability.
section_title: Identify Key Stakeholders to Participate in the Review
paragraphs: Because the achievement of outcomes may be complex and involve a  variety of contributors from within an agency, or include other federal  agencies, levels of government, and sectors, it is critical to consider which  key stakeholders should be involved in a strategic review. Each of these  stakeholders provides a unique perspective on their contribution or view  of progress of the outcome under review.
OMB’s guidance and our past work reinforce the importance of including  key stakeholders in the review. OMB’s guidance states that the analysis  of each objective should be conducted at the objective lead level, with  support from relevant bureaus and programs, and that the COO and PIO  office should be involved in analysis and decision making across all  objectives. Our prior report on effective practices for data-driven  performance reviews also indicated that performance review participants  should include high-level leaders and managers with an agency-wide  perspective, as well as those with programmatic knowledge and  responsibility for the specific performance issues likely to be raised.  Each of the six agencies covered by our work for this report developed  strategic review processes that involved relevant internal stakeholders,  from contributing program officials to the agency head or COO.
Agencies should also consider including the perspectives of relevant  third-party policy experts, academics, professional associations, end  users/clients or advocacy groups that represent them in the review  process. When outcomes are complex and involve multiple organizations,  it is also important to establish how existing collaboration mechanisms  can facilitate joint data collection, analysis, and reporting, or if new  networks should be established. In some cases, there may be an existing  interagency group, such as a task force, that has been formed to achieve  an outcome. Our prior work has shown that agencies that participated in  various planning and decision-making forums together—such as  interagency councils or planning bodies—reported that such interactions  contributed to achieving their goals. Specifically, agencies reported that  such participation opened lines of communication, fostered trust, and  helped build relationships, which can in turn lead to more effective  collaboration across agency lines.
In spite of the compelling rationale for all parties contributing to an  outcome to collaborate, our past work on GPRAMA implementation has  found that agencies generally have not included external stakeholders  when reviewing progress on an outcome. In our report on implementation  of data-driven reviews in February 2013, we concluded that as the  implementation of various GPRAMA provisions continues, agencies may  need to reevaluate the most effective way to engage outside stakeholders  in the performance review processes for APGs and other performance  goals that depend on other organizations to achieve desired outcomes.
We recommended that OMB work with the PIC and other relevant groups  to identify and share promising practices to help agencies extend their  quarterly performance reviews to include, as relevant, representatives  from outside organizations that contribute to achieving their agency  performance goals. OMB staff generally agreed with this  recommendation. As of April 2015, OMB staff told us that agencies  continue to find that most APG reviews are appropriately focused on  internal agency management, rather than involving external stakeholders.  Therefore, OMB and the PIC have focused recent efforts on developing  and sharing promising practices related to conducting reviews internal to  the agencies or on improving evidence/measurement. We will continue to  monitor progress on implementation of this recommendation.
subsections: section_title: HUD Considered Input from Contributors Internal and External to the Department Regarding Efforts to End Homelessness
paragraphs: GAO-13-228.
Office of the Secretary, who serves as the objective leader. The review  involved officials from the Office of Special Needs Assistance Programs  within the Office of Community Planning and Development, which  administers the department’s homelessness programs, such as the  Continuum of Care Program, which funds local networks of organizations  to quickly rehouse individuals and minimize the trauma and dislocation  caused to individuals, families, and communities by homelessness. In  addition, the review included participants from other HUD offices, such as  the Office of Public and Indian Housing and the Office of Multifamily  Housing, whose programs can assist in ending homelessness. For  example, the Office of Public and Indian Housing’s Housing Choice  Voucher Program provides rental subsidies for low-income families, which  may include families experiencing homelessness. This crosscutting and  inclusive approach reinforced one of HUD’s strategies supporting this  objective—to fully engage and leverage mainstream housing assistance  to build capacity among public housing agencies and multifamily owners  to admit homeless households into their units.
Although no one outside of HUD directly participated in the review, HUD  officials stated that they leveraged their existing relationship with officials  at the U.S. Interagency Council on Homelessness (Interagency Council)  and the 18 other federal agencies that comprise it to better understand  how other federal programs are contributing to progress in ending   This included attending and  homelessness for the target populations.participating in various meetings, including quarterly meetings with the  Interagency Council principals, staff-level coordinating meetings, and  targeted working groups, such as bimonthly meetings of the chronic and  family homelessness working group. In addition, as part of this objective,  HUD and the Department of Veterans Affairs (VA) share an APG to end  veterans homelessness. HUD officials described regular coordination  between the two agencies, in conjunction with Interagency Council  officials, to monitor progress towards the goal.
section_title: DHS Included Officials from Various Contributing Offices to Provide Differing Perspectives on Its Goal to Strengthen and Effectively Administer the Immigration System
paragraphs: In its strategic review of DHS strategic goal 3.1, “Strengthen and  Effectively Administer the Immigration System,” USCIS involved two  organizations that understand and promote the appropriate level of  attention to the rights and views of USCIS customers—DHS’s Office for  Civil Rights and Civil Liberties (CRCL) and the Office of the Citizenship  and Immigration Services Ombudsman (CISOMB). While both offices are  within DHS, organizationally they are located outside of USCIS. CRCL  supports the department’s mission to secure the nation while preserving  individual liberty, fairness, and equality under the law. CISOMB, which  was created by Congress in 2002, assists industry and other employers  with the services and benefits provided by USCIS. CISOMB maintains  neutrality and identifies issues where trends or policy could be corrected  with USCIS by making formal recommendations and providing an annual  report to Congress.
According to USCIS officials, strategic review participants from CISOMB  and CRCL were able to offer perspectives that reflected the views of  those who receive services and benefits provided by USCIS. The  presence of a CRCL representative helped to ensure that concerns  related to civil rights and civil liberties were given proper consideration  when discussing the administration of citizenship and immigration  benefits, according to those involved in the review. For example, the  CRCL representative shared that while reaching certain output or  outcome goals is important, it is also critical to clearly communicate the  various means through which USCIS customers can contest, appeal, or  seek reconsideration of certain adverse determinations involving DHS  employees or programs, or to correct outdated or otherwise incorrect  information that could impact determinations. According to USCIS  officials, the CRCL representative’s comment led them to evaluate, during  the strategic review, whether the agency was clearly communicating the  various avenues for customers to seek redress. They subsequently  determined that it was.
Overall, the USCIS officials involved in the strategic review told us that  the presence of CISOMB representatives helped ensure the review  accurately portrayed the views and experiences of customers and  employers that interacted with and received benefits from USCIS. A  representative from CISOMB told us that because of their institutional  knowledge regarding the impact of USCIS activities, CISOMB officials  involved in the strategic review were able to ask informed questions about  the evidence presented during the strategic review. In one instance,  CISOMB representatives encouraged USCIS participants to broaden their  assessment beyond quantitative output data to identify the impact of the  agency’s public engagement efforts. USCIS officials said this was  valuable input from the CISOMB representatives, and refocused the  review to also look at the quality and end results of USCIS’s services to  its customers.
section_title: EPA’s Office of Solid Waste and Emergency Response Collaborated with the Agency for Toxic Substances and Disease Registry to Improve Measures of Superfund Program’s Health Impacts
paragraphs: Officials in EPA’s Office of Solid Waste and Emergency Response  (OSWER) told us that, concurrent with the strategic review of an objective  related to the cleanup and reuse of contaminated sites, they launched a  working group with the Agency for Toxic Substances and Disease  Registry (ATSDR) located within the Center for Disease Control and  Prevention at the Department of Health and Human Services. This group  was created to collaborate in better understanding methodology to assess  human health at Superfund sites. OSWER is responsible for providing  policy, guidance, and direction for EPA’s emergency response and waste  programs, including the Superfund program. The Superfund program  responds to abandoned and active hazardous waste sites and accidental  chemical releases. ATSDR is responsible for performing specific  functions concerning the effect on public health of hazardous substances  in the environment, such as public health assessments of waste sites and  health consultations concerning specific hazardous substances.
The objectives of the working group were to develop measures to  estimate the number of people exposed to or potentially exposed to  contaminants at Superfund sites, as well as the number of people who  are now protected as a result of actions taken by OSWER and ATSDR.  From this collaboration, the working group made recommendations to  improve the methodology for determining measures to assess health  impacts and OSWER’s clean-up efforts at its clean-up sites. While the  collaborative effort was not completed in time to be incorporated into the  fiscal year 2014 strategic review findings, OSWER officials told us the  project had stronger internal support because the type of evidence the  working group was seeking to develop could help with reviewing progress  on the strategic objective. Going forward, OSWER officials told us that the  results of the EPA/ATSDR working group could help subsequent strategic  reviews by producing better evidence of the Superfund program’s  effectiveness in achieving the “Promote Sustainable and Livable  Communities” objective.
section_title: Identify and Assess Evidence Related to Strategic Objective Achievement
paragraphs: Given the long-term and complex nature of many outcomes, the strategic  review should be informed by a variety of evidence regarding the  implementation of strategies and their effectiveness in achieving the  outcome. OMB’s guidance states that the strategic review process should  consider multiple perspectives and sources of evidence to understand the  progress made on each strategic objective. This should include  progress made by the agency towards the performance goals and  measures related to the strategic objective as well as program  evaluations, research studies, data, and policy analysis relevant to the  objective or its related programs.evidence, studies conducted by external entities, such as academics,  think tanks, nonprofits, associations, and oversight entities (such as  ourselves or Inspectors General), may prove useful to the review.
While performance measurement and program evaluations can serve as  key evidence for assessing progress, our past work has identified issues  with agencies’ capacities to develop and use these types of evidence in  decision making.
Performance measurement is the ongoing monitoring and reporting of  program accomplishments, particularly progress toward preestablished  goals. Because of its ongoing nature, performance measurement can  serve as an early warning system to management and as a vehicle for  improving accountability to the public. Although our work on federal  performance measurement during the past 2 decades has found an  increase in the reported presence of different types of performance  measures across the government, it has not resulted in similar increases  in the reported use of performance information in decision making.  Moreover, in June 2013, we found that agencies continue to face  common, long-standing difficulties in measuring the performance of  various types of federal programs and activities—contracts, direct  services, grants, regulations, research and development, and tax  expenditures. We recommended that the Director of OMB work with the  PIC to develop a detailed approach to examine these difficulties across  agencies, including identifying and sharing any promising practices from  agencies that have overcome difficulties in measuring the performance of  these program types. OMB staff agreed with this recommendation. As of  April 2015, OMB and the PIC have taken some initial steps to address  this recommendation in a few areas, such as acquisition management  (contracts). In addition, according to information provided by OMB staff,  the PIC formed a working group on performance measurement that, in  part, is focusing on how to develop appropriate performance measures.  However, OMB has not yet developed a comprehensive and detailed  approach to address these issues as envisioned in our report. We will  continue to monitor progress on implementation of this recommendation.
Program evaluations are individual systematic studies conducted  periodically or on an ad hoc basis to assess how well a program is  working. A program evaluation’s typically more in-depth examination of  program performance and context allows for an overall assessment of  whether the program works and identification of adjustments that may  improve its results. However, as reported in June 2013 based on results  from a governmentwide survey, we found that most federal managers  lacked recent evaluations of their programs.reported that an evaluation had been completed within the past 5 years of  any program, operation, or project in which they were involved. Another  40 percent of managers reported that they did not know if an evaluation  had been completed. However, 80 percent of managers who did have  evaluations reported that those evaluations contributed to a moderate or  greater extent to improving program management or performance, and to  assessing program effectiveness or value.
Our past work has found that the capacity to collect and analyze useful  evidence is critical to successful reviews. To be useful to various decision makers, evidence must be accessible, accurate, complete,  credible, consistent, relevant, timely, and valid. In addition, having the  capacity to disaggregate data according to demographic, geographic, or  other relevant characteristics can aid in highlighting significant variation,  which can help meeting participants to pinpoint problems and identify  solutions. Agencies also need to plan for the time and resources required  to generate and communicate performance data and other evidence in a  timely manner. Easy access to relevant databases and systems- generated analysis, such as providing analysts with the ability to develop  performance reports without relying on information technology staff, can  streamline the data collection and analysis processes.
subsections: section_title: HUD’s Office of Policy Development and Research Supported Strategic Reviews by Identifying Relevant Research and Evaluation
paragraphs: HUD’s Office of Policy Development and Research (PD&R) is responsible  for maintaining current information on housing needs, market conditions,  and existing programs, as well as conducting research on priority housing  and community development issues. According to HUD performance  staff, PD&R supported the review of each strategic objective by providing  a template containing the most relevant research and evaluations related  to each objective. This included both HUD-funded and external evidence.  In addition, HUD’s performance staff asked objective leaders to  supplement the evidence provided by PD&R with any additional evidence  they thought would inform the review. Figure 6 provides examples of the  research and evaluations that informed the department’s review of its  objective to end homelessness for veterans, people experiencing chronic  homelessness, families, youth, and children.
HUD performance staff told us that when a strategy has a clear outcome  measure tied to departmental funding and support, identifying or  developing relevant research was a lower priority because the existing  measures provided an understanding of progress towards a goal or  objective. For example, they told us that HUD has outcome information  for its rental housing programs, in terms of individuals who are  subsequently housed. HUD performance staff told us they were more  concerned about developing new performance measures and identifying  relevant research to inform policy changes where existing strategies  lacked clear measures. For instance, HUD does not have broader  outcome information on how all of its rental housing programs help  individuals become more self-sufficient in terms of obtaining further  education or employment.
section_title: USCIS Officials Compiled and Categorized Key Performance Information and Evidence to Inform Its Review
paragraphs: Prior to kicking off the review for DHS’s goal to strengthen and administer  the immigration system, USCIS performance staff compiled relevant  evidence—including agency performance data, program evaluations, and  relevant reports by us and the DHS Inspector General—into a database  to allow strategic review team members to focus on analyzing the  evidence and determining progress in achieving the goal. In compiling the  database, performance staff summarized key findings from the evidence  and provided potential users with the source of the evidence so they  could obtain additional context, if necessary. Further, they categorized the  evidence to allow for easy sorting by users. For example, the evidence  could be sorted by the sub-goal to which it was related; DHS’s four  assessment areas it supported; whether it represented an  accomplishment, planned activity, challenge/recommendation, a study, or  other information; and key contributing organization within USCIS. Figure  7 provides an excerpt from this database, illustrating how agency  performance data on processing applications was categorized.
For its objective to promote sustainable and livable communities, OSWER  officials developed what they called a “ladder of evidence”—a framework  and inventory of relevant performance information, scientific studies,  academic research, and program evaluations, which they then assessed  and categorized by strength. Officials said the different levels (types) of  evidence allowed them to better assess and communicate the results of  OSWER’s programs.
The first level of evidence provides descriptive data, covering  information about what OSWER does, whom it serves and why, and  performance trends over time. For example, one performance  measure at this level is the number of Superfund sites with human  exposure to contamination under control.
The second level of evidence identifies a relationship between  OSWER’s activities and its outcomes. It provides evidence about the  effectiveness of program implementation which can help identify  promising practices or problematic areas for further study.
The third level of evidence establishes a causal link between  OSWER’s programs and the impact they are having on human health  and environmental outcomes.
Figure 8 provides additional information about each level of evidence  along with illustrative examples.
section_title: Assess Effectiveness in Achieving Strategic Objectives and Identify Actions Needed to Improve Implementation and Impact
paragraphs: Using relevant evidence, strategic review participants should assess  whether strategies are being implemented as planned and whether they  are having the desired effect, as well as whether other factors are  influencing results. The review may highlight areas where action is  needed to improve or enhance implementation and impact. The following  questions, based broadly on practices from OMB’s guidance and our past  work on performance management, could help participants focus and  facilitate this assessment and determine any needed actions.
If progress is lagging, why and what actions (strategy changes,  revised management practices, legislative or budgetary proposals,  etc.) could lead to better results?
Are there any potential gaps in strategy? Conversely, is there any  unnecessary overlap and duplication? Addressing such issues could  lead to improvements in effectiveness and efficiency.
Where progress is sufficient or exceeding expectations, are there  strategies or practices that could be replicated and/or scaled to further  enhance effectiveness?
Have there been recent changes in the agency’s operating  environment that need to be addressed?
Are there strengths/opportunities on which to capitalize?
Are there weaknesses/threats that need to be overcome?
If the review identified evidence gaps, what steps will the agency take  to develop sufficient evidence?
In addition, OMB’s guidance suggests additional actions that agencies  should consider, which could lead to enhanced performance. These  include benchmarking information from others trying to accomplish the  same or similar objectives or using the same or similar key process, and  identifying lessons learned from past efforts to continuously improve  service delivery and resolve management challenges.
subsections: section_title: Education Modified an Existing Program and Created a New One to Help Address Challenges Related to Turning around Schools and Closing Achievement Gaps
paragraphs: Education officials told us that, in addition to the Department’s policy  development and spending plan review, they used the strategic review  process to assess how recent changes to its School Improvement Grant  (SIG) program contributed to progress in one of the department’s  strategic objectives. Education’s SIG program is designed to fund  significant reforms in low-performing schools in support of the  department’s objective to “accelerate achievement by supporting states  and districts in turning around low-performing schools and closing  achievement gaps, and developing models of next-generation high  schools.” According to Education’s Fiscal Year 2014 Annual Performance  Report and Fiscal Year 2016 Performance Plan, turning around the  lowest-performing schools takes several years to show progress and  success. Education reported that since 2009, more than 1,700 schools  have received up to $2 million for 3 years through the SIG program to  implement intervention models intended to turn around the lowest- performing schools. While nearly two-thirds of the schools have made  progress, the remaining schools have either not shown progress or had  decreased performance.
Through their ongoing SIG program monitoring, Education officials told us  they learned about two challenges grantees reported facing that could be  hindering progress and developed new strategies intended to address  them. First, officials at state and local educational agencies expressed  concerns to Education officials about sustaining turnaround efforts, since  they are long term in nature and SIG program funds were only available  for 3 years.approaches to better support sustainability. Using waiver authority, the  department gave grantees flexibility to extend their use of existing funding  into a fourth year. In addition, beginning with its fiscal year 2014  appropriations, Education obtained additional authority for state  educational authorities to make school improvement grants for up to 5   Education officials told us they took two different  years. Education officials told us grantees also expressed concerns  about a lack of principals with knowledge about or experience in turning  around schools. Recognizing the importance of sustained leadership  commitment, the department launched a new grant program in 2014, the  Turnaround School Leaders program. This program provides funding for  3 years to local educational agencies to help ensure that leaders at  schools eligible for or receiving SIG program funds possess the  specialized skills needed to drive successful efforts to turn those schools  around.
section_title: DHS Is Developing New Performance Measures to Address Gaps in Data Identified during Strategic Review
paragraphs: Although CBP, TSA, ICE, and Coast Guard officials determined during  their review that sufficient progress was being made on DHS’s Goal 2.2 to  Safeguard and Expedite Lawful Trade and Travel, they also identified  gaps to address in performance monitoring. Officials told us that they  realized that while they tracked a number of performance measures  related to aspects of trade, they had none regarding the travel portion of  the objective (see table 3 below). Further, while most of the existing  measures addressed enforcement and security, they noted that they had  few measures that addressed the facilitation aspects of their mission— reducing barriers to the efficient flow of trade and travel.
According to one of the CBP officials who coordinated the review for this  goal from the Office of Planning, Program Analysis, and Evaluation  (PPAE), CBP is responsible for most of the activities that would be  covered by the gaps in performance information. He told us that CBP has  been working to address these weaknesses since they were identified  last year. For example, he shared that PPAE has been working with the  Trusted Traveler Division within the Office of Field Operations (OFO) to  develop travel-specific performance measures. These measures would  address the land border and air travel modes, the principal avenues by  which most international travelers enter the country. One or more of the  travel measures developed is to address the facilitation aspect of CBP’s  mission, as expressed in Goal 2.2. In addition, he told us that PPAE is  working with the Cargo and Conveyance Security Directorate within OFO  to develop a trade facilitation measure. CBP expects to complete the  formulation of these measures during calendar year 2015, and plans to  subsequently submit them to DHS as formal performance measures to  begin reporting in the second quarter of fiscal year 2016.
section_title: Develop a Process to Monitor Progress on Needed Actions
paragraphs: Much like we found for data-driven reviews, thorough and sustained  follow-up on issues identified during strategic reviews is critical to the  success of the reviews as a performance improvement tool. To ensure  that actions identified as a result of the strategic review are carried out in  the period between reviews, the agency should have a process to track  these actions and communicate the progress made towards them. Such a  process should identify, among other things for each action item, the  responsible party, target completion dates, and significant milestones. In  addition, agency leadership should hold responsible officials accountable  for taking the agreed upon actions and communicating what has been  done routinely. For example, agencies could use their existing quarterly  performance review processes to monitor progress on strategic review  action items, in line with the emphasis in OMB’s guidance for using  existing agency management processes for strategic reviews.
OMB’s guidance further reinforces this practice by stating that agencies  must incorporate actions to maintain or improve progress toward each  objective, along with related implementation activities, into their next  annual performance plan or other operating plans. For the fiscal year  2016 annual performance plan, this is to include, at a minimum, the  agency’s summary of plans to improve or maintain performance, key  milestones planned for the next year with completion dates, and efforts to  close evidence gaps, as appropriate.
subsections: section_title: HUD Is Using Its Existing Performance Review Process to Track Progress on Action Items Stemming from Strategic Reviews
paragraphs: HUD’s performance reviews for its agency priority goals, known as  HUDStat meetings, occur frequently and regularly (quarterly). To conduct  its strategic reviews, HUD broadened the focus of its HUDStat meetings  in one quarter to review progress toward its strategic objectives. For both  sets of meetings, HUD’s performance staff have developed a process for  identifying and tracking action items stemming from the reviews.
According to HUD performance staff, action items can be identified in a  number of ways, including by the Secretary or PIO during reviews of  materials prior to the HUDStat meeting, by meeting participants during  the HUDStat session, or in a postmeeting session among the Secretary,  PIO, and objective leads. HUD’s performance staff then compile and  share a list of action items by objective or goal to all participants via e- mail within a day of the HUDStat meeting to ensure agreement. These  are then added to a central tracking database for all action items. For  each action item, the tracking database identifies the responsible party, a  target completion date, any interim dates (milestones), and a status  update. For example, following the 2014 strategic review for HUD’s  objective to “end homelessness for veterans, people experiencing chronic  homelessness, families, youth, and children,” one action item identified  during the review was to establish targets for homeless family admissions  to public housing, tenant-based vouchers, and project-based vouchers. It  identifies the Office of Public and Indian Housing and the Office of  Multifamily Housing as the responsible parties. According to HUD  officials, as of April 2015, the Office of Public and Indian Housing is  working to understand the capacity of local partners and will subsequently  set targets. The Office of Multifamily Housing began collecting homeless  admissions data in late 2014 and requiring it in February 2015. However,  it is at least a year off from establishing and validating a baseline, and  subsequently setting a target. HUD performance staff told us they will use  the department’s 2015 strategic reviews to reinforce accountability for  setting these targets.
HUD’s performance staff told us they work with responsible parties to  update the status of each action item and provide a report to the Deputy  Secretary regularly. According to HUD performance staff, following the  2014 strategic review, these updates occurred either biweekly or monthly,  and for the 2015 strategic review they will occur biweekly.
section_title: USDA Instituted Quarterly Updates to Track Progress on Action Items Stemming from Strategic Reviews
paragraphs: USDA uses quarterly updates to the SOAR quad charts to keep the  Secretary and other senior leaders informed of ongoing progress towards  the objectives, as well as any related challenges. This includes providing  updated information on the status of actions that were identified in prior  quarters. For example, the Food Safety and Inspection Service (FSIS),  which is responsible for ensuring that the nation’s commercial supply of  meat, poultry, and egg products is safe, wholesome, and correctly labeled  and packaged, is the lead agency for USDA’s strategic objective to  “protect public health to ensure food is safe.” As part of the initial SOAR  quad chart, from the second quarter of 2014, one of the next steps FSIS  identified for this objective was to ensure continued progress in controlling  Salmonella by developing new performance standards targeting chicken  parts and ground poultry, and improving the agency’s verification  sampling plans. According to USDA, Salmonella is the leading known  cause of bacterial foodborne illness and death in the country, causing an  estimated 1.3 million illnesses, and between 400 and 500 deaths  annually. As part of its SOAR quad chart update for the fourth quarter of  2014, FSIS noted that it had developed a workplan for the Federal  Register to announce and seek public comment on draft performance  standards for Salmonella in chicken parts and ground chicken as part of  the progress update. However, FSIS also noted in the significant  challenges section that the draft rule was deemed “significant” by OMB,and FSIS was also responding to internal comments prior to moving  forward with publication.
section_title: Agency Comments
paragraphs: We provided a draft of the report to the Director of the Office of  Management and Budget, the Secretary of the Department of Agriculture,  the Secretary of the Department of Education, the Secretary of the  Department of Homeland Security, the Secretary of the Department of  Housing and Urban Development, the Administrator of the National  Aeronautics and Space Administration, and the Administrator of the  Environmental Protection Agency for comment.
OMB staff and officials from the six agencies generally agreed with the  findings presented in this report. In addition, DHS, Education, EPA, HUD,  NASA, and OMB provided technical comments, which we incorporated as  appropriate.
We are sending copies of this report to interested congressional  committees, the Director of the Office of Management and Budget, the  Secretary of the Department of Agriculture, the Secretary of the  Department of Education, the Secretary of the Department of Homeland  Security, the Secretary of the Department of Housing and Urban  Development, the Administrator of the National Aeronautics and Space  Administration, the Administrator of the Environmental Protection Agency,  and other interested parties. This report will also be available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of our report. Key contributors to this report are listed in appendix II.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: We are required to review implementation of the GPRA Modernization Act  of 2010 (GPRAMA) at several critical junctures. This report is part of our  response to that mandate. Our specific objective for this report was to  identify and illustrate, through case agency examples, practices that  facilitate effective strategic reviews by federal agencies.
To identify practices, we analyzed and synthesized information gathered  from a literature review we conducted, which covered public  administration and public policy journals, business administration  journals, our body of work on performance management and program  evaluation, and other sources on policies and practices that can facilitate  or challenge the effectiveness of strategic reviews as a decision-making  tool. We also conducted interviews with performance management and  evaluation experts representing different levels of government (local,  state, federal), sectors (e.g. public, non-profit, foundations), and nations,  who had experience with implementing elements of strategic reviews or  academic and/or consultative expertise in this area.and interviews experts based on the results of our literature review (i.e.,  the authors of relevant articles or books included in our review). Based on  suggestions from those individuals, we expanded our list of experts and  conducted a second round of interviews.
Using the information we obtained from our literature review and expert  interviews, we developed a broad set of practices for conducting effective  strategic reviews. We refined the practices through our audit work at  selected agencies (see next paragraphs). We also compared our  practices with legal requirements in GPRAMA, guidance from the Office  of Management and Budget (OMB), and a guide for conducting strategic  reviews developed by the Performance Improvement Council (PIC), and  found them to be broadly consistent.
To help illustrate and refine our draft practices, we selected a non- generalizeable sample of agencies based on several criteria and  analyses. We limited the initial population for selection to the 24 agencies  covered by the Chief Financial Officers Act of 1990 (CFO Act), as  amended, because GPRAMA directs us to periodically evaluate how  implementation of the act is affecting performance management at those  agencies. We further refined the list to exclude two agencies, the  Departments of Defense (DOD) and Veterans Affairs (VA), from selection.  We excluded DOD because the department had not published strategic  objectives related to its 2014 strategic goals at the time of our selection  process. We excluded VA because of ongoing corrective actions it was  taking to address significant shortcomings in the accuracy and reliability  of certain performance information.
Because agencies conducted their initial strategic reviews in 2014 as we  were selecting our sample, we could not use information about agencies’  strategic review processes to inform selection. As a proxy, we used  relevant agency-level results on selected items from our 2013 survey of  federal managers on performance and management issues to  approximate if agencies had robust review processes and selected  agencies with varying levels of robustness. These survey items covered the extent to which agency leadership was committed and involved in  performance management activities, as well as the use of performance  information.
We also considered the extent to which agency strategic review  processes had a greater chance of addressing areas of fragmentation,  overlap, and duplication, and high-risk issues identified in our past work.  We have previously reported that effective implementation of strategic  reviews could help identify opportunities to reduce, eliminate, or better  manage instances of fragmentation, overlap, and duplication because, as  part of the reviews, agencies are to identify the various organizations,  programs, regulations, tax expenditures, policies, and other activities that  contribute to each objective both within and outside the agency.addition, because agencies are to identify goals and strategies to resolve  major management challenges they face, strategic reviews could also  identify opportunities to better address issues on our High Risk List.
We also took into consideration agency size, based on the number full- time equivalent employees, given the potential for variation in review  practices due to organizational size and capacity. Based on the criteria  and analyses outlined above, we selected the Departments of Agriculture,  Education, Homeland Security, and Housing and Urban Development,  and the Environmental Protection Agency and National Aeronautics and  Space Administration. These selections were also in line with suggestions  we independently obtained from staff in OMB’s Office of Performance and  Personnel Management who had reviewed each of the agencies’ plans  for conducting their strategic reviews as well as the results of those  reviews.
To identify illustrative examples for each of our practices from the six  selected agencies and to further refine our practices, we reviewed  documentation about agencies’ strategic review processes and results,  including guidance, meeting agendas, relevant evidence used to inform  the review, and internal and published summaries of the results. We also  conducted interviews with officials involved in conducting strategic  reviews at the six selected agencies—which included agency  Performance Improvement Officers and their staff, strategic objective  leaders, and strategic review participants—and staff from OMB and the  PIC.
We conducted this performance audit from August 2013 to July 2015 in accordance with generally accepted government auditing standards.Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the above contact, Elizabeth Curda (Acting Director) and  Benjamin T. Licht (Assistant Director) supervised this review and the  development of the resulting report. Crystal Bernard, Virginia Chanley,  Jehan Chase, Carole J. Cimitile, Emily Gruenwald, and Katherine Wulff  made significant contributions to this report. Robert Robinson developed  the graphics for this report. Sandra Beattie, Ellen Grady, Adam Miles,  Jason Vassilicos, and Dan Webb verified the information contained in this  report.