You are given two reports by the government agency.  

Report 1:
While HUD has primary responsibility for addressing lead paint hazards in federally-assisted housing, EPA also has responsibilities related to setting federal lead standards for housing. EPA sets federal standards for lead hazards in paint, soil, and dust. Additionally, EPA regulates the training and certification of workers who remediate lead paint hazards. CDC sets a health guideline known as the “blood lead reference value” to identify children exposed to more lead than most other children. As of 2012, CDC began using a blood lead reference value of 5 micrograms of lead per deciliter of blood. For children whose blood lead level is at or above CDC’s blood lead reference value, health care providers and public health agencies can identify those children who may benefit the most from early intervention. CDC’s blood lead reference value is based on the 97.5th percentile of the blood lead distribution in U.S. children (ages 1 to 5), using data from the National Health and Nutrition Examination Survey. Children with blood lead levels above CDC’s blood lead reference value have blood lead levels in the highest 2.5 percent of all U.S. children (ages 1 to 5). HUD, EPA, and the Department of Health and Human Services (HHS) are members of the President’s Task Force on Environmental Health Risks and Safety Risks to Children. HUD co- chairs the lead subcommittee of this task force with EPA and HHS. The task force published the last national lead strategy in 2000. The primary federal legislation to address lead paint hazards and the related requirements for HUD is the Residential Lead-Based Paint Hazard Reduction Act (Title X of the Housing and Community Development Act of 1992). We refer to this law as Title X throughout this report. Title X required HUD to, among other things, promulgate lead paint regulations, implement the lead hazard control grant programs, and conduct research and reporting, as discussed throughout this report. The two key regulations that HUD has issued under Title X are the Lead Disclosure Rule and the Lead Safe Housing Rule: Lead Disclosure Rule. In 1996, HUD and EPA jointly issued the Lead Disclosure Rule. The rule applies to most housing built before 1978 and requires sellers and lessors to disclose any known information, available records, and reports on the presence of lead paint and lead paint hazards and provide an EPA-approved information pamphlet prior to sale or lease. Lead Safe Housing Rule. In 1999, HUD first issued the Lead Safe Housing Rule, which applies only to housing receiving federal assistance or federally-owned housing being sold. The rule established procedures for evaluating whether a lead paint hazard exists, controlling or eliminating the hazard, and notifying occupants of any lead paint hazards identified and related remediation efforts. The rule established an “elevated blood lead level” as a threshold that requires landlords and PHAs to take certain actions if a child’s blood test shows lead levels meeting or exceeding this threshold. In 2017, HUD amended the rule to align its definition of an “elevated blood lead level” with CDC’s blood lead reference value. This change lowered the threshold that generally required landlords and PHAs to act from 20 micrograms to 5 micrograms of lead per deciliter of blood. According to the rule, when a child under age 6 living in HUD-assisted housing has an elevated blood lead level, the housing provider must take several steps. These generally include testing the home and other potential sources of the child’s lead exposure within 15 days, ensuring that identified lead paint hazards are addressed within 30 days of receiving a report detailing the results of that testing, and reporting the case to HUD. Office of Lead Hazard Control and Healthy Homes (Lead Office). HUD’s Lead Office is primarily responsible for administering HUD’s two lead hazard control grant programs, providing guidance on HUD’s lead paint regulations, and tracking HUD’s efforts to make housing lead-safe. The Lead Office collaborates with HUD program offices on its oversight and enforcement of lead paint regulations. For instance, the Lead Office issues guidance, responds to questions about requirements of lead paint regulations, and provides training and technical assistance to HUD program staff, PHA staff, and property owners. The Lead Office’s oversight efforts also include maintaining email and telephone hotlines to receive complaints and tips from tenants or homeowners, among others, as they pertain to lead paint regulations. Additionally, the Lead Office, in collaboration with EPA, contributes to the operation of the National Lead Information Center––a resource that provides the general public and professionals with information about lead, lead hazards, and their prevention. Office of Public and Indian Housing (PIH). HUD’s PIH oversees and enforces HUD’s lead paint regulations for the rental assistance programs. As discussed earlier, this report focuses on the two largest rental assistance programs serving the most families with children––the Housing Choice Voucher and public housing programs. Housing Choice Voucher program. In the voucher program, eligible families and individuals are given vouchers as rental assistance to use in the private housing market. Generally, eligible families with vouchers live in the housing of their choice in the private market. The voucher generally pays the difference between the family’s contribution toward rent and the actual rent for the unit. Vouchers are portable; once a family receives one, it can take the voucher and move to other areas where the voucher program is administered. In 2017, there were roughly 2.5 million vouchers available. Public housing program. Public housing is reduced-rent developments owned and operated by the local PHA and subsidized by the federal government. PHAs receive several streams of funding from HUD to help make up the difference between what tenants pay in rent and what it costs to maintain public housing. For example, PHAs receive operating and capital funds through a formula allocation process. PHAs use operating funds to pay for management, administration, and day-to-day costs of running a housing development. Capital funds are used for modernization needs, such as replacing roofs or remediating lead paint hazards. According to HUD rules, generally families that are income-eligible to live in public housing pay 30 percent of their adjusted income toward rent. In 2017, there were roughly 1 million public housing units available. For both of these rental assistance programs, the Office of Field Operations (OFO) within PIH oversees PHAs’ compliance with lead paint regulations, in conjunction with HUD field office staff. The office has a risk-based approach to overseeing PHAs and performs quarterly risk assessments. Also within PIH, staff from the Real Estate Assessment Center are responsible for inspecting the physical condition of public housing properties. Office of Policy Development and Research (PD&R). HUD’s PD&R is the primary office responsible for data analysis, research, and program evaluations to inform the development and implementation of programs and policies across HUD offices. of the total grant amount, while the Lead Hazard Reduction Demonstration grant program has required at least a 25 percent match. For fiscal years 2013–2017, HUD awarded $527 million for its lead hazard control grants, which included 186 grants to state and local jurisdictions (see fig. 1). In these 5 years, about 40 percent of grants awarded went to jurisdictions in the Northeast and 31 percent to jurisdictions in the Midwest––regions of the country known to have a high prevalence of lead paint hazards. Additionally, in these 5 years, 90 percent of grant awards went to grantees at the local jurisdiction level (cities, counties, and the District of Columbia). The other 10 percent of grant awards went to state governments. During this time period, HUD awarded the most grants to jurisdictions in Ohio (17 grants), Massachusetts and New York (15 grants each), and Connecticut (14 grants). HUD’s Lead-Based Paint Hazard Control grant and the Lead Hazard Reduction Demonstration grant programs have incorporated Title X statutory requirements through recent annual funding notices and their grant processes. Title X contains applicant eligibility requirements and selection criteria HUD should use to award lead grants. To be eligible to receive a grant, applicants need to be a state or local jurisdiction, contribute matching funds to supplement the grant award, have an approved comprehensive affordable housing strategy, and have a certified lead abatement program (if the applicant is a state government). HUD has incorporated these eligibility requirements in its grant programs’ 2017 funding notices, which require applicants to demonstrate that they meet these requirements when they apply for a lead grant. According to the 2017 funding notices, applicants must detail the sources and amounts of their matching contributions in their applications. Similarly, applicants must submit a form certifying that the proposed grant activities are consistent with their local affordable housing strategy. HUD’s 2017 funding notices state that if applicants did not meet these eligibility requirements, HUD would not consider their applications. Additionally, Title X requires HUD to award lead grants according to the following applicant selection criteria: the extent to which an applicant’s proposed activities will reduce the risk of lead poisoning for children under the age of 6; the degree of severity and extent of lead paint hazards in the applicant’s jurisdiction; the applicant’s ability to supplement the grant award with state, local, or private funds; the applicant’s ability to carry out the proposed grant activities; and other factors determined by the HUD Secretary to ensure that the grants are used effectively. In its 2017 funding notices, HUD incorporated the Title X applicant selection criteria through five scoring factors that it used to assess lead grant applications. HUD allocated a certain number of points to each scoring factor. Applicants are required to develop their grant proposals in response to the scoring factors. When reviewing applications, HUD staff evaluated an applicant’s response to the factors and assigned points for each factor. See table 1 for a description of the 2017 lead grant programs’ scoring factors and points. As shown in table 1, HUD awarded the most points (46 out of 100) to the “soundness of approach” scoring factor, according to HUD’s 2017 funding notices. Through this factor, HUD incorporated Title X selection criteria on an applicant’s ability to carry out the proposed grant activities and supplement a grant award with state, local, or private funds. For example, HUD’s 2017 funding notices required applicants to describe their detailed plans to implement grant activities, including how the applicants will establish partnerships to make housing lead-safe. Specifically, HUD began awarding 2 of the 100 points to applicants who demonstrated partnerships with local public health agencies to identify families with children for enrollment in the lead grant programs. Additionally, HUD asked applicants to identify partners that can help provide assistance to complete the lead hazard control work for high-cost housing units. Furthermore, HUD required applicants to identify any nonfederal funding, including funding from the applicants’ partners. Appendix I includes examples of state, local, and nongovernmental funds that selected grantees planned to use to supplement their lead grants. In its lead grant programs, HUD has taken actions that were consistent with OMB’s requirements for competitively awarded grants. OMB generally requires federal agencies to: (1) establish a merit-review process for competitive grants that includes the criteria and process to evaluate applications; and (2) develop a framework to assess the risks posed by applicants for competitive grants, among other things. Through a merit-review process, an agency establishes and applies criteria to evaluate the merit of competitive grant applications. Such a process helps to ensure that the agency reviews grant applications in a fair, competitive, and transparent manner. Consistent with the OMB requirement to establish a merit review process, HUD has issued annual funding notices that communicate clear and explicit evaluative criteria. In addition, HUD has established processes for reviewing and scoring grant applications using these evaluative criteria, and selects grant recipients based on the review scores (see fig. 2). For example, applicants that score at or above 75 points are qualified to receive awards from HUD. Also, HUD awards funds beginning with the highest scoring applicant and proceeds by awarding funds to applicants in a descending order until funds are exhausted. Furthermore, consistent with the OMB requirement to develop a framework to assess applicant risks, HUD has developed a framework to assess the risk posed by lead grant applicants by, among other things, deeming ineligible those applicants with past performance deficiencies or those that do not have a financial management system that meets federal standards. However, HUD has not fully documented or evaluated its lead grant processes in reviewing and scoring the grants and making award decisions: Documenting grant processes and award decisions. While HUD has established processes for its lead grant programs, it lacks documentation, including detailed guidance to help ensure that staff carry out processes consistently and appropriately. Federal internal control standards state that agency management should develop and maintain documentation of its internal control system. Such documentation assists agency management by establishing and communicating the processes to staff. Additionally, documentation of processes can provide a means to retain organizational knowledge and communicate that knowledge as needed to external parties. The Lead Office’s Application Review Guide describes its grant application review and award processes at a high level but does not provide detailed guidance for staff as to how tasks should be performed. For example, the Guide notes that reviewers score eligible applications according to factors contained in the funding notices but does not describe how the reviewers should allocate points to the subfactors that make up each factor. Lead Office staff told us that creating detailed scoring guidance would be challenging because applicants’ proposed grant activities differ widely, and they said that scoring grant applications is a subjective process. While scoring grant applications may involve subjective judgments, improved documentation of grant review and scoring processes, including additional direction to staff, can help staff apply their professional judgment more consistently in evaluating applications. By better documenting processes, HUD can better ensure that staff evaluate applications consistently. Additionally, HUD has not fully documented its rationale for deciding which applicants receive lead grant awards and for deciding the dollar amounts of grant awards to successful applicants. In prior work examining federal grant programs, one recommended practice we identified is that agencies should document the rationale for award decisions, including the reasons individual applicants were selected or not and how award funding amounts were determined. While HUD’s internal memorandums listed the applicants selected and the award amounts, these memorandums did not document the rationale for these decisions or provide information sufficient to help applicants understand award outcomes. Lead Office staff told us that most grantees have received the amount of funding they requested in their applications, which was generally based on HUD’s maximum grant award amount. Lead Office staff said they could use their professional judgment to adjust award amounts to extend funding to more applicants when applicants received similar scores. However, the Lead Office’s documentation we reviewed did not explain this type of decision making. For example, in 2017, when two applicants received identical scores on their applications, HUD awarded each applicant 50 percent of the remaining available funds rather than awarding either applicant the amount they requested. Representatives of one of the two grantees told us they did not know why the Lead Office had not provided them the full amount they had requested. Lead Office staff told us that, to date, HUD has not considered alternative ways to award grant funding amounts. By fully documenting grant award processes, including the rationale for award decisions and amounts, HUD could provide greater transparency to grant applicants about its grant award decisions. Evaluating processes. HUD lacks a formal process for reviewing and updating its lead grant funding notices, including the factors and point allocations used to score applications. Federal internal control standards state that agencies should implement control activities through policies and that periodic review of policies and procedures can provide assurance of their effectiveness in achieving the agency’s objectives. Lead Office staff told us that previous changes to the factors and point allocation used to score applicants have been made based on informal discussions among staff. However, the Lead Office does not have a formal process to review and evaluate the relevance and appropriateness of the factors or points used to score applicants. Lead Office staff told us that they have never analyzed the scores applicants received for the factors to identify areas where applicants may be performing well or poorly or to help inform decisions about whether changes may be needed to the factors or points. Additionally, HUD has not changed the threshold criteria used to make award decisions since the threshold was established in 2003. As previously shown in figure 2, applicants who received at least 75 points (out of 100) have been qualified to receive a grant award. However, HUD grant documentation, including the funding notices and the Application Review Guide, does not explain the significance of this 75-point threshold. Lead Office staff stated that this threshold was first established in 2003 by HUD based on OMB guidance. A formal review of this 75-point threshold can help HUD determine whether it remains appropriate for achieving the grant programs’ objectives. Furthermore, by periodically evaluating processes for reviewing and scoring grant applications, HUD can better determine whether these processes continue to help ensure that lead grants reach areas of the country at greater risk for lead paint hazards. HUD has begun to develop analyses and tools to inform its efforts to target outreach and ensure that grant awards go to areas of the country that are at risk for lead paint hazards. However, HUD has not developed time frames for incorporating the results of the analyses into its lead grant programs’ processes. HUD has required jurisdictions applying for lead grants to include data on the need or extent of the problem in their jurisdiction (i.e., scoring factor 2). Additionally, Lead Office staff told us that HUD uses information from the American Healthy Homes Survey to obtain information on lead paint hazards across the country. However, the staff explained that the survey was designed to provide meaningful results at the regional level and did not include enough homes in its sample to provide information about housing conditions, such as lead paint hazards, at the state or local level. Because HUD awards lead grants to state and local jurisdictions, it cannot effectively use the survey results to help the agency make award decisions or inform decisions about areas for potential outreach. In early 2017, the Lead Office began working with PD&R to develop a model to identify local jurisdictions (at the census-tract level) that may be at heightened risk for lead paint hazards. Lead Office staff said that they hope to use results of this model to develop geographic tools to help target HUD funding to areas of the country at risk for lead paint hazards but not currently receiving a HUD lead grant. Lead Office staff said that they could reach out to these at-risk areas, help them build the capacity needed to administer a grant, and encourage them to apply. For example, HUD has identified that Mississippi and two major metropolitan areas in Florida (Miami and Tampa) had not applied for a lead grant. HUD has conducted outreach to these areas to encourage them to apply for a lead grant. In 2016, the City of Jackson, Mississippi, applied for and received a lead grant. Though the Lead Office has collaborated with PD&R on the model, HUD has not developed specific time frames to operationalize the model and incorporate the results of the model for using local-level data to help better identify areas at risk for lead paint hazards. Federal internal control standards require agencies to define objectives clearly to enable the identification of risks. This includes clearly defining time frames for achieving the objectives. Setting specific time frames could help to ensure that HUD operationalizes this model in a timely manner. By operationalizing a model that incorporates local data on lead paint hazard risk, HUD can better target its limited grant resources towards areas of the country with significant potential for lead hazard control needs. We performed a county-level analysis using HUD and Census Bureau data and found that most lead grants from 2013 through 2017 have gone to counties with at least one indicator of lead paint hazard risk. Information we reviewed, such as relevant literature, suggests that the two common indicators of lead paint hazard risk are the prevalence of housing built before the 1978 lead paint ban and the prevalence of individuals living below the poverty line. We defined areas with lead paint hazard risk as counties that had percentages higher than the corresponding national percentages for both of these indicators. The estimated average percentage nationwide of total U.S. housing stock constructed before 1980 was 56.9 percent and the estimated average percentage nationwide of individuals living below the poverty line was 17.5 percent. As shown in figure 3, our analysis estimated that 18 percent of lead grants from 2013 through 2017 have gone to counties with both indicators above the estimated national percentages, 59 percent of grants have gone to counties with estimated percentages of old housing above the estimated national percentage, and 7 percent of grants have gone to counties that had estimated poverty rates above the estimated national percentage. (For an interactive version of this map, click here.) When HUD finalizes its model and incorporates information into its lead grant processes, HUD will be able to better target its grant resources to areas that may be at heightened risk for lead paint hazards. In 2016, HUD began to incorporate new steps to monitor PHAs’ compliance with lead paint regulations for nearly 4,000 PHAs. Previously, according to PIH staff, HUD required only that PHAs annually self-certify their compliance with lead paint laws and regulations, and HUD’s Real Estate Assessment Center inspectors check for lead paint inspection reports and disclosure forms at public housing properties during physical inspections. Starting in June 2016, PIH began using new tools for HUD field staff to track PHAs’ compliance with lead paint requirements in the voucher and public housing programs. As shown in figure 4, PIH’s compliance oversight processes for the voucher and public housing programs include various monitoring tools for overseeing PHAs. Key components of PIH’s lead paint oversight processes include the following: Tools for tracking lead hazards and cases of elevated blood levels in children. HUD uses two databases to monitor PHAs’ compliance with lead paint regulations: (1) the Lead-Based Paint Response Tracker, which PIH uses to collect and monitor information on the status of lead paint-related documents, including lead inspection reports and disclosure forms, in public housing properties but not in units with voucher assisted households; and (2) the Elevated Blood Lead Level Tracker, which PIH uses to collect and monitor information reported by PHAs on cases of elevated blood levels in children living in voucher and public housing units. In June 2016, OFO began using the Lead-Based Paint Response Tracker database to store information on public housing units and to help HUD field office staff to follow up with PHAs that have properties missing required lead documentation. In July 2017, OFO began using information recorded in the Elevated Blood Lead Level Tracker to track whether PHAs started lead remediation activities in HUD- assisted housing within the time frames required by the Lead Safe Housing Rule. Lead paint hazards included in PHAs’ risk assessment scores. OFO assigns scores to PHAs based on their relative risk in four categories: physical condition, financial condition, management capacity, and governance. OFO uses these scores to identify high- and very high-risk PHAs that will receive on-site full compliance reviews. In July 2017, OFO incorporated data from the Real Estate Assessment Center into the physical condition category of its Risk Assessment Protocol to help account for potential lead paint hazards at public housing properties. Questions about lead paint included as part of on-site full compliance reviews. In fiscal year 2016, HUD field offices began conducting on-site full compliance reviews at high- and very high-risk PHAs as part of HUD’s compliance monitoring program to enhance oversight and accountability of PHAs. In fiscal year 2017, as part of the reviews, HUD field office staff started using a compliance monitoring checklist to determine if PHAs comply with major HUD rules and to gather additional information on the PHAs. This checklist included lead-related questions that PIH field office staff use to determine whether PHAs meet the requirements in lead paint regulations for both the voucher and public housing programs. In 2016, OFO and HUD field offices began using information from the new monitoring efforts to identify potential noncompliance by PHAs with lead paint regulations and help the PHAs resolve the identified issues. According to HUD data, as of November 2017, the Lead-Based Paint Response Tracker indicated that 9 percent (357) of PHAs were missing both lead inspection reports and lead disclosure forms for one or more properties. There were 973 PHAs missing one of the two required documents. OFO staff told us that they prioritized following up with PHAs that were missing both documents. According to OFO staff, PHAs can resolve potential noncompliance by submitting adequate lead documentation to HUD. OFO staff told us the agency considers missing lead documentation as “potential” noncompliance because PHAs may provide the required documentation or they may be exempt from certain requirements (e.g., HUD-designated elderly housing). While HUD has taken steps to strengthen compliance monitoring processes, it does not have a plan to identify and address the risks of noncompliance by PHAs with lead paint regulations. Federal internal control standards state that agencies should identify, analyze, and respond to risks related to achieving the defined objectives. Furthermore, when an agency has made significant changes to its processes—as HUD has done with its compliance monitoring processes—management review of changes to these processes can help the agency determine that its control activities are designed appropriately. Our review found that HUD does not have a plan to help mitigate and address risks related to noncompliance with lead paint regulations by PHAs (i.e., ensuring lead safety in assisted housing). Additionally, our review found several limitations with HUD’s new compliance monitoring approach, which include the following: Reliance on PHA self-certifications. HUD’s compliance monitoring processes rely in part on PHAs self-certifying that they are in compliance with lead paint regulations, but recent investigations have found that some PHAs may have falsely certified that they were in compliance. In November 2017, HUD filed a fraud complaint against two former officials of the Alexander County (Illinois) Housing Authority, alleging that the former official, among other things, falsely certified to HUD that the Housing Authority was in compliance with lead paint regulations. Further, PIH staff told us there are ongoing investigations related to potential noncompliance with lead paint regulations and false certifications at two other housing authorities. Lack of comprehensive data for the public housing program. OFO started to collect data for the public housing program in the Lead-Based Paint Response Tracker in June 2016 and the inventory of all public housing properties includes units inspected since 2012. In addition, HUD primarily relies on the presence of lead inspection reports but does not record in the database when inspections and remediation activities occurred and does not determine whether they are still effective. Because of this, the information contained in the lead inspection reports may no longer be up-to-date. For example, a lead inspection report from the 1990s may provide evidence that abatement work was conducted at that time, but according to PIH staff, the housing may no longer be lead-safe. Lack of readily available data for the voucher program. The voucher program does not have readily available data on housing units’ physical condition and compliance with lead paint regulations because data on the roughly 2.5 million units in the program are kept at the PHA level. According to PIH staff, HUD plans to adopt a new system for the voucher program that will include standardized, electronic data for voucher units. PIH staff said the new system (Uniform Physical Condition Standards for Vouchers Protocol) will allow greater oversight and provide HUD the ability to conduct data analysis for voucher units. Challenges identifying children with elevated blood lead levels. For several reasons, PHAs face ongoing challenges receiving information from state and local public health departments on the number of children identified with elevated blood lead levels. First, children across the U.S. are not consistently screened and tested for exposure to lead. Second, according to CDC data, many states use a less stringent health guideline to identify children compared to the health standard that HUD uses (i.e., CDC’s current blood lead reference value). PIH staff told us that some public health departments may not report children with elevated blood levels to PHAs because they do not know that a child is living in a HUD- assisted unit and needs to be identified using the more stringent HUD standard. Lastly, Lead Office staff told us that privacy laws in some states may impose restrictions on public health departments’ ability to share information with PHAs. Limited coverage of on-site compliance reviews. While full on-site compliance reviews can be used to determine if PHAs are in compliance with lead paint regulations, OFO conducts a limited number of these reviews annually. For example, in Fiscal Year 2017, OFO conducted 72 reviews of the roughly 4,000 total PHAs. Based on OFO information, there are 973 PHAs that are missing either lead inspection reports or lead disclosure forms indicating some level of potential noncompliance. HUD’s steps since June 2016 to enhance monitoring of PHAs’ compliance with lead paint regulations have some limitations that create risks in its new compliance monitoring approach. By developing a plan to help mitigate and address the various limitations associated with the new compliance monitoring approach, HUD could further strengthen its oversight and help ensure that PHAs maintain lead-safe housing units. HUD does not have detailed procedures to address PHA noncompliance with lead paint regulations or to determine when enforcement decisions may be needed. Lead Office staff told us that their enforcement program aims to ensure that PHAs have the information necessary to remain in compliance with lead paint regulations. According to federal internal control standards, agencies should implement control activities through policies and procedures. Effective design of procedures to address noncompliance would include documenting specific actions to be performed by agency staff when deficiencies are identified and related time frames for these actions. While HUD staff stated that they address PHA noncompliance through ongoing communication and technical assistance to PHAs, HUD has not documented specific actions to be performed by staff when deficiencies are identified. OFO staff told us that in general, PIH has not needed to take many enforcement actions because field offices are able to resolve most lead paint regulation compliance concerns with PHAs through ongoing communication and technical assistance. For example, HUD field offices sent letters to PHAs when Real Estate Assessment Center inspectors could not locate required lead inspection reports and lead disclosure forms, and requested that the PHA send the missing documentation within 30 days. However, OFO’s fiscal years 2015–2017 internal memorandums on monitoring and oversight guidance for HUD field offices did not contain detailed procedures, including time frames or criteria HUD staff would use to determine when to consider whether a more formal enforcement action might be warranted. Additionally, Lead Office staff said if efforts to bring a PHA into compliance are unsuccessful, the Lead Office would work in conjunction with PIH and HUD’s Office of General Counsel’s Departmental Enforcement Center to determine if an enforcement action is needed, such as withholding or delaying funds from a PHA or imposing civil money penalties on a PHA. Lead Office staff also told us that instead of imposing a fine on a PHA, HUD would rather work with the PHA to resolve the lead paint hazard. However, the Lead Office provided no documentation detailing the specific steps or time frames HUD staff would follow to determine when a noncompliance case is escalated to the Office of General Counsel. In a March 2018 report to Congress, HUD noted that children continued to test positive for lead in HUD-assisted housing in 2017. In the same report, HUD notes PIH and the Lead Office will continue to work with PHAs to ensure compliance with lead paint regulations. By adopting procedures that clearly describe when lead paint hazard compliance efforts are no longer sufficient and enforcement decisions are needed, HUD can better keep PHAs accountable in a consistent and timely manner. The standard HUD uses to identify children with elevated blood lead levels and initiate lead hazard control activities in its rental assistance aligns with the health guideline set by CDC in 2012. HUD also uses CDC’s health guideline in its lead grant programs. In HUD’s January 2017 amendment to the Lead Safe Housing Rule, HUD made its standard for lead in a child’s blood more stringent by lowering it from 20 micrograms to 5 micrograms of lead per deciliter of blood, matching CDC’s health guideline (i.e., blood lead reference value). Specifically, HUD’s stronger standard allows the agency to respond more quickly when children under 6 years old are exposed to lead paint hazards in voucher and public housing units. The January 2017 rule also established more comprehensive testing for children and evaluation procedures for HUD assisted housing. According to HUD’s press release that accompanied the rule, by aligning HUD’s standard with CDC’s guidance, HUD can respond more quickly in cases when a child who lives in HUD assisted housing shows early signs of lead in their blood. The 2017 rule notes HUD will revise the agency’s elevated blood lead level to align with future changes HHS may make to its recommended environmental intervention level. HUD’s standards for lead dust levels align with EPA standards for its rental assistance programs and exceed EPA standards for the lead grant programs. In 2001, EPA published a final rule on lead paint hazard standards, including lead dust clearance standards. The rule established standards to help property owners, contractors, and government agencies identify lead hazards in residential paint, dust, and soil and address these hazards in and around homes. Under these standards, lead is considered a hazard when equal to or exceeding 40 micrograms of lead in dust per square foot sampled on floors and 250 micrograms of lead in dust per square foot sampled on interior window sills. In 2004, HUD amended the Lead Safe Housing Rule to incorporate the 2001 EPA lead dust standards as HUD’s standards. Since this time, HUD has used EPA’s 2001 lead hazard standards in its rental assistance programs. In February 2017, HUD released policy guidance for its lead grantees requiring them to meet new and more protective requirements for identifying and addressing lead paint hazards in the lead grant programs than those imposed by EPA’s 2001 standards that HUD uses in the rental assistance programs. For example, the policy guidance requires grantees to consider lead dust a hazard on floors at 10 micrograms per square foot sampled (down from 40) and on window sills at 100 micrograms per square foot sampled (down from 250). The policy guidance noted that the new requirements are supported by scientific evidence on the adverse effects of lead exposure at low blood lead levels in children. Further, the policy guidance established a standard for porch floors––an area that EPA has not covered––because porch floors can be both a direct exposure source for children and a source of lead dust that can be tracked into the home. On December 27, 2017, the United States Court of Appeals for the Ninth Circuit ordered EPA to issue a proposed rule updating its lead dust hazard standard and the definition of lead-based paint within 90 days of the decision becoming final and a final rule within 1 year of the proposed rule. Because HUD’s Lead Safe Housing Rule generally defines lead paint hazards and lead dust hazards to mean the levels promulgated by EPA, if EPA changes its 2001 standards those new standards would be used in HUD’s rental assistance programs. On March 16, 2018, EPA filed a request to the court asking for clarification for when EPA is required to issue the proposed rule and followed up with a motion seeking clarification or an extension. In response to EPA’s motion, on March 26, 2018, the court issued an order clarifying time frames and ordered that the proposed rule be issued within 90 days from March 26, 2018. HUD’s Lead Safe Housing Rule requires a stricter lead inspection standard for public housing than for voucher units. According to HUD staff, HUD does not have the authority to require the more stringent inspection in the voucher program. While HUD has acknowledged that moving to a stricter inspection standard for voucher units would provide greater assurance that these units are lead-safe and expressed its plan to support legislative change to authorize it to impose a more stringent inspection standard, HUD has not requested authority from Congress to amend its inspection standard for the voucher program. For voucher units, HUD requires PHAs to ensure that trained inspectors conduct visual assessments to identify deteriorated paint for housing units inhabited by a child under 6 years old. In a visual assessment, an inspector looks for deteriorated paint and visible surface dust but does not conduct any testing of paint chips or dust samples from surfaces to determine the presence of lead in the home’s paint. By contrast, for public housing units, HUD requires a stronger inspection process. Lead- based paint inspections are required for pre-1978 public housing units. If that inspection identifies lead-based paint, PHAs must then perform a risk assessment. In a risk assessment, in addition to conducting a visual inspection, an inspector tests for the presence of lead paint by collecting and testing samples of paint chips and surface dust, and typically using a specialized device (an X-ray fluorescence analyzer) to measure the amount of lead in the paint on a surface, such as a wall, door, or window sill. Staff from HUD’s Lead Office and the Office of General Counsel told us that Title X did not include specific risk assessment requirements for voucher units, and HUD does not believe, therefore, that it has the statutory authority to require an assessment more thorough than a visual assessment of voucher units. As of May 2018, HUD had not requested statutory authority to change the visual assessment standard used in the voucher program. However, HUD previously acknowledged the limitation of the weaker inspection standard in a June 2016 publication titled Lead- Safe Homes, Lead-Free Kids Toolkit. In this publication, HUD noted its plans to support legislative change to strengthen lead safety in voucher units by eliminating reliance on visual-only inspections. Staff from HUD’s Lead Office and Office of General Counsel told us the agency recognizes that risk assessments are more comprehensive than visual assessments. The staff noted that, by definition, a risk assessment is a stronger inspection standard than a visual-only assessment because it includes additional identification and testing. In responding to a draft of this report, HUD cited the need to conduct and evaluate the results of a statistically rigorous study on the impacts of requiring a lead risk assessment versus a visual assessment, such as the impact on leasing times and the availability of housing for low-income families. HUD further noted that such a study could explore whether alternative options to the full risk assessment standard (such as targeted dust sampling) could achieve similar levels of protection for children in the voucher program. Requesting and obtaining authority to amend the standard for the voucher program would not preclude HUD from doing such a study. Such analysis might support a range of options based on consideration of health effects for children, housing availability, and other relevant factors. Because HUD’s Lead Safe Housing Rule contains a weaker lead inspection standard for the voucher program children living in voucher units may be less protected from lead paint hazards than children living in public housing. By requesting and obtaining statutory authority to amend the voucher program inspection standard, HUD would be positioned to take steps to ensure that children in the voucher program are provided better protection as indicated by analysis of the benefits and costs from amending the standard. HUD has taken limited steps to measure, evaluate, and report on the performance of its programmatic efforts to ensure that housing is lead- safe. First, HUD has tracked one performance measure for its lead grant programs but lacks comprehensive performance goals and measures. Second, while HUD has evaluated the effectiveness of its Lead-Based Paint Hazard Control grant program, it has not formalized plans and does not have a time frame for evaluating its lead paint regulations. Third, HUD has not issued an annual report on the results of its lead efforts since 1997. A key aspect to promoting improved federal management and greater efficiency and effectiveness is that agencies set goals and report on performance. We have previously reported that a program performance assessment contains three key elements––program goals, performance measures, and program evaluations (see fig. 5). In our prior work, we have noted that both the executive branch and congressional committees need evaluative information to help them make decisions about the programs they oversee––information that tells them whether, and why, a program is working well or not. Program goals and performance measures. HUD has tracked one performance measure for making private housing units lead-safe as part of its lead grant programs but lacks goals and performance measures that more fully cover the range of its lead efforts. In addition to our prior work on program goals and performance measures, federal internal control standards state that management should define objectives clearly and that defining objectives in measurable terms allows agency management to assess performance toward achieving objectives. According to Lead Office staff, HUD provides information on its goals and performance measures related to its lead efforts in the agency’s annual performance reports. For example, the fiscal year 2016 report contains information about the number of private housing units made lead-safe as part of HUD’s lead grant programs but does not include any performance measures on HUD’s lead efforts for the voucher and public housing programs. Lead Office staff told us HUD does not have systems to count the number of housing units made lead-safe in these two housing programs. The staff said the Lead Office and PIH recently began discussing whether data from an existing HUD database could be used to count units made lead-safe within these programs. However, they could not provide additional details on the status of all these efforts. Without comprehensive goals and performance measures, HUD does not know the results it is achieving with all its lead paint hazard reduction efforts. Moreover, HUD may be missing opportunities to use performance information to improve the results of its lead efforts. Program evaluations. HUD has evaluated the effectiveness of its Lead- Based Paint Hazard Control grant program but has not taken similar steps to evaluate the Lead Safe Housing Rule or Lead Disclosure Rule. As previously stated, our prior work on program performance assessment has noted the importance of program evaluations to know how well a program is working relative to its objectives. Additionally, Title X required HUD to conduct research to evaluate the long-term cost-effectiveness of interim lead hazard control and abatement strategies. For its Lead-Based Paint Hazard Control Grant program, HUD has contracted with outside experts to conduct evaluations. For example, the National Center for Healthy Housing and the University of Cincinnati’s Department of Environmental Health evaluated whether the lead hazard control methods used by grantees continued to be effective 1, 3, 6, and 12 years later. The evaluations concluded that the lead hazard control activities used by grantees substantially reduced lead dust levels and the original evaluation and those completed 1 and 3 years later were also associated with substantial declines in the blood lead levels of children living in the housing remediated using lead grant program funds. HUD has general plans to conduct evaluations of the Lead Safe Housing Rule and the Lead Disclosure Rule, but Lead Office and PD&R staff said they did not know when or if the studies will begin. In a 2016 publication, HUD noted its plans to evaluate the Lead Safe Housing Rule requirements and noted that such an evaluation would contribute toward policy recommendations and program improvements. Additionally, in its 2017 Research Roadmap, PD&R outlined HUD’s plans for two studies to evaluate the effectiveness of requirements within the Lead Safe Housing and Lead Disclosure Rules. However, PD&R and Lead Office staff were not able to provide a time frame for when the studies would begin. PD&R staff told us that the plans noted within the Research Roadmap were HUD’s first step in research planning and prioritization but that appropriations for research have been prescriptive in recent years (i.e., tied to specific research topics) and fell short of the agency’s research needs. By studying the effectiveness of requirements included within the Lead Safe Housing and Lead Disclosure Rules, including the cost- effectiveness of the various lead hazard control methods, HUD could have more complete information to assess how effectively it uses federal dollars to make housing units lead-safe. Reporting. HUD has not reported on its lead efforts as required since 1997. Title X includes annual and biennial reporting requirements for HUD. Staff from HUD’s Lead Office and General Counsel told us that in 1998 the agency agreed with the congressional committees of jurisdiction that HUD could satisfy this reporting requirement by including the required information in its annual performance reports. Lead Office staff told us HUD’s recent annual performance reports do not contain specific information required by law and that HUD has not issued other publicly available reports that contain the Title X reporting requirements. Title X requires HUD to annually provide Congress information on its progress in implementing the lead grant programs; a summary of studies looking at the incidence of lead poisoning in children living in HUD-assisted housing; the results of any required lead technical studies; and estimates of federal funds spent on lead hazard evaluation and reduction in HUD-assisted housing. As previously stated, the annual performance reports have provided information on the number of housing units made lead-safe through the agency’s lead grant programs, but not through the voucher or public housing programs. In March 2018, Lead Office staff told us HUD plans to submit separate reports on the agency’s lead effort, covering the Title X reporting requirements, starting in fiscal year 2019. By HUD complying with Title X statutory reporting requirements, Congress and the public will be in a position to better know the progress HUD is making toward ensuring that housing is lead-safe. Lead exposure can cause serious, irreversible cognitive damage that can impair a child for life. Through its lead grant programs and oversight of lead paint regulations, HUD is helping to address lead paint hazards in housing. However, our review identified specific areas where HUD could improve the effectiveness of its efforts to identify and address lead paint hazards and protect children in low-income housing from lifelong health problems: Documenting and evaluating grant processes. HUD could improve documentation for its lead grant programs’ processes by providing more specific direction to staff and documenting grant award rationale. In doing so, HUD could better ensure that grant program staff score grant applications consistently and appropriately and provide greater transparency about its award decisions. Additionally, periodically evaluating its grant processes and procedures could help HUD better ensure that its lead grants reach areas most at risk for lead paint hazards. Identifying areas at risk for lead hazards. By developing specific time frames to finalize and incorporate the results of its model to more fully identify areas at risk for lead paint hazards, HUD can better identify and conduct outreach to at-risk localities that its lead grant programs have not yet reached. Overseeing compliance with lead paint regulations. False self- certifications of compliance by some PHAs and other limitations in HUD’s compliance monitoring approach make it essential for HUD to develop a plan to mitigate and address limitations, as well as establish procedures to determine when enforcement decisions are needed. These actions could further strengthen HUD’s oversight and keep PHAs accountable for ensuring that housing units are lead-safe. Amending inspection standard in the voucher program. Children living in voucher units may receive less protection from lead paint hazards than children living in public housing units because HUD applies different lead inspection standards to the two programs. HUD could ensure that children in the voucher program are provided better protection from lead by requesting and obtaining statutory authority to amend the voucher program inspection standard as indicated by analysis of the benefits and costs of amending the standard. Assessing and reporting on performance. Fully incorporating key elements of performance assessment—by developing comprehensive goals, improving performance measures, and adhering to reporting requirements—could better enable HUD to assess its own progress and target its resources toward lead efforts that maximize impact. Additionally, HUD may be missing opportunities to inform the Congress and the public about how HUD’s lead efforts have helped reduce lead poisoning in children. We are making the following nine recommendations to HUD: The Director of HUD’s Lead Office should ensure that the office more fully documents its processes for scoring and awarding lead grants and its rationale for award decisions. (Recommendation 1) The Director of HUD’s Lead Office should ensure that the office periodically evaluates its processes for scoring and awarding lead grants. (Recommendation 2) The Director of HUD’s Lead Office, in collaboration with PD&R, should set time frames for incorporating relevant data on lead paint hazard risks into the lead grant programs’ processes. (Recommendation 3) The Director of HUD’s Lead Office and the Assistant Secretary for PIH should collaborate to establish a plan to mitigate and address risks within HUD’s lead paint compliance monitoring processes. (Recommendation 4) The Director of HUD’s Lead Office and the Assistant Secretary for PIH should collaborate to develop and document procedures to ensure that HUD staff take consistent and timely steps to address issues of PHA noncompliance with lead paint regulations. (Recommendation 5) The Secretary of HUD should request authority from Congress to amend the inspection standard to identify lead paint hazards in the Housing Choice Voucher program as indicated by analysis of health effects for children, the impact on landlord participation in the program, and other relevant factors. (Recommendation 6) The Director of the Lead Office should develop performance goals and measures to cover the full range of HUD’s lead efforts, including its efforts to ensure that housing units in its rental assistance programs are lead-safe. (Recommendation 7) The Director of the Lead Office, in conjunction with PD&R, should finalize plans and develop a time frame for evaluating the effectiveness of the Lead Safe Housing and Lead Disclosure Rules, including an evaluation of the long-term cost effectiveness of the lead remediation methods required by the Lead Safe Housing Rule. (Recommendation 8) The Director of the Lead Office should complete statutory reporting requirements, including but not limited to its efforts to make housing lead-safe through its lead grant programs and rental-assistance programs, and make the report publicly available. (Recommendation 9) We provided a draft of this report to HUD for review and comment. We also provided the relevant excerpts of the draft report to CDC and EPA for their review and technical comments. In written comments, reproduced in appendix III, HUD disagreed with one of our recommendations and generally agreed with the remaining eight. HUD and CDC also provided technical comments, which we incorporated as appropriate. EPA did not have any comments on the relevant excerpts of the draft report provided to them. In its general comments, HUD noted that the lead grant programs and HUD’s compliance assistance and enforcement of lead paint regulations have contributed significantly to, among other things, the low prevalence of lead-based paint hazards in HUD-assisted housing. Further, HUD said the lead grant programs and compliance assistance and enforcement of lead paint regulations have played a critical part in developing and maintaining the national lead-based paint safety infrastructure. HUD asked that this contextual information be included in the background of the report. The draft report included detailed information on the purpose and scope of HUD’s lead grant programs, two key regulations related to lead paint hazards, and efforts to make housing lead-safe. Furthermore, the draft report provided context on other federal agencies’ role in establishing relevant standards and guidelines for lead paint hazards. We made no changes in response to this comment because we did not think it was necessary for background purposes. HUD disagreed with the draft report’s sixth recommendation to request authority from Congress to use the risk assessment inspection standard to identify lead paint hazards in the Housing Choice Voucher program. As discussed in the report, HUD’s Lead Safe Housing Rule requires a more stringent lead inspection standard (risk assessments) for public housing than for Housing Choice Voucher units, for which a weaker inspection standard is used (visual assessments). In its written comments, HUD said that before deciding whether to request the statutory authority to implement risk assessments for voucher units, it would need to conduct and evaluate the results of a statistically rigorous study on the impacts of requiring a lead risk assessment versus a visual assessment, such as the impact on leasing times and the availability of housing for low-income families. HUD further noted that such a study could explore whether alternative options to the full risk assessment standard (such as targeted dust sampling) could achieve similar levels of protection for children in the voucher program. We note that requesting and obtaining authority to amend the standard for the Housing Choice Voucher program would not preclude HUD from doing such a study. We acknowledge that the results of such a study might support a range of options. Therefore, we revised our recommendation to provide HUD with greater flexibility in how it might amend the lead inspection standard for the voucher program based on consideration of not only leasing time and availability of housing, as HUD emphasized in its written comments, but also based on the health effects on children. The need for HUD to review the lead inspection standard for the voucher program is underscored by the greater number of households with children served by the voucher program compared to public housing, as well as recent information indicating that more children with elevated blood lead levels are living in voucher units than in public housing. HUD generally agreed with our remaining eight recommendations and provided specific information about planned steps and other considerations related to implementing them. For example, in response to our first three recommendations on the lead grant programs, HUD outlined specific steps it plans to take, such as updating its guidance for scoring grant applications and reviewing its grant application scoring methods to identify potential improvements. In response to our fourth and fifth recommendations to the Director of HUD’s Lead Office on compliance monitoring and enforcement of lead paint regulations, HUD noted that PIH should be the primary office for these recommendations with the Lead Office providing support. While these recommendations had already recognized the need for the Lead Office to collaborate with PIH, we reworded them to clarify that it is not necessary for the Lead Office to have primary responsibility for their implementation. HUD generally agreed with our seventh and eighth recommendations, but noted some considerations for implementing them. For our seventh recommendation about performance goals and measures, HUD noted that it will re-examine the availability of information from the current housing databases to determine whether data on housing unit production can be added to the existing data collected. HUD noted if that information is not sufficient, it would need to obtain Office of Management and Budget approval and have sufficient funds for such an information technology project. For our eighth recommendation about evaluating the Lead Safe Housing and Lead Disclosure Rules, HUD noted if its own resources are insufficient, the time frame for implementing this recommendation may depend on the availability of funding for contracted resources. Finally, in response to our ninth recommendation, HUD said that it will draft and submit annual and biennial reports to the congressional authorizing and appropriations committees and then post the reports on the Lead Office’s public website. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Housing and Urban Development, the Administrator of the Environmental Protection Agency, and the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Under the Department of Housing and Urban Development’s (HUD) Lead-Based Paint Hazard Control and the Lead Hazard Reduction Demonstration grant programs, HUD competitively awards grants to state and local jurisdictions, as authorized by the Residential Lead-Based Paint Hazard Reduction Act (Title X of the Housing and Community Development Act of 1992). Title X requires each grant recipient to make matching contributions with state, local, and private funds (i.e., nonfederal) toward the total cost of activities. For the Lead-Based Paint Hazard Control grant and the Lead Hazard Reduction Demonstration grant programs, the matching contribution has been set at no less than 10 percent and 25 percent, respectively, of the total grant amount. For example, if the total grant amount is $3 million, then state or local jurisdictions must provide at least $300,000 and $750,000, respectively, for each grant program, in additional funding toward the cost of activities. HUD requires lead grant applicants to include information on the sources and amounts of grantees’ matching contributions as part of their grant applications. Additionally, Title X requires HUD to award grants in part based on an applicant’s ability to leverage state, local, and private funds to supplement the federal grant funds. To identify the nonfederal funding sources grantees used in the lead hazard control grants, we selected and reviewed the lead grant applications of 20 HUD grantees and interviewed representatives from 10 of these. We selected these grantees based on their geographic locations; the number of HUD lead grants they had previously received; experience with HUD’s lead hazard control grants; and whether they have received both grants from 2013 through 2017. Grantees we selected included entities at the state, municipality, and county levels. Information from our grant application reviews and interviews of grantees cannot be generalized to all HUD grantees. Based on our review of the selected grant applications and interviews of selected grantees, we found that grantees planned to use the following types of nonfederal funding sources as their matching contributions to support their lead grants activities: State and local funds. Eighteen of the 20 grantees we selected noted that they planned to use state or local funding sources to supplement HUD’s grant funds. The state and local funding sources included state or local general funds and local property taxes or fees. For example, grantees in Connecticut, Baltimore, and Philadelphia used state or local general funds to cover personnel and operating costs. Additionally, grantees in Alameda County (California), Hennepin County (Minnesota), Malden, St. Louis, and Winnebago County (Illinois) planned to use local taxes, including property taxes or fees, such as real estate recording and building permit fees, to cover some costs associated with their lead hazard control grants activities. Community Development Block Grant funds. Ten of the 20 grantees we selected indicated that they planned to use Community Development Block Grant (CDBG) program funds to cover part of the costs of their lead hazard control grants. CDBG program funds can be used by states and local communities for housing; economic development; neighborhood revitalization; and other community development activities. For example, grantees in Baltimore and Memphis noted in their grant applications that they planned to use the funds to cover costs related to personnel, operations, and training. Nongovernmental contributions or discounts. Eight of 20 grantees we selected stated that they anticipated some forms of nongovernmental contributions from nonprofit organizations or discounts from contractors to supplement the lead grants. For example, all eight grantees stated that they expected to receive matching contributions from nonprofit organizations. Table 2 summarizes the nonfederal funds by source that the 20 selected grantees planned to use, based on our review of these grantees’ applications. Furthermore, almost all of the selected grantees stated in their grant applications or told us that they expected to receive or have received other nonfederal funds in excess of their matching contributions. For example, 15 grantees stated that they generally required or encouraged property owners or landlords to contribute toward the lead hazard remediation costs. Also, grantees in Baltimore, District of Columbia, Lewiston, and Providence indicated that they expected to receive monetary or in-kind donations from organizations to help carry out lead hazard remediation, blood lead-level testing, or training. Additionally, the grantee in Alameda County (California) told us that they have received nonfederal funds from a litigation settlement with a private paint manufacturer. This report examines the Department of Housing and Urban Development’s (HUD) efforts to (1) incorporate statutory requirements and other relevant federal standards in its lead grant programs; (2) monitor and enforce compliance with lead paint regulations for its rental assistance programs; (3) adopt federal health guidelines and environmental standards for lead hazards in its lead grant and rental assistance programs; and (4) measure and report on its performance related to making housing lead-safe. In this report, we examine lead paint hazards in housing, and we focus on HUD’s lead hazard control grant programs and its two largest rental assistance programs that serve the most families with children: the Housing Choice Voucher (voucher) and public housing programs. To address all four objectives, we reviewed relevant laws, such as the Residential Lead-Based Paint Hazard Reduction Act (Title X of the Housing and Community Development Act of 1992, referred to as Title X throughout this appendix) and relevant HUD regulations, such as the Lead Safe Housing Rule and a January 2017 amendment to this rule. To examine trends in funding for HUD’s lead grant programs for the past 10 years, we also reviewed HUD’s budget information for fiscal years 2008 through 2017. We interviewed HUD staff from the Office of Lead Hazard Control and Healthy Homes (Lead Office), Office of Public and Indian Housing (PIH), Office of Policy Development and Research (PD&R), and other relevant HUD program and field offices. Finally, we reviewed our prior work and those of HUD’s Office of Inspector General. To address the first objective, we reviewed HUD’s Notices of Funding Availability (funding notices), policies, and procedures to identify HUD’s grant award processes for the Lead-Based Paint Hazard Control grant and Lead Hazard Reduction Demonstration grant programs. For example, we reviewed HUD’s annual notices of funding availability from 2013 through 2017 to identify HUD’s scoring factors for evaluating grant applications. We compared HUD’s grant award processes in 2017 with Title X statutory requirements, the Office of Management and Budget (OMB) requirements for awarding federal grants, and relevant federal internal control standards. We also interviewed HUD staff about the agency’s grant application review and award processes. To determine the extent to which HUD’s grants have gone to counties in the United States potentially at high risk for lead paint hazards, we compared grantee locations from HUD’s lead grant data for grants awarded from 2013 through 2017 with county-level data on two indicators of lead paint hazard risk from the 2011–2015 American Community Survey—a continuous survey of households conducted by the U.S. Census Bureau. We analyzed HUD’s grant data to determine the number and dollar amount of grants received by each grantee, and the grantees’ addresses. We then conducted a geographic analysis to determine whether each HUD lead grant went to a county that met at least one, both, or neither of the two commonly known indicators of lead paint hazard risk—the age of housing and poverty level. We identified these two indicators through a review of relevant academic literature, agency research, and state lead modelling methodologies. We used data from the 2011–2015 American Community Survey because the data covered a time frame that best aligned with the 5 years of lead grant data (2013 through 2017). Using its county-level data, we calculated an estimated average percentage nationwide of housing units built before 1980 (56.9 percent) and an estimated average percentage nationwide of individuals living below the poverty level (17.5 percent). We used 1980 as a benchmark for age of housing because the American Community Survey data for age of housing is separated by the decade of construction and 1980 was closest in time to the 1978 federal lead paint ban. We categorized counties based on whether their levels of pre-1980 housing and poverty were above one, both, or neither of the respective national average percentage for each indicator. The estimated average nationwide and county-level percentages of the two indicators (e.g., older housing and poverty rate) are expressed as a range of values. For the lower and upper ends of the range, we generated a 95 percent confidence interval that was within plus or minus 20 percentage points. We classified a county as above the estimated average percentages nationwide if the county’s confidence interval was higher and did not overlap with the nationwide estimate’s confidence interval. We omitted the data for 12 counties that we determined were unreliable for our purposes. We analyzed data starting in 2013 because that was the first year for which these grant data were available electronically. We also interviewed HUD staff to understand their efforts and plans to perform similar analyses using indicators of lead paint hazard risk. To assess the reliability of HUD’s grant data, we reviewed documentation of HUD’s grant database, interviewed Lead Office staff on the processes HUD used to collect and ensure the reliability of the data, and tested the data for missing values, outliers, and obvious errors. To assess the reliability of the American Community Survey data, we reviewed statistical information from the Census Bureau and other publicly available documentation on the survey and conducted electronic testing of the data. We determined that the HUD grant data and American Community Survey county-level data on age of housing and poverty were sufficiently reliable for identifying areas at risk of lead paint hazards and determining the extent to which lead grants from 2013 through 2017 have gone to at-risk areas. Furthermore, to obtain information about how HUD works with grantees to achieve program objectives, we conducted in-person site visits to five grantees located in five localities (Alameda County, California; Atlanta, Georgia; Baltimore, Maryland; District of Columbia; and San Francisco, California); and interviewed an additional five grantees on the telephone (Hennepin County, Minnesota; Lewiston, Maine; Malden, Massachusetts; Providence, Rhode Island; and Winnebago County, Illinois). In addition, we reviewed the grant applications of the 10 grantees we spoke to and an additional 10 grantees from 10 additional jurisdictions (State of Connecticut; Cuyahoga County, Ohio; Denver, Colorado; Monroe County, New York; Philadelphia, Pennsylvania; Memphis, Tennessee; San Antonio, Texas; St. Louis, Missouri; Tucson, Arizona; and State of Vermont). We selected the 10 grantees for site visits or interviews based on the following criteria: geographic variation, number of years the grantees had HUD’s lead grants, and grantees that have received both types of lead grants from 2013 through 2017. We selected the 10 additional grantees’ applications for review based on geographic diversity and to achieve a total of two applications for each year during our 5-year time frame, with at least one application from each of the two HUD lead grant programs. As part of our review of selected grant applications, we identified nonfederal funding sources used by grantees, such as local tax revenues, contractor discounts, and property owner contributions. Information from the selected grantees and grant applications review cannot be generalized to those grantees we did not include in our review. Additionally, we interviewed representatives from housing organizations to obtain additional examples of any nonfederal funding sources, such as state or local bond measures, or low-interest loans to homeowners. To address the second objective, we also reviewed HUD guidance and internal memorandums related to its efforts to monitor and enforce compliance with lead paint regulations for public housing agencies (PHA), the entities that manage HUD’s voucher and public housing rental assistance programs. In addition, we reviewed HUD’s documentation of databases it uses to monitor compliance, including the Lead-Based Paint Response Tracker and the Elevated Blood Lead Level Tracker, and observed HUD staff’s demonstrations of these databases. HUD staff also provided a demonstration of the Record and Process Inspection Data database (known as “RAPID”) used by HUD’s Real Estate Assessment Center to collect physical inspection data for public housing units. We obtained and reviewed information from HUD about instances of potential noncompliance with lead paint regulations by PHAs as of November 2017 and enforcement actions HUD has taken. We compared HUD’s regulatory compliance monitoring and enforcement approach to federal internal control standards. We interviewed staff from HUD’s Lead Office, Office of General Counsel, Office of Field Operations, and field staff, including four HUD regional directors in areas of the country known to have a high prevalence of lead paint hazards, about internal procedures for monitoring and enforcing compliance with lead paint regulations by the PHAs within their respective regions. To address the third objective on HUD’s adoption of federal health guidelines and environmental standards for lead paint hazards in its lead grant and rental assistance programs, we reviewed relevant rules and HUD documentation. To identify relevant federal health guidelines and environmental standards, we reviewed guidelines and regulations from the Centers for Disease Control and Prevention (CDC) and the Environmental Protection Agency (EPA) and interviewed staff from each agency. To identify state and local laws with different requirements than these federal guidelines and standards, we obtained information from and interviewed staff from CDC’s Public Health Law Program and the National Conference of State Legislatures. We compared HUD’s requirements to CDC’s health guideline known as the “blood lead reference value” and EPA’s standards for lead-based paint hazards and lead-dust clearance standards. Finally, we reviewed information in HUD’s 2017 funding notices and lead grant programs’ policy guidance about requirements for grantees as they pertain to health guidelines and environmental standards. We also interviewed HUD staff about how HUD has used the findings from lead technical study grants to consider changes to HUD’s requirements and processes regarding identifying and addressing lead paint hazards for the grant programs. To address the fourth objective, we reviewed HUD documentation related to performance goals and measures, program evaluations, and reporting. For example, we reviewed HUD’s recent annual performance reports to identify goals and performance measures related to HUD’s efforts to make housing lead-safe. Further, we reviewed Title X to identify requirements related to evaluating and reporting on HUD’s lead efforts. We reviewed program evaluations and related studies completed by outside experts for the lead grant programs and interviewed staff from one of the organizations that conducted the evaluations. In addition, we interviewed Lead Office and PD&R staff about the agency’s plans to evaluate the requirements in the Lead Safe Housing Rule and reviewed corresponding agency documentation about these plans. Additionally, we reviewed the Lead Office’s most recent strategic plan (2009) and annual report (1997) on the agency’s lead efforts. We compared HUD’s use of performance goals and measures, program evaluations, and reporting against leading practices for assessing program performance and federal internal control standards. Finally, we interviewed staff from HUD to understand goals and performance measures used by the agency to assess their lead efforts. We conducted this performance audit from March 2017 to June 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, John Fisher (Assistant Director), Beth Faraguna (Analyst in Charge), Enyinnaya David Aja, Farah Angersola, Carol Bray, William R. Chatlos, Anna Chung, Melinda Cordero, Elizabeth Dretsch, Christopher Lee, Marc Molino, Rebecca Parkhurst, Tovah Rom, Tyler Spunaugle, and Sonya Vartivarian made key contributions to this report.

 

 Report 2:
GPRAMA significantly enhances GPRA, the centerpiece of a statutory framework that Congress put in place during the 1990s to help resolve longstanding performance and management problems in the federal government and provide greater accountability for results. Congress passed GPRAMA in 2010 to address a number of persistent federal performance challenges, including focusing attention on crosscutting issues and enhancing the use and usefulness of performance information. OMB and agencies are to establish various government-wide and agency-specific performance goals, in line with GPRAMA requirements or OMB guidance. These include the following: Cross-agency priority (CAP) goals: CAP goals are crosscutting and include outcome-oriented goals covering a limited number of policy areas as well as goals for management improvements needed across the government. OMB is to coordinate with agencies to establish CAP goals at least every 4 years. OMB is also required to coordinate with agencies to develop annual federal government performance plans to, among other things, define the level of performance to be achieved toward the CAP goals. Strategic objectives: A strategic objective is the outcome or impact the agency is intending to achieve through its various programs and initiatives. Agencies establish strategic objectives in their strategic plans and may update the objectives during the annual update of performance plans. Agency priority goals (APG): At the agency level, every 2 years, GPRAMA requires that the heads of certain agencies, in consultation with OMB, identify a subset of agency performance goals as APGs. These goals are to reflect the agencies’ highest priorities. They should be informed by the CAP goals as well as consultations with relevant congressional committees and other interested parties. In a schedule established by GPRAMA, OMB and agencies are to develop and publish new CAP goals, APGs, and strategic plans (with updated strategic objectives) in February 2018. GPRAMA and related OMB guidance require agencies to regularly assess their progress in achieving goals and objectives through performance reviews. Data-driven reviews: Agency leaders and managers are to use regular meetings, at least quarterly, to review data and drive progress toward key performance goals and other management-improvement priorities. For each APG, GPRAMA requires agency leaders to conduct reviews at least quarterly to assess progress toward the goal, determine the risk of the goal not being met, and develop strategies to improve performance. Similarly, the Director of OMB, with relevant parties, is to review progress toward each CAP goal. Strategic reviews: OMB guidance directs agency leaders to annually assess progress toward achieving each strategic objective using a broad range of evidence. GPRAMA establishes certain senior leadership positions and a council, as described below. Chief Operating Officer (COO): The deputy agency head, or equivalent, is designated COO, with overall responsibility for improving agency management and performance. Performance Improvement Officer (PIO): Agency heads are to designate a senior executive within the agency as the PIO. The PIO reports directly to the COO and assists the agency head and COO with various performance management activities. Goal leaders: Goal leaders are responsible for developing strategies to achieve goals, managing execution, and regularly reviewing performance. GPRAMA requires goal leaders for CAP goals and agency performance goals, including APGs. OMB guidance directs agencies to designate goal leaders for strategic objectives. Performance Improvement Council (PIC): The PIC is charged with assisting OMB to improve the performance of the federal government and achieve the CAP goals. The PIC is chaired by the Deputy Director for Management at OMB and includes agency PIOs from each of the 24 CFO Act agencies as well as other PIOs and individuals designated by the chair. Among its responsibilities, the PIC is to work to resolve government-wide or crosscutting performance issues, and facilitate the exchange among agencies of practices that have led to performance improvements within specific programs, agencies, or across agencies. GPRAMA includes several provisions related to providing the public and Congress with information, as described below. Performance.gov: GPRAMA calls for a single, government-wide performance website to communicate government-wide and agency performance information. Among other things, the website— implemented by OMB as Performance.gov—is to include (1) quarterly progress updates on CAP goals and APGs; (2) an inventory of all federal programs; and (3) agency strategic plans, annual performance plans, and annual performance reports. Reporting burden: GPRAMA establishes a process to reexamine the usefulness of certain existing congressional reporting requirements. Specifically, GPRAMA requires an annual review (including congressional consultation), based on OMB guidance, of agencies’ reporting requirements to Congress. Additionally, OMB is to include in the budget a list of plans and reports determined to be outdated or duplicative and may submit legislation to eliminate or consolidate such plans or reports. In early 2017, the administration announced several efforts that are intended to improve government performance. The 2018 Budget Blueprint states that the President’s Management Agenda will seek to improve the federal government’s effectiveness by using evidence-based approaches, balancing flexibility with accountability to better achieve results, improving mission support functions, and developing and monitoring critical performance measures. In addition, OMB issued several memoranda detailing the administration’s plans to improve government performance by reorganizing the government, reducing the federal workforce, and reducing federal agency burden. A number of these efforts, which are to leverage GPRAMA and our past work, have the potential to further progress in addressing key governance challenges. As part of reorganization efforts, OMB and agencies are developing government-wide and agency reform plans, respectively, that are to leverage various GPRAMA provisions. For example, an April 2017 memorandum states that OMB intends to monitor implementation of the reform plans using CAP goals, APGs, annual strategic reviews, and Performance.gov. The government-wide plan also is to include crosscutting reform proposals, such as merging agencies or programs that have similar missions. To that end, the memorandum states agencies should consider our reports, including our work on fragmentation, overlap, and duplication, as well as inspectors general reports. Many of the meaningful results that the federal government seeks to achieve, such as those related to ensuring public health, providing homeland security, and promoting economic development, require the coordinated efforts of more than one federal agency, level of government, or sector. For more than 2 decades, we have reported on agencies’ missed opportunities for improved collaboration through the effective implementation of GPRA and, more recently, GPRAMA. Our reports also have demonstrated that collaboration across agencies is critical to address issues of fragmentation, overlap, and duplication as well as many of the areas on our High-Risk List. Fragmentation, Overlap, and Duplication: Since 2011, our annual reports have identified 133 crosscutting areas that require the coordinated effort of more than one federal organization, level of government, or sector. For instance, for the area of federal grant awards, we found in January 2017 that the National Park Service (NPS), Fish and Wildlife Service, Food and Nutrition Service, and Centers for Disease Control and Prevention (CDC) had not established guidance and formal processes to ensure their grant-management staff review applications for potential duplication and overlap among grants in their agencies before awarding. We recommended that these agencies do so, and they agreed. As of August 2017, these agencies had taken several actions to address the recommendation. For example, the Department of the Interior (Interior) provided documentation showing that the Fish and Wildlife Service now requires discretionary grant applicants to provide a statement that addresses whether there is any overlap or duplication of proposed projects or activities to be funded by the grant. Fish and Wildlife also updated its guidance to grant awarding offices instructing them to perform a potential overlap and duplication review of all selected applicants prior to award. Our Action Tracker provides details on the status of actions from our annual reports. Within the 133 crosscutting areas, since 2011 we have identified 315 targeted actions where opportunities exist to better manage fragmentation, overlap, and duplication, including 29 new actions in our most recent report issued in April 2017. We found that the executive branch and Congress addressed 145 (46 percent) of the 315 actions. For example, in November 2014, we recommended that the U.S. Coast Guard and Consumer Product Safety Commission establish a formal approach to coordination (such as a memorandum of understanding) to facilitate information sharing; better leverage their resources; and address challenges, including those related to fragmentation and overlap that we identified. In response to this recommendation, the two agencies signed a formal policy document to govern their coordination in May 2015. This policy document outlined procedures for determining jurisdictional authority for recreational boat-associated equipment and marine safety items. Specifically, the procedures clarified that upon receiving notice of a possible defect, the agency receiving such notice shall determine whether the item properly falls within its jurisdiction, and if not, initiate discussions to determine the appropriate jurisdiction. These new procedures should help the agencies share information and leverage each other’s resources so they can better ensure that recreational boat-associated equipment and marine safety items are fully regulated. However, more work is needed on the remaining 170 actions (54 percent) that have not been fully addressed. For example, in July 2016, we reported that four federal agencies—the Departments of Defense, Education, Health and Human Services, and Justice—manage at least 10 efforts to collect data on sexual violence, which differ in target population, terminology, measurements, and methodology. We found that data collection efforts use 23 different terms to describe sexual violence. Data collection efforts also differed in how they categorized particular acts of sexual violence, the context in which data were collected, data sources, units of measurement, and time frames. We recommended that OMB convene an interagency forum to better manage fragmentation of efforts to collect sexual violence data. In commenting on that report, OMB stated it would consider implementing the action in the future but did not believe it was the most effective use of resources at that time, in part because the agencies were not far enough along in their research. In response, we stated that given the number of federal data collection efforts, the range of differences across them, and the potential for causing confusion, it would be beneficial for agencies to discuss these differences and determine whether they are, in fact, necessary. As of July 2017, OMB had not provided an update on the status of this recommendation. High-Risk List: Since the early 1990s, our high-risk program has focused attention on government operations with greater vulnerabilities to fraud, waste, abuse, and mismanagement or that are in need of transformation to address economy, efficiency, or effectiveness challenges. As of February 2017, there were 34 high-risk areas covering a wide range of issues including human capital management, modernizing the U.S. financial regulatory system, and ensuring the security of federal information systems and cyber critical infrastructure. Many of these high- risk areas require a coordinated response from more than one branch of government, agency, or sector. In the time between our 2015 and 2017 High-Risk Updates, many of these high-risk areas on our list demonstrated solid progress. During that period, 15 high-risk areas fully met at least one of the five criteria required for removal from the High-Risk List. In many cases, progress was possible through the joint efforts of Congress and leadership and staff in agencies. For example, Congress passed over a dozen laws following our 2015 High-Risk Update to help address high-risk issues. In addition, in 2017, we removed one high-risk area on managing terrorism-related information, because significant progress had been made to strengthen how intelligence on terrorism, homeland security, and law enforcement is shared among federal, state, local, tribal, international, and private sector partners. Despite this progress, continued oversight and attention is also warranted given the issue’s direct relevance to homeland security as well as the constant evolution of terrorist threats and changing technology. Our February 2017 High-Risk Update also highlighted a number of long- standing high-risk areas that require additional attention. We also added three new crosscutting areas to incorporate the management of federal programs that serve tribes and their members, the government’s environmental liabilities, and the 2020 decennial census. Based on our body of work on federal programs that serve tribes and their members, we concluded that federal agencies had (1) ineffectively administered Indian education and health care programs and (2) inefficiently fulfilled their responsibilities for managing the development of Indian energy resources. For example, we identified numerous challenges facing Interior’s Bureau of Indian Education (BIE) and Bureau of Indian Affairs, and the Department of Health and Human Services’ (HHS) Indian Health Service (IHS), in administering education and health care services. We concluded that these challenges put the health and safety of American Indians served by these programs at risk. In May 2017, we issued two additional reports on accountability for school construction and safety at schools funded by BIE. Although these agencies have taken some actions to address recommendations we made related to Indian programs, about 50 recommendations have yet to be fully resolved. We are monitoring federal efforts to address the unresolved recommendations. We also are reviewing IHS’s workforce, and tribal nations’ management and use of their energy resources. Many of the crosscutting areas highlighted by our annual reports on fragmentation, overlap, and duplication and designated as high-risk would benefit from enhanced collaboration among the federal agencies involved in them. GPRAMA establishes a framework aimed at taking a more crosscutting and integrated approach to focusing on results and improving government performance. Our survey results and past work demonstrate that agencies continue to face difficulties when working together on crosscutting issues, but also that implementing certain GPRAMA requirements can have a positive effect on collaboration. An item related to coordination in our survey of federal managers is statistically significantly lower in 2017, relative to our previous survey in 2013 and our initial survey in 1997. In 2017, an estimated 43 percent of managers agreed that they use information obtained from performance measurement to a great or very great extent when coordinating program efforts with internal or external organizations (compared to an estimated 50 percent in 2013 and an estimated 57 percent in 1997). Moreover, our past work has found that agencies face a variety of challenges when working across organizational boundaries to deliver programs and improve performance. For example, our work has found that interagency groups have, at times, encountered difficulty clarifying roles and responsibilities or developing shared outcomes and performance measures. In contrast, our past work demonstrates that implementing GPRAMA provisions can improve collaboration. For example, in May 2016, we found that OMB and the PIC updated the governance structure for CAP goals to include both agency-level and Executive Office of the President goal leaders and held regular, senior-level reviews on CAP goal progress. Moreover, CAP goal teams told us that the CAP goal designation increased leadership attention and improved interagency collaboration on their crosscutting issues. Furthermore, our prior work has found that priority goals and related data-driven reviews have also been used to help manage crosscutting issues and enhance collaboration. Various GPRAMA requirements are aimed at improving agencies’ coordination of efforts to address crosscutting issues. As with our 2013 survey, our 2017 survey continues to show that CAP goals, APGs, and related data-driven reviews—also called quarterly performance reviews (QPR)—are associated with reported higher levels of collaboration with internal and external stakeholders. For example, our 2017 survey data indicate that about half of federal managers (an estimated 54 percent) reported they were somewhat or very familiar with CAP goals. Among these individuals, those who viewed their programs as contributing to CAP goals to a great or very great extent (36 percent) were more likely to report collaborating outside their program to a great or very great extent to help achieve CAP goals (62 percent), as shown in figure 2. Our analysis shows a similar pattern exists for APGs and QPRs. Our past work also has highlighted ways in which OMB and agencies could better implement GPRAMA’s crosscutting provisions—many of which have been addressed. A continued focus on fully and effectively implementing these provisions will be important as OMB and agencies establish new CAP goals and APGs, and assess progress toward them through related QPRs. Cross-agency priority (CAP) goals: In May 2012 and June 2013, we found that OMB had not always identified relevant agencies and program activities as contributors to the initial set of CAP goals. OMB took actions in response to our recommendations to include relevant contributors. Our most recent review, in May 2016, found that all relevant contributors had been identified for a subsequent set of CAP goals. In that report, we also found that OMB and the PIC had improved implementation of the CAP goals, in part, by helping agencies build their capacity to contribute to implementing the goals. Appendix II summarizes our past recommendations related to GPRAMA and the actions agencies have taken to address them. Agency priority goals (APGs): In April 2013, we found that agencies did not fully explain the relationship between their APGs and crosscutting efforts. Identify contributors: Similar to OMB’s responsibilities with the CAP goals, agencies are to identify the various organizations and programs that contribute to each of their performance goals, including APGs. We found that agencies identified internal contributors for their APGs, but did not list external contributors in some cases. We recommended that the Director of OMB ensure that agencies adhere to OMB’s guidance for website updates by providing complete information about the organizations, program activities, regulations, tax expenditures, policies, and other activities—both within and external to the agency— that contribute to each APG. In response, in April 2015, OMB asked agencies to identify organizations, program activities, regulations, policies, tax expenditures, and other activities contributing to their 2014-2015 APGs. Based on an analysis of the final quarterly updates for those APGs, published in December 2015, we found that agencies made progress in identifying external organizations and programs for their APGs. Describe how agency goals contribute to CAP goals: Agencies generally did not identify how their APGs contributed to CAP goals. We recommended that OMB direct agencies to describe in their performance plans how the agency’s performance goals—including APGs—contribute to any of the CAP goals as required by GPRAMA. In response, in July 2013, OMB updated its guidance directing agencies to include a list of the CAP goals to which the agency contributes and explain the agency’s contribution to them in their strategic plans, performance plans, and performance reports. Data-driven reviews: For their data-driven reviews of agency priority goals, agencies are to include, as appropriate, relevant personnel within and outside the agency who contribute to the accomplishment of each goal. However, in February 2013, we found that most Performance Improvement Officers (PIO) we surveyed (16 of 24) indicated that there was little to no involvement in these reviews from external officials who contribute to achieving agency goals. We recommended that OMB and the PIC help agencies extend their QPRs to include, as relevant, representatives from outside organizations that contribute to achieving their APGs. OMB staff told us that they generally concurred with the recommendation, but believed it would not always be appropriate to regularly include external representatives in agencies’ data-driven reviews, which they considered to be internal management meetings. In a subsequent review, we found in July 2015 that PIOs at 21 of the 22 agencies we surveyed said that their data-driven reviews had a positive effect on collaboration among officials from different offices or programs within the agency. Despite the positive effects, most agency PIOs (17) indicated that there continued to be little to no involvement in the reviews from external officials who contribute to achieving agency goals. In May 2016, OMB and PIC staff reported that, in response to our earlier recommendation, they were working with agencies to identify examples where agencies included representatives from outside organizations in data-driven reviews, and to identify promising practices based on those experiences. PIC staff told us they would disseminate any promising practices identified through the PIC Internal Reviews Working Group and other venues. In August 2017, OMB staff told us they plan to hold a summit with agencies later in the year to discuss implementing various performance management requirements, which could include agencies highlighting experiences and promising practices related to involving external officials in their data-driven reviews. We continue to believe data- driven reviews should include any relevant contributors from outside organizations and will continue to monitor progress. Despite the important role priority goals and related reviews can play in addressing crosscutting issues and enhancing collaboration, OMB recently removed the priority status of the current sets of priority goals. According to OMB staff, removing the priority designation from CAP goals and APGs returned them to regular performance goals, which are not subject to quarterly data-driven reviews or updates on the results of those reviews on Performance.gov. In a June 2017 memorandum, OMB stated that CAP goals and APGs are intended to focus efforts toward achieving the priorities of current political leadership, and therefore reporting on the priority goals of the previous administration on Performance.gov was discontinued for the remainder of the period covered by the goals (through September 30, 2017, the end of fiscal year 2017). The memorandum further noted that agencies and teams working on those goals should continue working on the current goals where they align with the priorities of the current administration. Moreover, the memorandum states that agencies have flexibility in structuring their data-driven reviews, but they should continue such reviews focused on agency priorities. When asked about these actions, OMB staff told us that they believed they were working in line with the intentions of GPRAMA, which realigned the timing of goal setting with presidential terms, to better take into account changes in priorities. This is the first presidential transition since GPRAMA was enacted, and OMB staff told us they thought the act was unclear on how to handle priority goals during the changes in administrations and priorities. They stated that it was not practical to continue reporting on the priority goals of the prior administration as agencies worked to develop new strategic plans and priority goals for publication in February 2018. Hence, they told us OMB ended the current round of CAP goals and directed agencies to remove the priority designation from the APGs, returning them to regular performance goals. OMB staff further told us that although the guidance was published in a June 2017 memorandum, these decisions had been made and previously communicated to agencies during the transition in administrations. Therefore, reporting on the fiscal year 2014-2017 CAP goals, fiscal year 2016-2017 APGs, and related reviews stopped much earlier in the year, well before goal cycles were planned to be completed on September 30, 2017. OMB staff further stated that although the goals no longer had priority designations, work towards them largely continued in 2017. For example, one of the prior administration’s CAP goals was to modernize the federal permitting and review process for major infrastructure projects. OMB staff told us that they and agencies have continued many of the activities intended to achieve that goal, but they are no longer subject to quarterly data-driven reviews or updates on the results of these reviews on Performance.gov. Moreover, they expect most of this work will continue towards a new and refocused CAP goal on infrastructure permitting modernization. OMB staff reaffirmed to us their intentions to resume implementation of CAP goals, APGs, and related data-driven reviews when the new planning and reporting cycle begins in February 2018. This is in line with stated plans to leverage various GPRAMA provisions to track progress of proposed government-wide and agency-specific reforms, as outlined in OMB’s April 2017 memorandum on the reform plans. In addition, OMB’s July 2017 update to its guidance for implementing GPRAMA similarly focuses on continued implementation of the act. Additional aspects of GPRAMA implementation could similarly help improve the management of crosscutting issues. Strategic reviews: OMB’s 2012 guidance implementing GPRAMA established a process in which agencies, beginning in 2014, were to conduct leadership-driven, annual reviews of their progress toward achieving each strategic objective established in their strategic plans. As we found in July 2015, effectively implementing strategic reviews could help identify opportunities to reduce, eliminate, or better manage instances of fragmentation, overlap, and duplication. Under OMB’s guidance, agencies are to identify the various organizations, program activities, regulations, tax expenditures, policies, and other activities that contribute to each objective, both within and outside the agency. Where progress in achieving an objective is lagging, the reviews are intended to identify strategies for improvement, such as strengthening collaboration to better address crosscutting challenges, or using evidence to identify and implement more effective program designs. If successfully implemented in a way that is open, inclusive, and transparent—to Congress, delivery partners, and a full range of stakeholders—this approach could help decision makers assess the relative contributions of various programs to a given objective. Successful strategic reviews could also help decision makers identify and assess the interplay of public policy tools that are being used to ensure that those tools are effective and mutually reinforcing, and that results are being efficiently achieved. In July 2017, OMB released guidance which updated the status of the 2017 strategic reviews. Because agencies are currently developing new strategic goals and objectives, OMB stated that agencies may forego the reporting and categorization requirements for any current strategic objectives that an agency determines will be substantively different or no longer aligned with the current administration’s policy, legislative, regulatory, or budgetary priorities. In addition, OMB stated that while there will be no formal meetings between OMB and the agencies to discuss findings and related progress from the 2017 strategic reviews, it expects that agencies will continue to conduct strategic reviews or assess progress made toward strategic goals and objectives aligned with administration policy. Furthermore, OMB stated that during this transition year, updates of progress on agency strategic objectives will only be published in the agency’s annual performance report and will not be reported to Performance.gov. Full reporting through Performance.gov is to resume after new agency strategic plans are published in February 2018. Agencies are to include a progress update for strategic objectives as part of their progress update in their fiscal year 2017 annual performance reports. Agencies also must address next steps for performance improvement as part of their fiscal year 2019 annual performance plans. Program inventories: GPRAMA requires OMB to publish a list of all federal programs, along with related budget and performance information, on a central government-wide website. Such a list could help decision makers and the public fully understand what the federal government does, how it does it, and how well it is doing. An inventory of federal programs could also be a critical tool to help decision makers better identify and manage fragmentation, overlap, and duplication across the federal government. Agencies developed initial program inventories in May 2013, but since then have not updated or more fully implemented these inventories. In October 2014, we found several issues limited the completeness, comparability, and usefulness of the May 2013 program inventories. OMB and agencies did not take a systematic approach to developing comprehensive inventories. For example, OMB’s guidance in Circular No. A-11 presented five possible approaches agencies could take to define their programs and noted that agencies could use one or more of those approaches in doing so. We found that because the agencies used inconsistent approaches to define their programs, the comparability of programs was limited within agencies as well as government-wide. In addition, we found that the inventories had limited usefulness for decision making, as they did not consistently provide the program and related budget and performance information required by GPRAMA. Moreover, we found that agencies did not solicit feedback on their inventories from external stakeholders—which can include Congress, state and local governments, third party service providers, and the public. Doing so would have provided OMB and agencies an opportunity to ensure they were presenting useful information for stakeholder decision making. We concluded that the ability to tag and sort information about programs through a more dynamic, web-based presentation could make the inventory more useful. In October 2014, we made several recommendations to OMB to update relevant guidance to help develop a more coherent picture of all federal programs and to better ensure relevant information is useful for decision makers. For example, we recommended that OMB revise its guidance to direct agencies to consult with relevant congressional committees and stakeholders on their approach to defining and identifying programs when developing or updating their inventories. OMB staff generally agreed with these recommendations, but have not yet taken any actions to implement them. OMB’s guidance for the program inventory has largely remained unchanged since 2014, when OMB postponed further development of the program inventory and eliminated portions of the guidance. For example, the guidance no longer describes, or provides directions for agencies to meet, GPRAMA’s requirements for presenting related budget or performance information for each program. OMB decided to postpone implementing a planned May 2014 update to the program inventory in order to coordinate with the implementation of the public spending reporting required by the Digital Accountability and Transparency Act of 2014 (DATA Act). OMB subsequently stated that it would not begin implementing the program inventory until after the DATA Act was implemented in May 2017, despite requirements for regular updates to the program inventory to reflect current budget and performance information. The DATA Act is now being implemented, but OMB has postponed resuming the development of the program inventory. In July 2017, OMB staff told us that they are now considering how to align GPRAMA’s program inventory provisions with future implementation of the Program Management Improvement Accountability Act (PMIAA). This was reflected in OMB’s July 2017 update to its guidance, which states that OMB is working with agencies to determine the right strategy to merge the implementation of the DATA Act and PMIAA with GPRAMA’s program inventory requirements to the extent possible to avoid duplicating efforts. For example, PMIAA requires OMB to coordinate with agency Program Management Improvement Officers to conduct portfolio reviews of agency programs to assess the quality and effectiveness of program management. GPRAMA requires OMB to issue guidance for implementing the program inventory requirements, among other things. Moreover, federal internal control standards state that organizations should clearly define what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement. As described above, OMB’s current guidance for the program inventory lacks some of those details—such as describing and providing direction to meet GPRAMA’s requirements for budget and performance information—in part because OMB is working with agencies to determine a strategy for implementation. Ensuring all GPRAMA requirements are covered and taking action on our past recommendations would help OMB improve its guidance to more fully implement the program inventory and improve its usefulness. To that end, in a report issued earlier this month, we identified a series of iterative steps that OMB could use in directing agencies to develop a useful inventory, as described in figure 3. A useful inventory would consist of all programs identified, information about each program, and the organizational structure of the programs. Our work showed that the principles and practices of information architecture—a discipline focused on organizing and structuring information—offer an approach for developing such an inventory to support a variety of uses, including increased transparency for federal programs. Such a systematic approach to planning, organizing, and developing the inventory that centers on maximizing the use and usefulness of information could help OMB ensure the inventory is implemented in line with GPRAMA requirements and meets the needs of decision makers and the public, among others. OMB’s guidance also lacks specific time frames, with associated milestones for resuming implementation of the program inventory requirements. As part of PMIAA’s requirements, OMB is to issue standards, policies, and guidelines for program and project management for agencies by December 2017. OMB staff told us that, within a year after that, they expect to issue further guidance on moving forward with resuming the program inventory. However, that general time frame was not reflected in the July 2017 update to OMB’s guidance. Providing specific time frames and associated milestones would bring the program inventory guidance in line with other portions of OMB’s guidance for implementing GPRAMA requirements, which contains a timeline of various performance planning and reporting requirements, including specific dates for meeting those requirements and related descriptions of required actions. For example, OMB’s July 2017 guidance identifies over 30 actions agencies should take between June 2017 and December 2018 to implement various GPRAMA provision. More specific time frames and milestones related to the program inventory requirements would help agencies prepare for resumed implementation by allowing them to know what actions they would be expected to take and by when. Moreover, publicly disclosing planned implementation time frames and associated milestones also would help ensure that external stakeholders are prepared to engage with agencies as they develop and update their program inventories. Effectively implementing various GPRAMA tools could help inform assessments of the performance of tax expenditures, which are reductions in tax liabilities that result from preferential provisions (figure 4). In fiscal year 2016, tax expenditures represented an estimated $1.4 trillion in forgone revenue, an amount greater than total discretionary spending that year. Despite the magnitude of these investments, our work has also shown that little has been done to determine how well specific tax expenditures work to achieve their stated purposes and how their benefits and costs compare to those of spending programs with similar goals. GPRAMA requires OMB to identify tax expenditures that contribute to the CAP goals. In addition, OMB guidance directs agencies to identify tax expenditures that contribute to their strategic objectives and APGs. However, our past work reviewing GPRAMA implementation found that OMB and agencies rarely identified tax expenditures as contributors to these goals. Fully implementing our recommendation to identify how tax expenditures contribute to various goals could help the federal government establish a process for evaluating the performance of tax expenditures. To that end, in May 2017, we provided the Director of OMB with three priority recommendations that require attention: Develop framework for reviewing performance: In June 1994, and again in September 2005, we recommended that OMB develop a framework for reviewing tax expenditure performance. We explained that the framework should (1) outline leadership responsibilities and coordination among agencies with related responsibilities, (2) set a review schedule, (3) identify review methods and ways to address the lack of credible tax expenditure performance information, and (4) identify resources needed for tax expenditure reviews. Since their initial efforts in 1997 and 1999 to outline a framework for evaluating tax expenditures and preliminary performance measures, OMB and the Department of the Treasury (Treasury) have ceased to make progress and retreated from setting a schedule for evaluating tax expenditures. Inventory tax expenditures: In October 2014, we found that OMB had not included tax expenditures in the federal program inventory, and therefore was missing an opportunity to increase the transparency of tax expenditures and the outcomes to which they contribute. We recommended that OMB should designate tax expenditures as a program type in relevant guidance, and develop, in coordination with the Secretary of the Treasury, a tax expenditure inventory that identifies each tax expenditure and provides a description of how the tax expenditure is defined, its purpose, and related budget and performance information. OMB staff said they neither agreed nor disagreed with these recommended actions. As noted earlier, OMB has not resumed updates to the program inventory. Therefore, OMB had not taken any actions in response to this recommendation, according to OMB staff as of July 2017. Identify contributions to agency goals: In July 2016, we found that agencies had made limited progress identifying tax expenditures’ contribution to agency goals, as directed by OMB guidance. As of January 2016, 7 of the 24 CFO Act agencies identified tax expenditures as contributing to their missions or goals. The 11 tax expenditure they identified—out of the 169 tax expenditures included in the President’s Budget for Fiscal Year 2017—represented approximately $31.9 billion of the $1.2 trillion in estimated forgone revenues for fiscal year 2015. (See figure 5.) To help address this issue, we recommended that OMB, in collaboration with the Department of the Treasury, work with agencies to identify which tax expenditures contribute to their agency goals, as appropriate. In particular, we recommended that they identify which specific tax expenditures contribute to specific strategic objectives and APGs. In July 2017, OMB staff said they had taken no actions to address the recommendation. Our July 2016 report also identified options for policymakers to further incorporate tax expenditures into federal budgeting processes, several of which options align with the recommendations discussed above. These options could help achieve various benefits, but we also reported that policymakers would need to consider challenges and tradeoffs in deciding whether or how to implement them. For example, one option was to require that all tax expenditures, or some subset of them, expire after a finite period. This option could result in greater oversight, requiring policymakers to explicitly decide whether to extend more or all tax expenditures. One consideration with this option is that it could lead to frequent changes in the tax code, such as from extended or expired tax expenditures, which can create uncertainty and make tax planning more difficult. Our previous work has shown that using performance information in decision making is essential to improving results. Performance information can be used across a range of management activities, such as setting priorities, allocating resources, or identifying problems to be addressed. However, our work continues to show that agencies can better use performance information in decision making, as shown in the example in the text box below. Department of Justice (DOJ) Could Better Analyze Performance Information to Reduce Backlog in Immigration Courts In June 2017, we found that the case backlog—cases pending from previous years that remain open at the start of a new fiscal year—at DOJ’s Executive Office for Immigration Review (EOIR) courts more than doubled from fiscal years 2006 through 2015. Stakeholders identified various factors that potentially contributed to the backlog, including continuances—temporary case adjournments until a different day or time. Our analysis of continuance records showed that the use of continuances increased by 23 percent from fiscal years 2006 through 2015. We found that EOIR collects continuance data but does not systematically assess them. Systematically analyzing the use of continuances could provide EOIR officials with valuable information about challenges the immigration courts may be experiencing, such as with operational issues like courtroom technology malfunctions, or areas that may merit additional guidance for immigration judges. Further, using this information to potentially address operational challenges could help that office meet its goals for completing cases in a timely manner. We recommended that the Director of EOIR systematically analyze immigration court continuance data to identify and address any operational challenges faced by courts or areas for additional guidance or training. EIOR agreed with this recommendation. EOIR stated that it supports conducting additional analysis of immigration court continuance data and recognizes that additional guidance or training regarding continuances may be beneficial to ensure that immigration judges use continuances appropriately in support of EOIR’s mission to adjudicate immigration cases in a careful and timely manner. We will monitor EOIR’s progress in taking these actions. Our 2017 survey of federal managers shows little change in their reported use of performance information. Using a set of survey questions, we previously developed an index that reflects the extent to which managers reported that their agencies used performance information for various management activities and decision making. The index suggests that government-wide use of performance information did not change significantly between 2013 and 2017, and it is statistically significantly lower relative to our 2007 survey, when we created the index. Figure 6 shows the questions included in the index and the government-wide results. In regard to individual survey items, in 2017 federal managers reported no changes or decreases in their use of performance information when compared to our last survey and when those survey items were first introduced. These results are generally consistent with our last few surveys. For example, in 2008 we found that there had been little change in federal managers’ reported use of performance information government-wide from 1997 to our 2007 survey. Citing those results, the Senate Committee on Homeland Security and Governmental Affairs report accompanying the bill that would become GPRAMA stated that agencies were not consistently using performance information to improve their management and results. The report further stated that provisions in GPRAMA are intended to address those findings and increase the use of performance information to improve performance and results. However, five items that were highlighted in our 2008 statement on the 2007 survey results generally show no improvement when compared to the 2017 results, as shown in figure 7. The one exception is for managers’ reported use of performance information to refine program performance measures. While this item was statistically significantly higher in 2013 relative to 2007—an estimated 46 percent to 53 percent—the 2017 result (43 percent) is a statistically significant decrease relative to 2013 and is not statistically different from the 2007 results. Another item, the use of performance information to adopt new program approaches or change work processes, also was statistically significantly lower in 2017 (47 percent) when compared to 2007 and 2013 (53 and 54 percent, respectively). This is of particular concern as agencies are developing their reform plans. Moreover, when compared to our 1997 survey, the 2017 results show four of the five items are statistically significantly lower, and the remaining item—allocating resources—has not changed. Similarly, we found there was no improvement in 2017 for more recent survey items on other uses of performance information compared to the years in which they were introduced, as shown in figure 8. Although one item, on the use of performance information to develop program strategy, was statistically significantly higher in 2013 relative to 2007 (an estimated 58 and 51 percent, respectively), the 2017 result (53 percent) does not represent a statistically significant change from either of those years. Another item, on the use of performance information to streamline programs to reduce duplicative activities, is statistically significantly lower relative to 2013, when it was introduced (from 44 to 33 percent in 2017). This is especially concerning because streamlining and reducing duplication are to be key parts of agencies’ reform plans. There is one area in the survey where we saw improvement: an estimated 46 percent of managers agreed to a great or very great extent that employees who report to them pay attention to their agency’s use of performance information in management decision making. That is statistically significantly higher relative to 2013 (40 percent), as well as when compared to when the item was introduced in 2007 (37 percent). For a new and related item in the 2017 survey that asked managers the amount of attention their employees pay to the use of performance information in decision making when compared to 3 years ago, we found an estimated 48 percent reported that employees pay about the same 33 percent reported that employees pay somewhat or a great deal more attention. In September 2005, we identified five practices that agencies can apply to enhance the use of performance information in their decision making and improve results: demonstrating management commitment; communicating performance information frequently and efficiently; improving the usefulness of performance information, such as by ensuring the accessibility of the information; developing the capacity to use performance information; and aligning agency-wide goals, objectives, and measures. Many of the requirements put in place by GPRAMA reinforce the importance of these practices. Our 2017 survey of federal managers includes a number of items related to these practices. However, the 2017 results suggest that managers have not effectively adopted them. In the following sections, we examine several of the practices to enhance the use of performance information and their related survey items further. In doing so, we also highlight a subset of six survey items related to these practices that, while separate from those in our use of performance information index, we found in September 2014 to have a statistically significant and positive relationship with it. The commitment of agency leaders to results-oriented management is critical to increased use of performance information for policy and program decisions. GPRAMA requires top leadership involvement in performance management, including leading data-driven performance reviews. However, we have previously reported that improvements are needed to strengthen leadership’s commitment to use performance information, as discussed in the text box below. Department of Defense Should Strengthen Leadership Responsibilities for Using Performance Information In January 2005, we designated the Department of Defense’s (DOD) approach to business transformation as high-risk because DOD had not taken the necessary steps to achieve and sustain business reform on a broad, strategic, department-wide, and integrated basis. In the February 2017 update to our High-Risk List, we found that DOD had taken some positive steps to improve its business transformation efforts.continuing to hold business function leaders accountable for diagnosing performance problems and identifying strategies for improvement, and leading regular DOD performance reviews regarding transformation goals and associated metrics and ensuring that business function leaders attend these reviews to facilitate problem solving. In July 2017, DOD officials told us that the department’s performance reviews have been put on hold until after the new Agency Strategic Plan is issued. We will review DOD’s updated Agency Strategic Plan when it is issued (expected in February 2018, as required by GPRAMA) to see if it addresses continuing to hold business function leaders accountable for diagnosing performance problems and identifying strategies for improvement. We will continue to monitor the status of these actions. GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others, GAO-17-317 (Washington, D.C.: Feb. 15, 2017). Results from our 2017 survey show no statistically significant difference relative to 2013 in managers’ perceptions of leaders’ and supervisors’ attention and commitment to the use of performance information. (See figure 9.) Three items are statistically significantly different from the years when they were introduced. Two items increased between 1997 and 2017: changes by management to my program(s) are based on results-oriented information (from an estimated 16 to 25 percent), and the individual I report to periodically reviews with me the outcomes of my program(s) (from 42 to 54 percent). For the third item, top leadership demonstrates a strong commitment to using performance information to guide decision making, results decreased from 49 percent in 2007 to 42 percent in 2017. New items in the 2017 survey show some improvement in management commitment to the use of performance information in decision making. An estimated 36 percent of federal managers reported that, when compared to 3 years ago, the individual they report to pays somewhat or a great deal more attention to the use of performance information in decision making, while 46 percent said they pay about the same amount of attention. Additionally, an estimated 21 percent of federal managers said that, when compared to 3 years ago, the head of their agency pays somewhat or a great deal more attention to the use of performance information in decision making, while 33 percent said they pay about the same amount of attention. Communicating performance information frequently and effectively throughout an agency can help to achieve the agency’s goals. GPRAMA includes requirements for communicating performance information, such as reporting progress updates for APGs at least quarterly. However, our prior work has found that some agencies could continue to improve in the communication of performance information, as illustrated by the example in the text box below. Department of Education (Education) Could Better Share Effective Practices across States in Grant Program Education awards 21st Century Community Learning Centers grants to states, which in turn competitively award funds to local organizations that use them to offer academic enrichment and other activities to improve students’ academic and behavioral outcomes. In April 2017, we found that states are experiencing substantial difficulty in sustaining their programs after 21st Century funding ends. We further found that Education was missing opportunities in its monitoring efforts to collect information on states’ strategies and practices for program sustainability—information that could be useful for sharing promising practices across states. We recommended that Education use the information it collects from its monitoring visits and ongoing interactions with states to share effective practices across states for sustaining their 21st Century programs once program funding ends. Education neither agreed nor disagreed with the recommendation but outlined steps it is taking to address it. We will continue to monitor progress on the implementation of this recommendation. There is no difference for two survey items on federal managers communicating performance information relative to 2013 or since those items were introduced in 2007. In 2017, we estimate that 44 percent of federal managers agreed to a great or very great extent that agency managers at their level effectively communicate performance information on a routine basis. In addition, 34 percent agreed to a great or very great extent that managers at their level use performance information to share effective program approaches with others. Our 2017 survey data also indicate that agencies may not be effectively communicating to their employees about contributions to CAP goals or progress toward achieving APGs. Of the estimated 54 percent of federal managers who indicated they were familiar with CAP goals, 23 percent reported that their agency has communicated to its employees on those goals to a great or very great extent. Of the 74 percent of federal managers who indicated familiarity with APGs, 44 percent reported that their agency has communicated on progress toward achieving those goals to great or very great extent. Our prior work has shown that agencies should consider users’ differing needs—for accessibility, accuracy, completeness, consistency, ease of use, timeliness, and validity, among others things—to ensure that performance information will be both useful and used. GPRAMA introduced several requirements that could help to address aspects of usefulness, such as requiring agencies to disclose more information about the accuracy and validity of their performance data and actions to address limitations to the data. However, agencies face challenges in ensuring their performance information is useful, with one instance from our past work described in the text box below. The Environmental Protection Agency (EPA) Could Improve Usefulness of Information in Planned Grantee Portal EPA monitors performance reports and program-specific data from grantees to ensure that grants achieve environmental and other program results. However, in July 2016, we found that EPA’s 2014 internal analysis of its grants management business processes identified improvements that, if implemented into EPA’s planned web-based portal, could improve the accessibility and usefulness of information in grantee performance reports for EPA, grantees, and other users. We recommended, among other actions, that EPA incorporate expanded search capability features, such as keyword searches, into its proposed web- based portal for collecting and accessing performance reports to improve their accessibility. EPA agreed with our recommendation but stated that it is a long- term initiative, subject to the agency’s budget process and replacement of its existing grants management system. As of May 2017, EPA officials said that they have not begun work on the web-based portal project, which is subject to the availability of funds. Federal managers generally responded similarly in 2017 on a variety of survey items related to usefulness, relative to earlier surveys. On a broadly worded item, less than half of managers agreed to a great or very great extent that agency managers at their level take steps to ensure that performance information is useful and appropriate. At an estimated 43 percent in 2017, this represents no statistically significant change compared to our last surveys in 2013 or 2007, when the item was introduced. Responses to four survey items indicate no changes in hindrances related to the usefulness of performance information. There is no statistically significant change in managers reporting hindrances compared to 1997 or 2013, as shown in figure 10. In addition, there was a statistically significant increase when compared to 2013 on only one of six items about managers’ views on the usefulness of performance information, as shown in figure 11. As the figure shows, approximately one-third to half of managers agreed to a great or very great extent on each item related to the usefulness of performance information. Although less than half of managers reported having sufficient information on validity of performance data used to make decisions, this represents a statistically significant increase to an estimated 42 percent in 2017 compared to 36 percent in 2013, and from 28 percent in 2000, when this item was introduced. This is a notable improvement because our September 2014 report found that the strongest driver of the use of performance information was whether federal managers had confidence in its validity. Our analysis suggests that easy access to performance information is related to the effective communication of performance information. Of the estimated 49 percent of federal managers in 2017 who agreed to a great or very great extent that performance information is easily accessible to managers at their level, 63 percent also agreed that agency managers at their level effectively communicate performance information on a routine basis to a great or very great extent. Conversely, of the 20 percent that agreed to a small or no extent that performance information is easily accessible to managers at their level, 12 percent also agreed that agency managers at their level effectively communicate performance information on a routine basis to a great or very great extent. Our prior work has shown that building capacity—including analytical tools and staff expertise—is critical to using performance information in a meaningful manner. GPRAMA lays out specific requirements that reinforce the importance of staff capacity to use performance information. GPRAMA directed the Office of Personnel Management (OPM) to take certain actions to support agency hiring and training of performance management staff. Specifically, by January 2012, OPM was to identify skills and competencies needed by government personnel for setting goals, evaluating programs, and analyzing and using performance information for improving government efficiency and effectiveness. By January 2013, OPM was to incorporate these skills and competencies into relevant position classifications and to work with each agency to incorporate the identified skills into employee training. In April 2013, we found that OPM had completed its work on the first two responsibilities and taken steps to work with agencies to incorporate performance management staff competencies into training. However, OPM did not assess competency gaps among agency performance management staff to inform its work. Without this information, OPM, working with the PIC, was not well-positioned to focus on the most- needed resources and help other agencies use them. We recommended that the Director of OPM, in coordination with the PIC and the Chief Learning Officer Council, work with agencies to take the following three actions: 1. Identify competency areas needing improvement within agencies. 2. Identify agency training that focuses on needed performance management competencies. 3. Share information about available agency training on competency areas needing improvement. In July 2017, PIC staff stated they have not focused on identifying competency areas because the competencies do not resonate strongly with the performance community. Instead, staff said they identified a need for introductory training on performance management, which they have developed and piloted. They said that they are not sure when they will implement the training, since the PIC is reviewing priorities with its new executive director. We continue to believe that identifying the competency areas would be useful, and will monitor the PIC’s efforts to identify and share training. The need for performance management training is further highlighted by our survey results. Our 2017 survey shows no statistically significant change in managers’ responses about the availability of training on various performance management activities relative to 2013, including the use of performance information to make decisions. However, the response to each of the six questions related to specific training is statistically significantly higher relative to the year in which it was introduced, as shown in figure 12. Similarly, in 2017 there was no statistically significant change on four survey items related to agencies’ analysis and evaluation tools and staff’s skills and competencies when compared to 2013 or when these items were introduced. We estimate that in 2017 29 percent of managers agreed to a great or very great extent that their agencies were investing in resources to improve the agencies’ capacity to use performance information; 28 percent of managers agreed to a great or very great extent that their agencies were investing the resources needed to ensure that performance data are of sufficient quality; 33 percent of managers reported that they agreed to a great or very great extent that their agencies have sufficient analytical tools for managers at their levels to collect, analyze, and use performance information; and 33 percent of managers reported that they agree to a great or very great extent that the programs they are involved with have sufficient staff with the knowledge and skills needed to analyze performance information. Performance reviews can serve as a strategy to bring leadership and other responsible parties together to review performance information and identify important opportunities to drive performance improvements. Our prior work has examined how different types of performance reviews—strategic reviews, data-driven reviews, and retrospective regulatory reviews—can contribute to agencies assessing progress toward desired results. Strategic reviews: As previously mentioned, in implementing GPRAMA, OMB established a review process in which agencies are to annually assess their progress in achieving each strategic objective in their strategic plans, known as strategic reviews. Given the long-term and complex nature of many outcomes, the strategic review should be informed by a variety of evidence regarding the implementation of strategies and their effectiveness in achieving outcomes. OMB’s guidance states that the strategic review process should consider multiple perspectives and sources of evidence to understand the progress made on each strategic objective. It further states that the results of these reviews should inform many of the decision-making processes at the agency, as well as decision making by the agency’s stakeholders, in areas such as long-term strategy, budget formulation, and risk management. In 2017, agencies are completing their fourth round of these reviews. Our prior work has identified ways in which agencies can effectively conduct these reviews and leverage the results that come from them. In July 2015, we identified seven practices federal agencies can employ to facilitate effective strategic reviews. (See sidebar.) In addition, earlier this month we reported on selected agencies’ experiences in implementing these reviews. Specifically, we found that (1) strategic reviews helped direct leadership attention to progress on strategic objectives, (2) agencies used existing management and performance processes to conduct the reviews, and (3) agencies refined their reviews by capturing lessons learned. Data-driven reviews: GPRAMA requires agencies to review progress toward APGs at least once a quarter. The Senate Committee on Homeland Security and Governmental Affairs report accompanying the bill that would become GPRAMA stated that this approach is aimed at increasing the use of performance information to improve performance and results. In February 2013, we identified nine leading practices to promote successful data-driven performance reviews in the federal government. (See sidebar.) In July 2015, we found that most of the 24 CFO Act agencies were conducting their reviews in line with GPRAMA requirements and our leading practices. Moreover, agencies reported that their data-driven performance reviews had positive effects on progress toward agency goals, collaboration between agency officials, the ability to hold officials accountable for progress, and efforts to improve the efficiency of operations. Our 2017 survey shows that federal managers remain largely unfamiliar with their agency’s data-driven performance reviews, also known as quarterly performance reviews (QPRs). An estimated 35 percent of managers reported familiarity with their agency’s QPRs. Survey results show that a greater percentage of Senior Executive Service (SES) managers than non-SES managers reported that they were familiar with QPRs. Approximately 50 percent of SES managers reported being somewhat or very familiar with QPRs; 34 percent of non-SES reported the same. However, for the estimated 35 percent of managers who reported familiarity with QPRs, the more they viewed their programs being subject to a QPR, the more likely they were to report their agency’s QPRs were driving results and conducted in line with our leading practices. Figure 13 shows several illustrative examples of these survey items. For example, of the estimated 48 percent of federal managers who reported their programs being subject to QPRs to a great or very great extent, 83 percent also reported their agencies use QPRs to identify problems or opportunities associated with agency performance goals. Conversely, for the 24 percent of managers who reported their programs were subject to QPRs to a small or no extent, 22 percent also reported the reviews were used for these purposes to a great or very great extent. Being subject to a QPR is also positively related to viewing QPRs as having led to similar meetings at lower levels. An estimated 62 percent of federal managers who reported being subject to QPRs to a great or very great extent also reported their agencies have similar meetings at lower levels to a great or very great extent. An estimated 16 percent of federal managers subject to QPRs to a small or no extent reported the same. Despite the reported benefits of and results achieved through QPRs, as found by our past work and survey data, these reviews are not necessarily widespread. GPRAMA requires agencies to conduct QPRs for APGs, which represent a small subset of goals—generally 2 to 8 priority goals at each designated agency, with approximately 100 total government-wide. Moreover, these required reviews are at the department (or major independent agency) level. These reasons may explain why most managers reported they were not familiar with the reviews. As was described previously, our 2017 survey data show that the reported use of performance information in decision making generally has not improved and in some cases is lower than it was 20 years ago. Survey data also show that managers generally have not reported increases in their employment of practices that further promote the use of performance information in decision making. This suggests that agencies could increase the use of performance information in decision making and the likelihood of achieving desired results by going beyond the specific GPRAMA requirements and expanding their use of data-driven performance reviews—in line with leading practices—to more broadly cover other agency-wide performance goals, as well as goals at lower levels within the agency. For example, such reviews at the program level could help inform the previously mentioned portfolio reviews required by the Program Management Improvement Accountability Act (PMIAA). We have already suggested expanding reviews to other performance goals. Our management agenda for the presidential and congressional transition includes a key action to expand the use of data-driven performance reviews to assess progress toward meeting agency performance goals. Our prior work has stated that although GPRAMA’s requirements apply at the agency-wide level, they can also serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects. In addition, federal internal control standards call for the design of appropriate control activities, such as top-level reviews of actual performance and reviews by management at the functional or activity level. The standards also recommend that management design control activities at the appropriate levels in the organizational structure. The July 2017 update to OMB’s guidance states that agency leaders, including various chief officer positions, are to conduct frequent data- driven reviews to drive improvements on various management functions. For example, the agency Chief Human Capital Officer is to conduct quarterly data-driven reviews (known as HRStat) to monitor the progress of human capital goals and measures contained in the human capital operating plan. Beyond these management areas, OMB’s guidance also states that agencies may expand quarterly progress reviews beyond APGs to include other goals and priorities. However, OMB’s guidance does not identify practices for agencies to expand the use of these reviews to other goals, such as other agency-wide performance goals or those at lower levels within the agency. As mentioned previously, one of the responsibilities of the Performance Improvement Council (PIC) is to facilitate the exchange among agencies of practices that have led to performance improvements within specific programs, agencies, or across agencies. By working with the PIC to identify and share among agencies practices to expand the use of data- driven reviews, OMB could help agencies increase the use of performance information in decision making and achieve results. Retrospective regulatory reviews: In retrospective reviews, agencies evaluate how existing regulations are working in practice and whether they are achieving expected outcomes. GPRAMA requires agencies to identify and assess how their various program activities and other activities, including regulations, contribute to APGs. However, in April 2014, we found that agencies reported mixed experiences linking retrospective analyses to APGs. We recommended that OMB strengthen these reviews by issuing guidance for agencies to take actions to ensure that contributions made by regulations toward achieving APGs are properly considered, and improve how retrospective regulatory reviews can be used to help inform assessments of progress toward these APGs. OMB staff agreed with this recommendation and stated that the agency was working on strategies to help facilitate agencies’ ability to use retrospective reviews to inform APGs. To that end, in April 2017, OMB issued guidance to agencies that, among other things, emphasized the importance of performance measures related to evaluating and improving the net benefits of their respective regulatory programs. OMB included explicit references to section 6 of Executive Order 13563, which directed agencies’ efforts to conduct retrospective regulatory reviews. Specifically, the updated guidance encourages agencies to establish and report “meaningful performance indicators and goals for the purpose of evaluating and improving the net benefits of their respective regulatory programs.” The guidance further states that agencies’ efforts to improve such net benefits may be conducted as part of developing agency strategic and performance plans and priority goals. In July 2017, OMB confirmed that the updated guidance was issued, in part, to address our April 2014 recommendation. For several years, OMB has encouraged agencies to expand their use of evidence—performance measures, program evaluation results, and other relevant data analytics and research studies—in budget, management, and policy decisions with the goal of improving government effectiveness. In particular, OMB has encouraged agencies to strengthen their program evaluations—systematic studies that use research methods to address specific questions about program performance. Evaluation is closely related to performance measurement and reporting. Evaluations can be designed to better isolate the causal impact of programs from other external economic or environmental conditions in order to assess a program’s effectiveness. Thus, an evaluation study can provide a valuable supplement to ongoing performance reporting by measuring results that are too difficult or expensive to assess annually, explaining the reasons why performance goals were not met, or assessing whether one approach is more effective than another. Despite the valuable insights and information that program evaluations can provide, we continue to find that most federal managers lack access to or awareness of such studies. Our 2017 survey shows that an estimated 40 percent of managers reported that an evaluation had been completed within the past 5 years of any program, operation, or project in which they were involved—comparable to the results in our 2013 survey, when questions about program evaluations were added. In recent years, OMB has encouraged agencies to explore evidence-based tools to strengthen agency and grantee evaluation capacity, consider the effectiveness of their programs, and foster innovation rooted in research and rigorous evaluation. During the past 2 years, we examined several of those tools, as described below. Pay for success: Also known as social impact bonds, pay for success is a contracting mechanism under which investors provide the capital the government uses to provide a social service. The government specifies performance outcomes in pay for success contracts and generally includes a requirement that a program’s impact be independently evaluated. The evaluators also are to regularly review performance data, while those managing and investing in a project focus on performance and accountability, as shown in the figure 14. In September 2015, we found that the federal government’s involvement in pay for success had been limited. In addition, a formal mechanism for federal agencies to collaborate on pay for success did not exist. We concluded that, given the evolving nature of pay for success, a mechanism for federal agencies to collaborate would increase access to leading practices. We therefore recommended that OMB establish a formal means for federal agencies to collaborate on pay for success. OMB concurred and, in February 2016, announced that it had developed the Pay for Success Interagency Learning Network with representatives from 10 federal agencies to share lessons, hone policy, and strengthen implementation. Tiered evidence grants: Tiered evidence grants seek to incorporate evidence of effectiveness into grant making. Federal agencies establish tiers of grant funding based on the level of evidence grantees provide on their approaches to deliver social, educational, health, or other services. (See figure 15.) Smaller awards are used to test new and innovative approaches, while larger awards are used to scale up approaches that have strong evidence of effectiveness. This creates incentives for grantees to use approaches supported by evidence and helps them build the capacity to conduct evaluations. In September 2016, we found that interagency collaboration had helped federal agencies that administer tiered evidence grants address challenges and share lessons learned. At that time, such collaborative efforts relied on informal networks. We recommended that OMB establish a formal means for agencies to collaborate on tiered evidence grants. OMB had no comment on the recommendation. In July 2017, OMB staff told us that they had established an interagency working group and other mechanisms to facilitate collaboration and disseminate information on tiered evidence grants. Performance partnerships: Performance partnerships allow federal agencies to provide grant recipients flexibility in how they use funding across two or more programs along with additional flexibilities. In exchange, the recipient commits to improve and assess progress toward agreed-upon outcomes. Figure 16 provides an overview of the performance partnership model. In April 2017, we examined two performance partnership initiatives authorized by Congress: the Environmental Protection Agency’s Performance Partnership Grants and the Performance Partnership Pilots for Disconnected Youth, which allows funding from multiple programs across multiple agencies to be combined into pilot programs serving disconnected youth. For the Performance Partnership Pilots for Disconnected Youth, we found that the agencies involved in the initiative had not fully identified the key financial and staff resources each agency would need to contribute over the lifetime of the initiative in line with leading practices for interagency collaboration. This was because agencies primarily had been focused on meeting near-term needs to support design and implementation. We also found that agencies had not developed criteria to help determine whether, how, and when to implement the flexibilities tested by the pilots in a broader context. (This is known as scalability.) Officials involved in the pilots told us it was too early in pilot implementation to determine such criteria. However, by not identifying these criteria while designing the pilots, they were risking not collecting needed data during pilot implementation. We recommended that OMB coordinate with federal agencies to identify (1) agency resource contributions needed for the lifetime of the pilots and (2) criteria and related data for assessing scalability. OMB neither agreed nor disagreed with these recommendations. We continue to monitor progress on these recommendations. In 2003, we identified nine key practices for effective performance management that collectively create a “line of sight” between individual performance and organizational success. (See sidebar on next page.) Our recent work and the results of our 2017 survey of federal managers highlight areas where agencies have made progress but could take additional action to better reflect several of these practices, thereby better instilling results-oriented cultures. Align individual performance expectations with organizational goals: Our 2003 report found that high-performing organizations use their performance management systems to help individuals see the connection between their daily activities and organizational goals. The executive branch has taken several steps to link individual and organizational results. For example, in October 2000, OPM issued guidance to link SES performance expectations with GPRA-required goals. In January 2012, OPM and OMB released a government-wide SES performance appraisal system that provided agencies with a standard framework to manage the performance of SES members. However, our work continues to identify areas for improvement. Goal leaders and deputy goal leaders are responsible for achieving APGs, but our July 2014 review found that the performance plans for a sample of goal and deputy goal leaders generally did not link their individual performance and the broader goal. We recommended that OMB ensure that those plans demonstrate a clear connection with APGs. OMB staff generally agreed with our recommendation. In July 2017, OMB staff stated that components of both OMB and OPM guidance support accountability for agency priority goals. Despite this, we continue to believe that ensuring an explicit connection in performance plans to APGs will improve accountability, and that additional action is needed to do so. In May 2016, we found that the Federal Emergency Management Agency (FEMA) had not aligned Federal Disaster Recovery Coordinators’ performance expectations with its organizational goals for implementing the National Disaster Recovery Framework. We concluded that without this linkage, FEMA could not evaluate how effectively the coordinators performed in implementing the framework. We recommended that FEMA align performance expectations consistent with leading practices. The Department of Homeland Security concurred with our recommendation. In July 2017, FEMA stated that it is preparing the Field Leader Manual, which will define the core competencies and duties of coordinators. We will continue to monitor FEMA’s actions to implement this recommendation. Our 2017 survey also shows that this linkage could be improved for other federal employees. An estimated 58 percent of federal managers reported using performance information to a great or very great extent in setting expectations for employees they manage or supervise. The 2017 responses do not represent a statistically significant change when compared to our last survey in 2013 (62 percent) or to 1997 (61 percent), the year this survey item was introduced. Address organizational priorities: Our prior work showed that, by requiring and tracking follow-up actions on performance gaps, high- performing organizations underscore the importance of holding individuals accountable for making progress on their priorities. Our past and 2017 surveys have identified differences in responses between SES and non-SES managers reporting being held accountable for results. For example, in 2017, our survey results indicate that there was a statistically significant difference between SES and non-SES managers reporting to a great or very great extent that they were held accountable for results of the programs for which they are responsible. However, our 2017 survey shows no change compared to our last survey in either SES or non-SES managers reporting they were held accountable for results. There are statistically significant increases when compared to 1997, when these survey items were introduced. For example, an estimated 79 percent of SES managers and 64 percent of non-SES managers reported being held accountable to a great or very great extent for results of the programs for which they are responsible in 2017. This does not represent a statistically significant change from our 2013 survey (80 percent and 67 percent, respectively), but it is statistically significantly higher than the 62 percent of SES managers and 54 percent of non-SES managers in 1997. (See figure 17.) Similarly, as shown in figure 18, an estimated 71 percent of SES managers reported being held accountable to a great or very great extent for accomplishing agency strategic goals in 2017. This represents no statistical change since 2013 (73 percent), but it is a statistically significant increase compared to when this item was introduced in 2003 (61 percent). Additionally, as figure 18 shows, a gap between being held accountable for strategic goals and having the decision-making authority needed to help accomplish those goals has nearly closed, due to an increase in the latter survey item. The estimated 69 percent of SES managers who reported having such authority to a great or very great extent in 2017 is a statistically significant increase relative to both 2013 (61 percent) and 1997 (51 percent). As noted earlier, GPRAMA requires goal leaders for CAP goals and APGs. Our past work has generally found that they are in place. GPRAMA also requires agencies to identify an agency official responsible for resolving major management challenges, which can help ensure accountability. (See sidebar.) However, in June 2016 we found that 17 of the 24 CFO Act agencies had not identified an agency official responsible for resolving each of their challenges, partly because OMB guidance was not clear that major management challenges should be identified in agency performance plans. We recommended that the 17 agencies identify such officials in their performance plans, and that OMB clarify its guidance. OMB revised its guidance accordingly in July 2016, and, as of July 2017, 7 of the 17 agencies had identified officials responsible for resolving major management challenges. Link pay to individual and organizational performance: High- performing organizations seek to create pay, incentive, and reward systems that clearly link employee knowledge, skills, and contributions to organizational results. Our work has found that agencies have made progress in this area. For example, in July 2013, we found that the Securities and Exchange Commission (SEC) lacked mechanisms to monitor how supervisors used its performance management system to recognize and reward performance. To help enhance the credibility of SEC’s performance management system, we recommended that it create mechanisms to monitor how supervisors use the performance management system. In a subsequent (December 2016) report, we found that, in response to our recommendation, SEC began monitoring how supervisors provide feedback, recognize and reward staff, and address poor performance. However, federal managers generally reported no change on three items related to recognizing and rewarding employee performance since our last survey in 2013 (figure 19). One of those items—managers agreeing to a great or very great extent that employees in their agency receive positive recognition for helping the agency to accomplish its strategic goals—had a statistically significant increase between 1997 and 2017 (from an estimated 26 percent to 46 percent). Make meaningful distinctions in performance: Effective performance management requires the organization’s leadership to meaningfully distinguish between acceptable and outstanding performance of individuals and to appropriately reward those who perform at the highest level. For example, in January 2015, we found disparities in performance ratings for SES among agencies. Across the 24 CFO Act agencies, the percent of SES rated at the highest level ranged from about 22 percent to 95 percent in fiscal year 2013. To help address these disparities, we recommended that the Director of OPM consider the need to refine the performance certifications guidelines addressing distinctions in performance. To address this recommendation, OPM informed us, in June 2015, that it had convened a cross-agency working group that developed a standard template for agencies to complete and post on a website to more transparently justify their SES ratings distributions. In May 2016, we found that about 74 percent of non-SES employees under a five-level appraisal system—the most commonly used system— were rated in the top two of five performance categories in 2013. We explored this issue further in our December 2016 review of human capital challenges at the Veterans Health Administration (VHA), which illustrates the importance of making meaningful distinctions in performance for non- SES employees. We found that in fiscal year 2014, about 73 percent of VHA employees were rated in the top two of five performance categories. This may have been due, in part, to a policy that did not require standards to be defined for each level of performance. We recommended that VHA ensure that meaningful distinctions are being made in employee performance ratings by reviewing and revising performance management policies consistent with leading practices, among other actions. The Department of Veterans Affairs partially concurred with our recommendation. In May 2017, the department stated that it had begun piloting a new performance management process and would analyze results at the end of fiscal year 2017. One key aspect of connecting daily operations to results is aligning program performance measures to agency-wide goals and objectives. However, in 2017, an estimated 50 percent of federal managers agreed to a great or very great extent that managers at their level took steps to create such an alignment. There has been no statistically significant change since this item was introduced in 2007. In addition, GPRAMA calls for agencies to develop a balanced set of performance measures, which reinforces the need for agencies to have a variety of measures across program areas. Our 2017 survey shows that managers have not reported any difference in the availability of performance measures for their programs when compared to the 2013 results. However, the 2017 result (an estimated 87 percent) represents a statistically significant increase when compared to 1997 (76 percent). When asked about the availability of certain types of performance measures, three of the five types (outcome, output, and efficiency) were statistically significantly higher in 2017 when compared to our initial 1997 survey. However, when comparing 2017 results to those in 2013, two of the five types (output and quality) showed a statistically significant decrease, and the other types did not change. These are illustrated in figure 20. Beyond the survey results, our work has found that some agencies had not developed or used outcome measures, but have taken steps to do so. Agencies have been responsible for measuring program outcomes since GPRA was enacted in 1993. The text box below describes two illustrative examples from our past work. Examples of Agencies That Did Not Develop or Use Outcome Measures Patient access to electronic health information: In March 2017, we found that the Department of Health and Human Services (HHS) had invested over $35 billion since 2009 to enhance patient access to electronic health information, among other things. HHS had not developed outcome measures to gauge the effectiveness of these efforts, which meant the department did not have information to determine whether the efforts were contributing to its overall goals. We recommended that HHS develop relevant outcome measures and HHS concurred. Safety interventions: According to the Federal Motor Carrier Safety Administration (FMCSA), between 2011 and 2015, over 4,000 people died in crashes involving motor carriers each year. GAO, Motor Carriers: Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions, GAO-17-49 (Washington, D.C.: Oct. 27, 2016). Further OMB actions could also help agencies make progress in measuring the performance of different program types. In our June 2013 report on initial GPRAMA implementation, we found that agencies experienced common issues in measuring the performance of various types of programs, such as contracts and grants. We recommended that OMB work with the PIC to develop a detailed approach to examine those difficulties. Although they took some actions, OMB and the PIC have not yet developed a comprehensive and detailed approach to address these issues. We concluded that, without such an approach, it would be difficult for the PIC and agencies to fully understand these measurement issues and develop a crosscutting approach to help address them. In August 2017, OMB staff stated that efforts related to the future implementation of the Program Management Improvement Accountability Act (PMIAA) could help address this recommendation. As highlighted in table 1, our work continues to show why it is important for OMB and the PIC to take actions to more fully address our recommendation. Congress has passed legislation to increase the transparency and accessibility of federal performance and financial data. For example, GPRAMA modernized agency reporting requirements to ensure that they make timely, relevant data available to inform decision making by Congress and agency officials as well as improve transparency for the public. Results of our 2017 survey, however, show the need for improvements in the public availability of agency performance information. An estimated 17 percent of managers reported that their agency’s performance information is easily accessible to the public to a great or very great extent, the same percentage as in 2013. Moreover, of the 87 percent of managers that reported there are performance measures for the programs they are involved in, 25 percent reported that they use information obtained from performance measurement when informing the public about how programs are performing to a great or very great extent. This is not statistically different from the 30 percent estimated in 2013. The DATA Act, enacted in 2014, built on previous transparency legislation by expanding what federal agencies are required to report regarding their spending. The act significantly increases the types of data that must be reported, requires government-wide data standards, and regular reviews of data quality to help improve the transparency and accountability of federal spending data. OMB provides websites and guidance to make agency performance and financial information available to the public; however, our prior work has identified a number of areas related to Performance.gov and the DATA Act where OMB action is needed to improve the transparency and accessibility of this information. Performance.gov: Since 2013, our work has identified a number of issues with Performance.gov, the website intended to serve as a central source of information on the federal government’s goals and performance. Over time, we have recommended that OMB take a number of specific actions to improve the website. For example, in June 2013, we found that the website offered an inconsistent user experience and presented accessibility and navigation challenges. To clarify the purpose of the website and enhance its usability, we recommended that OMB take steps to systematically collect customer input. In August 2016, we reported that OMB was not meeting all of the reporting requirements for Performance.gov, and did not have a plan to develop and improve the website. We recommended that OMB ensure that information presented on Performance.gov consistently complies with reporting requirements and develop a plan for the website that includes, among other things, a customer outreach plan. OMB agreed with these recommendations and, in July 2017, OMB staff informed us that they will be partnering with a vendor to redesign Performance.gov to improve the accessibility of information on the website. To inform this redesign, OMB staff said that they will consider our previous recommendations and plan to engage a wide group of stakeholders, including Congress, agency staff, and interested members of the public and outside organizations. OMB staff anticipated releasing updated agency reporting guidance in the fall of 2017 and the redesigned website in February 2018. Under GPRAMA, OMB is required to make available, through Performance.gov, quarterly updates on progress toward CAP goals and APGs. As described earlier, in June 2017 OMB announced that reporting to Performance.gov has been discontinued through the end of fiscal year 2017 as agencies develop new priority goals. However, Performance.gov does not state that it will not be updated, nor does it provide the location of the final progress updates for these goals. OMB’s guidance states that agencies should report the results of progress on their previous APGs in their annual performance reports for fiscal year 2017. Moreover, OMB staff told us that the existing updates on Performance.gov for CAP goals, last updated in December 2016, represent the final updates on those goals, although they are not labeled as such on the website. As a result, those interested in progress updates and reported results for the previous priority goals may not know where they will be able to find this information, limiting the transparency and accessibility of those results for decision makers and the public. DATA Act: The DATA Act requires federal agencies to disclose their spending and link this to program activities so that policymakers and the public can more effectively track federal spending. The act has the potential to improve the accuracy and transparency of federal spending information and increase its usefulness for government decision making and oversight. Since the DATA Act became law, OMB and Treasury have taken significant steps to make more complete and accurate federal spending data available. These have included standardizing data element definitions to make it easier to compare different federal agencies’ financial information, and issuing guidance to help agencies submit required data. In May 2017, federal agencies started to report data under the standardized definitions developed under the act. We have made a number of recommendations to address challenges that could affect the consistency and quality of the data. Addressing these recommendations could help ensure that financial data are provided to the public in a transparent and useful manner. For example, in January 2016, we found some standardized data element definitions were imprecise or ambiguous, which could result in inconsistent or potentially misleading reporting. We recommended that OMB provide agencies with additional guidance to address potential issues with the clarity, consistency, and quality of reported data. OMB released guidance in May and November 2016, but in April 2017 we found that additional guidance was needed to help agencies implement certain data definitions to produce data that would be consistent and comparable across agencies. We are in the process of examining the quality of the data that was submitted by agencies in May 2017 and was made available to the public on an early version of the USAspending.gov website. We expect to issue the results of this work in fall 2017. Our past work also identified a number of actions agencies need to take to make performance information more transparent. Increasing the accessibility of this information could enhance oversight and accountability of agency performance and results. CAP goals: In May 2016, we found that while selected CAP goal teams were working to develop performance measures to track progress, they were not consistently reporting on their efforts to develop these measures. We recommended that OMB report on Performance.gov the actions that CAP goal teams are taking to develop performance measures and quarterly targets to help ensure that measures are aligned with major activities, and ensure that it is possible to track teams’ progress toward establishing measures. While OMB agreed with this recommendation, it did not address it before reporting on the CAP goals was discontinued, as discussed earlier. Customer service standards: As we described earlier, in 2017, an estimated 48 percent of federal managers that indicated they have performance measures for the programs they are involved in also agreed to a great or very great extent that they have customer service performance measures. There has been no statistically significant change relative to our last survey in 2013, or the initial survey in 1997. Relatedly, in October 2014, we reviewed customer service standards at five federal agencies. Customer service standards inform customers about what they have a right to expect when they request services, and the standards should include goals for the quality and timeliness of a service an agency provides to its customers. They should also be easily available to the public so that customers know what to expect, when to expect it, and from whom. In our review of standards at five agencies, however, we found that only Customs and Border Protection had standards that were easily available to the public. We recommended the other four agencies—the United States Forest Service, Federal Student Aid, the National Park Service (NPS), and the Veterans Benefits Administration (VBA)—make their standards more easily accessible to the public. As of July 2017, only VBA had done so. Major management challenges: In June 2016, we found that 14 of the 24 CFO Act agencies did not describe their major management challenges in their performance plans, as required by GPRAMA. Furthermore, 22 of the 24 agencies reviewed did not report complete performance information for each of their major management challenges, including performance goals, milestones, indicators, and planned actions that they have developed to address such challenges. As a result, it was not always transparent what these agencies considered to be their major management challenges or how they planned to resolve these challenges. We recommended that the 22 agencies describe their major management challenges in their agency performance plans and include goals, measures, milestones, and information on planned actions and responsible officials. As of August 2017, 8 agencies—the U.S. Agency for International Development, Small Business Administration, Nuclear Regulatory Commission, OPM, National Aeronautics and Space Administration (NASA), and the Departments of Education, State, and Veterans Affairs—had fully implemented our recommendations; the other 14 agencies had not. Quality of performance information: In September 2015, we found that six selected agencies reported limited information on the actions they are taking to ensure the quality of their performance information for selected APGs, as required by GPRAMA. We recommended that all six of the agencies work with OMB to fully report this information. In response, the Department of Homeland Security and NASA described how they ensure reliable performance information is reported to external audiences. As of June 2017, the Departments of Agriculture, Defense, the Interior, and Labor had not yet taken actions to address this recommendation by providing more specific explanations of how they ensure reliable performance information is reported for their APGs. Unnecessary reports: GPRAMA requires that OMB guide an annual review of agencies’ plans and reports for Congress and include in the President’s budget a list of those plans and reports determined to be outdated or duplicative. However, in July 2017, we found that OMB did not implement the report review process on an annual basis, as required. We also found that OMB published the list of agency plans and reports on Performance.gov, rather than in the President’s annual budget, where they may be more visible and useful to congressional decision makers and others. Therefore, we recommended that OMB instruct agencies to identify outdated or duplicative reports on an annual basis and submit or reference the list of identified plans and reports with the President’s annual budget. OMB agreed with these recommendations. In July 2017, OMB stated it would include a list of report modification proposals in the President’s fiscal year 2019 budget as required by GPRAMA. For all of the unimplemented recommendations described above, we will continue to monitor agencies’ actions. In addition to providing access to performance and financial information, federal agencies can directly engage and collaborate with citizens, nonprofits, academic institutions, and other levels of government using open innovation strategies. Open innovation involves using various tools and approaches to harness the ideas, expertise, and resources of those outside an organization to address an issue or achieve specific goals. In October 2016, we found that in recent years agencies had frequently used five open innovation strategies—singularly or in combination—to collaborate with citizens and encourage their participation in agency initiatives. (See figure 21.) Our October 2016 report found that agencies can use these strategies for a variety of purposes. To develop new ideas, solutions to specific problems, or new products: For example, from April 2015 to November 2016, the Department of Energy held a prize competition to create more efficient devices that would double the energy captured from ocean waves. According to the competition’s website, the winning team achieved a five-fold improvement. To enhance collaboration and agency capacity by leveraging external resources, knowledge, and expertise: For example, every 2 years since 2009, the Federal Highway Administration has regularly engaged stakeholders to identify and implement innovative ideas that have measurably improved the execution of highway construction projects. To collect the perspectives and preferences of a broad group of citizens and external stakeholders: For example, the Food and Drug Administration used in-person and online dialogue to engage outside stakeholders in the development of an online platform designed to make key datasets easily accessible to the public. Subsequently, in June 2017, we found that OMB, the Office of Science and Technology Policy (OSTP), and the General Services Administration (GSA) developed resources to support the use of open innovation strategies by federal agencies. These resources included guidance, staff to assist agencies in implementing initiatives, and websites to improve access to relevant information. For example, GSA developed a step-by-step implementation guide, program management team, and website to help agency staff carry out prize competitions and challenges. Agencies have also developed their own resources, including guidance, staff positions, and websites, to reach specific audiences and to provide tailored support for open innovation strategies they use frequently. For example, NASA’s Solve website provides a central location for the public to find the agency’s challenges and citizen science projects, as well as links to relevant resources. We also evaluated key government-wide guidance for the five strategies listed above to determine the extent to which the guidance reflects leading practices for effectively implementing open innovation initiatives. We identified these practices in our October 2016 report. We found that the guidance for each strategy reflected these practices to differing extents, as shown in figure 22. We made 22 recommendations to GSA, OMB, and OSTP to enhance the guidance. GSA and OMB generally agreed with these recommendations and OSTP neither agreed nor disagreed. We will monitor their progress toward implementing these recommendations. GPRAMA provides important tools that can help decision makers better achieve results and address the federal government’s significant and long-standing governance challenges. Although OMB and agencies have made progress in improving implementation of the act over the years, our work has highlighted numerous opportunities for further improvements. In 2017, OMB removed the priority designation of CAP goals and APGs. For those goals, this action stopped related data-driven reviews and quarterly updates of progress on Performance.gov until new priority goals are published next year. What OMB considers to be the final results of CAP goals for fiscal years 2014 to 2017 already are on Performance.gov (although not labeled as such). In addition, agencies may report on their former APGs in their annual fiscal year 2017 performance reports. However, Performance.gov does not state that it will not be updated or provide the location of the final progress updates for these goals, limiting transparency and its value to the public. OMB has stated its plans to restart implementation of those provisions in February 2018, with the start of a new goal cycle. We believe it is critical for OMB to do so, given the important role those tools play in addressing key governance challenges and the results we have seen in better managing crosscutting areas and driving performance improvements across the government. In addition, OMB has postponed implementation of the federal program inventory. To date, the inventory has only been developed once, in 2013, despite requirements for regular updates to reflect current budget and performance information. OMB has given a variety of reasons for the delays over the past 4 years—most recently, to determine the right strategy to merge implementation of the DATA Act and PMIAA with GPRAMA’s program inventory requirements. Although OMB staff told us that they expect to issue guidance by the end of 2018 to resume implementation of the program inventory requirements, they have not provided more specific time frames and milestones related to the program inventory requirements. Doing so would help agencies prepare for resumed implementation. Moreover, publicly disclosing planned implementation time frames and associated milestones would help ensure that interested stakeholders, such as federal decision makers and the public, are prepared to engage with agencies as they develop and update their program inventories, which in turn could help ensure the inventories meet stakeholders’ needs. A well-developed inventory would provide key program, budget, and performance information in one place to help federal decision makers better understand the federal investment and results in given policy areas, and better identify and manage fragmentation, overlap, and duplication. Information architecture offers one approach to developing an inventory. As OMB determines a strategy for implementing the program inventory and develops its guidance, considering such a systematic approach to planning, organizing, and developing the inventory that centers on maximizing the use and usefulness of information could help it ensure the inventory meets GPRAMA requirements as well as the needs of decision makers and the public. Moreover, such an approach could also help OMB implement our past recommendations related to the program inventory, which are intended to ensure the inventory provides more complete information and is useful to various stakeholders. Our survey of federal managers continues to generally show no improvement in their reported use of performance information in decision making, nor in the employment of practices that can enhance such use. One area where our survey data and past work show promise is through the use of regular, leadership-driven reviews of performance data at agencies, especially when conducted in line with related leading practices. However, GPRAMA only requires these data-driven reviews for APGs, which represent a small subset of goals, both within individual agencies as well as across the government. This is probably why most federal managers were not familiar with the reviews. Identifying and sharing practices for expanding the use of those reviews—such as for additional agency-wide performance goals and at lower levels within agencies—could significantly enhance the use of performance information and drive to better and greater results. We are making the following four recommendations to OMB: The Director of OMB should update Performance.gov to explain that quarterly reporting on the fiscal year 2014 through 2017 CAP goals and fiscal year 2016 and 2017 APGs was suspended, and provide the location of final progress updates for these goals. (Recommendation 1) The Director of OMB should revise and publicly issue OMB guidance— through an update to its Circular No. A-11, a memorandum, or other means—to provide time frames and associated milestones for implementing the federal program inventory. (Recommendation 2) The Director of OMB should consider—as OMB determines its strategy for resumed implementation of the federal program inventory—using a systematic approach, such as the information architecture framework, to help ensure that GPRAMA requirements and our past recommendations for the inventory are addressed. (Recommendation 3) The Director of OMB should work with the Performance Improvement Council to identify and share among agencies practices for expanding the use of data-driven performance reviews beyond APGs, such as for other performance goals and at lower levels within agencies, that have led to performance improvements. (Recommendation 4) We provided a draft of this report to the Director of the Office of Management and Budget for review and comment. In comments provided orally and via email, OMB staff agreed with the recommendations in this report. OMB staff also asked us to (1) consider revising the draft title of the report, to better reflect progress in GPRAMA implementation, and (2) clarify our recommendations on issuing guidance for implementing the federal program inventory and expanding the use of data-driven performance reviews, by describing possible actions that could be taken to implement them. We agreed and made revisions accordingly. We are sending copies of this report to interested congressional committees, the Director of the Office of Management and Budget, and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix III. The GPRA Modernization Act (GPRAMA) includes a statutory provision for us to periodically evaluate implementation of the act. Since 2012, we have issued over 30 products in response to this provision; this is the third summary report. This report assesses how implementation of GPRAMA has affected the federal government’s progress in resolving key governance challenges in (1) addressing crosscutting issues, (2) ensuring performance information is useful and used in decision making, (3) aligning daily operations with results, and (4) building a more transparent and open government. We reviewed relevant statutory requirements, related Office of Management and Budget (OMB) guidance, and our recent work related to GPRAMA implementation and the four key governance challenges included in our reporting objectives. Specifically, since our last summary report in September 2015, we examined various aspects of GPRAMA implementation in 12 products that covered 35 agencies, including the 24 agencies covered under the Chief Financial Officers (CFO) Act of 1990, as amended (identified in table 2). We interviewed OMB and Performance Improvement Council staff to obtain (1) their perspectives on GPRAMA implementation and progress on the four governance challenges, and (2) updates on the status of our past recommendations. We also received updates from other agencies on the status of our past recommendations to them related to GPRAMA implementation. To supplement this review, we administered our periodic survey of federal managers on organizational performance and management issues from November 2016 through March 2017. This survey is comparable to five previous surveys we conducted in 1997, 2000, 2003, 2007, and 2013. We selected a stratified random sample of 4,395 people from a population of approximately 153,779 mid-level and upper-level civilian managers and supervisors working in the 24 executive branch agencies covered by the CFO Act, as shown in table 2. We obtained the sample from the Office of Personnel Management’s (OPM) Enterprise Human Resources Integration (EHRI) database as of September 30, 2015, which was the most recent fiscal year data available at the time. We used file designators indicating performance of managerial and supervisory functions. In reporting survey data, we use the term “government-wide” and the phrases “across the government” or “overall” to refer to the 24 CFO Act executive branch agencies. We use the terms “federal managers” and “managers” to collectively refer to both managers and supervisors. We designed the questionnaire to obtain the observations and perceptions of respondents on various aspects of results-oriented management topics. These topics include the presence and use of performance measures, any hindrances to measuring performance and using performance information, agency climate, and program evaluation use. To assess implementation of GPRAMA, the questionnaire included questions to collect respondents’ views on various provisions of GPRAMA, such as cross-agency priority goals, agency priority goals, and related quarterly performance reviews. Similar to the five previous surveys, the sample was stratified by agency and by whether the manager or supervisor was a member of the Senior Executive Service (SES). The management levels covered general schedule (GS) or equivalent schedules at levels comparable to GS-13 through GS-15 and career SES or equivalent. Stratifying the sample in this way ensured that the population from which we sampled covered at least 90 percent of all mid- to upper-level managers and supervisors at the departments and agencies we surveyed. Most of the items on the questionnaire were closed-ended, meaning that depending on the particular item, respondents could choose one or more response categories or rate the strength of their perception on a 5-point extent scale ranging from “no extent” to “very great extent.” On most items, respondents also had an option of choosing the response category “no basis to judge/not applicable.” A few items had other options, such as “yes,” “no,” or “do not know,” or a 3-point familiarity scale (“not familiar,” “somewhat familiar,” and “very familiar”). We asked many of the items on the questionnaire in our earlier surveys, though we introduced a number of new items in 2013, including the sections about GPRAMA and program evaluations. For 2017, we added a new question on use of performance information (question 12) and a new question on program evaluation (question 24). Before administering the survey, questions were reviewed by our staff, including subject matter experts, a survey specialist, and a research methodologist. We also conducted pretests of the new questions with federal managers in several of the 24 CFO Act agencies. We changed the wording of subquestions or added clarifying examples based on pretester feedback. To administer the survey, we e-mailed managers in the sample to notify them of the survey’s availability on our website and we included instructions on how to access and complete the survey. To follow up with managers in the sample who did not respond to the initial notice, we emailed or called multiple times to encourage survey participation or provide technical assistance, as appropriate. Similar to our last survey, we worked with OPM to obtain the names of the managers and supervisors in our sample, except for those within selected subcomponents whose names were withheld from the EHRI database. Since Foreign Service officials from the Department of State (State) are not in the EHRI database, we drew a sample for that group with the assistance from State. We worked with officials at the Department of Homeland Security (DHS) and the Department of the Treasury (Treasury) to gain access to these individuals to maintain continuity of the population of managers surveyed from previous years. The Department of Justice (DOJ) was concerned about providing identifying information (e.g., names, e-mail addresses, and phone numbers) of federal agents to us, so we administered the current survey to DOJ managers in our sample through DOJ officials. To identify the sample of managers whose names were withheld from the EHRI database, we provided DOJ with the last four digits of Social Security numbers, the subcomponent, duty location, and pay grade information. To ensure that DOJ managers received the same survey administration process as the rest of the managers in our sample to the extent possible, we provided DOJ with text for the survey activation and reminder e-mails similar to ones we emailed to managers at other agencies. DOJ administered the survey to these managers and emailed them one reminder to complete the survey. To help determine the reliability and accuracy of the EHRI data elements used to draw our sample of federal managers, we checked the data for reasonableness and the presence of any obvious or potential errors in accuracy and completeness and reviewed past analyses of the reliability of this database. For example, we identified cases where the managers’ names were withheld and contacted OPM to discuss this issue. We also checked the names of the managers in our selected sample provided by OPM with the applicable agency contacts to verify these managers were still employed with the agency. We noted discrepancies when they occurred and excluded them from our population of interest, as applicable. On the basis of these procedures, we believe the data we used from the EHRI database are sufficiently reliable for the purpose of the survey. Of the 4,395 managers selected for the 2017 survey, we found that 388 of the sampled managers had retired, separated, or otherwise left the agency or had some other reason that excluded them from the population of interest. These exclusions included managers that the agency could not locate, and therefore we were unable to request that they participate in the survey. We received usable questionnaires from 2,726 sample respondents, for a weighted response rate of about 67 percent of the remaining eligible sample. The weighted response rate across 23 of the 24 agencies ranged from 57 percent to 82 percent, while DOJ had a weighted response rate of 36 percent. See the supplemental material for each agency’s response rate. We conducted a nonresponse bias analysis using information from the survey and sampling frame as available. The analysis confirmed discrepancies in the tendency to respond to the survey related to agency and SES status. The analysis also revealed some differences in response propensity by age and GS level; however, the direction and magnitude of the differences on these factors were not consistent across agencies or strata. Our data may be subject to bias from unmeasured sources for which we cannot control. Results, and in particular estimates from agencies with low response rates such as DOJ, should be interpreted with caution because these estimates are associated with a higher level of uncertainty. The overall survey results are generalizable to the government-wide population of managers as described above. The responses of each eligible sample member who provided a usable questionnaire were weighted in the analyses to statistically account for all members of the population. All results are subject to some uncertainty or sampling error as well as nonsampling error, including the potential for nonresponse bias as noted above. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. The magnitude of sampling error will vary across the particular surveys, groups, or items being compared because we (1) used complex survey designs that differed in the underlying sample sizes, usable sample respondents, and associated variances of estimates, and (2) conducted different types of statistical analyses. For example, the 2000 and 2007 surveys were designed to produce agency-level estimates and had effective sample sizes of 2,510 and 2,943, respectively. However, the 1997 and 2003 surveys were designed to obtain government-wide estimates only, and their sample sizes were 905 and 503, respectively. Consequently, in some instances, a difference of a certain magnitude may be statistically significant. In other instances, depending on the nature of the comparison being made, a difference of equal or even greater magnitude may not achieve statistical significance. Because each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. The percentage estimates presented in this report based on our sample for the 2017 survey have 95 percent confidence intervals within plus or minus 5.5 percentage points of the estimate itself, unless otherwise noted. We also note in this report when we are 95 percent confident that changes from 1997 or 2013 relative to 2017 are statistically significant. Online supplemental material shows the questions asked on the survey along with the percentage estimates and associated 95 percent confidence intervals for each item for each agency and government-wide. In a few instances, we report estimates with larger margins of error because we deemed them reliable representations of given findings due to the statistical significance of larger differences between comparison groups. In all cases, we report the applicable margins of error. In addition to sampling errors, the practical difficulties of conducting any survey may also introduce other types of errors, commonly referred to as nonsampling errors. For example, difficulties in how a particular question is interpreted, in the sources of information available to respondents, or in how the data were entered into a database or analyzed can introduce unwanted variability into the survey results. With this survey, we took a number of steps to minimize these nonsampling errors. For example, our staff with subject matter expertise designed the questionnaire in collaboration with our survey specialists. As noted earlier, the new questions added to the survey were pretested to ensure they were relevant and clearly stated. When the data were analyzed, a second independent analyst on our staff verified the analysis programs to ensure the accuracy of the code and the appropriateness of the methods used for the computer-generated analysis. Since this was a web-based survey, respondents entered their answers directly into the electronic questionnaire, thereby eliminating the need to have the data keyed into a database, thus avoiding a source of data entry error. To supplement descriptive analysis of the survey questions, we generated an index to gauge government-wide use of performance information. The index, which was identical to one we reported in 2014, averaged manager’s responses to 11 questions deemed to relate to the concept of performance information use. The index runs from 1 (corresponding to an average value of “to no extent”) to 5 (corresponding to an average value of “to a very great extent”). We used Cronbach’s alpha to assess the internal consistency of the scale. Our government- wide index score weights each agency’s contribution equally, and provides a relative measure of the use of performance information over time rather than an absolute indicator of the government-wide level of use of performance information. We conducted this performance audit from January 2016 to September 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Office of Management and Budget (OMB) and agencies have taken some actions to address our recommendations related to implementation of the GPRA Modernization Act of 2010 (GPRAMA); however, the majority of recommendations remain open. Since GPRAMA was enacted in January 2011, we have made 100 recommendations in 18 reports to OMB and agencies aimed at improving the act’s implementation (table 3). Of those 100, OMB and the agencies have implemented 42 recommendations. Fifty-eight recommendations require additional action. Nearly half (47) of our recommendations are directed to OMB. For the 23 recommendations that OMB has implemented, many represent revisions to guidance to better reflect GPRAMA’s requirements or to enhance implementation. Many of the 24 recommendations to OMB that are not implemented deal with long-standing or complex challenges, on which OMB has taken limited action to date. Of those, we have designated 3 as priorities for OMB to address. Agencies have also taken some action on our recommendations, implementing 19 of the 53 recommendations we have made. The following tables present each of the 100 recommendations along with a summary of actions taken to address it. Tables 4 and 5 provide information about our recommendations to OMB that are implemented and not implemented, respectively. Tables 6 and 7 provide information about our recommendations to other agencies that are implemented and not implemented, respectively. In addition to the above contact, Benjamin T. Licht (Assistant Director) and Shannon Finnegan (Assistant Director) supervised this review and the development of the resulting report. Leah Q. Nash (Assistant Director), Elizabeth Fan (Analyst-in-Charge), and Adam Miles (Analyst-in- Charge) supervised the development and administration of the Federal Managers Survey and the resulting supplemental material. Peter Beck, Valerie Caracelli, Karin Fangman, Steven Flint, Robert Gebhart, Ricky Harrison Jr., John Hussey, Jill Lacey, Won Lee, Krista Loose, Meredith Moles, Anna Maria Ortiz, Steven Putansu, Alan Rozzi, Cindy Saunders, Stephanie Shipman, Shane Spencer, Andrew J. Stephens, and Brian Wanlass also made key contributions. Ann Czapiewski and Donna Miller developed the graphics for this report. John Ahern, Divya Bali, Jeff DeMarco, Alexandra Edwards, Ellen Grady, Jyoti Gupta, Erinn L. Sauer, and Katherine Wulff verified the information presented in this report. Managing for Results: Implementation of GPRA Modernization Act Has Yielded Mixed Progress in Addressing Pressing Governance Challenges. GAO-15-819. Washington, D.C.: September 30, 2015. Managing For Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges. GAO-13-518. Washington, D.C.: June 26, 2013. Supplemental Material for GAO-17-775: 2017 Survey of Federal Managers on Organizational Performance and Management Issues. GAO-17-776SP. Washington, D.C.: September 29, 2017. Program Evaluation: Annual Agency-wide Plans Could Enhance Leadership Support for Program Evaluations. GAO-17-743. Washington, D.C.: September 29, 2017. Managing for Results: Agencies’ Trends in the Use of Performance Information to Make Decisions. GAO-14-747. Washington, D.C.: September 26, 2014. Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges. GAO-13-518. Washington, D.C.: June 26, 2013. Managing for Results: 2013 Federal Managers Survey on Organizational Performance and Management Issues, an E-supplement to GAO-13-518. GAO-13-519SP. Washington, D.C.: June 26, 2013. Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making. GAO-13-570. Washington, D.C.: June 26, 2013. Government Performance: Lessons Learned for the Next Administration on Using Performance Information to Improve Results. GAO-08-1026T. Washington, D.C.: July 24, 2008. Government Performance: 2007 Federal Managers Survey on Performance and Management Issues, an E-supplement to GAO-08-1026T. GAO-08-1036SP. Washington, D.C.: July 24, 2008. Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results. GAO-04-38. Washington, D.C.: March 10, 2004. Managing for Results: Federal Managers’ Views on Key Management Issues Vary Widely Across Agencies. GAO-01-592. Washington, D.C.: May 25, 2001. Managing for Results: Federal Managers’ Views Show Need for Ensuring Top Leadership Skills. GAO-01-127. Washington, D.C.: October 20, 2000. The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven. GAO/GGD-97-109. Washington, D.C.: June 2, 1997. Federal Programs: Information Architecture Offers a Potential Approach for Inventory Development. GAO-17-739. Washington, D.C.: September 28, 2017. Managing for Results: Selected Agencies’ Experiences in Implementing Strategic Reviews. GAO-17-740R. Washington, D.C.: September 7, 2017. Federal Reports: OMB and Agencies Should More Fully Implement the Process to Streamline Reporting Requirements. GAO-17-616. Washington, D.C.: July 14, 2017. Open Innovation: Executive Branch Developed Resources to Support Implementation, but Guidance Could Better Reflect Leading Practices. GAO-17-507. Washington, D.C.: June 8, 2017. Performance Partnerships: Agencies Need to Better Identify Resource Contributions to Sustain Disconnected Youth Pilot Programs and Data to Assess Pilot Results. GAO-17-208. Washington, D.C.: April 18, 2017. Open Innovation: Practices to Engage Citizens and Effectively Implement Federal Initiatives. GAO-17-14. Washington, D.C.: October 13, 2016. Tiered Evidence Grants: Opportunities Exist to Share Lessons from Early Implementation and Inform Future Federal Efforts. GAO-16-818. Washington, D.C.: September 21, 2016. Performance.gov: Long-Term Strategy Needed to Improve Website Usability. GAO-16-693. Washington, D.C.: August 30, 2016. Tax Expenditures: Opportunities Exist to Use Budgeting and Agency Performance Processes to Increase Oversight. GAO-16-622. Washington, D.C.: July 7, 2016. Managing for Results: Agencies Need to Fully Identify and Report Major Management Challenges and Actions to Resolve them in their Agency Performance Plans. GAO-16-510. Washington, D.C.: June 15, 2016. Managing for Results: OMB Improved Implementation of Cross-Agency Priority Goals, But Could Be More Transparent About Measuring Progress. GAO-16-509. Washington, D.C.: May 20, 2016. Managing for Results: Greater Transparency Needed in Public Reporting on the Quality of Performance Information for Selected Agencies’ Priority Goals. GAO-15-788. Washington, D.C.: September 10, 2015. Pay for Success: Collaboration among Federal Agencies Would Be Helpful as Governments Explore New Financing Mechanisms. GAO-15-646. Washington, D.C.: September 9, 2015. Managing for Results: Practices for Effective Agency Strategic Reviews. GAO-15-602. Washington, D.C.: July 29, 2015. Managing for Results: Agencies Report Positive Effects of Data-Driven Reviews on Performance but Some Should Strengthen Practices. GAO-15-579. Washington, D.C.: July 7, 2015. Program Evaluation: Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity. GAO-15-25. Washington, D.C.: November 13, 2014. Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories. GAO-15-83. Washington, D.C.: October 31, 2014. Managing for Results: Selected Agencies Need to Take Additional Efforts to Improve Customer Service. GAO-15-84. Washington, D.C.: October 24, 2014. Managing for Results: Enhanced Goal Leader Accountability and Collaboration Could Further Improve Agency Performance. GAO-14-639. Washington, D.C.: July 22, 2014. Managing for Results: OMB Should Strengthen Reviews of Cross-Agency Goals. GAO-14-526. Washington, D.C.: June 10, 2014. Managing for Results: Implementation Approaches Used to Enhance Collaboration in Interagency Groups. GAO-14-220. Washington, D.C.: February 14, 2014. Managing for Results: Leading Practices Should Guide the Continued Development of Performance.gov. GAO-13-517. Washington, D.C.: June 6, 2013. Managing for Results: Agencies Should More Fully Develop Priority Goals under the GPRA Modernization Act. GAO-13-174. Washington, D.C.: April 19, 2013. Managing for Results: Agencies Have Elevated Performance Management Roles, but Additional Training Is Needed. GAO-13-356. Washington, D.C.: April 16, 2013. Managing for Results: Data-Driven Performance Reviews Show Promise But Agencies Should Explore How to Involve Other Relevant Agencies. GAO-13-228. Washington, D.C.: February 27, 2013. Managing for Results: A Guide for Using the GPRA Modernization Act to Help Inform Congressional Decision Making. GAO-12-621SP. Washington, D.C.: June 15, 2012. Managing for Results: GAO’s Work Related to the Interim Crosscutting Priority Goals under the GPRA Modernization Act. GAO-12-620R. Washington, D.C.: May 31, 2012. Managing for Results: Opportunities for Congress to Address Government Performance Issues. GAO-12-215R. Washington, D.C.: December 9, 2011.

Question: Write a one-page summary of the report 1.

Answer: Lead paint in housing is the most common source of lead exposure for U.S. children. HUD awards grants to state and local governments to reduce lead paint hazards in housing and oversees compliance with lead paint regulations in its rental assistance programs. The 2017 Consolidated Appropriations Act, Joint Explanatory Statement, includes a provision that GAO review HUD’s efforts to address lead paint hazards. This report examines HUD’s efforts to (1) incorporate statutory requirements and other relevant federal standards in its lead grant programs, (2) monitor and enforce compliance with lead paint regulations in its rental assistance programs, (3) adopt federal health guidelines and environmental standards for its lead grant and rental assistance programs, and (4) measure and report on the performance of its lead efforts. GAO reviewed HUD documents and data related to its grant programs, compliance efforts, performance measures, and reporting. GAO also interviewed HUD staff and some grantees. The Department of Housing and Urban Development’s (HUD) lead grant and rental assistance programs have taken steps to address lead paint hazards, but opportunities exist for improvement. For example, in 2016, HUD began using new tools to monitor how public housing agencies comply with lead paint regulations. However, HUD could further improve efforts in the following areas: Lead grant programs. While its recent grant award processes incorporate statutory requirements on applicant eligibility and selection criteria, HUD has not fully documented or evaluated these processes. For example, HUD’s guidance is not sufficiently detailed to ensure consistent and appropriate grant award decisions. Better documentation and evaluation of HUD’s grant program processes could help ensure that lead grants reach areas at risk of lead paint hazards. Further, HUD has not developed specific time frames for using available local-level data to better identify areas of the country at risk for lead paint hazards, which could help HUD target its limited resources. Oversight. HUD does not have a plan to mitigate and address risks related to noncompliance with lead paint regulations by public housing agencies. We identified several limitations with HUD’s monitoring efforts, including reliance on public housing agencies’ self-certifying compliance with lead paint regulations and challenges identifying children with elevated blood lead levels. Additionally, HUD lacks detailed procedures for addressing noncompliance consistently and in a timely manner. Developing a plan and detailed procedures to address noncompliance with lead paint regulations could strengthen HUD’s oversight of public housing agencies. Inspections. The lead inspection standard for the Housing Choice Voucher program is less strict than that of the public housing program. By requesting and obtaining statutory authority to amend the standard for the voucher program, HUD would be positioned to take steps to better protect children in voucher units from lead exposure as indicated by analysis of benefits and costs. Performance assessment and reporting. HUD lacks comprehensive goals and performance measures for its lead reduction efforts. In addition, it has not complied with annual statutory reporting requirements, last reporting as required on its lead efforts in 1997. Without better performance assessment and reporting, HUD cannot fully assess the effectiveness of its lead efforts. GAO makes nine recommendations to HUD including to improve lead grant program and compliance monitoring processes, request authority to amend its lead inspection standard in the voucher program, and take additional steps to report on progress. HUD generally agreed with eight of the recommendations. HUD disagreed that it should request authority to use a specific, stricter inspection standard. GAO revised this recommendation to allow HUD greater flexibility to amend its current inspection standard as indicated by analysis of the benefits and costs.