Moving to Work Demonstration: Improved Information and Monitoring Could Enhance Program Assessment
Released: 2013-06-26T00:00:00

Letter

Background

HUD implemented the MTW demonstration program in 1999. As of June 2013, 35 public housing agencies (PHA) were participating through the end of their fiscal year 2018. To put in place the innovations intended under the program’s authorizing legislation, agencies may request waivers of certain provisions in the United States Housing Act of 1937, as amended. For example, housing agencies may combine the funding they are awarded annually from different programs—such as public housing capital funds, public housing operating funds, and voucher funds—into a single, authoritywide funding source.

In addition to addressing the program’s three statutory purposes—reduce costs and achieve greater cost-effectiveness in federal housing expenditures, give families with children incentives to obtain employment and become self-sufficient, and increase housing choices for low-income families—MTW agencies must meet five requirements. The agencies must (1) serve substantially the same total number of eligible low-income families that they would have served had funding amounts not been combined; (2) maintain a mix of families (by family size) comparable to those they would have served without the demonstration; (3) ensure that at least 75 percent of households served are very low-income;establish a reasonable rent policy to encourage employment and self- sufficiency; and (5) assure that the housing provided meets HUD’s housing quality standards. (4)

A standard agreement (between HUD and each MTW agency) governs the conditions of participation in the program. The agreement includes an attachment that sets out reporting requirements, as well as the information that MTW agencies must include in annual reports. For example, these reports must include detailed information on the impact of each activity.

MTW agencies also must self-certify that they are in compliance with three of the five statutory requirements: assisting substantially the same total number of eligible low-income families that they would have served had funding amounts not been combined; maintaining a mix (by family size) comparable to those they would have served had funding amounts not been combined under the demonstration; and ensuring that at least 75 percent of households served are very low-income.

HUD Lacked Standard Data and Indicators That Would Aid Program Evaluation

In our 2012 report, we identified a number of weaknesses related to MTW data, performance indicators, and identification of lessons learned—all of which resulted in a limited ability to determine program outcomes as they related to statutory purposes. Although MTW agencies reported annually on their activities, which included efforts to reduce administrative costs and encourage residents to work, the usefulness of this information was limited because it was not consistently outcome-oriented. For example, for similar activities designed to promote family self-sufficiency, one MTW agency reported only the number of participants, which is generally considered an output, and another did not provide any performance information. In contrast, a third agency reported on the average income of program graduates, which we consider an outcome. To be consistent with GPRAMA, HUD’s guidance on reporting performance information should indicate the importance of outcome-oriented information.specific guidance on the reporting of performance information—for Without more example, to report quantifiable and outcome-oriented information—HUD could not be assured of collecting information that reflected the outcomes of individual activities.

As we reported in 2012, HUD had not identified the performance data needed to assess the results of similar MTW activities or of the program as a whole. Obtaining performance information from demonstration programs is critical—because the purpose of a demonstration is to test which approach obtains positive results. Although HUD started collecting additional data from MTW agencies (including household size, income, and educational attainment) in its MTW database, it had not analyzed the data. And since 2009, HUD had required agencies to provide information on the impact of activities, including benchmarks and metrics, in their annual MTW reports. While these reports were informative, they did not lend themselves to quantitative analysis because the reporting requirements did not call for standardized data, such as the number of residents who found employment. Whether these data would be sufficient to assess similar activities and the program as a whole was not clear, and as of April 2012 HUD had not identified the data it would need for such an assessment.

HUD also had not established performance indicators for MTW. According to GPRAMA, federal agencies should establish efficiency, output, and outcome indicators for each program activity as appropriate. Federal internal control standards also require the establishment of performance indicators. As we noted in 2012, specific performance indicators for the MTW program could be based on the three statutory purposes of the program. For example, agencies could report on the savings achieved (reducing costs). However, without performance indicators HUD could not demonstrate the results of the program.

The shortage of standard performance data and performance indicators had hindered comprehensive evaluation efforts, which are key to determining the success of any demonstration program. We recommended in 2012 that HUD (1) improve its guidance to MTW agencies on providing performance information in their annual reports by requiring that such information be quantifiable and outcome-oriented, (2) develop and implement a plan for quantitatively assessing the effectiveness of similar activities and for the program, and (3) establish performance indicators for the program. HUD partially agreed with these recommendations. Since our report, HUD has revised the performance reporting requirements for MTW agencies. The Office of Management and Budget (OMB) approved these revisions on May 31, 2013. The new requirements state that MTW agencies are to report standard metrics and report outcome information on the effects of MTW policy changes on residents. HUD also provided a standard format to allow analysis and aggregation across agencies for similar activities. We are currently assessing the extent to which these new requirements address our recommendations.

Furthermore, as we indicated in our 2012 report, while HUD had identified some lessons learned on an ad hoc basis, it did not have a systematic process in place for identifying such lessons. As previously noted, obtaining impact information from demonstration programs is critical. Since 2000, HUD had identified some activities that could be replicated by other housing agencies. For example, a HUD-sponsored contractor developed five case studies to describe issues and challenges involved in implementing MTW. However, these and subsequent efforts had shortcomings. In most cases, the choice of lessons learned was based on the opinions of HUD or contracted staff and largely involved anecdotal (or qualitative) data rather than quantitative data. Because HUD had not developed criteria and a systematic process for identifying lessons learned, we reported in 2012 that it was limited in its ability to promote useful practices for broader implementation. Thus, we recommended that HUD create a process to systematically identify lessons learned. HUD agreed and in response, stated that once its revised reporting requirements were implemented, the resulting data would inform an effort to establish lessons learned. Consistent with this, HUD noted that one purpose of the revised reporting requirements that OMB approved in May 2013 was to identify promising practices learned through the MTW demonstration.

HUD Could Strengthen Some Monitoring Policies and Procedures

HUD had policies and procedures in place to monitor MTW agencies but could have done more to ensure that MTW agencies demonstrated compliance with statutory requirements and to identify possible risks relating to each agency’s activities. For example, as noted in our 2012 report, HUD had not issued guidance to MTW agencies clarifying key program terms, including definitions of the purposes and statutory requirements of the MTW program. Federal internal control standards require the establishment of clear, consistent goals and objectives. Agencies also must link each of their activities to one of the three program purposes cited in the MTW authorizing legislation. However, at that time HUD had not clearly defined what some of the statutory language meant, such as “increasing housing choices for low-income families.” HUD officials acknowledged that the guidance could be strengthened. At the time, they told us that they planned to update the guidance to more completely collect information related to the program’s statutory purposes and requirements. As discussed later, HUD has since updated its guidance.

Additionally, we reported in 2012 that HUD had only recently assessed agencies’ compliance with two (self-certified) requirements—to serve substantially the same total number of eligible low-income families that they would have served had funding amounts not been combined and ensure that at least 75 percent of households served were very low- income. Also, HUD had not assessed compliance with the third (also self- certified) requirement—to maintain a comparable mix of families. Federal internal control standards require control activities to be in place to address program risks. formulate an approach for assessing compliance with program requirements. Without a process for systematically assessing compliance with statutory requirements, HUD lacked assurance that agencies were complying with them.

GAO/AIMD-00-21.3.1. program offices to perform an annual risk assessment of their programs or administrative functions using a HUD risk-assessment worksheet. By not performing annual risk assessments or tailoring its monitoring efforts to reflect the perceived risk of each MTW agency, HUD lacked assurance that it had properly identified and addressed risks that may prevent agencies from addressing program purposes and meeting statutory requirements. HUD also lacked assurance that it had been using its limited monitoring resources efficiently.

Finally, we reported that HUD did not have policies or procedures in place to verify the accuracy of key information that agencies self-report, such as the number of program participants and the average income of residents “graduating” from MTW programs. Internal control standards and guidance emphasize the need for federal agencies to have control activities in place to help ensure that program participants report information accurately. reported performance information during their reviews of annual reports or annual site visits. GAO guidance on data reliability recommends tracing a sample of data records to source documents to determine whether the data accurately and completely reflect the source documents.performance information, it lacked assurance that this information was accurate. To the extent that HUD relied on this information to assess program compliance with statutory purposes and requirements, its analyses were limited.

GAO/AIMD-00-21.3.1 and GAO-01-1008G. information that MTW agencies self-report. HUD partially agreed with our recommendations, citing potential difficulties in verifying MTW performance data. HUD also described steps it was taking to improve its guidance to MTW agencies and implement risk-based monitoring procedures. In May 2013, OMB approved revised reporting guidance to MTW agencies. The guidance requires agencies to report information related to the program’s statutory purposes and requirements. For example, it includes a template for data on compliance with the requirement to maintain a comparable mix of families. Additionally, according to a HUD official, the recently approved reporting requirements will result in more standardized data that HUD can verify either through audits or during site visits. As noted above, we are assessing this guidance.

An MTW Expansion May Offer Benefits but Has Raised Questions

Legislation has been proposed to expand the number of PHAs that can participate in the MTW program, and a 2010 HUD report recommended expanding the program to up to twice its size. We reported in 2012 that HUD and some stakeholders believed that expansion could provide information on the effect of the MTW program and allow more PHAs to test innovative ideas, but questions remained about the lack of performance information on current MTW activities. Since our report was issued, four additional agencies were admitted into the program. HUD required these agencies to implement and study rent reform activities through partnerships with local universities and a research organization.

HUD, Moving to Work (2010). had occurred in some of the communities affected by the MTW program and indicated that expansion could enable more PHAs to address local needs and therefore benefit additional communities. Similarly, officials from MTW agencies that we contacted stated that expansion of the program would provide a broader testing ground for new approaches and best practices. Finally, information from a private research organization, affordable housing advocates, and MTW agencies suggested that allowing additional PHAs to participate in the program could result in additional opportunities to test innovative ideas and tailor housing programs and activities to local conditions. In 2004, the Urban Institute reported that the local flexibility and independence permitted under MTW appeared to allow strong, creative PHAs to experiment with innovative solutions to local challenges. We have reported separately on cost savings that could be realized from allowing additional housing authorities to implement some of the reforms MTW agencies have tested.

Some proponents of expansion that we interviewed also noted that expanding the MTW program could provide more PHAs with the ability to use funding from different sources more flexibly than possible without MTW status. As we have seen, MTW agencies may request waivers of certain provisions of the 1937 Housing Act in order to combine annual funding from separate sources into a single authoritywide funding source. HUD field office staff with responsibility for monitoring MTW agencies observed that the single-fund flexibility was beneficial because it enabled participating agencies to develop supportive service programs, such as job training or educational programs, which help move families toward self sufficiency. Further, officials from the MTW agencies we interviewed agreed that this flexibility was beneficial. For example, officials from one MTW agency stated it had been able to use the single fund to organize itself as a business organization, develop a strategic plan based on the housing needs of low-income families in the community, leverage public funds and public and private partnerships, and develop mixed-income communities.

However, a lack of performance information (which creates a limited basis for judging what lessons could be taken from the program to date), limited HUD oversight, and concerns about the program’s impact on residents raised questions about expanding the MTW program. In its 2010 report to Congress, HUD acknowledged that the conclusive impacts of many MTW activities, particularly as they relate to residents, were not yet known.

For example, the report noted that the rent reforms implemented under MTW varied greatly and were not implemented using a controlled experimental methodology. As a result, it was not clear which aspects of rent reforms should be recommended for all PHAs. The report also noted the limitations relating to evaluating the outcomes of MTW—limitations that stemmed from the weak initial reporting requirements and lack of a research design. The report concluded that, given these limitations, expansion should occur only if newly admitted PHAs structured their programs for high-quality evaluations that permitted lessons learned to be generalized for other PHAs. Similarly, representatives of affordable housing advocates and legal aid organizations that we interviewed stated that because lessons had not been learned from MTW, there was no basis for expanding the program.

Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004). agencies were added under the current program design, HUD might need additional resources.

Researchers and representatives of several affordable housing advocates and legal aid agencies with whom we met also suggested that an expanded program could negatively affect residents. For example, two research organizations had stated that some voucher policies could reduce portability—that is, residents’ ability to use their rental vouchers outside the area that the voucher-issuing PHA served. One of these organizations stated that differences in the way voucher programs were implemented across MTW agencies could reduce residents’ ability to use vouchers outside of the area where they received assistance. Officials from the other organization noted that some MTW agencies prohibited vouchers from being used outside of the originating jurisdictions, thereby limiting housing choices. According to HUD officials, MTW agencies with policies that limit portability could make exceptions. For example, these agencies had made exceptions for residents seeking employment opportunities.

Until more complete information on the program’s effectiveness and the extent to which agencies adhered to program requirements is available, it will be difficult for Congress to know whether an expanded MTW would benefit additional agencies and the residents they serve.

Mr. Chairman, Ranking Member Capuano, and Members of the Subcommittee, this concludes my prepared statement. I would be happy to respond to any questions that you may have at this time.

GAO Contact and Staff Acknowledgments

For further information about this testimony, please contact me at (202) 512-8678 or sciremj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this testimony include Paige Smith, Assistant Director; Emily Chalmers; John McGrail; Lisa Moore; Daniel Newman; Lauren Nunnally; Barbara Roesmann; and Andrew Stavisky.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.