Moving to Work Demonstration: Improved Information and Monitoring Could Enhance Program Assessment
Published: 2013-06-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Implemented in 1999, HUDs MTW demonstration program gives participating PHAs the flexibility to create innovative housing strategies. MTW agencies must create activities linked to three statutory purposesreducing costs, providing incentives for self-sufficiency, and increasing housing choicesand meet five statutory requirements. Congress has been considering expanding MTW.
This testimony discusses (1) the programs progress in addressing the three purposes, (2) HUDs monitoring efforts, and (3) potential benefits of and concerns about expansion.
This testimony draws from a prior report on the MTW program ( GAO-12-490 ). For that report, GAO analyzed the most current annual reports for 30 MTW agencies; compared HUDs monitoring efforts with internal control standards; and interviewed agency officials, researchers, and industry officials. For this testimony, GAO also reviewed actions HUD has taken in response to the reports recommendations.
section_title: What GAO Found
paragraphs: Opportunities existed to improve how the Department of Housing and Urban Development (HUD) evaluated the Moving to Work (MTW) program, which is intended to give participating public housing agencies (PHA) flexibility to design and test innovative strategies for providing housing assistance. GAO reported in April 2012 that HUD had not (1) developed guidance specifying that performance information collected from MTW agencies be outcome-oriented, (2) identified the performance data needed to assess results, or (3) established performance indicators for the program. The shortage of such standard performance data and indicators had hindered comprehensive evaluation efforts; such evaluations are key to determining the success of any demonstration program. In addition, HUD had not developed a systematic process for identifying lessons learned from the program, which limited HUD's ability to promote useful practices for broader implementation. Since the GAO report, HUD has revised reporting requirements for MTW agencies. These requirements were approved by the Office of Management and Budget in May 2013. GAO is reviewing this new guidance.
In 2012, GAO also reported that HUD had not taken key monitoring steps set out in internal control standards, such as issuing guidance that defines program terms or assessing compliance with all the program's statutory requirements. As a result, HUD lacked assurance that MTW agencies were complying with statutory requirements. Additionally, HUD had not done an annual assessment of program risks, although it had a requirement to do so, and had not developed risk-based monitoring procedures. Without taking these steps, HUD lacked assurance that it had identified all risks to the program. Finally, HUD did not have policies or procedures in place to verify the accuracy of key information that MTW agencies self-report. For example, HUD staff did not verify self-reported performance information during their reviews of annual reports or annual site visits. Without verifying at least a sample of information, HUD could not be sure that self-reported information was accurate. According to HUD, the recently approved reporting requirements will result in more standardized data that HUD can verify either through audits or during site visits.
Finally, GAO noted in 2012 that expanding the MTW program might offer benefits but also raised questions. According to HUD, affordable housing advocates, and MTW agencies, expanding MTW to additional PHAs would allow agencies to develop more activities tailored to local conditions and produce more lessons learned. However, data limitations and monitoring weaknesses raised questions about expansion. HUD had reported in 2010 that expansion should occur only if newly admitted PHAs structured their programs to permit high-quality evaluations and ensure that lessons learned could be generalized. Since the GAO report was issued, four additional agencies were admitted into the program. HUD required these agencies to implement and study rent reform activities through partnerships with local universities and a research organization. Until more complete information on the program's effectiveness and the extent to which agencies adhered to program requirements is available, it will be difficult for Congress to know whether an expanded MTW would benefit additional agencies and the residents they serve.
section_title: What GAO Recommends
paragraphs: GAO recommended that HUD improve MTW information and monitoring. HUD partially agreed with these recommendations and has since issued new guidance to MTW agencies.

section_title: Letter
paragraphs: I am pleased to be here today to discuss our work on the Department of  Housing and Urban Development’s (HUD) Moving to Work (MTW)  demonstration program. The MTW program has three statutory purposes:  to reduce costs and achieve greater cost-effectiveness in federal housing  expenditures, to give families with children incentives to obtain  employment and become self-sufficient, and to increase housing choices  for low-income families. MTW was conceived to help test new ideas for  providing and administering housing assistance that could be replicated  on a broader scale. However, HUD only recently has begun to put in  place policies and practices that could help realize that potential.
My statement today is based on an April 2012 report on the MTW  program. Specifically, I will discuss (1) what was known at that time  about the extent to which the program had addressed statutory purposes,  (2) HUD’s monitoring of the way MTW agencies addressed statutory  purposes and met program requirements, and (3) potential benefits of  and concerns about expanding the number of public housing agencies in  the program. I also will provide information on actions that HUD has taken  in response to the report’s recommendations.
According to HUD data, as of January 2012, 35 agencies were in the MTW program. Of  these, 30 had submitted an annual report to HUD and the other 5 had not been in the  program long enough to report on their accomplishments. officials, officials from a sample of seven MTW agencies, and  organizations that advocate on behalf of residents, among others. Our  prior report includes a detailed description of our scope and methodology.  For this testimony, we reviewed some recent changes to the program.
We performed the work on which this statement was primarily based from  July 2011 to April 2012 in accordance with generally accepted  government auditing standards. Those standards require that we plan  and perform the audit to obtain sufficient, appropriate evidence to provide  a reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: HUD implemented the MTW demonstration program in 1999. As of June  2013, 35 public housing agencies (PHA) were participating through the  end of their fiscal year 2018. To put in place the innovations intended  under the program’s authorizing legislation, agencies may request  waivers of certain provisions in the United States Housing Act of 1937, as  amended. For example, housing agencies may combine the funding they  are awarded annually from different programs—such as public housing  capital funds, public housing operating funds, and voucher funds—into a  single, authoritywide funding source.
In addition to addressing the program’s three statutory purposes—reduce  costs and achieve greater cost-effectiveness in federal housing  expenditures, give families with children incentives to obtain employment  and become self-sufficient, and increase housing choices for low-income  families—MTW agencies must meet five requirements. The agencies  must (1) serve substantially the same total number of eligible low-income  families that they would have served had funding amounts not been  combined; (2) maintain a mix of families (by family size) comparable to  those they would have served without the demonstration; (3) ensure that  at least 75 percent of households served are very low-income;establish a reasonable rent policy to encourage employment and self- sufficiency; and (5) assure that the housing provided meets HUD’s  housing quality standards.   (4)
A standard agreement (between HUD and each MTW agency) governs  the conditions of participation in the program. The agreement includes an  attachment that sets out reporting requirements, as well as the  information that MTW agencies must include in annual reports. For  example, these reports must include detailed information on the impact of  each activity.
MTW agencies also must self-certify that they are in compliance with  three of the five statutory requirements: assisting substantially the same  total number of eligible low-income families that they would have served  had funding amounts not been combined; maintaining a mix (by family  size) comparable to those they would have served had funding amounts  not been combined under the demonstration; and ensuring that at least  75 percent of households served are very low-income.
section_title: HUD Lacked Standard Data and Indicators That Would Aid Program Evaluation
paragraphs: In our 2012 report, we identified a number of weaknesses related to MTW  data, performance indicators, and identification of lessons learned—all of  which resulted in a limited ability to determine program outcomes as they  related to statutory purposes. Although MTW agencies reported annually  on their activities, which included efforts to reduce administrative costs  and encourage residents to work, the usefulness of this information was  limited because it was not consistently outcome-oriented. For example,  for similar activities designed to promote family self-sufficiency, one MTW  agency reported only the number of participants, which is generally  considered an output, and another did not provide any performance  information. In contrast, a third agency reported on the average income of  program graduates, which we consider an outcome. To be consistent with  GPRAMA, HUD’s guidance on reporting performance information should  indicate the importance of outcome-oriented information.specific guidance on the reporting of performance information—for   Without more  example, to report quantifiable and outcome-oriented information—HUD  could not be assured of collecting information that reflected the outcomes  of individual activities.
As we reported in 2012, HUD had not identified the performance data  needed to assess the results of similar MTW activities or of the program  as a whole. Obtaining performance information from demonstration  programs is critical—because the purpose of a demonstration is to test  which approach obtains positive results. Although HUD started collecting  additional data from MTW agencies (including household size, income,  and educational attainment) in its MTW database, it had not analyzed the  data. And since 2009, HUD had required agencies to provide information  on the impact of activities, including benchmarks and metrics, in their  annual MTW reports. While these reports were informative, they did not  lend themselves to quantitative analysis because the reporting  requirements did not call for standardized data, such as the number of  residents who found employment. Whether these data would be sufficient  to assess similar activities and the program as a whole was not clear, and  as of April 2012 HUD had not identified the data it would need for such an  assessment.
HUD also had not established performance indicators for MTW.  According to GPRAMA, federal agencies should establish efficiency,  output, and outcome indicators for each program activity as appropriate.  Federal internal control standards also require the establishment of  performance indicators. As we noted in 2012, specific performance indicators for the MTW program could be based on the three statutory  purposes of the program. For example, agencies could report on the  savings achieved (reducing costs). However, without performance  indicators HUD could not demonstrate the results of the program.
The shortage of standard performance data and performance indicators  had hindered comprehensive evaluation efforts, which are key to  determining the success of any demonstration program. We  recommended in 2012 that HUD (1) improve its guidance to MTW  agencies on providing performance information in their annual reports by  requiring that such information be quantifiable and outcome-oriented, (2)  develop and implement a plan for quantitatively assessing the  effectiveness of similar activities and for the program, and (3) establish  performance indicators for the program. HUD partially agreed with these  recommendations. Since our report, HUD has revised the performance  reporting requirements for MTW agencies. The Office of Management  and Budget (OMB) approved these revisions on May 31, 2013. The new  requirements state that MTW agencies are to report standard metrics and  report outcome information on the effects of MTW policy changes on  residents. HUD also provided a standard format to allow analysis and  aggregation across agencies for similar activities. We are currently  assessing the extent to which these new requirements address our  recommendations.
Furthermore, as we indicated in our 2012 report, while HUD had identified  some lessons learned on an ad hoc basis, it did not have a systematic  process in place for identifying such lessons. As previously noted,  obtaining impact information from demonstration programs is critical.  Since 2000, HUD had identified some activities that could be replicated  by other housing agencies. For example, a HUD-sponsored contractor  developed five case studies to describe issues and challenges involved in  implementing MTW. However, these and subsequent efforts had  shortcomings. In most cases, the choice of lessons learned was based on  the opinions of HUD or contracted staff and largely involved anecdotal (or  qualitative) data rather than quantitative data. Because HUD had not  developed criteria and a systematic process for identifying lessons  learned, we reported in 2012 that it was limited in its ability to promote  useful practices for broader implementation. Thus, we recommended that  HUD create a process to systematically identify lessons learned. HUD  agreed and in response, stated that once its revised reporting  requirements were implemented, the resulting data would inform an effort  to establish lessons learned. Consistent with this, HUD noted that one  purpose of the revised reporting requirements that OMB approved in May  2013 was to identify promising practices learned through the MTW  demonstration.
section_title: HUD Could Strengthen Some Monitoring Policies and Procedures
paragraphs: HUD had policies and procedures in place to monitor MTW agencies but  could have done more to ensure that MTW agencies demonstrated  compliance with statutory requirements and to identify possible risks  relating to each agency’s activities. For example, as noted in our 2012  report, HUD had not issued guidance to MTW agencies clarifying key  program terms, including definitions of the purposes and statutory  requirements of the MTW program. Federal internal control standards  require the establishment of clear, consistent goals and objectives.  Agencies also must link each of their activities to one of the three  program purposes cited in the MTW authorizing legislation. However, at  that time HUD had not clearly defined what some of the statutory  language meant, such as “increasing housing choices for low-income  families.” HUD officials acknowledged that the guidance could be  strengthened. At the time, they told us that they planned to update the  guidance to more completely collect information related to the program’s  statutory purposes and requirements. As discussed later, HUD has since  updated its guidance.
Additionally, we reported in 2012 that HUD had only recently assessed  agencies’ compliance with two (self-certified) requirements—to serve  substantially the same total number of eligible low-income families that  they would have served had funding amounts not been combined and  ensure that at least 75 percent of households served were very low- income. Also, HUD had not assessed compliance with the third (also self- certified) requirement—to maintain a comparable mix of families. Federal  internal control standards require control activities to be in place to  address program risks. formulate an approach for assessing compliance with program  requirements. Without a process for systematically assessing  compliance with statutory requirements, HUD lacked assurance that  agencies were complying with them.
GAO/AIMD-00-21.3.1. program offices to perform an annual risk assessment of their programs  or administrative functions using a HUD risk-assessment worksheet. By  not performing annual risk assessments or tailoring its monitoring efforts  to reflect the perceived risk of each MTW agency, HUD lacked assurance  that it had properly identified and addressed risks that may prevent  agencies from addressing program purposes and meeting statutory  requirements. HUD also lacked assurance that it had been using its  limited monitoring resources efficiently.
Finally, we reported that HUD did not have policies or procedures in place  to verify the accuracy of key information that agencies self-report, such as  the number of program participants and the average income of residents  “graduating” from MTW programs. Internal control standards and  guidance emphasize the need for federal agencies to have control  activities in place to help ensure that program participants report  information accurately. reported performance information during their reviews of annual reports  or annual site visits. GAO guidance on data reliability recommends  tracing a sample of data records to source documents to determine  whether the data accurately and completely reflect the source  documents.performance information, it lacked assurance that this information was  accurate. To the extent that HUD relied on this information to assess  program compliance with statutory purposes and requirements, its  analyses were limited.
GAO/AIMD-00-21.3.1 and GAO-01-1008G. information that MTW agencies self-report. HUD partially agreed with our  recommendations, citing potential difficulties in verifying MTW  performance data. HUD also described steps it was taking to improve its  guidance to MTW agencies and implement risk-based monitoring  procedures. In May 2013, OMB approved revised reporting guidance to  MTW agencies. The guidance requires agencies to report information  related to the program’s statutory purposes and requirements. For  example, it includes a template for data on compliance with the  requirement to maintain a comparable mix of families. Additionally,  according to a HUD official, the recently approved reporting requirements  will result in more standardized data that HUD can verify either through  audits or during site visits. As noted above, we are assessing this  guidance.
section_title: An MTW Expansion May Offer Benefits but Has Raised Questions
paragraphs: Legislation has been proposed to expand the number of PHAs that can  participate in the MTW program, and a 2010 HUD report recommended  expanding the program to up to twice its size. We reported in 2012 that  HUD and some stakeholders believed that expansion could provide  information on the effect of the MTW program and allow more PHAs to  test innovative ideas, but questions remained about the lack of  performance information on current MTW activities. Since our report was  issued, four additional agencies were admitted into the program. HUD  required these agencies to implement and study rent reform activities  through partnerships with local universities and a research organization.
HUD, Moving to Work (2010). had occurred in some of the communities affected by the MTW program  and indicated that expansion could enable more PHAs to address local  needs and therefore benefit additional communities. Similarly, officials  from MTW agencies that we contacted stated that expansion of the  program would provide a broader testing ground for new approaches and  best practices. Finally, information from a private research organization,  affordable housing advocates, and MTW agencies suggested that  allowing additional PHAs to participate in the program could result in  additional opportunities to test innovative ideas and tailor housing  programs and activities to local conditions. In 2004, the Urban Institute  reported that the local flexibility and independence permitted under MTW  appeared to allow strong, creative PHAs to experiment with innovative  solutions to local challenges. We have reported separately on cost  savings that could be realized from allowing additional housing authorities  to implement some of the reforms MTW agencies have tested.
Some proponents of expansion that we interviewed also noted that  expanding the MTW program could provide more PHAs with the ability to  use funding from different sources more flexibly than possible without  MTW status. As we have seen, MTW agencies may request waivers of  certain provisions of the 1937 Housing Act in order to combine annual  funding from separate sources into a single authoritywide funding source.  HUD field office staff with responsibility for monitoring MTW agencies  observed that the single-fund flexibility was beneficial because it enabled  participating agencies to develop supportive service programs, such as  job training or educational programs, which help move families toward  self sufficiency. Further, officials from the MTW agencies we interviewed  agreed that this flexibility was beneficial. For example, officials from one  MTW agency stated it had been able to use the single fund to organize  itself as a business organization, develop a strategic plan based on the  housing needs of low-income families in the community, leverage public  funds and public and private partnerships, and develop mixed-income  communities.
However, a lack of performance information (which creates a limited basis  for judging what lessons could be taken from the program to date), limited  HUD oversight, and concerns about the program’s impact on residents  raised questions about expanding the MTW program. In its 2010 report to  Congress, HUD acknowledged that the conclusive impacts of many MTW  activities, particularly as they relate to residents, were not yet known.
For example, the report noted that the rent reforms implemented under  MTW varied greatly and were not implemented using a controlled  experimental methodology. As a result, it was not clear which aspects of  rent reforms should be recommended for all PHAs. The report also noted  the limitations relating to evaluating the outcomes of MTW—limitations  that stemmed from the weak initial reporting requirements and lack of a  research design. The report concluded that, given these limitations,  expansion should occur only if newly admitted PHAs structured their  programs for high-quality evaluations that permitted lessons learned to be  generalized for other PHAs. Similarly, representatives of affordable  housing advocates and legal aid organizations that we interviewed stated  that because lessons had not been learned from MTW, there was no  basis for expanding the program.
Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004). agencies were added under the current program design, HUD might need  additional resources.
Researchers and representatives of several affordable housing advocates  and legal aid agencies with whom we met also suggested that an  expanded program could negatively affect residents. For example, two  research organizations had stated that some voucher policies could  reduce portability—that is, residents’ ability to use their rental vouchers  outside the area that the voucher-issuing PHA served. One of these  organizations stated that differences in the way voucher programs were  implemented across MTW agencies could reduce residents’ ability to use  vouchers outside of the area where they received assistance. Officials  from the other organization noted that some MTW agencies prohibited  vouchers from being used outside of the originating jurisdictions, thereby  limiting housing choices. According to HUD officials, MTW agencies with  policies that limit portability could make exceptions. For example, these  agencies had made exceptions for residents seeking employment  opportunities.
Until more complete information on the program’s effectiveness and the  extent to which agencies adhered to program requirements is available, it  will be difficult for Congress to know whether an expanded MTW would  benefit additional agencies and the residents they serve.
Mr. Chairman, Ranking Member Capuano, and Members of the  Subcommittee, this concludes my prepared statement. I would be happy  to respond to any questions that you may have at this time.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For further information about this testimony, please contact me at   (202) 512-8678 or sciremj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this testimony  include Paige Smith, Assistant Director; Emily Chalmers; John McGrail;  Lisa Moore; Daniel Newman; Lauren Nunnally; Barbara Roesmann; and  Andrew Stavisky.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.