Moving to Work Demonstration: Opportunities Exist to Improve Information and Monitoring
Published: 2012-04-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: HUDs MTW demonstration program gives participating PHAs the flexibility to create innovative housing strategies through their fiscal year 2018. MTW agencies must create activities linked to three statutory purposesreducing costs, providing incentives for self-sufficiency, and increasing housing choicesand meet five statutory requirements. Congress is considering expanding MTW and has asked GAO to examine what is known about (1) the programs success in addressing the three purposes, (2) HUDs monitoring efforts, and (3) the potential benefits of and concerns about expansion. GAO analyzed the most current annual reports for 30 MTW agencies; compared HUDs monitoring efforts with internal control standards; and interviewed agency officials, researchers, and industry officials.
section_title: What GAO Found
paragraphs: Public housing agencies (PHA) that participate in the Moving to Work (MTW) program report annually on the performance of their activities, which include efforts to reduce administrative costs and encourage residents to work. But this performance information varies, and the Department of Housing and Urban Developments (HUD) guidance does not specify that it be quantifiable and outcome oriented. Further, HUD has not identified the performance data that would be needed to assess the results of similar MTW activities or the program as a whole and has not established performance indicators for the program. The shortage of such analyses and indicators has hindered comprehensive evaluation efforts, although such evaluations are key to determining the success of any demonstration program. Further, while HUD has identified some lessons learned from the program, it has no systematic process to identify them and thus has relied primarily on ad hoc information. The absence of a systematic process for identifying lessons learned limits HUDs ability to promote useful practices that could be more broadly implemented to address the purposes of the program.
HUD generally follows its MTW monitoring policies and procedures, but they could be strengthened. HUD staff review and approve each MTW agencys annual plan to ensure that planned activities are linked to program purposes and visit each MTW agency annually to provide technical assistance. But HUD has not taken key monitoring steps set out in internal control standards, such as issuing guidance that defines program terms or assessing compliance with all of the requirements. Without clarifying key terms and establishing a process for assessing compliance with statutory requirements, HUD lacks assurance that agencies are actually complying with the statute. Additionally, HUD has not done an annual assessment of program risks despite its own requirement to do so and has not developed risk-based monitoring procedures. Without taking these steps, HUD lacks assurance that it has identified all risks to the program. Finally, HUD does not have policies or procedures in place to verify the accuracy of key information that agencies self-report. For example, HUD staff do not verify self-reported performance information during their reviews of annual reports or annual site visits. Without verifying at least some information, HUD cannot be sure that self-reported information is accurate.
Expanding the MTW program may offer benefits but also raises questions. According to HUD, affordable housing advocates, and MTW agencies, expanding MTW to additional PHAs would allow agencies to develop more activities tailored to local conditions and result in more lessons learned. However, data limitations and monitoring weaknesses raise questions about expansion. HUD recently reported that expansion should occur only if newly admitted PHAs structured their programs to permit high-quality evaluations and ensure that lessons learned could be generalized. Until more complete information on the programs effectiveness and the extent to which agencies are adhering to program requirements is available, it will be difficult for Congress to know whether an expanded MTW would benefit additional agencies and the residents they serve. Some researchers and MTW agencies suggested alternatives to expansion, including implementing a program that was more limited in scope.
section_title: What GAO Recommends
paragraphs: GAO makes eight recommendations to HUD: that HUD improve its guidance on reporting performance information, develop a plan for identifying and analyzing standard performance data, establish performance indicators, systematically identify lessons learned, clarify key terms, implement a process for assessing compliance with statutory requirements, do annual assessments of program risks, and verify the accuracy of self-reported data. HUD generally or in part agreed with seven of them. HUD disagreed with our recommendation that it create overall performance indicators. GAO believes, however, that they are critical to demonstrating program results and thus maintains its recommendation.

section_title: Letter
paragraphs: The Department of Housing and Urban Development (HUD) provides  over $25 billion each year to public housing agencies (PHA) to make  rental housing affordable to about 3.2 million low-income households  through a variety of programs. As of January 2012, 35 of the  approximately 3,300 PHAs operating in the United States were  participating in the Moving to Work (MTW) demonstration program until  the end of their fiscal year 2018. The MTW program has three statutory  purposes: to reduce costs and achieve greater cost-effectiveness in  federal housing expenditures, to give families with children incentives to  obtain employment and become self-sufficient, and to increase housing  choices for low-income families. It gives participating PHAs the flexibility  to design and test innovative strategies for providing and administering  housing assistance in their communities. The agencies (MTW agencies)  must implement activities that are linked to one or more of the program’s  statutory purposes but that are also designed specifically for the  communities the agencies serve. In fiscal year 2011, MTW agencies  received combined funding of $3.3 billion for housing activities. This  represented more than 13 percent of HUD’s affordable housing  expenditures. Legislation has recently been proposed to expand the  number of MTW agencies. However, researchers and organizations that  advocate on behalf of residents, including legal aid groups, have raised  concerns about expanding it due to the lack of information on the effect of  the program as well as concerns about residents.
In response to your request for information on the effectiveness of the  MTW program, this report discusses (1) what is known about the extent to  which the MTW demonstration program is addressing the program’s  statutory purposes, (2) HUD’s monitoring of MTW agencies’ efforts to  address these purposes and meet statutory requirements, and (3) the  potential benefits of and concerns about expanding the number of PHAs  that can participate in the demonstration program.
To determine what is known about the extent to which the MTW program  is addressing statutory purposes, we analyzed the most recent annual  reports as of January 2012 for 30 agencies.of selected information in the reports by reviewing supporting  documentation for a sample of seven MTW agencies and interviewing  officials responsible for preparing and reviewing this information. (We  selected the sample of MTW agencies to provide diversity in factors such  as geography, agency size, and length of time participating in the  program.) We determined that the reports were sufficiently reliable for the  purposes of our review. We analyzed these reports and corresponding  annual plans to determine the number of ongoing activities and the  statutory purpose(s) to which each activity was linked. We compared  HUD’s guidance for MTW agencies on the type of performance  information they should report with the GPRA (Government Performance  and Results Act) Modernization Act of 2010. We also reviewed   We assessed the reliability  evaluations of the program and summarized the challenges that HUD and  others have faced in conducting such evaluations. Finally, we assessed  HUD’s past efforts to identify lessons learned from the MTW program.
To assess HUD’s monitoring of MTW agencies’ efforts to address the  program’s statutory purposes and meet requirements, we reviewed  relevant policies and procedures, including HUD’s desk guide for the  program, the Standard Agreement that HUD executed in 2008 to govern  participation in the program, and documentation of the steps HUD had  taken to help ensure compliance with its policies and procedures. We  also examined the relevant internal control standards that apply to federal  agencies and HUD’s guidance on internal controls and compared HUD’s  monitoring policies and procedures with these standards. To discuss the  potential benefits and concerns associated with expanding the number of  PHAs that can participate in the program, we reviewed studies, reports,  and testimonies by researchers, affordable housing advocates, and  resident advocates. For all three objectives, we interviewed HUD officials,  officials from our sample of seven MTW agencies, research  organizations, affordable housing advocates, and organizations that  advocate on behalf of residents of federally subsidized housing.
We conducted this performance audit from July 2011 to April 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I contains additional  information on our scope and methodology.
subsections: section_title: Background
paragraphs: The MTW demonstration program was authorized by the Omnibus  Consolidated Rescissions and Appropriations Act of 1996. The program  is intended to give participating agencies the flexibility to design and test  innovative strategies for providing and administering housing assistance.  To implement such strategies, participating agencies may request  waivers of certain provisions in the United States Housing Act of 1937, as  amended.provisions in order to combine the funding they are awarded annually  from different programs into a single, authoritywide funding source.  Requirements outside of the 1937 Housing Act, such as fair housing  rules, cannot be waived under MTW. In addition, certain sections of the  act, including those that cover labor requirements and the demolition and  disposition of public housing, cannot be waived. The 1996 act that  created the program requires participating agencies to address three  purposes and meet five requirements. Specifically, the three statutory  purposes are to (1) reduce costs and achieve greater cost-effectiveness  in federal housing expenditures, (2) give families with children incentives  to obtain employment and become self-sufficient, and (3) increase  housing choices for low-income families. For example, to reduce  administrative costs MTW agencies can reduce the frequency of income  verifications for households with fixed incomes. In making these changes,  MTW agencies must   For example, agencies may request to waive certain  1.  serve substantially the same total number of eligible low-income  families that they would have served had funding amounts not been  combined; 2.  maintain a mix of families (by family size) comparable to those they  would have served without the demonstration; 3.  ensure that at least 75 percent of households served are very low  4.  establish a reasonable rent policy to encourage employment and self- 5.  assure that the housing they provide meets HUD’s housing quality  standards.
The program’s ultimate goal is to identify successful approaches that can  be applied to PHAs nationwide.
The 1996 act authorized MTW for 30 agencies. HUD invited PHAs to  apply for the program and selected an initial cohort of 24 PHAs from  among the respondents. Six more were added as a result of the Jobs- Plus initiative. Since then, some agencies have opted not to participate,  and additional agencies have been added to replace them. Other  agencies have been added through specific appropriations language (see  fig. 1). In addition, some agencies have completed their participation and  exited the program. As of January 2012, a total of 39 PHAs were  authorized to participate, and 35 were participating.
MTW agencies do not receive special funding allocations. Rather, they  receive funds from the three traditional primary funding sources (public  housing capital funds, public housing operating funds, and Housing  Choice Vouchers). Traditionally, PHAs have been required to use the funds from each separate source only for specific purposes, but MTW  agencies may combine the money from the three sources and use them  for a variety of HUD-approved activities. This fungibility is intended to give  MTW agencies greater flexibility. For example, public housing operating  funds are traditionally used to make up the difference between the rents  charged for units and the actual cost of operating them. Capital funds are  used for modernization efforts and management improvements, while  voucher funds provide rental assistance in the private market. However,  by combining funds an MTW agency may use public housing capital  funds to issue additional vouchers or use voucher funds to develop more  public housing to better fit the needs of its community. MTW agencies  also have the authority to use their funds to implement innovative  activities that differ from traditional housing assistance. For instance, an  MTW agency can use funds to replace public housing with mixed-income  communities or reach special-needs populations, such as the homeless,  using vouchers paired with supportive services.
A Standard Agreement, executed in 2008 to replace individual contracts  between HUD and participating PHAs, governs the conditions of  participation in the program. HUD enters into this agreement with each  MTW agency. HUD created the agreement to standardize the language in  its contracts and its reporting requirements and to help create a more  operationally sound program. The Standard Agreement includes a  termination date (the end of each agency’s 2018 fiscal year) and an  attachment that sets out reporting requirements (Attachment B). While  much of the Standard Agreement is the same for all MTW agencies, two  sections are tailored to individual agencies: a description of the formulas  for determining the amounts of funding each agency will receive and an  optional section that may include some agency-specific authorizations.
MTW agencies with less than 10 percent of their housing stock in the MTW program  continue to submit the 5-Year and annual plans required by Section 5A of the 1937  Housing Act. Only information not included in these documents would need to be included  in a supplemental annual MTW plan. contents that lists the information that agencies must include in their  annual plans and reports. For example, the plan must include, among  other things, a description of how each planned activity relates to at least  one of the three purposes of the program; baselines, proposed  benchmarks, and proposed metrics for assessing the outcomes of each  activity; citations of the authorizations that give the agency the flexibility to  conduct the activity; and descriptions of required rent reform activities.  In addition, the plan must include a certification that the agency published  a notice of plans to hold a public hearing on the plan, made the agency’s  annual plan available for public inspection, and conducted a public  hearing to discuss the annual plan prior to its approval. Similarly, the  Standard Agreement outlines the information that MTW agencies are  required to include in annual MTW reports. These reports must include,  for example, detailed information on the impact of each activity, including  comparisons of actual outcomes to the benchmarks proposed in the  annual plan. If the agencies do not achieve the benchmarks or the  activities are determined to be ineffective, the MTW agency is required to  describe the challenges, and, if possible, identify alternative activities that  may be more effective. MTW agencies also are required to self-certify  that they are in compliance with three statutory requirements: assisting  substantially the same total number of eligible low-income families that  they would have served had funding amounts not been combined;  maintaining a mix of families (by family size) comparable to those they  would have served had funding amounts not been combined under the  demonstration; and ensuring that at least 75 percent of households  served are very low income.
As well as meeting the requirements in the Standard Agreement, MTW  agencies must submit tenant-related data into the Moving to Work section  of the Public and Indian Housing Information Center (MTW-PIC).  According to HUD officials, the MTW-PIC module was created because  the standard PIC system that non-MTW agencies use to report tenant  data could not accommodate some of the activities allowed under MTW,  such as less frequent tenant income recertifications and rent calculations  that vary from HUD’s standard calculations. The MTW-PIC module was  created in 2007, and most MTW agencies had transitioned to it by 2008.
The Office of Public Housing Investments within the Office of Public and  Indian Housing at HUD headquarters is the designated program office for  the MTW demonstration program. Within the Office of Public Housing  Investments is an MTW Office that includes a program director and four  coordinators who are each assigned to a specific group of MTW  agencies. The MTW Office is responsible for, among other things,  processing, reviewing, and approving all annual plans submitted by MTW  agencies; establishing guidelines for MTW agencies; monitoring approved  activities and accomplishments; and accepting annual reports. Individual  MTW coordinators facilitate the reviews of planned and implemented  activities and are responsible for coordinating with other HUD offices,  including local HUD field offices, to obtain additional input on MTW  agencies’ planned activities and accomplishments. In January 2011, the  Office of Public Housing Investments signed a Memorandum of  Understanding with HUD’s Office of Field Operations to increase  collaboration and formally describe the roles and responsibilities of the  MTW Office and HUD field office staff. Per this memorandum, field office  staff assist the MTW Office by reviewing and providing input on annual  plans and reports, ensuring that agencies are reporting tenant  information, and participating in annual site visits.
section_title: HUD Has Not Identified Standard Performance Data and Indicators Needed to Evaluate the Program
paragraphs: MTW agencies provide descriptions of their activities and performance  information in their annual reports to HUD. They show how the activities  link to the program’s statutory purposes in their annual plans, as required,  and sometimes also in their annual reports. However, the type of  performance information they provide varies, and HUD has provided  limited guidance. While varied information on individual activities is  available, a comprehensive evaluation of the MTW program is lacking, in  part because HUD does not have a plan for identifying and analyzing  standard performance data and has not established performance  indicators for the MTW program as a whole. Further, HUD does not  have a systematic process for identifying lessons learned by individual  MTW agencies that can be replicated at other PHAs.
subsections: section_title: Although Information on Activities Is Generally Available at the Agency Level, It Varies Due to HUD’s Limited Guidance
paragraphs: MTW agencies report information on specific activities, including  descriptions, in their annual reports. Agencies are required in their annual  plans to link each of their proposed activities to one of the program’s  three statutory purposes, and some agencies also show links between  ongoing activities and statutory purposes in their annual reports. The  three statutory purposes are to reduce costs and achieve greater cost- effectiveness in federal housing expenditures, give families with children  incentives to obtain employment and become self-sufficient, and increase  housing choices for low-income families. According to the most recently  available annual reports, 30 agencies have over 360 ongoing activities,  including rent reform initiatives and work requirements (see table 1).
According to the most recent annual reports (and corresponding plans)  for 30 MTW agencies, agencies associated the largest percentage of  ongoing activities (41 percent) with the statutory purpose of reducing  costs and improving cost-effectiveness (see fig. 2). For example,  agencies associated changes in certification schedules, inspection  protocols, and medical deductions with reduced costs. The agencies  linked 30 percent of their ongoing activities to the statutory purpose of  increasing housing choices and 24 percent to encouraging self- sufficiency. The agencies did not link 4 percent of their ongoing activities  to any purpose in either their most recent annual plan or report.
In its Standard Agreement, HUD requires agencies to include in their  annual reports performance information on the impact of each  implemented activity, including describing the metrics used to assess  outcomes and comparing actual performance with proposed  benchmarks.defined them in 2009 training materials. In these materials, HUD defined  a metric as the “unit of measure that quantifies the changes that might  occur as a result of the MTW activity” and a benchmark as the “projected  outcome of the MTW activity.” Further, in these 2009 training materials  HUD defined an outcome as the “actual, measured result of the  implemented activity.” As examples, the training materials stated that a  metric could be the hours of staff time saved, a benchmark could be the  number of anticipated staff hours saved, and the outcome could be staff  hours actually saved. HUD directs agencies to develop their own metrics   While HUD did not define these terms in the agreement, it  and benchmarks for each activity based on local and community  standards. Our analysis of the most recent annual reports for 30 MTW  agencies showed that the agencies reported performance information for  91 percent of the ongoing activities included in the reports and used over  1,000 metrics to assess these activities. MTW agencies collectively met  the benchmarks associated with 40 percent of these metrics and fell short  of meeting 17 percent of them. For 30 percent of the metrics, it was too  soon to determine if the benchmarks had been met because the activities  were not yet completed. For the remaining 13 percent, information (either  the benchmark or performance data) was lacking to determine whether  the benchmark was met.
While MTW agencies are generally devising their own metrics for  activities and reporting performance information, the usefulness of this  information is limited because, in some cases, it is not outcome-oriented.  Our analysis of the most recent annual reports for 30 MTW agencies  showed that the type of information that agencies reported on the impact  of their activities varied. For example, for similar activities designed to  promote family self-sufficiency, one MTW agency reported only the  number of participants, which is generally considered an output, and  another did not provide any performance information. In contrast, a third agency reported on, among other things, the average income of program  graduates, which we consider an outcome.
Internal control standards state that good guidance (information and  communication) is a key component of a strong internal control  framework and that there is a need for clear documentation. To be  consistent with the GPRA Modernization Act of 2010, HUD’s guidance on  reporting performance information should indicate the importance of  outcome-oriented information. Specifically, the act states that an agency  should establish efficiency, output, and outcome indicators for each  program activity. Furthermore, Office of Management and Budget  guidance on implementation of the act states that quantitative and  outcome-focused measures are preferred.
At the time of our review, HUD’s guidance did not specify that agencies  should report quantifiable and outcome-oriented performance information.  According to the Director of the MTW Office, Attachment B of the  Standard Agreement is the most current guidance on the information that  agencies should report in their annual reports. It simply states that  agencies are to provide detailed information on the impact of the activity  and compare it against the proposed benchmarks to assess outcomes,  including whether an activity is on schedule. The attachment does not  define terms or set expectations for the type of information to be reported.  After the Standard Agreement was executed in 2008, HUD conducted  training for participating agencies. As previously discussed, the 2009  training materials defined key terms such as a metric and outcome and  outlined steps agencies could take to evaluate their activities. HUD also  encouraged the MTW agencies to use metrics and benchmarks that did  not focus on the number of individuals participating in an activity but  rather on the objectives of the activity and to report quantifiable  information. While HUD has posted the 2009 training materials on its  website, these materials have not been incorporated into Attachment B of  the Standard Agreement.
According to the Director of the MTW Office, HUD has not made its  guidance more specific because agencies are implementing a wide  variety of activities and thus require some reporting flexibility. We  acknowledge the need for flexibility, but it is important that HUD require  agencies to report at least some outcome-oriented performance  information. Without more specific guidance on the reporting of  performance information, HUD cannot be assured of collecting  information that reflects the outcomes of individual activities. Such  information would help HUD assess the demonstration program and  whether the activities are furthering program purposes.
section_title: Design and Data Weaknesses Have Prevented a Comprehensive Evaluation of MTW
paragraphs: As we have previously reported, obtaining performance information from  demonstration programs that are intended to test whether an approach  (or any of several approaches) can obtain positive results is critical. This  information is needed to help determine whether the program has led to  improvements consistent with its purposes. HUD has sponsored three  broad reviews of the MTW program, but these studies are not  comprehensive evaluations because of data limitations, among other  things. A 2004 Urban Institute evaluation of the MTW program found  that most MTW agencies reported modest benefits from activities related  to administrative streamlining and that these results were often not as  dramatic as the agencies had anticipated.difficulty in determining whether MTW activities related to employment   The study also noted the  and income had any independent effect and that MTW activities resulted  in both greater and more limited housing choice.
A second study, conducted in 2007 by Applied Real Estate Analysis, Inc.  and the Urban Institute, reviewed eight MTW agencies that had placed  limits on the length of time that residents could receive housing  assistance. concluded that only limited information was available with which to  evaluate outcomes or establish cause-and-effect relationships between  agencies’ policies and recipients’ experiences. It noted that there were  significant limitations to what could be learned from these experiences  because no evaluative framework had been built into the program.  Finally, a 2010 HUD Report to Congress found that the effects of many  MTW activities, especially as they related to residents, could not be  conclusively identified because of the variety of and differences in the  activities and metrics that MTW agencies were implementing. However,  the report did identify some results concerning agencies’ ability to more  efficiently allocate resources and engage in strategic long-term  planning. For instance, the study noted that some agencies had seen  positive effects from combining their traditional sources of funding and  streamlining their operations—for example, by simplifying their housing  quality inspections.
The study found that each of the approaches varied and  These three studies of the MTW program and our work have identified  several challenges that have hindered evaluation efforts. These  challenges include the way the program was initially designed and the  resulting lack of standard performance data as well as the lack of  performance indicators for the program as a whole.
Robert Miller, Martin D. Abravanel, Helene Berlin, Elizabeth Cove, Maria-Alicia  Newsome, Carlos A. Manjarrez, Lipi Saikia, Robin E. Smith, and Maxine V. Mitchell, The  Experiences of Public Housing Agencies That Established Time Limits Policies Under the  MTW Demonstration (Applied Real Estate Analysis, Inc. and The Urban Institute, May  2007).
subsections: section_title: Lack of Program Evaluation Design and Standard Performance Data
paragraphs: HUD has taken steps to address the problems noted with MTW’s initial  design and the lack of standard data; however, it has not analyzed the  data it currently collects or determined whether these data are sufficient  to evaluate similar activities and the program as a whole. As we have  previously reported, comparable data are essential to a full analysis of  programs that incorporate a variety of activities. We also noted that  obtaining performance information from demonstration programs that are  intended to test whether an approach (or any of several approaches) can  obtain positive results is critical. Finally, we have reported that agencies  need to identify any data that will be needed to assess the effectiveness  of program regulations.
Researchers and others have noted the limitations that the program’s  initial design posed to evaluation. In the early years of MTW, rigorous  evaluation strategies were not required, and the program lacked a  research design that would have helped in determining baseline  information. The 2004 Urban Institute review of MTW concluded that  there were limits to what could be learned from its review for a variety of  reasons, such as the inability to separate individual components of  agencies’ MTW activities for analysis. As a result, the report is mainly  descriptive and qualitative. HUD’s 2010 Report to Congress noted that  because rent reform activities varied greatly and were not implemented  using a controlled experimental methodology, the authors were unable to  recommend specific reforms as best practices.
To help evaluate aspects of the MTW program moving forward, MTW  officials have added requirements for new agencies. According to HUD’s  2010 Report to Congress, the three agencies admitted to the MTW  program in 2009 had strong evaluation components. Two of these  agencies have commitments from local universities to evaluate their  programs. Additionally, HUD has required the two newest agencies to  participate in a controlled rent reform study. However, these  improvements will not help evaluate the program as a whole or the  activities implemented by the 30 other MTW agencies.
Likely due to the absence of an evaluation framework for the MTW  program, researchers have noted the lack of standard performance data  needed to evaluate similar activities and the program as a whole. The  2004 Urban Institute study noted that the lack of consistent data on  resident characteristics, incomes, and rent payments prevented the  authors from being able to determine whether individual agencies were  able to achieve the goal of increasing self-sufficiency. Similarly, the  HUD Inspector General reported in 2005 that HUD lacked the empirical  data needed to assess the program as a whole.
Since these reports, HUD has started collecting some additional data  from MTW agencies, but it has not yet analyzed the data. HUD created  the MTW-PIC module to collect tenant characteristics such as household  size, income, and educational attainment. However, according to MTW  officials, HUD has not used these data to analyze the program’s effects,  such as changes in resident income. In addition, HUD’s Standard  Agreement has required agencies since 2009 to provide information in  their annual reports on the impact of activities, including benchmarks and  metrics. While these reports are informative, they do not lend themselves  to quantitative analysis because the reporting requirements do not call for  standardized data, such as the number of residents that found  employment.
In addition, whether these data are sufficient to assess similar activities  and the program as a whole is not clear, and HUD has not identified the  data it would need for such an assessment. For example, neither MTW- PIC nor annual reports capture standard data on implemented activities.  Further, according to the Director of the MTW Office, MTW-PIC does not  include information on individuals who receive nontraditional services  from an MTW agency, such as homeless assistance or case  management. Representatives from MTW agencies have suggested that  HUD should collect some standard data for similar activities. For  example, they noted that if HUD required all agencies that implemented  rent reform activities to report standard data, the results of these efforts  could be analyzed even if the specific activities varied. The Director of the  MTW Office also noted that MTW-PIC was a potential tool for collecting  and analyzing standard demographic data.
The MTW Office has recently developed a statement of work for an  evaluation of the program, but HUD has not allocated funding for the  study, according to the Director of the MTW Office. Among other things,  the proposed evaluation is intended to assess the current state of the  MTW demonstration and determine the extent to which the three statutory  purposes have been addressed. The study is also expected to include an  analysis of outcomes associated with specific activities and the  demonstration as a whole to identify which MTW activities are appropriate  for expansion to all PHAs. However, the approach envisioned may be  limited because it would primarily rely on existing data sources. Until HUD  develops and implements a plan (that includes the identification of  standard data) to quantitatively assess similar activities and the MTW  program as a whole, HUD cannot determine their effectiveness. While  such analyses may be challenging, they would enhance HUD’s ability to  rigorously assess the demonstration.
HUD has not established performance indicators for the MTW program as  a whole. The GPRA Modernization Act of 2010 requires that federal  agencies establish efficiency, output, and outcome indicators for each  program activity as appropriate. Internal control standards also require  the establishment of performance indicators. In addition, we have  previously reported that successful performance indicators demonstrate  results and provide useful information for decision making in order to  track how programs and activities can contribute to attaining an  organization’s goals and mission, among other things.
As previously discussed, MTW agencies set their own performance  metrics for activities, but HUD has not established performance indicators  for the program as a whole. HUD’s Fiscal Year 2011 Annual Performance  Plan established agencywide performance indicators but did not explicitly  connect the MTW program to any of them. specific targets for the program, the Director of the MTW Office noted that  the program’s activities support some of the agencywide indicators.  Specific performance indicators for the MTW program could be based on  the statutory purposes. For example, agencies could report on the  savings achieved (reducing costs) and the number of additional  households served (increasing housing choices). Without performance  indicators for the MTW program, however, HUD cannot demonstrate the  results of the program as a whole.
section_title: HUD Lacks a Systematic Process for Identifying Lessons Learned
paragraphs: While HUD has identified some lessons learned on an ad hoc basis, it  does not have a systematic process in place for identifying such lessons.  We have previously reported that obtaining impact information from  demonstration programs that are intended to test whether an approach  (or any of several approaches) can obtain positive results is critical. This  information should be gathered to help determine whether programs have  led to improvements consistent with their purposes.
HUD, Fiscal Year 2011 Annual Performance Plan (Washington, D.C.: 2011).
PHAs nationwide. These practices, which are posted on HUD’s website,  included implementing savings accounts for public housing and voucher  recipients to promote resident savings. Most recently, HUD’s 2010 Report  to Congress described promising policies, practices, and concerns. In  addition, officials from some of the MTW agencies we interviewed noted  that HUD officials had shared information on activities that had shown  positive effects during site visits, quarterly phone calls, newsletters, and  annual conferences. Finally, HUD’s statement of work for its proposed  evaluation of the MTW program includes the creation of five case studies  that would review MTW flexibilities.
However, these efforts have shortcomings. In most cases, the practices  chosen were based on the opinions of HUD or contracted staff and  largely involved anecdotal (or qualitative) data rather than quantitative  data. The lack of standard performance data has affected HUD’s ability to  systematically identify lessons learned. In its 2005 report on the MTW  program, the HUD Inspector General noted that the lack of data on the  program made it difficult to identify activities that could be considered  models for addressing the three statutory purposes or that could be used  to show the importance of individual policy changes.  Further, HUD has  not established criteria, such as demonstrated performance, for  identifying lessons learned. Finally, HUD has not made regular efforts to  review and identify lessons learned. Because HUD does not currently  have a systematic process for identifying lessons learned, it is limited in  its ability to promote useful practices that could be implemented more  broadly.
HUD, Office of Inspector General, 2005-SE-001.
section_title: HUD Generally Follows Its Monitoring Policies and Procedures, but Could Strengthen Them
paragraphs: HUD has policies and procedures in place to monitor MTW agencies.  First, HUD requires program staff to review and comment on agencies’  annual plans and reports. Second, staff review tenant data submitted by  MTW agencies. Third, program staff conduct annual site visits at each  participating agency to provide technical assistance and program  updates. HUD generally follows these policies and procedures, which  focus on technical assistance rather than compliance. Due in part to this  focus, HUD’s policies and procedures have several key weaknesses.  Specifically, HUD has not clarified program terminology, ensured that  each MTW agency is meeting statutory requirements, performed an  annual risk assessment, or developed policies and procedures to verify  the accuracy of key information that MTW agencies self-report.
subsections: section_title: HUD Reviews Agency Information and Conducts Annual Site Visits
paragraphs: HUD’s monitoring policies and procedures for the MTW program are  contained in a desk guide, which describes the roles and responsibilities  of HUD staff in reviewing annual plans and reports and data submissions,  making site visits, and performing other monitoring activities. In January  2011, HUD’s Office of Public Housing Investments and Office of Field  Operations signed a Memorandum of Understanding documenting the  framework for headquarters and field staff to follow in overseeing MTW  agencies. According to the memorandum, the MTW Office (within the  Office of Public Housing Investments) is responsible for oversight of the  MTW program. In many cases, the MTW Office works with field offices to  jointly develop responses to MTW agency issues. Further, a MTW  Working Group—consisting of representatives from Public and Indian  Housing programs, the Real Estate Assessment Center, and the Office of  Policy Development and Research—was established to assist with the  annual plan and report review process. As part of the memorandum, the  Offices of Public Housing Investments and Field Operations agreed to the  protocols set forth in the desk guide.
HUD staff from the MTW Office and field offices and the MTW Working  Group share responsibility for reviewing and commenting on participating  agencies’ annual plans and reports. The Standard Agreement  (Attachment B) outlines the requirements for annual plans and reports  that agencies must submit. MTW coordinators, who are each responsible  for a specific number of MTW agencies, have the lead role in reviewing  annual plans and reports to determine if they meet the requirements of  Attachment B and obtaining input from other HUD staff, including field  offices and the MTW Working Group. Field offices are required to review  the annual plans and reports submitted by the MTW agencies in their  jurisdictions and provide their assessment to the MTW coordinator.  Similarly, the MTW Working Group reviews and provides comments to  the MTW coordinator. Coordinators summarize the comments from the  field offices and MTW Working Group and send them to the agencies.  The coordinators and field office staff work with MTW agencies to resolve  any outstanding issues. Once such issues have been resolved, the MTW  Office approves annual plans and accepts annual reports.
Soliciting public comments on the plan is important because, unlike other PHAs, MTW  agencies are not subject to the Public Housing Assessment System, which includes a  customer satisfaction survey that promotes resident participation. five statutory requirements.MTW coordinators to verify each agency’s certification that it has met the  three statutory requirements.
However, these procedures do not require  Interviews with MTW coordinators and field staff and documentation for  our sample of seven MTW agencies indicated that HUD generally  followed these procedures. Documentation we reviewed for the agencies  in our sample showed that the coordinators generally completed  checklists while reviewing annual plans and reports. For example,  coordinators verified that all ongoing activities were reported, ensured the  agency included its certification that it had met three of the five statutory  requirements, and made certain the agency certified that it had held a  public hearing on its annual plan, among other requirements.  Coordinators also provided comments to agency staff on annual plans  and reports. of the comments, the MTW Office notified the agency in writing that its  plan had been approved and report accepted.
Field office staff that we interviewed said that they reviewed annual plans and reports  and sent their comments and concerns to MTW coordinators.  MTW agencies are subject to a variety of other reporting requirements. For example,  MTW agencies are required to report voucher utilization in the Voucher Management  System. They also must procure a public accountant to perform an Office of Management  and Budget Circular A-133 compliance audit and submit unaudited financial statements. In  addition, they are subject to HUD physical and management inspections of public housing  and on-site monitoring reviews related to voucher reporting. example, in January 2012, MTW agencies overall achieved a 100-percent  tenant data submission rate.
In addition, HUD conducts annual site visits to provide technical  assistance to each MTW agency. The MTW Office and the local field  office conduct these visits jointly. The MTW Office (in particular the  coordinator assigned to the agency) takes the lead role in conducting the  visit, including preparing the agenda, coordinating with the local HUD field  office, and working with the MTW agency to select properties to visit.  According to HUD officials, the primary objective of the site visit is to  provide technical assistance and build a working relationship with the  participating agencies, not to assess compliance with statutory  requirements. However, HUD officials stated that if compliance issues  with statutory purposes are found, HUD staff address these issues during  the site visit, and coordinators often develop timelines for the agency to  come into compliance.
Our analysis of documentation of site visits to participating agencies  indicated that MTW Office and field staff generally followed HUD’s annual  site visit procedures. Specifically, analysis of site visit reports indicated  that HUD officials generally discussed the effectiveness of activities and  helped resolve any outstanding issues. For example, as a result of site  visits, HUD staff recommended that an agency include cost-saving  measures in its annual plan, requested clarification of output measures,  and encouraged one agency to submit articles to the MTW newsletter to  share its experiences on how rent reform encouraged self-sufficiency.
Interviews with our sample of MTW agencies and corresponding field  office officials also indicated that HUD was following its policies and  procedures for annual site visits. MTW agency officials we spoke with  indicated that the site visits were generally beneficial because they  provided an opportunity for in-person discussions that helped facilitate  communication with HUD. HUD’s field office staff noted their active  involvement over the years, which had become more defined with the  issuance of the desk guide in 2011. According to the Director of the MTW  Office, the office is considering conducting future site visits using a risk- based approach. Using this approach, HUD would conduct site visits less  frequently but would focus on larger agencies that had implemented a  wide range of complex activities and newly admitted agencies that were  implementing new activities.
To foster information sharing across agencies and provide technical  assistance, HUD employs a number of additional strategies. For example,  HUD hosts annual conferences to share information with MTW agencies  and facilitate information sharing among agencies. The conferences  cover a variety of topics, and all participating MTW agencies are invited to  attend. For example, the 2011 conference focused on effectively  managing funds in a challenging budgetary environment. HUD also has  engaged participating agencies in quarterly conference calls and other  training related to program changes such as the conversion from PIC to  MTW-PIC and the transition to the Standard Agreement. Further, HUD  issues notices on various topics, such as MTW reporting requirements,  and publishes quarterly newsletters that highlight activities relating to  each statutory purpose, among other topics. HUD also publishes each  agency’s annual report and researchers’ evaluations of MTW activities on  its website.
section_title: HUD’s Monitoring Policies and Procedures Have Several Key Weaknesses
paragraphs: Although HUD follows the policies and procedures that it has in place, it  could do more to ensure that MTW agencies are demonstrating  compliance with statutory requirements and to identify possible risks  relating to activities implemented by each agency, among other things.  First, HUD has not issued guidance to participating agencies clarifying  key program terms, including definitions of the purposes and statutory  requirements of the MTW program. Internal control standards require the  establishment of clear, consistent goals and objectives. As previously  noted, MTW authorizing legislation established three purposes for the  program, and agencies must link each of their activities to one of these  purposes. However, HUD has not clearly defined what the language in  some of these purposes means, such as “increasing housing choices for  low-income families.” MTW agencies have linked activities to this purpose  that range from using block grant funding to support homeownership  programs to requiring applicants to complete a renter education program  to establishing a prisoner re-entry housing program. HUD noted the lack  of a clear definition in its 2010 Report to Congress but continued to  require that MTW agencies link activities to this purpose. According to  MTW officials, they have not defined what is meant by “increasing  housing choices” so that agencies have the ability to define this term in a  manner that fits their local needs. In addition, HUD has not clarified what  is meant by “serving a comparable mix of families” but also requires  agencies to comply with this requirement. MTW agencies we spoke with  described varying interpretations of this requirement. For example,  officials from one agency told us that they observed how family sizes  changed in their community and compared those changes to changes in  families within the MTW program, using community survey data and data  from the agency’s internal system. Officials from another agency we  spoke with said that over time it had become increasingly difficult to  determine compliance with this statutory requirement.
HUD has recently taken steps to clarify some terminology, explaining how  agencies can certify that at least 75 percent of the families they serve  have very low incomes and that they are serving substantially the same  number of households under MTW as they did before the program. In  addition, HUD is revising its reporting requirements for MTW agencies. As  part of this process, HUD officials told us that they plan to update their  guidance to more completely collect information related to the program’s  statutory purposes and requirements. They acknowledged that the  guidance could be strengthened to require MTW agencies to provide their  agency-specific definition for the three statutory purposes. As a first step,  they noted that they planned to require agencies to define “self- sufficiency” by either choosing one of the definitions provided by HUD or  creating their own. Similarly, the officials stated that they would consider  requiring MTW agencies to choose between using HUD’s definition of  increasing housing choices or creating their own definition. Although a  step in the right direction, allowing MTW agencies to create their own  definitions of key terms would make it difficult to assess the effectiveness  of efforts to address statutory purposes. HUD officials also said that the  revised guidance would provide standardized tables for agencies to report  data related to the requirement to serve a comparable mix of families.  Until HUD clearly defines what is meant by all of the statutory purposes  and requirements of the MTW program, HUD cannot effectively determine  whether agencies are addressing these purposes and meeting  requirements.
Second, HUD has only recently assessed agencies’ compliance with two  self-certified requirements and has not assessed compliance with the  third. Internal control standards require control activities to be in place to  address program risks. In addressing these risks, internal control  guidance states that management should formulate an approach for  assessing compliance with program requirements. While HUD has  recently made efforts to assess agencies’ compliance with two of the  three self-certified requirements, it does not have a process in place to  systematically review compliance with all three requirements. In 2011,  HUD for the first time assessed participating agencies’ compliance with  the requirement to assist “substantially the same” number of eligible  families that would have been served in the absence of MTW. HUD  collected data from MTW-PIC, the Voucher Management System, and  each participating agency’s most recent annual report on the number of  public housing units occupied, vouchers utilized, and other families  housed and used a formula to compare these data with similar data  reported before MTW. HUD and MTW agency staff we interviewed told  us that they worked together to discuss discrepancies in the calculations.  According to the Director of the MTW Office, agencies were in  compliance with this requirement if they were serving at least 95 percent  of the number of families in their baseline figure. HUD’s recent review of  each agency’s baseline calculation indicated that all but one of the  agencies were in compliance. Also in 2011, HUD reviewed MTW-PIC  data for the first time to determine agencies’ compliance with the  requirement that at least 75 percent of assisted residents be very low  income. HUD’s analysis of MTW-PIC data showed that, as of September  2011, 91 percent of the residents served by MTW agencies fell into this  category.
While HUD has taken steps to assess compliance with these two  statutory requirements, it has not yet developed a methodology for  assessing agencies’ compliance with the requirement to maintain a  comparable mix of families. The Director of the MTW Office  acknowledged that self-certifications were not the best means of ensuring  compliance and told us that the planned revisions to the reporting  requirements for MTW agencies would help assess compliance with the  requirements to maintain a comparable mix of families and ensure that at  least 75 percent of families assisted are very low income. Without a  process for systematically assessing compliance with statutory  requirements, HUD lacks assurance that agencies are complying with  them.
Third, HUD has not performed an annual assessment of program risks.  Internal control standards state that an agency should have a risk  assessment plan that considers internal and external risk factors and  establishes a control structure to address those risks. The standards  also state that managers should focus on control activities to address  risks that may involve verifications, performance reviews, and  documentation, among other things. HUD’s own internal control  standards also require its program offices to perform an annual risk  assessment of their programs or administrative functions using a HUD  risk-assessment worksheet. These standards also stress the  importance of performing a risk assessment when there are significant  program changes. According to the Director of the MTW Office, the office  has not performed an annual risk assessment for the MTW program  because it was not aware of this requirement.
MTW agencies are exempt from scoring in the Public Housing Assessment System and  the Section 8 Management Assessment System. However, MTW agencies are subject to  physical inspections conducted by the Real Estate Assessment Center under HUD  guidelines and issued a score. This score is entered into the Public Housing Assessment  System and can be viewed by MTW staff at any time. A score of 22 or below is flagged by  the Real Estate Assessment Center (the maximum score is 30) and reported to the  appropriate field office. their perceived level of risk. While monitoring procedures are not risk- based, the Director of the MTW Office stated that his office would become  aware of risks from HUD’s field office staff, which have routine  responsibility for reviewing financial audits and Office of Management and  Budget compliance audits. As previously discussed, HUD is considering  moving toward conducting risk-based site visits. In addition, according  to HUD officials, the office is considering other methods to more  rigorously analyze MTW agency risk factors. By not performing an annual  risk assessment and implementing a risk-based approach to monitoring  MTW agencies, HUD lacks assurance that it has properly identified and  addressed risks that may prevent participating agencies from addressing  program purposes and meeting statutory requirements. HUD also lacks  assurance that it is efficiently using its limited monitoring resources.
GAO/AIMD-00-21.3.1 and GAO-01-1008G. reflect the source documents.accuracy of any reported performance information, it lacks assurance that  this information is accurate. To the extent that HUD relies on this  information to assess program compliance with statutory purposes and  requirements, its analyses are limited.
section_title: Expanding MTW May Offer Benefits but Also Raises Questions
paragraphs: Legislation has been proposed to expand the number of PHAs that can  participate in the MTW program, and a recent HUD report recommended  expanding the program up to twice its size. As of March 2012, a  maximum of 39 PHAs could participate in the program, but a 2011 Senate  bill would direct HUD to increase that number up to 250. In addition,  legislation has been drafted that would establish MTW as a permanent  program and eliminate the current restrictions on the number of agencies  that can participate. HUD’s 2010 Report to Congress recommends  increasing the number of participating agencies to about 60. HUD and  some stakeholders believe that expansion could provide the needed  information on the effect of the MTW program and allow more PHAs to  test innovative ideas, but questions remain about the lack of performance  information on current MTW activities. In addition, alternatives to  expansion exist, including implementing a more narrowly focused  program.
subsections: section_title: Current MTW Participants and Affordable Housing Advocates Indicated that Expanding MTW Could Help Demonstrate the Program’s Effect and Increase Innovation
paragraphs: According to HUD, some affordable housing advocates, and MTW  agencies we interviewed, expanding the MTW program could help  demonstrate the program’s effect and increase the number of lessons  that can be learned from the program. HUD has reported that doubling  the number of MTW agencies with the use of strategic criteria and  program implementation could help demonstrate the effects of MTW on a  broader scale and enable the housing industry to learn even more from  the demonstration. For example, expansion could provide more  information on how MTW flexibilities would affect a broader group of  PHAs. The Director of the MTW Office noted that some MTW activities,  specifically those related to administrative streamlining, had influenced  the draft Affordable Housing and Self-Sufficiency Improvement Act of  2012 (AHSSIA). Some affordable housing advocates that we met with  emphasized the value of the changes, such as decreases in concentrated  poverty, that have occurred in some of the communities affected by the  MTW program and indicated that expansion could enable more PHAs to  address local needs and therefore benefit additional communities.  Similarly, officials from MTW agencies that we contacted stated that  expansion of the program would provide a broader testing ground for new  approaches and best practices.
Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004). implementation innovations. Officials from an organization that advocates  on behalf of large PHAs and supports expansion noted that affordable  housing needs varied by locality and that the MTW program enabled  participating agencies to design effective approaches based on local  needs. Similarly, another affordable housing advocacy organization told  us that they supported expanding MTW not only because it enabled  participating agencies to tailor activities to local needs but also because it  involved local communities in the process.
Officials from several of the MTW agencies we interviewed also noted  that the MTW program had empowered them to create and implement  strategies that addressed local issues and said that expanding the  program would give more PHAs the same flexibility. For example, in one  northeastern state where the housing stock was relatively old, the MTW  agency was able to focus on developing new affordable housing. Another  MTW agency in a western state with mostly newer housing stock chose to  reduce the frequency of inspections of its properties and focus its efforts  on administrative streamlining and the disposition of its older units.  Further, several MTW agencies that we interviewed described how they  implemented the requirement to establish a rent policy that encouraged  employment and self sufficiency. For example, officials from one MTW  agency told us that they believed the traditional requirement that  residents pay 30 percent of their adjusted income in rent was a  disincentive to work, because as resident income increases so would the  payment toward rent. To encourage residents to seek work, this agency  implemented work requirements and a minimum rent. Additionally, some  agencies have used their MTW status to establish programs that focus on  specific populations, including working families with children, the elderly  and disabled, and the homeless.
Some proponents of expansion that we interviewed also noted that  expanding the MTW program could provide more PHAs with the ability to  use funding from different sources more flexibly than possible without  MTW status. Agencies without MTW status have to implement their  activities while adhering to the regulations associated with three different  funding streams, evidence of the fragmented nature of housing  assistance.certain provisions of the 1937 Housing Act in order to combine annual  funding from separate sources into a single authoritywide funding source.  HUD field office staff with responsibility for monitoring MTW agencies  observed that the single-fund flexibility was beneficial because it enabled  participating agencies to develop supportive service programs, such as  job training or educational programs, which help move families toward  self-sufficiency. One HUD field office official stated that this flexibility  would be a significant benefit for other PHAs. An affordable housing  advocate we met with also noted that this ability to use different kinds of  funds interchangeably was beneficial because it enabled MTW agencies  to shift funds based on local priorities. Further, officials from the MTW  agencies we interviewed agreed that this flexibility was beneficial. For  example, officials from one MTW agency stated it had been able to use  the single fund to organize itself as a business organization, develop a  strategic plan based on the housing needs of low-income families in the  community, leverage public funds and public and private partnerships,  and develop mixed-income communities. Two of the MTW agencies that  we interviewed also stated that the single-fund flexibility had enabled  them to fund programs that encouraged self-sufficiency among residents.  For example, officials explained that they had used funding for coaching  and counseling services, job training support, and education programs.  Finally, officials from three of the MTW agencies we interviewed noted a  related benefit of participation. They said that their MTW status had  enabled them to respond more quickly to real estate opportunities  because they do not have to wait for HUD approvals to purchase  properties.
section_title: Lack of Performance Information and Other Issues Raise Questions about Expansion
paragraphs: A lack of performance information, limited HUD oversight, and concerns  about the program’s impact on residents raise questions about expanding  the MTW program. As we noted previously, conclusive information about  the effectiveness of the MTW program is limited in part because HUD  does not have a plan for identifying and analyzing standard performance  data, has not established performance indicators for the program as  whole, and does not have a systematic process for identifying lessons  learned.
HUD’s 2010 Report to Congress noted that the conclusive impacts of  many MTW activities, particularly as they relate to residents, could not yet  be known. For example, the report noted that the rent reforms  implemented under MTW varied greatly and were not implemented using  a controlled experimental methodology. As a result, which aspects of rent  reforms should be recommended for all PHAs were not clear. The report  also noted the limitations that exist when evaluating the outcomes of  MTW—limitations that stem from the weak initial reporting requirements  and lack of a research design. The report concluded that, given these  limitations, expansion should occur only if newly admitted PHAs structure  their programs for high-quality evaluations that permit lessons learned to  be generalized beyond a single PHA experience.
Similarly, affordable housing advocates and legal aid organizations that  we interviewed stated that because lessons had not been learned from  MTW, there was no basis for expanding the program. For example,  officials from a national affordable housing advocacy organization stated  that some MTW agencies have used their flexibility to establish limits on  the length of time someone can live in assisted housing, but there is little  research on the effect of such efforts. The officials stated that there was  no evidence that this policy had helped anyone become self-sufficient and  move out of public housing. The officials added that data were not  available on the extent to which MTW agencies have provided incentives  for residents to become self-sufficient or have increased housing choices.  Similarly, an official from a national housing law advocacy organization  stated that data were not available to determine the effect of the MTW  program, particularly at the national level.
In addition, our own work, some research organizations, and affordable  housing advocates question HUD’s ability to effectively manage an  expanded MTW program. As previously noted, HUD’s current monitoring  procedures have several key weaknesses, including the lack of a  systematic process for assessing agencies’ compliance with statutory  requirements and an assessment of program risks. Some research  organizations also have questioned HUD’s capacity to oversee additional  MTW agencies. For example, the Urban Institute reported that the  approval process that HUD was using at the time of the institute’s 2004  review would not be feasible for an expanded program because of the  administrative burden involved. At the time of the 2004 study as well as  our review, HUD reviewed each individual request to waive specific  provisions of the 1937 Housing Act before approving annual plans. Staff  from another research organization questioned whether HUD has the  capacity to oversee additional agencies. Similarly, one affordable housing  advocate that we interviewed stated that HUD’s capacity to oversee an  expanded program is not clear, in part because current monitoring  activities are not transparent. At the time of our review, HUD had four full- time MTW coordinators, who each managed from 6 to 10 MTW  agencies. According to the Director of the MTW Office, it takes more  resources for HUD to oversee MTW agencies than non-MTW agencies.  Thus, if additional agencies were added under the current program  design, HUD would likely need additional resources.
Researchers and several of the affordable housing advocates and legal  aid agencies that we met with also raised concerns that the current  program, and therefore also an expanded program, could negatively  affect residents of MTW agencies. For example, two research  organizations have stated residents could be negatively affected by MTW  agencies that implement voucher policies that reduce portability—that is,  residents’ ability to use their vouchers in an area outside of the area  where they received it. One of these research organizations stated that  the differences in the way voucher programs were implemented across  MTW agencies could reduce residents’ ability to use vouchers outside of  the area where they received the assistance. Officials from the other  organization noted that some MTW agencies had instituted policies that  prohibited vouchers from being transported out of the originating  jurisdictions, thereby limiting housing choices. According to HUD officials,  MTW agencies with policies that limit portability can make exceptions. For  example, these agencies have made exceptions for residents seeking  employment opportunities.
Legal aid organizations that have worked with residents of MTW agencies  as well as affordable housing advocates told us some of the requirements  that MTW agencies have implemented, such as work requirements, were  potentially harmful to residents. For example, legal aid representatives  from one community told us that the work requirement was not  consistently enforced across various mixed-income properties that  included public housing as well as market-rate units. According to these  officials, they have had clients who have been evicted for not working,  even though the client was in school or disabled—both exceptions to the  work requirement. These officials also stated that property managers in  the city’s various mixed-income developments did not implement MTW  policies consistently. For example, the officials stated that residents have  been told by property managers that they would be in compliance with the  work requirement if they were in school or another training program, only  to have the MTW agency determine that they were not in compliance.  According to HUD officials, inconsistent enforcement of policies is not  unique to MTW agencies, and residents would have recourse. Legal aid  representatives that worked with residents of another MTW agency also  told us that the work requirement was a punitive policy that negatively  affected the poorest residents. The officials stated that there were better  methods for encouraging work and self-sufficiency, such as job training.  Officials from a national affordable housing advocacy organization agreed  that work requirements are punitive and stated that they disagreed with a  policy of making housing assistance contingent on other factors, such as  having a job. In their view, housing assistance should be a stable form of  assistance for low-income households.
section_title: Alternatives Exist to Expanding the MTW Program
paragraphs: Alternatives to expansion include implementing a program that is targeted  more to specific activities and waiving some regulations for all PHAs as  described in proposed legislation. According to the Urban Institute, an  alternative to expanding MTW could be to systematically test a limited  number of programmatic alternatives—such as flat rents, time limits, or  debt financing of capital improvements.individual agencies as much discretion to design combinations of reforms  around local conditions and priorities. However, this approach could yield  more systematic evidence about the costs and benefits of particular   This approach would not allow  program reforms if it included a rigorous evaluation design and mandatory  data collection on key outcomes, such as the number and characteristics  of participating households. Collection of such information in a  standardized format would need to be a minimum requirement for  participation if the point was to learn from the experiences of those testing  activities. In addition, an official from an affordable housing advocacy  organization that we met with stated that testing the effectiveness of  discrete activities on a smaller scale would be useful. HUD also noted  that altering the scope of the demonstration for new participants could  improve what was learned from specific activities. For example, its 2010  Report to Congress stated that data on MTW could be strengthened if the  scope of the demonstration were altered for new participants by selecting  agencies committed to testing a particular activity, such as rent reform,  and requiring rigorous evaluation. In December 2011, HUD issued a  request for proposals for a demonstration that would test alternatives to  the current rent structure in the voucher program. According to the  proposal, the demonstration would most likely be undertaken at select  MTW agencies. In addition, HUD’s 2012 appropriations act authorized a  Rental Assistance Demonstration that would enable HUD to authorize  and evaluate new approaches to preserving affordable rental housing,  including converting public housing to project-based rental assistance.   AHSSIA includes authorization for a revised version of the Rental  Assistance Demonstration.
Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).
Congress supported allowing more PHAs to participate in the program.  Finally, we recently reported on cost savings that could be realized from  allowing additional housing authorities to implement some of the reforms  MTW agencies have tested.
section_title: Conclusions
paragraphs: The MTW demonstration is designed to provide participating agencies  with the flexibility to develop and test activities that achieve cost- efficiency, encourage residents with children to obtain employment and  become self-sufficient, and increase housing choices for low-income  families. While this flexibility has allowed participating agencies to  implement hundreds of activities, HUD has not done all that it can to  evaluate the program’s effectiveness, identify successful approaches that  could be applied to public housing agencies more broadly, or ensure that  MTW agencies comply with program requirements. Because Congress is  considering expanding the program to many more PHAs, the absence of  information needed to conduct a comprehensive program evaluation and  compliance reviews is significant. HUD has recognized the importance of  rigorous evaluation by requiring newly admitted agencies to have strong  evaluation components. However, these improvements will not help  evaluate the program as a whole. Without more complete knowledge of  the program’s effectiveness and the extent to which agencies are  adhering to program requirements, it is difficult for Congress to know  whether an expanded MTW will benefit additional agencies and the  residents they serve.
Recognizing that it needed to do more to improve what was known about  the program’s effectiveness, HUD started requiring MTW agencies to  describe the impact of each implemented activity in their annual reports  beginning in 2009. However, the information that MTW agencies reported  did not always reflect outcomes, and HUD’s guidance does not require  that information on activities be quantifiable and outcome-oriented to the  extent possible. Without more specific guidance on reporting performance  information, HUD cannot be assured of collecting data that reflects the  outcomes of activities.
Further, challenges such as the lack of analysis of standard data and the  absence of performance indicators have prevented comprehensive  evaluations of similar activities and the overall MTW program. HUD has  recently started collecting additional information through MTW-PIC and  annual reports, but has not yet analyzed the data. Further, whether the  data collected are sufficient to assess similar activities and the program  as a whole is not clear, and HUD has not identified the performance data  it needs to undertake such analysis. Until HUD has a plan (that includes  the identification of standard data) to quantitatively assess similar  activities and the MTW program as a whole, HUD cannot determine their  effectiveness. Additionally, HUD has not established performance  indicators specific to MTW. Indicators linked to the statutory purposes of  reducing costs, encouraging self-sufficiency, and increasing housing  choices would help HUD demonstrate that the program has produced  desired results.
Similarly, HUD does not have a systematic process in place to identify  lessons learned from the MTW demonstration. Identifying activities that  could be replicated more broadly is a goal of the MTW program and could  be aided by the analysis of some standard performance data. The  absence of a criteria-based, regular process for identifying lessons  learned complicates efforts to determine which MTW activities are most  effective and should be replicated more broadly.
At the same time, HUD’s monitoring efforts are not as strong as they  could be. First, because HUD has not clarified key terms related to the  three statutory purposes and five requirements, it cannot effectively  determine whether MTW agencies are actually addressing these  purposes and meeting requirements. Second, HUD does not have a  process in place to systematically review compliance with all  requirements. Such a review is especially important to a program like  MTW that allows participants to self-certify their compliance with some  program requirements. HUD has begun assessing compliance with two of  the MTW requirements that call for self-certification, but not the third, and  thus lacks assurance that agencies are complying with all three.
Moreover, HUD’s procedures for monitoring MTW agencies are not risk- based. It does not conduct an annual assessment of risks and provides  the same level of monitoring for all agencies, even though some may  pose greater risks than others. A risk-based approach to monitoring  would provide greater assurance that HUD has addressed all risks,  particularly those that may prevent participating agencies from addressing  program purposes and meeting statutory requirements. Further, unless it  implements a risk-based approach (such as that currently being  considered for annual site visits) to monitoring MTW agencies, HUD  cannot be assured that it is using its limited monitoring resources most  efficiently. Finally, just as HUD does not assess compliance with all three  self-certified requirements, it does not verify the accuracy of key  information that agencies self-report, including information on the impact  of MTW activities. Annual site visits have been used primarily to provide  technical assistance rather than to assess self-reported information. By  not verifying the accuracy of any performance information, HUD lacks  assurance that this information is accurate.
section_title: Recommendations for Executive Action
paragraphs: To improve what is known about the effectiveness of the MTW program,  we recommend that the Secretary of the Department of Housing and  Urban Development  improve HUD’s guidance to MTW agencies on providing performance  information in their annual reports by requiring that such information  be quantifiable and outcome-oriented to the extent possible; develop and implement a plan for quantitatively assessing the  effectiveness of similar activities and the program as a whole  including the identification of standard performance data needed; and establish performance indicators for the MTW program as a whole.
To enhance the ability to identify MTW practices that could be applied  more broadly, we recommend that the Secretary of the Department of  Housing and Urban Development create a process to systematically  identify lessons learned.
To improve HUD’s oversight of the MTW program, we recommend that  the Secretary of the Department of Housing and Urban Development  issue guidance that clarifies key program terms, such as the three  statutory purposes of the program and the five statutory requirements  that MTW agencies must meet; develop and implement a systematic process for assessing  compliance with statutory requirements;  conduct an annual risk assessment for the MTW program and  implement risk-based monitoring policies and procedures such as  those currently being considered for site visits; and implement control activities designed to verify the accuracy of a  sample of the performance information that MTW agencies self-report.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to HUD. The Assistant Secretary for  Public and Indian Housing provided written comments, which are  reprinted in appendix II. HUD disagreed with our recommendation that the  agency develop performance indicators for the MTW program as a whole,  was in partial agreement with four recommendations, and generally  agreed with three. The agency said that developing programwide  performance measures could be difficult and might be contrary to the  nature of the demonstration. In addition, HUD emphasized the  improvements that it had made to its reporting requirements in order to  collect more consistent, outcome-oriented data. We acknowledged these  improvements in the draft report, but as our recommendations indicated,  we saw opportunities for additional improvements. HUD also noted that  some of our recommendations might be a good fit for the existing  program but that others might be more appropriate for a future expanded  demonstration.
In disagreeing with our recommendation that it establish performance  indicators for the MTW program as a whole, HUD emphasized the  difficulty of measuring all activities against the same standard. The  agency noted that because each MTW agency had implemented a unique  combination of activities, developing programwide performance measures  would make determining the impacts of specific activities unclear and  prevent the identification of individual policies that could be applied more  broadly. However, the purpose of programwide indicators would not be to  isolate the impact of individual activities but to demonstrate programwide  results—including showing the extent to which the program was  addressing its statutory purposes of achieving greater cost-effectiveness  in federal housing expenditures, giving families with children incentives to  obtain employment and become self-sufficient, and increasing housing  choices for low-income families. HUD also stated that applying  programwide performance measures would be complicated by the fact  that activities that advance one statutory purpose might conflict with other  purposes. We agree that it is important to evaluate similar activities and  have a separate recommendation addressing this need. But, the purpose  of programwide assessment is to demonstrate whether the provision of  flexibility in itself results in the intended benefits of the MTW program,  such as cost savings or increased family self-sufficiency. Demonstrating  that the increased flexibility the program offers has produced the intended  results is critical, particularly as Congress considers whether to expand  the program. We continue to believe in the importance of demonstrating  program results and therefore continue to recommend that HUD develop  performance indicators for the MTW program as a whole.
HUD was in partial agreement with four recommendations.
First, HUD said that proposed revisions to the reporting requirements  for MTW agencies had addressed our recommendation that the  agency improve its guidance to MTW agencies on providing  performance information in annual reports. HUD’s draft guidance is in  line with our recommendation that HUD require agencies to report  quantifiable and outcome-oriented information. However, because  these proposed revisions have yet to be finalized, we did not revise  our recommendation.
Second, HUD agreed that quantitatively assessing the effectiveness  of similar activities was an important step but noted the difficulties  associated with assessing the effectiveness of the program as a  whole. However, as noted above, we continue to believe in the  importance of demonstrating program results. Consequently, we did  not revise our recommendation.
Third, HUD stated that providing a menu of standard metrics may be  the best way to clarify the program’s statutory purposes and that it  had made progress in recent years in addressing our  recommendation that it issue guidance that clarifies the statutory  requirements. HUD also noted that the proposed revisions to the  reporting requirements would provide additional clarification on the  statutory requirements. These efforts, which were acknowledged in  the draft report, are a step in the right direction, and we encourage  HUD to continue finalizing this guidance. As noted above, because  these proposed revisions have yet to be finalized, we did not revise  our recommendation.
Fourth, HUD described recent efforts to assess compliance with two  statutory requirements and analysis that it could conduct once  proposed revisions to reporting requirements for MTW agencies were  finalized. Because the process used to assess compliance with one of  the requirements has not been formalized in policy and the proposed  revisions have not been finalized, we did not revise our   recommendation that HUD develop and implement a systematic  process for assessing compliance with statutory requirements.
HUD generally agreed with the three remaining recommendations. For  example, HUD agreed that it should proactively identify lessons learned  and described some of its recent efforts to do so. We acknowledged  these efforts in our draft report but noted the absence of a criteria-based,  regular process for identifying lessons learned. HUD also described plans  to develop a formal risk-based strategy for monitoring and, when we  asked for further clarification, stated that it agreed with our  recommendation to conduct an annual risk assessment for the MTW  program. Finally, HUD discussed potential strategies for verifying the  information that MTW agencies report using existing or planned HUD  systems.
HUD also requested that we consolidate four separate recommendations  into two, but we continue to believe that maintaining distinctions between  the separate recommendations is important. First, HUD requested that we  combine two recommendations: that it create a plan to quantitatively  assess the effectiveness of similar activities and the program as a whole  (including identifying the standard performance data needed), and that it  establish performance indicators for the program as a whole. Although  related, the two recommendations are distinct because the first focuses  on the need for program evaluation and the second on performance  measurement. Program evaluations typically examine a broad range of  information on program performance, while performance measurement  generates outcomes that show whether a program has achieved specific  objectives. As a result, we did not combine the recommendations.
Second, HUD requested that we combine the recommendation that it  issue guidance clarifying key program terms (such as the three statutory  purposes and five statutory requirements) with the recommendation that it  implement a systematic process for assessing compliance with statutory  requirements. However, defining program requirements and assessing  them are separate and distinct activities. Therefore, we did not combine  the recommendations.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of Housing  and Urban Development and other interested committees. In addition, the  report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-8678 or sciremj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: Our objectives were to examine (1) what is known about the extent to  which the Moving to Work (MTW) demonstration program is addressing  the program’s statutory purposes, (2) the Department of Housing and  Urban Development’s (HUD) monitoring of MTW agencies’ efforts to  address these purposes and meet statutory requirements, and (3)  potential benefits of and concerns about expanding the number of public  housing agencies (PHA) that can participate in the demonstration  program.
To evaluate what is known about the extent to which the MTW program is  addressing the program’s statutory purposes, we reviewed the most  recent annual reports as of January 2012 for 30 MTW agencies. We  reviewed these annual reports and the corresponding annual plans to  identify the ongoing activities that the agencies were implementing,  determine the extent to which these activities were linked with one or  more of the program’s statutory purposes, and assess the performance  information provided for each activity. To assess the performance  information provided by MTW agencies, we examined HUD’s reporting  guidance and compared it with internal control standards for federal  agencies. We assessed the reliability of selected information in the  reports by reviewing supporting documentation from a sample of seven  MTW agencies and interviewing the officials responsible for preparing  and reviewing this information. These seven agencies were
Cambridge Housing Authority (Cambridge, Massachusetts),
Chicago Housing Authority (Chicago, Illinois),
Housing Authority of the City of Atlanta (Atlanta, Georgia),
Housing Authority of the City of Pittsburgh (Pittsburgh, Pennsylvania),
Housing Authority of the County of Santa Clara/Housing Authority of  the City of San Jose (Santa Clara County and San Jose, California),
Lawrence-Douglas County Housing Authority (Lawrence, Kansas),
Vancouver Housing Authority (Vancouver, Washington).
We selected these agencies to provide diversity in geography, agency  size, and length of time participating in the program. We determined that  the reports were sufficiently reliable for the purposes of our review.
Through interviews and a literature search, we identified three studies of  the MTW program as a whole. We reviewed these studies to identify  information on the program’s effectiveness and any challenges  associated with assessing it. We determined that these studies were  methodologically sound and reliable for our purposes. We examined  HUD’s recent efforts to collect data from MTW agencies, including  documentation on the reporting requirements for MTW agencies. In  addition, we reviewed HUD’s fiscal year 2010-15 strategic plan and Fiscal  Year 2011 Annual Performance Plan for any performance indicators for  the MTW program. We also reviewed the GPRA (Government  Performance and Results Act) Modernization Act of 2010, Office of  Management and Budget guidance, internal control standards, and a  GAO report on attributes of successful performance measures. Further,  we identified five studies of specific MTW agencies or activities identified  by HUD and representatives of the sample of MTW agencies we  interviewed. Finally, we reviewed published reports and HUD’s website  for information on HUD’s efforts to identify lessons learned.
To assess HUD’s monitoring of MTW agencies’ efforts to address the  program’s statutory purposes and meet requirements, we obtained and  reviewed documentation of monitoring policies and procedures, including  the Standard Agreement that HUD executed with MTW agencies in 2008,  the MTW Desk Guide, a 2011 Memorandum of Understanding between  HUD’s Office of Public Housing Investments and Office of Field  Operations, and other HUD guidance. Based on these documents and  interviews with HUD staff, we identified three key monitoring processes:  the review of annual plans and reports, reviews of data entered into the  Moving to Work section of the Public and Indian Housing Information  Center (MTW-PIC), and annual site visits to each MTW agency.
To assess the extent to which HUD staff were following these monitoring  policies and procedures, we reviewed documentation of monitoring  activities for our sample of seven MTW agencies. For example, to verify  the steps HUD had taken to review annual plans and reports, we  reviewed the checklists that the MTW coordinators used to document  their review of these plans and reports. We also reviewed HUD’s  comment letters for fiscal year 2011. To verify the steps HUD had taken  to review data submitted into the MTW-PIC system, we reviewed monthly  reports that showed the degree to which MTW agencies overall complied  with reporting requirements from August 2011 through January 2012.  Finally, to verify that both headquarters and field office staff made site  visits and the extent to which they made annual visits, we reviewed the  most recently available site visit reports completed by the MTW Office for  all agencies as of October 2011. In addition, we interviewed the MTW  agencies in our sample and the corresponding HUD field office officials to  discuss the annual site visits.
We also compared HUD’s monitoring policies and procedures to internal  control standards for the federal government and HUD’s own program  management guidance. As a part of this analysis, we compared HUD’s  guidance to MTW agencies with the internal control requirement for clear  goals and objectives. We also reviewed information on HUD’s efforts to  clarify how agencies could certify compliance with the requirement to  assist “substantially the same” number of eligible families that would have  been served in the absence of MTW. In addition, we compared HUD’s  efforts to assess agencies’ compliance with statutory requirements with  the internal control standard related to assessing compliance with  program requirements. Further, we reviewed internal control standards for  the federal government and HUD’s own internal control standards and  identified the requirement that programs have an annual risk assessment.  We interviewed HUD regarding any risk assessment performed for the  MTW program. Finally, we interviewed HUD officials to determine  whether any of the performance information that MTW agencies reported  had been verified. We compared HUD’s lack of verification with the  internal control standards and guidance that emphasized the need for  control activities to ensure that program participants report information  accurately.
To discuss the potential benefits and concerns associated with expanding  the number of PHAs that can participate in the program, we reviewed  studies, reports, and testimonies by researchers, affordable housing  advocates, resident advocates, and the HUD Office of Inspector General.  For all three objectives, we interviewed officials from the seven MTW  agencies in our sample and representatives from affordable housing  advocacy organizations such as the Council of Large Public Housing  Agencies, the National Association of Housing and Redevelopment  Officials, the National Leased Housing Association, and the Public  Housing Authorities Directors Association. We spoke with resident  advocacy organizations such as the National Low-Income Housing  Coalition, the National Housing Law Project, and legal aid agencies that  represented residents serviced by five of our sample MTW agencies. We  also interviewed staff from the Center for Budget Policy and Priorities, a  research organization that has studied and written about the MTW  program; researchers who had evaluated the MTW program; and HUD  officials from the MTW office and the field offices that corresponded to our  sample of agencies. During our interviews, we discussed the potential  benefits of expansion and the concerns of these organizations. Based on  our review of available studies and reports and interviews with the above  mentioned stakeholders, we identified key benefits and concerns. We  also made observations based on our findings related to the availability of  performance information for the program and HUD’s monitoring efforts.
We conducted this performance audit from July 2011 to April 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Housing and Urban Development
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Paige Smith (Assistant Director),  Anna Carbino, Emily Chalmers, John McGrail, Marc Molino, Lisa Moore,  Daniel Newman, Lauren Nunnally, and Andrew Stavisky made key  contributions to this report.