Moving to Work Demonstration: Opportunities Exist to Improve Information and Monitoring
Released: 2012-05-21T00:00:00

Letter

Background

The MTW demonstration program was authorized by the Omnibus Consolidated Rescissions and Appropriations Act of 1996. The program is intended to give participating agencies the flexibility to design and test innovative strategies for providing and administering housing assistance. To implement such strategies, participating agencies may request waivers of certain provisions in the United States Housing Act of 1937, as amended.provisions in order to combine the funding they are awarded annually from different programs into a single, authoritywide funding source. Requirements outside of the 1937 Housing Act, such as fair housing rules, cannot be waived under MTW. In addition, certain sections of the act, including those that cover labor requirements and the demolition and disposition of public housing, cannot be waived. The 1996 act that created the program requires participating agencies to address three purposes and meet five requirements. Specifically, the three statutory purposes are to (1) reduce costs and achieve greater cost-effectiveness in federal housing expenditures, (2) give families with children incentives to obtain employment and become self-sufficient, and (3) increase housing choices for low-income families. For example, to reduce administrative costs MTW agencies can reduce the frequency of income verifications for households with fixed incomes. In making these changes, MTW agencies must For example, agencies may request to waive certain 1. serve substantially the same total number of eligible low-income families that they would have served had funding amounts not been combined; 2. maintain a mix of families (by family size) comparable to those they would have served without the demonstration; 3. ensure that at least 75 percent of households served are very low 4. establish a reasonable rent policy to encourage employment and self- 5. assure that the housing they provide meets HUD’s housing quality standards.

The program’s ultimate goal is to identify successful approaches that can be applied to PHAs nationwide.

The 1996 act authorized MTW for 30 agencies. HUD invited PHAs to apply for the program and selected an initial cohort of 24 PHAs from among the respondents. Six more were added as a result of the Jobs- Plus initiative. Since then, some agencies have opted not to participate, and additional agencies have been added to replace them. Other agencies have been added through specific appropriations language (see fig. 1). In addition, some agencies have completed their participation and exited the program. As of January 2012, a total of 39 PHAs were authorized to participate, and 35 were participating.

MTW agencies do not receive special funding allocations. Rather, they receive funds from the three traditional primary funding sources (public housing capital funds, public housing operating funds, and Housing Choice Vouchers). Traditionally, PHAs have been required to use the funds from each separate source only for specific purposes, but MTW agencies may combine the money from the three sources and use them for a variety of HUD-approved activities. This fungibility is intended to give MTW agencies greater flexibility. For example, public housing operating funds are traditionally used to make up the difference between the rents charged for units and the actual cost of operating them. Capital funds are used for modernization efforts and management improvements, while voucher funds provide rental assistance in the private market. However, by combining funds an MTW agency may use public housing capital funds to issue additional vouchers or use voucher funds to develop more public housing to better fit the needs of its community. MTW agencies also have the authority to use their funds to implement innovative activities that differ from traditional housing assistance. For instance, an MTW agency can use funds to replace public housing with mixed-income communities or reach special-needs populations, such as the homeless, using vouchers paired with supportive services.

A Standard Agreement, executed in 2008 to replace individual contracts between HUD and participating PHAs, governs the conditions of participation in the program. HUD enters into this agreement with each MTW agency. HUD created the agreement to standardize the language in its contracts and its reporting requirements and to help create a more operationally sound program. The Standard Agreement includes a termination date (the end of each agency’s 2018 fiscal year) and an attachment that sets out reporting requirements (Attachment B). While much of the Standard Agreement is the same for all MTW agencies, two sections are tailored to individual agencies: a description of the formulas for determining the amounts of funding each agency will receive and an optional section that may include some agency-specific authorizations.

MTW agencies with less than 10 percent of their housing stock in the MTW program continue to submit the 5-Year and annual plans required by Section 5A of the 1937 Housing Act. Only information not included in these documents would need to be included in a supplemental annual MTW plan. contents that lists the information that agencies must include in their annual plans and reports. For example, the plan must include, among other things, a description of how each planned activity relates to at least one of the three purposes of the program; baselines, proposed benchmarks, and proposed metrics for assessing the outcomes of each activity; citations of the authorizations that give the agency the flexibility to conduct the activity; and descriptions of required rent reform activities. In addition, the plan must include a certification that the agency published a notice of plans to hold a public hearing on the plan, made the agency’s annual plan available for public inspection, and conducted a public hearing to discuss the annual plan prior to its approval. Similarly, the Standard Agreement outlines the information that MTW agencies are required to include in annual MTW reports. These reports must include, for example, detailed information on the impact of each activity, including comparisons of actual outcomes to the benchmarks proposed in the annual plan. If the agencies do not achieve the benchmarks or the activities are determined to be ineffective, the MTW agency is required to describe the challenges, and, if possible, identify alternative activities that may be more effective. MTW agencies also are required to self-certify that they are in compliance with three statutory requirements: assisting substantially the same total number of eligible low-income families that they would have served had funding amounts not been combined; maintaining a mix of families (by family size) comparable to those they would have served had funding amounts not been combined under the demonstration; and ensuring that at least 75 percent of households served are very low income.

As well as meeting the requirements in the Standard Agreement, MTW agencies must submit tenant-related data into the Moving to Work section of the Public and Indian Housing Information Center (MTW-PIC). According to HUD officials, the MTW-PIC module was created because the standard PIC system that non-MTW agencies use to report tenant data could not accommodate some of the activities allowed under MTW, such as less frequent tenant income recertifications and rent calculations that vary from HUD’s standard calculations. The MTW-PIC module was created in 2007, and most MTW agencies had transitioned to it by 2008.

The Office of Public Housing Investments within the Office of Public and Indian Housing at HUD headquarters is the designated program office for the MTW demonstration program. Within the Office of Public Housing Investments is an MTW Office that includes a program director and four coordinators who are each assigned to a specific group of MTW agencies. The MTW Office is responsible for, among other things, processing, reviewing, and approving all annual plans submitted by MTW agencies; establishing guidelines for MTW agencies; monitoring approved activities and accomplishments; and accepting annual reports. Individual MTW coordinators facilitate the reviews of planned and implemented activities and are responsible for coordinating with other HUD offices, including local HUD field offices, to obtain additional input on MTW agencies’ planned activities and accomplishments. In January 2011, the Office of Public Housing Investments signed a Memorandum of Understanding with HUD’s Office of Field Operations to increase collaboration and formally describe the roles and responsibilities of the MTW Office and HUD field office staff. Per this memorandum, field office staff assist the MTW Office by reviewing and providing input on annual plans and reports, ensuring that agencies are reporting tenant information, and participating in annual site visits.

HUD Has Not Identified Standard Performance Data and Indicators Needed to Evaluate the Program

MTW agencies provide descriptions of their activities and performance information in their annual reports to HUD. They show how the activities link to the program’s statutory purposes in their annual plans, as required, and sometimes also in their annual reports. However, the type of performance information they provide varies, and HUD has provided limited guidance. While varied information on individual activities is available, a comprehensive evaluation of the MTW program is lacking, in part because HUD does not have a plan for identifying and analyzing standard performance data and has not established performance indicators for the MTW program as a whole. Further, HUD does not have a systematic process for identifying lessons learned by individual MTW agencies that can be replicated at other PHAs.

Although Information on Activities Is Generally Available at the Agency Level, It Varies Due to HUD’s Limited Guidance

MTW agencies report information on specific activities, including descriptions, in their annual reports. Agencies are required in their annual plans to link each of their proposed activities to one of the program’s three statutory purposes, and some agencies also show links between ongoing activities and statutory purposes in their annual reports. The three statutory purposes are to reduce costs and achieve greater cost- effectiveness in federal housing expenditures, give families with children incentives to obtain employment and become self-sufficient, and increase housing choices for low-income families. According to the most recently available annual reports, 30 agencies have over 360 ongoing activities, including rent reform initiatives and work requirements (see table 1).

According to the most recent annual reports (and corresponding plans) for 30 MTW agencies, agencies associated the largest percentage of ongoing activities (41 percent) with the statutory purpose of reducing costs and improving cost-effectiveness (see fig. 2). For example, agencies associated changes in certification schedules, inspection protocols, and medical deductions with reduced costs. The agencies linked 30 percent of their ongoing activities to the statutory purpose of increasing housing choices and 24 percent to encouraging self- sufficiency. The agencies did not link 4 percent of their ongoing activities to any purpose in either their most recent annual plan or report.

In its Standard Agreement, HUD requires agencies to include in their annual reports performance information on the impact of each implemented activity, including describing the metrics used to assess outcomes and comparing actual performance with proposed benchmarks.defined them in 2009 training materials. In these materials, HUD defined a metric as the “unit of measure that quantifies the changes that might occur as a result of the MTW activity” and a benchmark as the “projected outcome of the MTW activity.” Further, in these 2009 training materials HUD defined an outcome as the “actual, measured result of the implemented activity.” As examples, the training materials stated that a metric could be the hours of staff time saved, a benchmark could be the number of anticipated staff hours saved, and the outcome could be staff hours actually saved. HUD directs agencies to develop their own metrics While HUD did not define these terms in the agreement, it and benchmarks for each activity based on local and community standards. Our analysis of the most recent annual reports for 30 MTW agencies showed that the agencies reported performance information for 91 percent of the ongoing activities included in the reports and used over 1,000 metrics to assess these activities. MTW agencies collectively met the benchmarks associated with 40 percent of these metrics and fell short of meeting 17 percent of them. For 30 percent of the metrics, it was too soon to determine if the benchmarks had been met because the activities were not yet completed. For the remaining 13 percent, information (either the benchmark or performance data) was lacking to determine whether the benchmark was met.

While MTW agencies are generally devising their own metrics for activities and reporting performance information, the usefulness of this information is limited because, in some cases, it is not outcome-oriented. Our analysis of the most recent annual reports for 30 MTW agencies showed that the type of information that agencies reported on the impact of their activities varied. For example, for similar activities designed to promote family self-sufficiency, one MTW agency reported only the number of participants, which is generally considered an output, and another did not provide any performance information. In contrast, a third agency reported on, among other things, the average income of program graduates, which we consider an outcome.

Internal control standards state that good guidance (information and communication) is a key component of a strong internal control framework and that there is a need for clear documentation. To be consistent with the GPRA Modernization Act of 2010, HUD’s guidance on reporting performance information should indicate the importance of outcome-oriented information. Specifically, the act states that an agency should establish efficiency, output, and outcome indicators for each program activity. Furthermore, Office of Management and Budget guidance on implementation of the act states that quantitative and outcome-focused measures are preferred.

At the time of our review, HUD’s guidance did not specify that agencies should report quantifiable and outcome-oriented performance information. According to the Director of the MTW Office, Attachment B of the Standard Agreement is the most current guidance on the information that agencies should report in their annual reports. It simply states that agencies are to provide detailed information on the impact of the activity and compare it against the proposed benchmarks to assess outcomes, including whether an activity is on schedule. The attachment does not define terms or set expectations for the type of information to be reported. After the Standard Agreement was executed in 2008, HUD conducted training for participating agencies. As previously discussed, the 2009 training materials defined key terms such as a metric and outcome and outlined steps agencies could take to evaluate their activities. HUD also encouraged the MTW agencies to use metrics and benchmarks that did not focus on the number of individuals participating in an activity but rather on the objectives of the activity and to report quantifiable information. While HUD has posted the 2009 training materials on its website, these materials have not been incorporated into Attachment B of the Standard Agreement.

According to the Director of the MTW Office, HUD has not made its guidance more specific because agencies are implementing a wide variety of activities and thus require some reporting flexibility. We acknowledge the need for flexibility, but it is important that HUD require agencies to report at least some outcome-oriented performance information. Without more specific guidance on the reporting of performance information, HUD cannot be assured of collecting information that reflects the outcomes of individual activities. Such information would help HUD assess the demonstration program and whether the activities are furthering program purposes.

Design and Data Weaknesses Have Prevented a Comprehensive Evaluation of MTW

As we have previously reported, obtaining performance information from demonstration programs that are intended to test whether an approach (or any of several approaches) can obtain positive results is critical. This information is needed to help determine whether the program has led to improvements consistent with its purposes. HUD has sponsored three broad reviews of the MTW program, but these studies are not comprehensive evaluations because of data limitations, among other things. A 2004 Urban Institute evaluation of the MTW program found that most MTW agencies reported modest benefits from activities related to administrative streamlining and that these results were often not as dramatic as the agencies had anticipated.difficulty in determining whether MTW activities related to employment The study also noted the and income had any independent effect and that MTW activities resulted in both greater and more limited housing choice.

A second study, conducted in 2007 by Applied Real Estate Analysis, Inc. and the Urban Institute, reviewed eight MTW agencies that had placed limits on the length of time that residents could receive housing assistance. concluded that only limited information was available with which to evaluate outcomes or establish cause-and-effect relationships between agencies’ policies and recipients’ experiences. It noted that there were significant limitations to what could be learned from these experiences because no evaluative framework had been built into the program. Finally, a 2010 HUD Report to Congress found that the effects of many MTW activities, especially as they related to residents, could not be conclusively identified because of the variety of and differences in the activities and metrics that MTW agencies were implementing. However, the report did identify some results concerning agencies’ ability to more efficiently allocate resources and engage in strategic long-term planning. For instance, the study noted that some agencies had seen positive effects from combining their traditional sources of funding and streamlining their operations—for example, by simplifying their housing quality inspections.

The study found that each of the approaches varied and These three studies of the MTW program and our work have identified several challenges that have hindered evaluation efforts. These challenges include the way the program was initially designed and the resulting lack of standard performance data as well as the lack of performance indicators for the program as a whole.

Robert Miller, Martin D. Abravanel, Helene Berlin, Elizabeth Cove, Maria-Alicia Newsome, Carlos A. Manjarrez, Lipi Saikia, Robin E. Smith, and Maxine V. Mitchell, The Experiences of Public Housing Agencies That Established Time Limits Policies Under the MTW Demonstration (Applied Real Estate Analysis, Inc. and The Urban Institute, May 2007).

Lack of Program Evaluation Design and Standard Performance Data

HUD has taken steps to address the problems noted with MTW’s initial design and the lack of standard data; however, it has not analyzed the data it currently collects or determined whether these data are sufficient to evaluate similar activities and the program as a whole. As we have previously reported, comparable data are essential to a full analysis of programs that incorporate a variety of activities. We also noted that obtaining performance information from demonstration programs that are intended to test whether an approach (or any of several approaches) can obtain positive results is critical. Finally, we have reported that agencies need to identify any data that will be needed to assess the effectiveness of program regulations.

Researchers and others have noted the limitations that the program’s initial design posed to evaluation. In the early years of MTW, rigorous evaluation strategies were not required, and the program lacked a research design that would have helped in determining baseline information. The 2004 Urban Institute review of MTW concluded that there were limits to what could be learned from its review for a variety of reasons, such as the inability to separate individual components of agencies’ MTW activities for analysis. As a result, the report is mainly descriptive and qualitative. HUD’s 2010 Report to Congress noted that because rent reform activities varied greatly and were not implemented using a controlled experimental methodology, the authors were unable to recommend specific reforms as best practices.

To help evaluate aspects of the MTW program moving forward, MTW officials have added requirements for new agencies. According to HUD’s 2010 Report to Congress, the three agencies admitted to the MTW program in 2009 had strong evaluation components. Two of these agencies have commitments from local universities to evaluate their programs. Additionally, HUD has required the two newest agencies to participate in a controlled rent reform study. However, these improvements will not help evaluate the program as a whole or the activities implemented by the 30 other MTW agencies.

Likely due to the absence of an evaluation framework for the MTW program, researchers have noted the lack of standard performance data needed to evaluate similar activities and the program as a whole. The 2004 Urban Institute study noted that the lack of consistent data on resident characteristics, incomes, and rent payments prevented the authors from being able to determine whether individual agencies were able to achieve the goal of increasing self-sufficiency. Similarly, the HUD Inspector General reported in 2005 that HUD lacked the empirical data needed to assess the program as a whole.

Since these reports, HUD has started collecting some additional data from MTW agencies, but it has not yet analyzed the data. HUD created the MTW-PIC module to collect tenant characteristics such as household size, income, and educational attainment. However, according to MTW officials, HUD has not used these data to analyze the program’s effects, such as changes in resident income. In addition, HUD’s Standard Agreement has required agencies since 2009 to provide information in their annual reports on the impact of activities, including benchmarks and metrics. While these reports are informative, they do not lend themselves to quantitative analysis because the reporting requirements do not call for standardized data, such as the number of residents that found employment.

In addition, whether these data are sufficient to assess similar activities and the program as a whole is not clear, and HUD has not identified the data it would need for such an assessment. For example, neither MTW- PIC nor annual reports capture standard data on implemented activities. Further, according to the Director of the MTW Office, MTW-PIC does not include information on individuals who receive nontraditional services from an MTW agency, such as homeless assistance or case management. Representatives from MTW agencies have suggested that HUD should collect some standard data for similar activities. For example, they noted that if HUD required all agencies that implemented rent reform activities to report standard data, the results of these efforts could be analyzed even if the specific activities varied. The Director of the MTW Office also noted that MTW-PIC was a potential tool for collecting and analyzing standard demographic data.

The MTW Office has recently developed a statement of work for an evaluation of the program, but HUD has not allocated funding for the study, according to the Director of the MTW Office. Among other things, the proposed evaluation is intended to assess the current state of the MTW demonstration and determine the extent to which the three statutory purposes have been addressed. The study is also expected to include an analysis of outcomes associated with specific activities and the demonstration as a whole to identify which MTW activities are appropriate for expansion to all PHAs. However, the approach envisioned may be limited because it would primarily rely on existing data sources. Until HUD develops and implements a plan (that includes the identification of standard data) to quantitatively assess similar activities and the MTW program as a whole, HUD cannot determine their effectiveness. While such analyses may be challenging, they would enhance HUD’s ability to rigorously assess the demonstration.

HUD has not established performance indicators for the MTW program as a whole. The GPRA Modernization Act of 2010 requires that federal agencies establish efficiency, output, and outcome indicators for each program activity as appropriate. Internal control standards also require the establishment of performance indicators. In addition, we have previously reported that successful performance indicators demonstrate results and provide useful information for decision making in order to track how programs and activities can contribute to attaining an organization’s goals and mission, among other things.

As previously discussed, MTW agencies set their own performance metrics for activities, but HUD has not established performance indicators for the program as a whole. HUD’s Fiscal Year 2011 Annual Performance Plan established agencywide performance indicators but did not explicitly connect the MTW program to any of them. specific targets for the program, the Director of the MTW Office noted that the program’s activities support some of the agencywide indicators. Specific performance indicators for the MTW program could be based on the statutory purposes. For example, agencies could report on the savings achieved (reducing costs) and the number of additional households served (increasing housing choices). Without performance indicators for the MTW program, however, HUD cannot demonstrate the results of the program as a whole.

HUD Lacks a Systematic Process for Identifying Lessons Learned

While HUD has identified some lessons learned on an ad hoc basis, it does not have a systematic process in place for identifying such lessons. We have previously reported that obtaining impact information from demonstration programs that are intended to test whether an approach (or any of several approaches) can obtain positive results is critical. This information should be gathered to help determine whether programs have led to improvements consistent with their purposes.

HUD, Fiscal Year 2011 Annual Performance Plan (Washington, D.C.: 2011).

PHAs nationwide. These practices, which are posted on HUD’s website, included implementing savings accounts for public housing and voucher recipients to promote resident savings. Most recently, HUD’s 2010 Report to Congress described promising policies, practices, and concerns. In addition, officials from some of the MTW agencies we interviewed noted that HUD officials had shared information on activities that had shown positive effects during site visits, quarterly phone calls, newsletters, and annual conferences. Finally, HUD’s statement of work for its proposed evaluation of the MTW program includes the creation of five case studies that would review MTW flexibilities.

However, these efforts have shortcomings. In most cases, the practices chosen were based on the opinions of HUD or contracted staff and largely involved anecdotal (or qualitative) data rather than quantitative data. The lack of standard performance data has affected HUD’s ability to systematically identify lessons learned. In its 2005 report on the MTW program, the HUD Inspector General noted that the lack of data on the program made it difficult to identify activities that could be considered models for addressing the three statutory purposes or that could be used to show the importance of individual policy changes. Further, HUD has not established criteria, such as demonstrated performance, for identifying lessons learned. Finally, HUD has not made regular efforts to review and identify lessons learned. Because HUD does not currently have a systematic process for identifying lessons learned, it is limited in its ability to promote useful practices that could be implemented more broadly.

HUD, Office of Inspector General, 2005-SE-001.

HUD Generally Follows Its Monitoring Policies and Procedures, but Could Strengthen Them

HUD has policies and procedures in place to monitor MTW agencies. First, HUD requires program staff to review and comment on agencies’ annual plans and reports. Second, staff review tenant data submitted by MTW agencies. Third, program staff conduct annual site visits at each participating agency to provide technical assistance and program updates. HUD generally follows these policies and procedures, which focus on technical assistance rather than compliance. Due in part to this focus, HUD’s policies and procedures have several key weaknesses. Specifically, HUD has not clarified program terminology, ensured that each MTW agency is meeting statutory requirements, performed an annual risk assessment, or developed policies and procedures to verify the accuracy of key information that MTW agencies self-report.

HUD Reviews Agency Information and Conducts Annual Site Visits

HUD’s monitoring policies and procedures for the MTW program are contained in a desk guide, which describes the roles and responsibilities of HUD staff in reviewing annual plans and reports and data submissions, making site visits, and performing other monitoring activities. In January 2011, HUD’s Office of Public Housing Investments and Office of Field Operations signed a Memorandum of Understanding documenting the framework for headquarters and field staff to follow in overseeing MTW agencies. According to the memorandum, the MTW Office (within the Office of Public Housing Investments) is responsible for oversight of the MTW program. In many cases, the MTW Office works with field offices to jointly develop responses to MTW agency issues. Further, a MTW Working Group—consisting of representatives from Public and Indian Housing programs, the Real Estate Assessment Center, and the Office of Policy Development and Research—was established to assist with the annual plan and report review process. As part of the memorandum, the Offices of Public Housing Investments and Field Operations agreed to the protocols set forth in the desk guide.

HUD staff from the MTW Office and field offices and the MTW Working Group share responsibility for reviewing and commenting on participating agencies’ annual plans and reports. The Standard Agreement (Attachment B) outlines the requirements for annual plans and reports that agencies must submit. MTW coordinators, who are each responsible for a specific number of MTW agencies, have the lead role in reviewing annual plans and reports to determine if they meet the requirements of Attachment B and obtaining input from other HUD staff, including field offices and the MTW Working Group. Field offices are required to review the annual plans and reports submitted by the MTW agencies in their jurisdictions and provide their assessment to the MTW coordinator. Similarly, the MTW Working Group reviews and provides comments to the MTW coordinator. Coordinators summarize the comments from the field offices and MTW Working Group and send them to the agencies. The coordinators and field office staff work with MTW agencies to resolve any outstanding issues. Once such issues have been resolved, the MTW Office approves annual plans and accepts annual reports.

Soliciting public comments on the plan is important because, unlike other PHAs, MTW agencies are not subject to the Public Housing Assessment System, which includes a customer satisfaction survey that promotes resident participation. five statutory requirements.MTW coordinators to verify each agency’s certification that it has met the three statutory requirements.

However, these procedures do not require Interviews with MTW coordinators and field staff and documentation for our sample of seven MTW agencies indicated that HUD generally followed these procedures. Documentation we reviewed for the agencies in our sample showed that the coordinators generally completed checklists while reviewing annual plans and reports. For example, coordinators verified that all ongoing activities were reported, ensured the agency included its certification that it had met three of the five statutory requirements, and made certain the agency certified that it had held a public hearing on its annual plan, among other requirements. Coordinators also provided comments to agency staff on annual plans and reports. of the comments, the MTW Office notified the agency in writing that its plan had been approved and report accepted.

Field office staff that we interviewed said that they reviewed annual plans and reports and sent their comments and concerns to MTW coordinators. MTW agencies are subject to a variety of other reporting requirements. For example, MTW agencies are required to report voucher utilization in the Voucher Management System. They also must procure a public accountant to perform an Office of Management and Budget Circular A-133 compliance audit and submit unaudited financial statements. In addition, they are subject to HUD physical and management inspections of public housing and on-site monitoring reviews related to voucher reporting. example, in January 2012, MTW agencies overall achieved a 100-percent tenant data submission rate.

In addition, HUD conducts annual site visits to provide technical assistance to each MTW agency. The MTW Office and the local field office conduct these visits jointly. The MTW Office (in particular the coordinator assigned to the agency) takes the lead role in conducting the visit, including preparing the agenda, coordinating with the local HUD field office, and working with the MTW agency to select properties to visit. According to HUD officials, the primary objective of the site visit is to provide technical assistance and build a working relationship with the participating agencies, not to assess compliance with statutory requirements. However, HUD officials stated that if compliance issues with statutory purposes are found, HUD staff address these issues during the site visit, and coordinators often develop timelines for the agency to come into compliance.

Our analysis of documentation of site visits to participating agencies indicated that MTW Office and field staff generally followed HUD’s annual site visit procedures. Specifically, analysis of site visit reports indicated that HUD officials generally discussed the effectiveness of activities and helped resolve any outstanding issues. For example, as a result of site visits, HUD staff recommended that an agency include cost-saving measures in its annual plan, requested clarification of output measures, and encouraged one agency to submit articles to the MTW newsletter to share its experiences on how rent reform encouraged self-sufficiency.

Interviews with our sample of MTW agencies and corresponding field office officials also indicated that HUD was following its policies and procedures for annual site visits. MTW agency officials we spoke with indicated that the site visits were generally beneficial because they provided an opportunity for in-person discussions that helped facilitate communication with HUD. HUD’s field office staff noted their active involvement over the years, which had become more defined with the issuance of the desk guide in 2011. According to the Director of the MTW Office, the office is considering conducting future site visits using a risk- based approach. Using this approach, HUD would conduct site visits less frequently but would focus on larger agencies that had implemented a wide range of complex activities and newly admitted agencies that were implementing new activities.

To foster information sharing across agencies and provide technical assistance, HUD employs a number of additional strategies. For example, HUD hosts annual conferences to share information with MTW agencies and facilitate information sharing among agencies. The conferences cover a variety of topics, and all participating MTW agencies are invited to attend. For example, the 2011 conference focused on effectively managing funds in a challenging budgetary environment. HUD also has engaged participating agencies in quarterly conference calls and other training related to program changes such as the conversion from PIC to MTW-PIC and the transition to the Standard Agreement. Further, HUD issues notices on various topics, such as MTW reporting requirements, and publishes quarterly newsletters that highlight activities relating to each statutory purpose, among other topics. HUD also publishes each agency’s annual report and researchers’ evaluations of MTW activities on its website.

HUD’s Monitoring Policies and Procedures Have Several Key Weaknesses

Although HUD follows the policies and procedures that it has in place, it could do more to ensure that MTW agencies are demonstrating compliance with statutory requirements and to identify possible risks relating to activities implemented by each agency, among other things. First, HUD has not issued guidance to participating agencies clarifying key program terms, including definitions of the purposes and statutory requirements of the MTW program. Internal control standards require the establishment of clear, consistent goals and objectives. As previously noted, MTW authorizing legislation established three purposes for the program, and agencies must link each of their activities to one of these purposes. However, HUD has not clearly defined what the language in some of these purposes means, such as “increasing housing choices for low-income families.” MTW agencies have linked activities to this purpose that range from using block grant funding to support homeownership programs to requiring applicants to complete a renter education program to establishing a prisoner re-entry housing program. HUD noted the lack of a clear definition in its 2010 Report to Congress but continued to require that MTW agencies link activities to this purpose. According to MTW officials, they have not defined what is meant by “increasing housing choices” so that agencies have the ability to define this term in a manner that fits their local needs. In addition, HUD has not clarified what is meant by “serving a comparable mix of families” but also requires agencies to comply with this requirement. MTW agencies we spoke with described varying interpretations of this requirement. For example, officials from one agency told us that they observed how family sizes changed in their community and compared those changes to changes in families within the MTW program, using community survey data and data from the agency’s internal system. Officials from another agency we spoke with said that over time it had become increasingly difficult to determine compliance with this statutory requirement.

HUD has recently taken steps to clarify some terminology, explaining how agencies can certify that at least 75 percent of the families they serve have very low incomes and that they are serving substantially the same number of households under MTW as they did before the program. In addition, HUD is revising its reporting requirements for MTW agencies. As part of this process, HUD officials told us that they plan to update their guidance to more completely collect information related to the program’s statutory purposes and requirements. They acknowledged that the guidance could be strengthened to require MTW agencies to provide their agency-specific definition for the three statutory purposes. As a first step, they noted that they planned to require agencies to define “self- sufficiency” by either choosing one of the definitions provided by HUD or creating their own. Similarly, the officials stated that they would consider requiring MTW agencies to choose between using HUD’s definition of increasing housing choices or creating their own definition. Although a step in the right direction, allowing MTW agencies to create their own definitions of key terms would make it difficult to assess the effectiveness of efforts to address statutory purposes. HUD officials also said that the revised guidance would provide standardized tables for agencies to report data related to the requirement to serve a comparable mix of families. Until HUD clearly defines what is meant by all of the statutory purposes and requirements of the MTW program, HUD cannot effectively determine whether agencies are addressing these purposes and meeting requirements.

Second, HUD has only recently assessed agencies’ compliance with two self-certified requirements and has not assessed compliance with the third. Internal control standards require control activities to be in place to address program risks. In addressing these risks, internal control guidance states that management should formulate an approach for assessing compliance with program requirements. While HUD has recently made efforts to assess agencies’ compliance with two of the three self-certified requirements, it does not have a process in place to systematically review compliance with all three requirements. In 2011, HUD for the first time assessed participating agencies’ compliance with the requirement to assist “substantially the same” number of eligible families that would have been served in the absence of MTW. HUD collected data from MTW-PIC, the Voucher Management System, and each participating agency’s most recent annual report on the number of public housing units occupied, vouchers utilized, and other families housed and used a formula to compare these data with similar data reported before MTW. HUD and MTW agency staff we interviewed told us that they worked together to discuss discrepancies in the calculations. According to the Director of the MTW Office, agencies were in compliance with this requirement if they were serving at least 95 percent of the number of families in their baseline figure. HUD’s recent review of each agency’s baseline calculation indicated that all but one of the agencies were in compliance. Also in 2011, HUD reviewed MTW-PIC data for the first time to determine agencies’ compliance with the requirement that at least 75 percent of assisted residents be very low income. HUD’s analysis of MTW-PIC data showed that, as of September 2011, 91 percent of the residents served by MTW agencies fell into this category.

While HUD has taken steps to assess compliance with these two statutory requirements, it has not yet developed a methodology for assessing agencies’ compliance with the requirement to maintain a comparable mix of families. The Director of the MTW Office acknowledged that self-certifications were not the best means of ensuring compliance and told us that the planned revisions to the reporting requirements for MTW agencies would help assess compliance with the requirements to maintain a comparable mix of families and ensure that at least 75 percent of families assisted are very low income. Without a process for systematically assessing compliance with statutory requirements, HUD lacks assurance that agencies are complying with them.

Third, HUD has not performed an annual assessment of program risks. Internal control standards state that an agency should have a risk assessment plan that considers internal and external risk factors and establishes a control structure to address those risks. The standards also state that managers should focus on control activities to address risks that may involve verifications, performance reviews, and documentation, among other things. HUD’s own internal control standards also require its program offices to perform an annual risk assessment of their programs or administrative functions using a HUD risk-assessment worksheet. These standards also stress the importance of performing a risk assessment when there are significant program changes. According to the Director of the MTW Office, the office has not performed an annual risk assessment for the MTW program because it was not aware of this requirement.

MTW agencies are exempt from scoring in the Public Housing Assessment System and the Section 8 Management Assessment System. However, MTW agencies are subject to physical inspections conducted by the Real Estate Assessment Center under HUD guidelines and issued a score. This score is entered into the Public Housing Assessment System and can be viewed by MTW staff at any time. A score of 22 or below is flagged by the Real Estate Assessment Center (the maximum score is 30) and reported to the appropriate field office. their perceived level of risk. While monitoring procedures are not risk- based, the Director of the MTW Office stated that his office would become aware of risks from HUD’s field office staff, which have routine responsibility for reviewing financial audits and Office of Management and Budget compliance audits. As previously discussed, HUD is considering moving toward conducting risk-based site visits. In addition, according to HUD officials, the office is considering other methods to more rigorously analyze MTW agency risk factors. By not performing an annual risk assessment and implementing a risk-based approach to monitoring MTW agencies, HUD lacks assurance that it has properly identified and addressed risks that may prevent participating agencies from addressing program purposes and meeting statutory requirements. HUD also lacks assurance that it is efficiently using its limited monitoring resources.

GAO/AIMD-00-21.3.1 and GAO-01-1008G. reflect the source documents.accuracy of any reported performance information, it lacks assurance that this information is accurate. To the extent that HUD relies on this information to assess program compliance with statutory purposes and requirements, its analyses are limited.

Expanding MTW May Offer Benefits but Also Raises Questions

Legislation has been proposed to expand the number of PHAs that can participate in the MTW program, and a recent HUD report recommended expanding the program up to twice its size. As of March 2012, a maximum of 39 PHAs could participate in the program, but a 2011 Senate bill would direct HUD to increase that number up to 250. In addition, legislation has been drafted that would establish MTW as a permanent program and eliminate the current restrictions on the number of agencies that can participate. HUD’s 2010 Report to Congress recommends increasing the number of participating agencies to about 60. HUD and some stakeholders believe that expansion could provide the needed information on the effect of the MTW program and allow more PHAs to test innovative ideas, but questions remain about the lack of performance information on current MTW activities. In addition, alternatives to expansion exist, including implementing a more narrowly focused program.

Current MTW Participants and Affordable Housing Advocates Indicated that Expanding MTW Could Help Demonstrate the Program’s Effect and Increase Innovation

According to HUD, some affordable housing advocates, and MTW agencies we interviewed, expanding the MTW program could help demonstrate the program’s effect and increase the number of lessons that can be learned from the program. HUD has reported that doubling the number of MTW agencies with the use of strategic criteria and program implementation could help demonstrate the effects of MTW on a broader scale and enable the housing industry to learn even more from the demonstration. For example, expansion could provide more information on how MTW flexibilities would affect a broader group of PHAs. The Director of the MTW Office noted that some MTW activities, specifically those related to administrative streamlining, had influenced the draft Affordable Housing and Self-Sufficiency Improvement Act of 2012 (AHSSIA). Some affordable housing advocates that we met with emphasized the value of the changes, such as decreases in concentrated poverty, that have occurred in some of the communities affected by the MTW program and indicated that expansion could enable more PHAs to address local needs and therefore benefit additional communities. Similarly, officials from MTW agencies that we contacted stated that expansion of the program would provide a broader testing ground for new approaches and best practices.

Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004). implementation innovations. Officials from an organization that advocates on behalf of large PHAs and supports expansion noted that affordable housing needs varied by locality and that the MTW program enabled participating agencies to design effective approaches based on local needs. Similarly, another affordable housing advocacy organization told us that they supported expanding MTW not only because it enabled participating agencies to tailor activities to local needs but also because it involved local communities in the process.

Officials from several of the MTW agencies we interviewed also noted that the MTW program had empowered them to create and implement strategies that addressed local issues and said that expanding the program would give more PHAs the same flexibility. For example, in one northeastern state where the housing stock was relatively old, the MTW agency was able to focus on developing new affordable housing. Another MTW agency in a western state with mostly newer housing stock chose to reduce the frequency of inspections of its properties and focus its efforts on administrative streamlining and the disposition of its older units. Further, several MTW agencies that we interviewed described how they implemented the requirement to establish a rent policy that encouraged employment and self sufficiency. For example, officials from one MTW agency told us that they believed the traditional requirement that residents pay 30 percent of their adjusted income in rent was a disincentive to work, because as resident income increases so would the payment toward rent. To encourage residents to seek work, this agency implemented work requirements and a minimum rent. Additionally, some agencies have used their MTW status to establish programs that focus on specific populations, including working families with children, the elderly and disabled, and the homeless.

Some proponents of expansion that we interviewed also noted that expanding the MTW program could provide more PHAs with the ability to use funding from different sources more flexibly than possible without MTW status. Agencies without MTW status have to implement their activities while adhering to the regulations associated with three different funding streams, evidence of the fragmented nature of housing assistance.certain provisions of the 1937 Housing Act in order to combine annual funding from separate sources into a single authoritywide funding source. HUD field office staff with responsibility for monitoring MTW agencies observed that the single-fund flexibility was beneficial because it enabled participating agencies to develop supportive service programs, such as job training or educational programs, which help move families toward self-sufficiency. One HUD field office official stated that this flexibility would be a significant benefit for other PHAs. An affordable housing advocate we met with also noted that this ability to use different kinds of funds interchangeably was beneficial because it enabled MTW agencies to shift funds based on local priorities. Further, officials from the MTW agencies we interviewed agreed that this flexibility was beneficial. For example, officials from one MTW agency stated it had been able to use the single fund to organize itself as a business organization, develop a strategic plan based on the housing needs of low-income families in the community, leverage public funds and public and private partnerships, and develop mixed-income communities. Two of the MTW agencies that we interviewed also stated that the single-fund flexibility had enabled them to fund programs that encouraged self-sufficiency among residents. For example, officials explained that they had used funding for coaching and counseling services, job training support, and education programs. Finally, officials from three of the MTW agencies we interviewed noted a related benefit of participation. They said that their MTW status had enabled them to respond more quickly to real estate opportunities because they do not have to wait for HUD approvals to purchase properties.

Lack of Performance Information and Other Issues Raise Questions about Expansion

A lack of performance information, limited HUD oversight, and concerns about the program’s impact on residents raise questions about expanding the MTW program. As we noted previously, conclusive information about the effectiveness of the MTW program is limited in part because HUD does not have a plan for identifying and analyzing standard performance data, has not established performance indicators for the program as whole, and does not have a systematic process for identifying lessons learned.

HUD’s 2010 Report to Congress noted that the conclusive impacts of many MTW activities, particularly as they relate to residents, could not yet be known. For example, the report noted that the rent reforms implemented under MTW varied greatly and were not implemented using a controlled experimental methodology. As a result, which aspects of rent reforms should be recommended for all PHAs were not clear. The report also noted the limitations that exist when evaluating the outcomes of MTW—limitations that stem from the weak initial reporting requirements and lack of a research design. The report concluded that, given these limitations, expansion should occur only if newly admitted PHAs structure their programs for high-quality evaluations that permit lessons learned to be generalized beyond a single PHA experience.

Similarly, affordable housing advocates and legal aid organizations that we interviewed stated that because lessons had not been learned from MTW, there was no basis for expanding the program. For example, officials from a national affordable housing advocacy organization stated that some MTW agencies have used their flexibility to establish limits on the length of time someone can live in assisted housing, but there is little research on the effect of such efforts. The officials stated that there was no evidence that this policy had helped anyone become self-sufficient and move out of public housing. The officials added that data were not available on the extent to which MTW agencies have provided incentives for residents to become self-sufficient or have increased housing choices. Similarly, an official from a national housing law advocacy organization stated that data were not available to determine the effect of the MTW program, particularly at the national level.

In addition, our own work, some research organizations, and affordable housing advocates question HUD’s ability to effectively manage an expanded MTW program. As previously noted, HUD’s current monitoring procedures have several key weaknesses, including the lack of a systematic process for assessing agencies’ compliance with statutory requirements and an assessment of program risks. Some research organizations also have questioned HUD’s capacity to oversee additional MTW agencies. For example, the Urban Institute reported that the approval process that HUD was using at the time of the institute’s 2004 review would not be feasible for an expanded program because of the administrative burden involved. At the time of the 2004 study as well as our review, HUD reviewed each individual request to waive specific provisions of the 1937 Housing Act before approving annual plans. Staff from another research organization questioned whether HUD has the capacity to oversee additional agencies. Similarly, one affordable housing advocate that we interviewed stated that HUD’s capacity to oversee an expanded program is not clear, in part because current monitoring activities are not transparent. At the time of our review, HUD had four full- time MTW coordinators, who each managed from 6 to 10 MTW agencies. According to the Director of the MTW Office, it takes more resources for HUD to oversee MTW agencies than non-MTW agencies. Thus, if additional agencies were added under the current program design, HUD would likely need additional resources.

Researchers and several of the affordable housing advocates and legal aid agencies that we met with also raised concerns that the current program, and therefore also an expanded program, could negatively affect residents of MTW agencies. For example, two research organizations have stated residents could be negatively affected by MTW agencies that implement voucher policies that reduce portability—that is, residents’ ability to use their vouchers in an area outside of the area where they received it. One of these research organizations stated that the differences in the way voucher programs were implemented across MTW agencies could reduce residents’ ability to use vouchers outside of the area where they received the assistance. Officials from the other organization noted that some MTW agencies had instituted policies that prohibited vouchers from being transported out of the originating jurisdictions, thereby limiting housing choices. According to HUD officials, MTW agencies with policies that limit portability can make exceptions. For example, these agencies have made exceptions for residents seeking employment opportunities.

Legal aid organizations that have worked with residents of MTW agencies as well as affordable housing advocates told us some of the requirements that MTW agencies have implemented, such as work requirements, were potentially harmful to residents. For example, legal aid representatives from one community told us that the work requirement was not consistently enforced across various mixed-income properties that included public housing as well as market-rate units. According to these officials, they have had clients who have been evicted for not working, even though the client was in school or disabled—both exceptions to the work requirement. These officials also stated that property managers in the city’s various mixed-income developments did not implement MTW policies consistently. For example, the officials stated that residents have been told by property managers that they would be in compliance with the work requirement if they were in school or another training program, only to have the MTW agency determine that they were not in compliance. According to HUD officials, inconsistent enforcement of policies is not unique to MTW agencies, and residents would have recourse. Legal aid representatives that worked with residents of another MTW agency also told us that the work requirement was a punitive policy that negatively affected the poorest residents. The officials stated that there were better methods for encouraging work and self-sufficiency, such as job training. Officials from a national affordable housing advocacy organization agreed that work requirements are punitive and stated that they disagreed with a policy of making housing assistance contingent on other factors, such as having a job. In their view, housing assistance should be a stable form of assistance for low-income households.

Alternatives Exist to Expanding the MTW Program

Alternatives to expansion include implementing a program that is targeted more to specific activities and waiving some regulations for all PHAs as described in proposed legislation. According to the Urban Institute, an alternative to expanding MTW could be to systematically test a limited number of programmatic alternatives—such as flat rents, time limits, or debt financing of capital improvements.individual agencies as much discretion to design combinations of reforms around local conditions and priorities. However, this approach could yield more systematic evidence about the costs and benefits of particular This approach would not allow program reforms if it included a rigorous evaluation design and mandatory data collection on key outcomes, such as the number and characteristics of participating households. Collection of such information in a standardized format would need to be a minimum requirement for participation if the point was to learn from the experiences of those testing activities. In addition, an official from an affordable housing advocacy organization that we met with stated that testing the effectiveness of discrete activities on a smaller scale would be useful. HUD also noted that altering the scope of the demonstration for new participants could improve what was learned from specific activities. For example, its 2010 Report to Congress stated that data on MTW could be strengthened if the scope of the demonstration were altered for new participants by selecting agencies committed to testing a particular activity, such as rent reform, and requiring rigorous evaluation. In December 2011, HUD issued a request for proposals for a demonstration that would test alternatives to the current rent structure in the voucher program. According to the proposal, the demonstration would most likely be undertaken at select MTW agencies. In addition, HUD’s 2012 appropriations act authorized a Rental Assistance Demonstration that would enable HUD to authorize and evaluate new approaches to preserving affordable rental housing, including converting public housing to project-based rental assistance. AHSSIA includes authorization for a revised version of the Rental Assistance Demonstration.

Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).

Congress supported allowing more PHAs to participate in the program. Finally, we recently reported on cost savings that could be realized from allowing additional housing authorities to implement some of the reforms MTW agencies have tested.

Conclusions

The MTW demonstration is designed to provide participating agencies with the flexibility to develop and test activities that achieve cost- efficiency, encourage residents with children to obtain employment and become self-sufficient, and increase housing choices for low-income families. While this flexibility has allowed participating agencies to implement hundreds of activities, HUD has not done all that it can to evaluate the program’s effectiveness, identify successful approaches that could be applied to public housing agencies more broadly, or ensure that MTW agencies comply with program requirements. Because Congress is considering expanding the program to many more PHAs, the absence of information needed to conduct a comprehensive program evaluation and compliance reviews is significant. HUD has recognized the importance of rigorous evaluation by requiring newly admitted agencies to have strong evaluation components. However, these improvements will not help evaluate the program as a whole. Without more complete knowledge of the program’s effectiveness and the extent to which agencies are adhering to program requirements, it is difficult for Congress to know whether an expanded MTW will benefit additional agencies and the residents they serve.

Recognizing that it needed to do more to improve what was known about the program’s effectiveness, HUD started requiring MTW agencies to describe the impact of each implemented activity in their annual reports beginning in 2009. However, the information that MTW agencies reported did not always reflect outcomes, and HUD’s guidance does not require that information on activities be quantifiable and outcome-oriented to the extent possible. Without more specific guidance on reporting performance information, HUD cannot be assured of collecting data that reflects the outcomes of activities.

Further, challenges such as the lack of analysis of standard data and the absence of performance indicators have prevented comprehensive evaluations of similar activities and the overall MTW program. HUD has recently started collecting additional information through MTW-PIC and annual reports, but has not yet analyzed the data. Further, whether the data collected are sufficient to assess similar activities and the program as a whole is not clear, and HUD has not identified the performance data it needs to undertake such analysis. Until HUD has a plan (that includes the identification of standard data) to quantitatively assess similar activities and the MTW program as a whole, HUD cannot determine their effectiveness. Additionally, HUD has not established performance indicators specific to MTW. Indicators linked to the statutory purposes of reducing costs, encouraging self-sufficiency, and increasing housing choices would help HUD demonstrate that the program has produced desired results.

Similarly, HUD does not have a systematic process in place to identify lessons learned from the MTW demonstration. Identifying activities that could be replicated more broadly is a goal of the MTW program and could be aided by the analysis of some standard performance data. The absence of a criteria-based, regular process for identifying lessons learned complicates efforts to determine which MTW activities are most effective and should be replicated more broadly.

At the same time, HUD’s monitoring efforts are not as strong as they could be. First, because HUD has not clarified key terms related to the three statutory purposes and five requirements, it cannot effectively determine whether MTW agencies are actually addressing these purposes and meeting requirements. Second, HUD does not have a process in place to systematically review compliance with all requirements. Such a review is especially important to a program like MTW that allows participants to self-certify their compliance with some program requirements. HUD has begun assessing compliance with two of the MTW requirements that call for self-certification, but not the third, and thus lacks assurance that agencies are complying with all three.

Moreover, HUD’s procedures for monitoring MTW agencies are not risk- based. It does not conduct an annual assessment of risks and provides the same level of monitoring for all agencies, even though some may pose greater risks than others. A risk-based approach to monitoring would provide greater assurance that HUD has addressed all risks, particularly those that may prevent participating agencies from addressing program purposes and meeting statutory requirements. Further, unless it implements a risk-based approach (such as that currently being considered for annual site visits) to monitoring MTW agencies, HUD cannot be assured that it is using its limited monitoring resources most efficiently. Finally, just as HUD does not assess compliance with all three self-certified requirements, it does not verify the accuracy of key information that agencies self-report, including information on the impact of MTW activities. Annual site visits have been used primarily to provide technical assistance rather than to assess self-reported information. By not verifying the accuracy of any performance information, HUD lacks assurance that this information is accurate.

Recommendations for Executive Action

To improve what is known about the effectiveness of the MTW program, we recommend that the Secretary of the Department of Housing and Urban Development improve HUD’s guidance to MTW agencies on providing performance information in their annual reports by requiring that such information be quantifiable and outcome-oriented to the extent possible; develop and implement a plan for quantitatively assessing the effectiveness of similar activities and the program as a whole including the identification of standard performance data needed; and establish performance indicators for the MTW program as a whole.

To enhance the ability to identify MTW practices that could be applied more broadly, we recommend that the Secretary of the Department of Housing and Urban Development create a process to systematically identify lessons learned.

To improve HUD’s oversight of the MTW program, we recommend that the Secretary of the Department of Housing and Urban Development issue guidance that clarifies key program terms, such as the three statutory purposes of the program and the five statutory requirements that MTW agencies must meet; develop and implement a systematic process for assessing compliance with statutory requirements; conduct an annual risk assessment for the MTW program and implement risk-based monitoring policies and procedures such as those currently being considered for site visits; and implement control activities designed to verify the accuracy of a sample of the performance information that MTW agencies self-report.

Agency Comments and Our Evaluation

We provided a draft of this report to HUD. The Assistant Secretary for Public and Indian Housing provided written comments, which are reprinted in appendix II. HUD disagreed with our recommendation that the agency develop performance indicators for the MTW program as a whole, was in partial agreement with four recommendations, and generally agreed with three. The agency said that developing programwide performance measures could be difficult and might be contrary to the nature of the demonstration. In addition, HUD emphasized the improvements that it had made to its reporting requirements in order to collect more consistent, outcome-oriented data. We acknowledged these improvements in the draft report, but as our recommendations indicated, we saw opportunities for additional improvements. HUD also noted that some of our recommendations might be a good fit for the existing program but that others might be more appropriate for a future expanded demonstration.

In disagreeing with our recommendation that it establish performance indicators for the MTW program as a whole, HUD emphasized the difficulty of measuring all activities against the same standard. The agency noted that because each MTW agency had implemented a unique combination of activities, developing programwide performance measures would make determining the impacts of specific activities unclear and prevent the identification of individual policies that could be applied more broadly. However, the purpose of programwide indicators would not be to isolate the impact of individual activities but to demonstrate programwide results—including showing the extent to which the program was addressing its statutory purposes of achieving greater cost-effectiveness in federal housing expenditures, giving families with children incentives to obtain employment and become self-sufficient, and increasing housing choices for low-income families. HUD also stated that applying programwide performance measures would be complicated by the fact that activities that advance one statutory purpose might conflict with other purposes. We agree that it is important to evaluate similar activities and have a separate recommendation addressing this need. But, the purpose of programwide assessment is to demonstrate whether the provision of flexibility in itself results in the intended benefits of the MTW program, such as cost savings or increased family self-sufficiency. Demonstrating that the increased flexibility the program offers has produced the intended results is critical, particularly as Congress considers whether to expand the program. We continue to believe in the importance of demonstrating program results and therefore continue to recommend that HUD develop performance indicators for the MTW program as a whole.

HUD was in partial agreement with four recommendations.

First, HUD said that proposed revisions to the reporting requirements for MTW agencies had addressed our recommendation that the agency improve its guidance to MTW agencies on providing performance information in annual reports. HUD’s draft guidance is in line with our recommendation that HUD require agencies to report quantifiable and outcome-oriented information. However, because these proposed revisions have yet to be finalized, we did not revise our recommendation.

Second, HUD agreed that quantitatively assessing the effectiveness of similar activities was an important step but noted the difficulties associated with assessing the effectiveness of the program as a whole. However, as noted above, we continue to believe in the importance of demonstrating program results. Consequently, we did not revise our recommendation.

Third, HUD stated that providing a menu of standard metrics may be the best way to clarify the program’s statutory purposes and that it had made progress in recent years in addressing our recommendation that it issue guidance that clarifies the statutory requirements. HUD also noted that the proposed revisions to the reporting requirements would provide additional clarification on the statutory requirements. These efforts, which were acknowledged in the draft report, are a step in the right direction, and we encourage HUD to continue finalizing this guidance. As noted above, because these proposed revisions have yet to be finalized, we did not revise our recommendation.

Fourth, HUD described recent efforts to assess compliance with two statutory requirements and analysis that it could conduct once proposed revisions to reporting requirements for MTW agencies were finalized. Because the process used to assess compliance with one of the requirements has not been formalized in policy and the proposed revisions have not been finalized, we did not revise our recommendation that HUD develop and implement a systematic process for assessing compliance with statutory requirements.

HUD generally agreed with the three remaining recommendations. For example, HUD agreed that it should proactively identify lessons learned and described some of its recent efforts to do so. We acknowledged these efforts in our draft report but noted the absence of a criteria-based, regular process for identifying lessons learned. HUD also described plans to develop a formal risk-based strategy for monitoring and, when we asked for further clarification, stated that it agreed with our recommendation to conduct an annual risk assessment for the MTW program. Finally, HUD discussed potential strategies for verifying the information that MTW agencies report using existing or planned HUD systems.

HUD also requested that we consolidate four separate recommendations into two, but we continue to believe that maintaining distinctions between the separate recommendations is important. First, HUD requested that we combine two recommendations: that it create a plan to quantitatively assess the effectiveness of similar activities and the program as a whole (including identifying the standard performance data needed), and that it establish performance indicators for the program as a whole. Although related, the two recommendations are distinct because the first focuses on the need for program evaluation and the second on performance measurement. Program evaluations typically examine a broad range of information on program performance, while performance measurement generates outcomes that show whether a program has achieved specific objectives. As a result, we did not combine the recommendations.

Second, HUD requested that we combine the recommendation that it issue guidance clarifying key program terms (such as the three statutory purposes and five statutory requirements) with the recommendation that it implement a systematic process for assessing compliance with statutory requirements. However, defining program requirements and assessing them are separate and distinct activities. Therefore, we did not combine the recommendations.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Housing and Urban Development and other interested committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-8678 or sciremj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Scope and Methodology

Our objectives were to examine (1) what is known about the extent to which the Moving to Work (MTW) demonstration program is addressing the program’s statutory purposes, (2) the Department of Housing and Urban Development’s (HUD) monitoring of MTW agencies’ efforts to address these purposes and meet statutory requirements, and (3) potential benefits of and concerns about expanding the number of public housing agencies (PHA) that can participate in the demonstration program.

To evaluate what is known about the extent to which the MTW program is addressing the program’s statutory purposes, we reviewed the most recent annual reports as of January 2012 for 30 MTW agencies. We reviewed these annual reports and the corresponding annual plans to identify the ongoing activities that the agencies were implementing, determine the extent to which these activities were linked with one or more of the program’s statutory purposes, and assess the performance information provided for each activity. To assess the performance information provided by MTW agencies, we examined HUD’s reporting guidance and compared it with internal control standards for federal agencies. We assessed the reliability of selected information in the reports by reviewing supporting documentation from a sample of seven MTW agencies and interviewing the officials responsible for preparing and reviewing this information. These seven agencies were

Cambridge Housing Authority (Cambridge, Massachusetts),

Chicago Housing Authority (Chicago, Illinois),

Housing Authority of the City of Atlanta (Atlanta, Georgia),

Housing Authority of the City of Pittsburgh (Pittsburgh, Pennsylvania),

Housing Authority of the County of Santa Clara/Housing Authority of the City of San Jose (Santa Clara County and San Jose, California),

Lawrence-Douglas County Housing Authority (Lawrence, Kansas),

Vancouver Housing Authority (Vancouver, Washington).

We selected these agencies to provide diversity in geography, agency size, and length of time participating in the program. We determined that the reports were sufficiently reliable for the purposes of our review.

Through interviews and a literature search, we identified three studies of the MTW program as a whole. We reviewed these studies to identify information on the program’s effectiveness and any challenges associated with assessing it. We determined that these studies were methodologically sound and reliable for our purposes. We examined HUD’s recent efforts to collect data from MTW agencies, including documentation on the reporting requirements for MTW agencies. In addition, we reviewed HUD’s fiscal year 2010-15 strategic plan and Fiscal Year 2011 Annual Performance Plan for any performance indicators for the MTW program. We also reviewed the GPRA (Government Performance and Results Act) Modernization Act of 2010, Office of Management and Budget guidance, internal control standards, and a GAO report on attributes of successful performance measures. Further, we identified five studies of specific MTW agencies or activities identified by HUD and representatives of the sample of MTW agencies we interviewed. Finally, we reviewed published reports and HUD’s website for information on HUD’s efforts to identify lessons learned.

To assess HUD’s monitoring of MTW agencies’ efforts to address the program’s statutory purposes and meet requirements, we obtained and reviewed documentation of monitoring policies and procedures, including the Standard Agreement that HUD executed with MTW agencies in 2008, the MTW Desk Guide, a 2011 Memorandum of Understanding between HUD’s Office of Public Housing Investments and Office of Field Operations, and other HUD guidance. Based on these documents and interviews with HUD staff, we identified three key monitoring processes: the review of annual plans and reports, reviews of data entered into the Moving to Work section of the Public and Indian Housing Information Center (MTW-PIC), and annual site visits to each MTW agency.

To assess the extent to which HUD staff were following these monitoring policies and procedures, we reviewed documentation of monitoring activities for our sample of seven MTW agencies. For example, to verify the steps HUD had taken to review annual plans and reports, we reviewed the checklists that the MTW coordinators used to document their review of these plans and reports. We also reviewed HUD’s comment letters for fiscal year 2011. To verify the steps HUD had taken to review data submitted into the MTW-PIC system, we reviewed monthly reports that showed the degree to which MTW agencies overall complied with reporting requirements from August 2011 through January 2012. Finally, to verify that both headquarters and field office staff made site visits and the extent to which they made annual visits, we reviewed the most recently available site visit reports completed by the MTW Office for all agencies as of October 2011. In addition, we interviewed the MTW agencies in our sample and the corresponding HUD field office officials to discuss the annual site visits.

We also compared HUD’s monitoring policies and procedures to internal control standards for the federal government and HUD’s own program management guidance. As a part of this analysis, we compared HUD’s guidance to MTW agencies with the internal control requirement for clear goals and objectives. We also reviewed information on HUD’s efforts to clarify how agencies could certify compliance with the requirement to assist “substantially the same” number of eligible families that would have been served in the absence of MTW. In addition, we compared HUD’s efforts to assess agencies’ compliance with statutory requirements with the internal control standard related to assessing compliance with program requirements. Further, we reviewed internal control standards for the federal government and HUD’s own internal control standards and identified the requirement that programs have an annual risk assessment. We interviewed HUD regarding any risk assessment performed for the MTW program. Finally, we interviewed HUD officials to determine whether any of the performance information that MTW agencies reported had been verified. We compared HUD’s lack of verification with the internal control standards and guidance that emphasized the need for control activities to ensure that program participants report information accurately.

To discuss the potential benefits and concerns associated with expanding the number of PHAs that can participate in the program, we reviewed studies, reports, and testimonies by researchers, affordable housing advocates, resident advocates, and the HUD Office of Inspector General. For all three objectives, we interviewed officials from the seven MTW agencies in our sample and representatives from affordable housing advocacy organizations such as the Council of Large Public Housing Agencies, the National Association of Housing and Redevelopment Officials, the National Leased Housing Association, and the Public Housing Authorities Directors Association. We spoke with resident advocacy organizations such as the National Low-Income Housing Coalition, the National Housing Law Project, and legal aid agencies that represented residents serviced by five of our sample MTW agencies. We also interviewed staff from the Center for Budget Policy and Priorities, a research organization that has studied and written about the MTW program; researchers who had evaluated the MTW program; and HUD officials from the MTW office and the field offices that corresponded to our sample of agencies. During our interviews, we discussed the potential benefits of expansion and the concerns of these organizations. Based on our review of available studies and reports and interviews with the above mentioned stakeholders, we identified key benefits and concerns. We also made observations based on our findings related to the availability of performance information for the program and HUD’s monitoring efforts.

We conducted this performance audit from July 2011 to April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Housing and Urban Development

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Paige Smith (Assistant Director), Anna Carbino, Emily Chalmers, John McGrail, Marc Molino, Lisa Moore, Daniel Newman, Lauren Nunnally, and Andrew Stavisky made key contributions to this report.