Public Housing: HUD Should Improve the Usefulness and Accuracy of Its Management Assessment Program
Released: 1997-01-29T00:00:00

Letter

Introduction

Because public housing represents a $90 billion investment on the part of the federal government since the program’s inception in 1937 and because the Department of Housing and Urban Development (HUD) currently spends $5.4 billion a year on operating subsidies and modernization grants for this housing, interest remains keen in knowing how well local public housing authorities (PHA) are managing their properties. The PHAs, through which HUD provides these subsidies and grants, house 3 million low-income people, many of whom are elderly or disabled. The Congress holds HUD responsible for ensuring that the authorities provide safe and decent housing, operate their developments efficiently, and protect the federal investment in their properties.

The National Affordable Housing Act of 1990 required HUD to develop indicators to assess the management performance of PHAs. This law became the framework through which HUD developed one of its primary oversight tools for housing authorities, the Public Housing Management Assessment Program (PHMAP). Primarily, PHMAP establishes objective standards for HUD to evaluate and monitor the management operations of all PHAs to identify those that are troubled. According to HUD, PHMAP also allows the Department to identify ways to reward high-performing PHAs as well as improve the management practices of troubled PHAs. The program also allows PHAs’ governing bodies, management officials, residents, and the local community to better understand and identify specific program areas needing improvement.

To help improve public housing management, the National Affordable Housing Act of 1990, as amended (the act), required HUD to develop indicators to assess the performance of PHAs in all the major aspects of their management operations. The act required HUD to use certain indicators as well as provided discretion for the Secretary of HUD to develop up to five additional indicators that the Department deemed appropriate. HUD implemented PHMAP by using the 12 indicators listed in table 1.1, the first seven of which are those required by statute.

Because some indicators are more important than others in measuring management performance, HUD assigns them added weight in determining the overall score. HUD considers the indicators for vacancies, rents uncollected, annual inspection and condition of units and systems, and resident initiatives most indicative of good property management and delivery of services to residents, so each one has a greater weight than other indicators.

After reviewing existing procedures and extensively consulting with a group of PHAs, public housing industry groups, private management firms, resident groups, and HUD staff in field offices, HUD has significantly revised the PHMAP indicators. HUD’s revisions to PHMAP, published December 30, 1996, eliminated three indicators; consolidated four other indicators into two; and added one new indicator, security. These revisions primarily address the performance indicators on which housing authorities report data, not HUD’s use of PHMAP data.

Indicator Grades Determine the PHMAP Score, Performance Designation, Required Follow-Up, and Incentives

Annually, PHAs receive a grade of “A” through “F” for each of the twelve indicators that apply to their operations. HUD uses a formula that reflects the weights assigned to each indicator, converts indicator grades into points, totals each PHA’s points, and divides that total by the maximum total the PHA could have achieved to arrive at a percentage. That percentage, a number between 0 and 100, is the PHMAP score.

HUD draws data on the performance of a housing authority from two sources to determine the authority’s PHMAP score. First, the housing authority submits data to HUD for about half of the PHMAP indicators and certifies that this information is accurate and complete. HUD assigns grades to each of these indicators according to a comparison of the authority’s data and HUD’s criteria for grades “A” through “F.” The balance of the information HUD uses comes from its own information system for tracking expenditures from major grants. This system contains the financial and other types of data the field offices need to grade the remaining indicators for which the PHAs do not provide data. The field offices use this data and the PHA-certified data to determine indicator scores, the PHMAP score, and the PHA’s performance designation.

The PHMAP score is HUD’s starting point for both the performance designation it assigns to a PHA and, depending on that designation, the extent of follow-up required of the PHA to correct deficiencies identified during the PHMAP assessment. Generally, HUD uses three designations to describe the performance of PHAs: troubled performers are those scoring less than 60 percent; standard performers are those scoring between 60 and less than high performers are those scoring 90 percent or more.

HUD has the discretion to withhold the troubled designation or award the high performer designation if a PHA’s score is within 10 points of the threshold for either designation and HUD determines that its score results from the physical condition and/or neighborhood environment of that authority’s units rather than from the PHA’s poor management practices.

If a housing authority is designated as troubled, it faces several mandatory follow-up activities and/or corrective actions to improve performance and remove the troubled designation. Specifically, the act requires HUD to perform an independent management assessment of the troubled PHA’s overall operations to identify the causes of the deficiencies that led to its poor PHMAP score. HUD uses private contractors to perform these independent assessments.

HUD expects the independent assessments to form the basis for the second requirement for troubled PHAs—the memorandum of agreement (MOA). A memorandum of agreement is a binding contract between HUD and a troubled PHA to identify solutions to its management problems and pursue those solutions in a way that is significant, expeditious, and lasting. Among other things, HUD requires that the MOA address the specific responsibilities of HUD and the PHA, the resources each will commit to resolving the authority’s problems, the annual and quarterly performance targets for improving its performance on PHMAP indicators, and the incentives for it to meet its performance targets as well as sanctions for failing to do so. A PHA’s initial MOA generally lasts 18 months so that it can complete a second-year agreement with HUD, if necessary, before the first expires.

HUD’s regulations for implementing PHMAP require standard- and high-performing PHAs to develop improvement plans for every PHMAP indicator on which the PHA received an “F,” unless the PHA can correct the deficiency within 90 days; HUD may also choose to require these plans for indicators receiving scores of “D” or “E” when failure to raise the grade might pose significant added risk. An improvement plan documents how and when the PHA plans to correct deficiencies. Although similar in content and scope to a memorandum of agreement, improvement plans differ in that (1) PHAs develop and submit them to HUD for approval rather than negotiate them with HUD officials and (2) they are not a binding contractual commitment between the PHA and HUD.

When HUD first implemented PHMAP, it offered high-performers a variety of incentives, primarily regulatory relief from various reporting requirements. These incentives included less frequent reviews of changes to a PHA’s operating budget and, for those performing well on the modernization indicator, no prior HUD review for architects’ or engineers’ contracts. In addition to regulatory relief, high-performing PHAs receive a HUD certificate of commendation and public recognition for their performance.

In its fiscal year 1997 budget request, HUD proposed an additional PHMAP-based incentive for high-performing PHAs when it sought to create a $500 million capital bonus fund (as part of the $3.2 billion it sought for its public housing capital fund). To be eligible for a bonus, a PHA would have to be a PHMAP high performer and have undertaken substantive efforts to obtain education and job training for its residents. However, the Congress chose not to fund the bonus proposal for public housing or any of HUD’s other major programs, in part because of concerns about HUD’s ability to accurately and reliably track the performance of bonus recipients.

HUD’s Field Offices Implement PHMAP

With nearly 800 staff devoted to oversight of housing authorities and implementation of the full range of HUD’s public housing programs, its field offices have the bulk of the Department’s responsibility for the day-to-day implementation of PHMAP. Field offices’ PHMAP responsibilities include determining the indicator grades and PHMAP scores, negotiating memorandums of agreement, approving PHAs’ improvement plans, and monitoring their progress in meeting the goals the MOA or improvement plan set forth. To determine a housing authority’s PHMAP score, a field office relies on that PHA to provide about half the data that leads to the overall PHMAP score and certify the data’s accuracy. As a result, the overall PHMAP score and everything it influences—from incentives for high performers to sanctions for troubled PHAs—are very much a joint effort and a shared responsibility.

A PHA may also request to exclude or modify the data HUD should consider in computing its PHMAP score. An exclusion means that the indicator (or one or more of its components) is entirely excluded from calculations to determine the PHMAP score. For example, PHAs with no ongoing modernization or development programs are automatically excluded from being assessed on those indicators. Modifying the data for an indicator allows HUD to consider unique or unusual circumstances by exempting some of the data HUD usually requires the PHA to consider. The PHA still receives a score for the indicator, but the score would not reflect the data associated with the PHA’s unique or unusual circumstances. For example, a PHA operating under a court order not to collect tenants’ rent at specific developments until it corrects deficiencies the court had identified can seek to exempt those units in its developments from being considered in its indicator score for rents uncollected. A PHA always has the right to appeal a field office’s decision about modifications, exclusions, indicator scores, or the performance designation. However, after those appeals have been exhausted, the field office certifies the PHA’s PHMAP score, assigns a final performance designation, and proceeds with any required improvement plans, MOAs, or other necessary follow-up.

When a troubled authority’s new PHMAP score is high enough to cause HUD to remove its troubled designation, HUD’s policy is to require the field office to verify the accuracy and completeness of the new data submitted by the housing authority. HUD also requires the field office to conduct a confirmatory review to verify the data the PHA had certified as well as the accuracy of the data HUD had obtained from its own information system. HUD’s guidance for implementing PHMAP stipulates that a confirmatory review must take place on-site at the PHA and cannot be accomplished through remote monitoring.

HUD’s field offices may choose to conduct some confirmatory reviews of standard- and high-performing PHAs’ PHMAP certifications. HUD expects its field offices to choose these PHAs according to the risk they pose and focus on those with the highest potential for fraud, waste, mismanagement, or poor performance. Some of the factors HUD field offices may consider in analyzing the risk associated with a PHA’s PHMAP certification include size (number of units), borderline troubled designation (5 percent above or below the percentage for the designation), and negative trends in overall or individual indicator scores over several years.

In May 1995, HUD expanded the scope of the annual independent audit each PHA receives in order to improve the Department’s ability to determine whether PHA-certified data are accurate. The annual audit, conducted pursuant to the requirements of the Single Audit Act, examines the housing authority’s financial statements, internal controls, and compliance with HUD’s rules and regulations. Housing authorities are responsible for selecting their own auditors and submitting the results of the audits to their HUD field office. Field offices are responsible for reviewing the audits to ensure they meet all of HUD’s requirements and, when they have approved the audit, reimbursing housing authorities for them. In fiscal year 1995, these independent audits cost HUD about $8 million for all housing authorities.

HUD now requires the independent auditors to determine whether a housing authority has adequate documentation for the data it submits to HUD for its PHMAP certification. According to HUD officials, because the Department’s resources are too limited to conduct annual confirmatory reviews of most housing authorities, they expected to use the results of these audits to better focus HUD’s attention, oversight, and technical assistance. In addition to paying for the audits, HUD expects its field offices to use the results as part of a risk assessment to determine which housing authorities should get the most sustained attention and technical assistance.

Objectives, Scope, and Methodology

Stressing the need for HUD to hold housing authorities accountable while making better use of the data that PHMAP produces, the Chairman of the Subcommittee on Housing and Community Opportunity, House Committee on Banking and Financial Services, asked GAO to review HUD’s use and implementation of PHMAP. As agreed with the Chairman’s office, we reviewed whether HUD’s field offices are using PHMAP and complying with the program’s statutory and regulatory requirements to monitor and provide technical assistance to housing authorities, whether PHMAP scores have increased and how HUD uses the program to inform HUD’s Secretary and the Congress about the performance of housing authorities, and whether PHMAP scores are consistently accurate and can be considered a generally accepted measure of good property management.

We developed information from several different sources to address questions concerning the usefulness of PHMAP to HUD and other uses for which PHMAP may not be appropriate. To determine PHMAP’s usefulness to HUD, we interviewed officials and collected information on technical assistance activities at both the Department’s headquarters and field offices. At HUD’s headquarters, we analyzed a variety of documents pertaining to PHMAP and discussed the program’s use as a basis for technical assistance with the Offices of the Deputy Assistant Secretaries under HUD’s Assistant Secretary for Public and Indian Housing.

At HUD’s field offices, our approach was twofold. First, we surveyed them via fax questionnaire to obtain data on the use of PHMAP, such as the number of confirmatory reviews each field office performs and how useful such program tools as improvement plans have been. This data reflect responses from all of HUD’s public housing field offices. Second, we visited five HUD field offices to review their use of PHMAP in more depth and to supplement the information we had gathered in our survey. We judgmentally selected the five field offices because of their geographic distribution, variations in the number of HUD staff in each office as well as the number of PHAs each oversees, and variations in average PHMAP scores for the PHAs reporting to those offices.

To provide information on PHAs’ PHMAP scores, we relied on existing data from HUD sources, including HUD’s primary public housing database, the System for Management Information Retrieval-Public Housing (SMIRPH). From this database, we extracted the module containing housing authorities’ PHMAP data, including the PHMAP scores and individual indicator grades. Our analysis covers federal fiscal years 1992 through 1995 because the first fiscal year in which the rules governing PHMAP took effect was 1992 and the most recent year for which all PHMAP scores were complete at the time of our review was 1995.

We did not systematically verify the accuracy of HUD’s data or conduct a reliability assessment of HUD’s database. In performing our analysis we found erroneous and incomplete information for a few PHAs, ranging from 1 to 3 percent of the total. We confirmed this with HUD officials, who attributed the errors to mistakes in data input or the field office’s having entered incomplete scores. However, because we used these data in context with additional evidence we obtained directly from HUD’s field offices and we did not focus on the scores of specific PHAs or small groups of PHAs, we believe our conclusions about overall trends in scores are valid.

Throughout the course of our work, because the number of PHAs reporting PHMAP scores is too great for us to visit a representative sample, we consulted with several prominent groups representing the public housing industry to discuss HUD’s uses for PHMAP as well as their perspectives on the program’s ability to measure the performance of public housing authorities. These groups include the Council of Large Public Housing Authorities, the National Association of Housing and Redevelopment Officials, and the Public Housing Authorities Directors Association.

We provided a draft of this report to HUD for review and comment. HUD’s comments appear in appendix V and are addressed at the end of each applicable chapter.

We performed our work from January through December 1996 in accordance with generally accepted government auditing standards.

Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited

HUD’s field offices use PHMAP scores for their primary intended purposes: as a standard, objective means to identify troubled housing authorities; to compare performance among PHAs; and to identify when, where, and how to target HUD’s limited resources for technical assistance. However, beyond identifying troubled authorities and what they need, the amounts and kinds of technical assistance HUD provides varies because its field offices interpret their responsibilities differently—some choose to be actively involved while others adopt a hands-off approach. Furthermore, HUD’s 1995 reorganization of its field offices adversely affected some offices’ ability to provide technical assistance while others adapted to changed expectations and resumed providing as much assistance as they did before the reorganization.

HUD Uses PHMAP to Identify Troubled Housing Authorities, but Technical Assistance Varies

As part of HUD’s oversight of public housing, the PHMAP score is an important tool for identifying troubled authorities so HUD can focus technical assistance and monitoring on them. The most common types of technical assistance that HUD’s 49 public housing field offices provided all PHAs were telephone consultations, training, and participation in conferences. However, we found differences in how field offices defined their roles in providing PHAs technical assistance as well as some innovations in how others provided that assistance. For example, some field offices have encouraged high-performing PHAs to provide “peer assistance” to lower performers. Many of the differences in assistance were due to variations in field offices’ interpretations of their roles and the impact of HUD’s 1995 reorganization of its field offices. HUD headquarters officials believe that more training for all field staff and leadership from field office managers would help achieve more quality and consistency among field offices in providing technical assistance.

HUD Uses PHMAP to Target PHAs for Technical Assistance

Officials in 40 of HUD’s 49 field offices rated PHMAP as being of “utmost” or “major” importance in identifying which housing authorities need the most technical assistance. According to field office staff, PHMAP provides standard indicators to objectively measure an authority’s performance. In addition, some staff said that because PHAs have a strong aversion to failing performance scores and try to avoid failure, they are confident that when PHAs report information that results in low scores or failing grades, the data and the resulting scores are accurate. Because an accumulation of low or failing scores results in a PHA’s being designated troubled, HUD staff are confident that those PHAs PHMAP identifies as the worst-performing housing authorities are accurately designated as troubled performers.

Some field office staff also use declining PHMAP scores to provide an early warning of management problems and to identify which PHAs could need additional technical assistance. In addition, the staff use PHMAP’s 12 individual indicator grades to better focus their limited technical assistance resources and thereby maximize the benefits PHAs receive from HUD’s assistance. For example, one field office developed a package of technical assistance for the “resident initiatives” indicator because many PHAs failed this indicator. The package of assistance included sample policies and procedures for operating resident programs. Another field office developed assistance specifically for small housing authorities because many of them were having trouble renting their units when they became vacant (thus failing PHMAP’s unit turnaround indicator). Among other things, that field office provided its small PHAs an extensive list of suggestions on how and where to better market their units.

Most technical assistance from HUD’s field offices consisted of telephone consultations, training sessions, and industry conferences. HUD also provided assistance—although limited because of time constraints—at the time of a PHMAP confirmatory review. During telephone consultations, several offices we visited answered questions from housing authority staff and helped the executive directors of new housing authorities better understand public housing regulations and operations. Training sessions covered these and other topics and provided more details than telephone discussions. In addition, to increase the amount of personal contact they have with housing authority staff and to provide technical assistance, field office staff said they regularly participate in conferences hosted by public housing industry associations.

Field Offices’ Interpretations of Their Role and Their Recent Reorganization Influence the Level and Types of Technical Assistance

Field offices’ interpretations of their obligation to improve the performance of housing authorities influences the type of technical assistance they provide. For example, officials in one field office did not believe that it was HUD’s role to manage PHAs’ operations. Instead, they believed that the role of their field office should be limited to providing information on compliance with federal rules and regulations and to suggesting solutions to management problems. This field office avoids showing PHAs how to manage their developments because the staff believe that they do not have sufficient expertise and that the housing authorities would view this advice as intrusive.

In contrast, staff at other field offices that we visited believed they are obligated to tell PHAs what must be done to correct management deficiencies because HUD is responsible for ensuring that PHAs use federal funds efficiently and effectively to provide safe, decent housing. For example, staff from one field office spent several days at a troubled authority to help it set up proper tenant rent records and waiting lists.

In addition to differences in how they view their role to directly assist PHAs, we found differences in the extent to which field offices use outside resources to help their housing authorities. Some field offices told us that to compensate for a shortage of resources from HUD, they help PHAs in their jurisdiction by encouraging technical assistance from other PHAs rather than providing it themselves. For example, some of the field offices arranged for high-performing PHAs to provide peer assistance to authorities with management problems. One field office persuaded staff from a high-performing PHA to temporarily manage a small authority that unexpectedly lost its executive director. Another field office recruited a high-performing PHA to help another one develop a system for inspecting its housing units.

In 1995, HUD reorganized the field offices and changed the responsibilities of the staff who oversee and assist PHAs. Before the reorganization, most field office staff were generalists and broadly understood federal housing regulations and PHA operations. After the reorganization, however, the responsibilities of individual field office staff became more specialized to focus on the rules and regulations of specific public housing operations.This specialization confused some staff in field offices and housing authorities as well as impaired the ability of some field offices to provide technical assistance. For example, field office staff we visited said that some specialists do not have the skills needed to do their jobs because many of them did not have the work experience or requisite training for the specialists’ positions; the staff also noted that HUD had not provided sufficient training for the staff to understand the reorganization and their new responsibilities. The staff also said that the reorganization was a source of confusion for PHAs. Before the reorganization, a housing authority could call one employee at HUD’s field office to answer all its questions; afterward, a housing authority generally needed to call several different staff at HUD’s field office to answer questions.

Adjusting to the reorganization differed across field offices. At one field office, staff resisted the reorganization because they did not want to become specialists and they recognized that technical assistance to the PHAs suffered as a result. For example, the staff now disagree over who is responsible for overseeing certain PHA operations. They also have resisted working together to provide technical assistance and have not been sharing PHMAP information to develop the best plan for correcting management deficiencies. Other field offices we visited adapted to the reorganization. Staff in these field offices worked cooperatively to build on the skills of the experienced staff. For example, one field office continues to assign each housing authority to only one staff member who provides or coordinates all technical assistance to that authority. The responsible staff member, however, belongs to a team of staff from all operational areas who work together to solve each PHA’s problems.

Officials at HUD headquarters, including the Deputy Assistant Secretary for Public and Assisted Housing Operations, acknowledged that some field offices had difficulty adjusting to the reorganization. They stated that although adequate training was crucial to the reorganization’s success, some field offices either did not seek it or did not take the need for it seriously, despite the availability of training funds for field staff. HUD officials continue to emphasize the importance and availability of training and expect field office management to assess the staff’s skills and expertise and request the appropriate training. These officials believe that because of limited staff resources, now and in the future, the reorganization is the best way for field offices to provide effective oversight and technical assistance to PHAs. Furthermore, they believe that managers of the field offices must take a more active leadership role in directing their staff to work together.

HUD’s Infrequent Use of Some Oversight Tools May Not Adequately Improve the Performance of PHAs or Target Technical Assistance

The act and HUD’s requirements for how field offices use PHMAP provide for several tools to guide improvements in a housing authority’s performance and thereby raise its indicator grades and PHMAP score. These tools include the memorandums of agreement (MOA), improvement plans, confirmatory reviews, and the annual independent audits. While such tools as MOAs and improvement plans generally apply to PHAs designated as troubled or failing specific indicators, a confirmatory review is mandatory for any PHA coming off HUD’s troubled list and an independent audit is mandatory for all PHAs. Nonetheless, we found that the compliance of field offices with statutory requirements and HUD’s guidance for using these tools has been inadequate and infrequent.

Furthermore, HUD has not determined whether these statutory or agency requirements are effective, adequately improve housing authority performance, or help the field offices better target limited technical assistance resources. As a result, HUD has little information to determine which of these tools best improve a PHA’s performance and which tools its field offices can use most effectively to offset their declining resources.

Field Offices Make Limited Use of Oversight Tools

Over 90 percent of the field offices we surveyed reported that on-site visits to the housing authorities were one of the most effective means to ensure compliance with PHMAP requirements and provide technical assistance. Officials at one field office responded that PHAs under its jurisdiction believed that on-site visits from HUD staff to provide technical assistance were essential to maintaining effective operations. Yet, most field office staff we visited made fewer personal visits to housing authorities than they felt were necessary because of limited staff resources and travel funds. Field office staff told us, for example, that their workload has increased because their offices have been unable to replace staff who have left the agency. With less time available for on-site visits, direct monitoring of the PHAs’ performance has occurred less frequently. In addition, some field office staff said that they could rarely justify to their management using limited staff and travel resources to visit a PHA that is more than a 1-day trip from the office unless that authority’s PHMAP score was below 60.

Memorandums of Agreement

Although HUD is required by law to enter into MOAs with troubled housing authorities to improve management performance, few field offices have done so. Figure 2.1 shows that the percentage of troubled PHAs operating under an MOA has been decreasing since 1992. Furthermore, in fiscal year 1995, only 3 of HUD’s 32 field offices that had troubled PHAs were fully in compliance with the requirement to enter into an MOA with each troubled authority.

The primary reason HUD’s field offices told us that they did not enter into these required agreements with troubled housing authorities is that the PHAs had already corrected or were in the process of correcting their management deficiencies. However, HUD headquarters officials told us they did not accept this as a valid reason for not meeting the requirement and questioned how the field offices could be sure the housing authorities were no longer troubled.

Improvement Plans

When a PHA fails any of PHMAP’s 12 performance indicators, HUD requires the responsible HUD field office to obtain a plan from that PHA for improving its performance and to track its progress against the plan. However, we found that nearly a third—31 percent—of HUD’s field offices had not ensured that local housing authorities had developed these plans. We also found examples of PHAs’ plans lacking specific strategies and time frames for correcting management deficiencies. For example, one PHA’s plan for a failing “rents uncollected” indicator simply stated that the housing authority would start collecting rent. Although field office staff acknowledged that the PHA also needed to update its standard tenant lease and develop a rent collection policy to improve this indicator grade, they said that they had not yet had the time to contact the PHA to revise its plan.

HUD requires its field offices to monitor the progress of housing authorities in implementing improvement plans to ensure PHAs meet the quarterly and annual performance targets in their plans. However, four of the five field offices we visited told us they do not follow up with the PHAs to determine the status of improvement plans or whether the plans had corrected the management deficiencies. Field office staff said that they did not have time to track the effectiveness of the plans because their workloads have been increasing due to decreasing numbers of staff.

HUD headquarters officials confirmed that systematic tracking of the field offices’ success in obtaining improvement plans or executing MOAs has not been done. They emphasized that responsibility for implementing PHMAP rests with the field offices and said that limited efforts were underway to ensure field offices do more to use these tools and measure their effectiveness. However, they could not tell us whether troubled PHAs without MOAs had improved their scores and left the troubled list without such oversight, nor could they tell us whether improvement plans are instrumental in improving indicator scores.

Field Offices Confirm Few PHMAP Scores

When a troubled housing authority receives a new PHMAP score that is high enough to remove that designation, HUD requires that the field office confirm the score’s accuracy by verifying that the PHA’s improvements have been effective before removing the troubled designation. However, we found most field offices are not meeting this requirement. In 1995, for example, HUD’s field offices confirmed less than 30 percent of the scores that should have been confirmed. HUD officials acknowledged that the infrequency of confirmatory reviews by its field offices hampers the program’s credibility and integrity. Because it has done so few confirmatory reviews, HUD cannot say that most scores are accurate, nor can it say that most troubled PHAs that raised their scores above 60 really are no longer troubled. The HUD Inspector General (IG) recently noted that without more confirmatory reviews, the self-reporting nature of PHMAP creates a temptation for PHAs to manipulate data to raise their scores. In fiscal year 1995, 24 of the 49 field offices had housing authorities with PHMAP scores high enough to remove them from HUD’s troubled list, but only 11 of the 24 field offices performed all or some of the required confirmatory reviews. The remaining 13 offices performed none of the required confirmatory reviews. Nonetheless, some of these same 13 field offices performed discretionary confirmatory reviews of other housing authorities that had not been classified as troubled. In one case, a field office had just one housing authority whose new PHMAP score was high enough to remove its troubled designation. Although the field office did not perform a confirmatory review for that authority until the next fiscal year, it did complete nine confirmatory reviews of standard- or high-performing housing authorities. HUD headquarters officials told us that although they encourage the field offices to do as many additional, discretionary confirmatory reviews as possible, they expect field offices to complete the mandatory reviews first. They also told us that limited resources kept them from monitoring the performance of field offices on these reviews.

In addition to the field offices’ lack of compliance with HUD’s requirement for performing confirmatory reviews, few offices are performing discretionary confirmatory reviews. Over the life of the program, HUD has confirmed 6.7 percent of all PHMAP scores. Table 2.1 shows that since the program began in 1992, HUD has confirmed no more than 8 percent of all PHMAP scores in any given year (see table 2.1).

To expand on fiscal year 1995 data, nine field offices performed no confirmatory reviews, over two thirds performed five or fewer, and 4 offices performed 10 or more confirmatory reviews (see fig. 2.2).

Recognizing that PHMAP scores may not be as accurate as they could or should be to give the program integrity and credibility, HUD has added new requirements and begun initiatives to improve the accuracy of the scores and strengthen the program. HUD currently requires its field offices to confirm the PHMAP scores of housing authorities whose scores have risen to 60 or above, thereby removing them from the troubled list. Recently, HUD formed a team of “expert” field office staff to develop review guidelines and to perform confirmatory reviews at selected housing authorities whose new PHMAP scores meet HUD’s criteria for a mandatory confirmatory review. HUD officials expect this team to perform as many as 12 confirmatory reviews in 1 year, during which they will focus primarily on large, high-risk housing authorities.

Field Offices Are Not Using Independent Audits to Verify Data Provided by PHAs

In May 1995, HUD expanded the scope of the mandatory annual financial audits of PHAs to require that auditors review the records underlying a PHA’s self-reported PHMAP data. HUD expects the financial audits to verify that the PHAs’ data are accurate and complete and that the PHAs have adequate documentation to support their submissions. HUD adopted this requirement because the field offices do not have sufficient resources to confirm each PHA’s score every year. Moreover, HUD officials told us that further departmental downsizing will limit its field offices’ ability to provide meaningful technical assistance, including confirmatory reviews. As a result, HUD expects that the PHMAP review in the annual audit can help ensure the integrity of housing authorities’ PHMAP data and should be a valuable tool for aiding the field offices to identify those housing authorities most needing technical assistance.

HUD does not consider the auditors’ analysis to be a confirmatory review because the auditors do not verify the information HUD maintains in its information system. Furthermore, even though the auditors certify that a housing authority has documentation to support the data it submitted to HUD, they do not verify that some of the activities reflected in that data were actually performed by that authority. For example, while the auditors verify that a PHA has data indicating it has met the requirements for the indicator on conducting annual inspections of all of its housing units and major systems (e.g., heating, plumbing, and electrical), the auditors do not verify that those inspections actually took place.

Although the independent audit requirement has been in place since May 1995, few of the staff in the five field offices we visited were aware of it. Before field offices authorize payment for an annual audit, HUD headquarters officials said that they expect field offices to review the audits for quality and completeness and verify that the audits addressed all appropriate areas of the PHAs’ operations, including the PHMAP. However, field office staff said that they had not seen an audit of a housing authority that tested the reliability of its PHMAP submission. HUD also expects the field offices to consider significant audit findings in deciding which PHAs need additional oversight or assistance. HUD officials acknowledged, however, that the independent auditors may need training to better understand HUD’s expectations of them, regulations, and PHMAP system as well as the operations of PHAs. Similarly, these officials noted that staff in HUD’s field offices need training and guidance in how to better use the annual independent audit.

Conclusions

One of the key challenges HUD faces in the coming years is effectively downsizing the Department while maintaining the needed level of oversight at public housing authorities. However, HUD is currently not maintaining a consistent, minimally acceptable level of oversight at all housing authorities because of the variance in how field offices interpret their roles to provide that oversight as well as their lack of systematic compliance with follow-up requirements. Furthermore, because field offices are not making enough use of the independent audits’ verification of PHMAP data to target their technical assistance, HUD is not using the resources it has to effectively determine which housing authorities’ scores are most likely to be inaccurate. As a result, HUD is not ensuring that the housing authorities most in need of oversight and assistance are receiving it and thereby improving their performance. Continued departmental downsizing likely will cause HUD to leverage its existing resources to achieve a minimally acceptable level of oversight. This oversight is needed for HUD to be reasonably confident that all housing authorities are using federal funds appropriately, managing and maintaining their developments properly, and reporting accurately their performance information.

Recommendation

To make better use of the limited resources it has to devote to the oversight of public housing, we recommend that HUD provide guidance to its field offices that clearly (1) articulates their minimally acceptable roles regarding oversight and assistance to housing authorities and (2) emphasizes the importance of using the results of the independent audits to better target HUD’s limited technical assistance resources.

Agency Comments

HUD agreed with our findings regarding oversight of public housing authorities and stated that it has begun taking steps to address this recommendation. These steps include a wide variety of training and other activities to (1) explain the revisions HUD is making to PHMAP; (2) reemphasize the need for and importance of statutory and agency follow-up requirements, such as memorandums of agreement, improvement plans, and confirmatory reviews; and (3) update HUD’s guidance to its field offices regarding their PHMAP and other oversight responsibilities.

Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use

According to a HUD database of PHMAP scores, average PHMAP scores have increased over the life of the program from an average of 83 in 1992 to 86 in 1995 (the last year of complete data). The number of high-performing housing authorities increased, with more than half of all authorities designated high performers in 1995, and the number of troubled authorities decreased. However, the smallest housing authorities—those with fewer than 100 units—now make up a greater proportion of those designated troubled than when the program began. During our analysis of this database, we found omissions of key data, such as the number of units under a PHA’s management and its performance designation. We also found inconsistencies between PHMAP scores and the assigned performance designations. Notwithstanding these weaknesses, the database represents the most complete data available on PHA performance over time.

Most PHMAP Scores Are Increasing and Fewer Housing Authorities Are Troubled

Nationwide, average PHMAP scores generally increased over the 4 years of the program for which we analyzed data. By 1995, over half of all public housing authorities were high performers. Subsequent analysis showed little regional variation in how well they scored on PHMAP. While the overall increases in PHMAP scores held true for all sizes of PHAs, the largest ones had scores consistently lower than the national average. With average scores increasing, the number of PHAs with scores low enough for HUD to designate them as troubled also decreased. The number of troubled authorities reached 83 in 1995, with half of that total consisting of the smallest housing authorities (those managing fewer than 100 units).

Average PHMAP Scores Increased

The average PHMAP score for all housing authorities rose from about 83 in 1992 to 86 in 1995. This increase held true for PHAs of all sizes, although large PHAs—those with more than 1,250 units—consistently scored lower than the national average (see table 3.1). In fiscal year 1995, 151 large PHAs accounted for approximately 5 percent of all PHAs reporting PHMAP scores, but they operated nearly 60 percent of all public housing units. Consequently, while more PHAs had higher scores, more units were under the control of PHAs with somewhat lower scores.

Appendix I provides average PHMAP scores for PHAs for all of HUD’s field offices for fiscal years 1992 through 1995.

The Majority of PHAs Were High Performers

By fiscal year 1995, more than half—about 57 percent—of all public housing authorities were designated as high performers. As shown in table 3.2, the number of high performing authorities grew each year, rising from 1,033 (33 percent) in 1992 to 1,791 (57 percent) in 1995. Also, by 1995, nearly 50 percent of all public housing units were under the management of high-performing authorities.

Little Variation Among Regions

Our analysis showed little regional variation in PHMAP scores. The regional differences we found were slightly greater than those associated with the size of housing authorities, but no region was significantly below the national average. Likewise, there was little variation among the regions in the percentage of troubled PHAs under their jurisdiction. For example, in fiscal year 1995, 5 percent of all PHAs nationwide were troubled, but within the 10 regions we analyzed, the percentage of troubled housing authorities ranged from 2 to 9 percent.

Appendixes I-IV provide detailed information on average PHMAP scores as well as the number of troubled, standard- and high-performing PHAs, respectively, for each HUD field office.

PHAs Consistently Failed Some Indicators

Despite some improvement in overall scores, some indicators were more problematic for PHAs than others. As shown in table 3.3, with the exception of 1 year, PHAs consistently had the most difficulty with the energy consumption indicator—which had the highest failure rate for 1992, 1994, and 1995. Similarly, the indicators for unit turnaround, tenants accounts receivable, and operating expenses proved troublesome, with 10 percent or more of all PHAs failing them in 1995.

A HUD official explained that the high failure rate in 1993 for the indicator measuring resident initiatives occurred because the PHAs were not paying attention to this indicator. In 1992, all PHAs received an automatic “C” for this indicator because HUD had not provided enough information on the requirements for grades “A” through “F” until after the assessment period started. This official said that many PHAs assumed they would receive an automatic “C” the next year as well, even though HUD had stated in 1992 that the automatic grade was a one-time occurrence. This official added that most field offices followed up by providing technical assistance to the PHAs with failing grades and were able to resolve the problems in the following year. This appears to be supported by the decline of the failure rate over the following 2 years to less than 6 percent in 1995.

Smaller PHAs Were More Likely to Be Troubled

While the total number of troubled housing authorities declined—130 were troubled in 1992 compared to 83 in 1995—more of those PHAs were concentrated among the smallest housing authorities than when the program began. The percentage of troubled PHAs that were small—managing fewer than 100 units each—grew from 32 percent of all troubled authorities in 1992 to 49 percent in 1995 (see fig. 3.1).

HUD Recognizes Database Flaws and Plans Corrections

We found missing, inaccurate, and inconsistent data in HUD’s SMIRPH database, the primary database for storing PHMAP scores. A HUD official attributed these problems to data input problems at the field offices. Although HUD headquarters makes regular, periodic use of this database, it must also manually verify much of the information before providing it to HUD’s Secretary, Members of Congress, and others. HUD’s General Deputy Assistant Secretary for Public and Indian Housing acknowledged that the SMIRPH database, as currently implemented, does not produce a complete, accurate list of troubled PHAs and that HUD is in the process of making it more reliable and useful.

We found that the number of troubled authorities (150) for fiscal year 1995 that we derived from the database was inaccurate when we compared it to the number reported (83) as of December 20, 1995, by HUD’s Management Assessment Coordinator. We also found performance designations that were inconsistent with PHMAP scores. In 1995, for the 150 PHAs we found to be troubled, HUD had designated 42 as high performers, 7 as standard, and 51 had no designation. Among high-performing PHAs in 1995, of the 1,791 PHAs that we found that had PHMAP scores of 90 or higher, HUD had designated one as troubled, 43 as standard, and 325 had no performance designation. We also found some omissions in the database. Data, such as the number of units and performance designations, had not been entered for all PHAs. For example, we found that the database did not have size information on 18 PHAs from fiscal years 1992 through 1995. We also found that no designations had been entered for 132 PHAs with scores less than 60 and 1,037 PHAs with scores 90 or higher.

HUD’s Management Assessment Coordinator stated that these problems with missing, inaccurate, and inconsistent data occurred because field offices either (1) did not enter the information at all or (2) entered it incorrectly. These instances of inconsistent or missing data suggest that basic system safeguards do not exist to prevent field offices from making these data entry errors or omitting essential PHMAP data.

While HUD officials who oversee PHMAP and the Department’s field offices acknowledged problems with the database, they added that the program’s redesign includes changes that will address the problems with data accuracy and reliability. HUD officials told us they plan to change procedures for entering information on PHAs into the database to allow field offices to update PHA data on a real-time basis and to make immediate corrections when they find errors or omissions. These procedural changes will also enable HUD headquarters staff to access field office data directly and allow ongoing reviews of the information for accuracy and completeness. HUD officials also believe that the changes will increase control over the information from the field offices and help ensure that the information in the SMIRPH database is accurate.

Agency Comments

HUD expressed concern that our draft report used data from the SMIRPH database that HUD had not verified for accuracy. HUD noted that it is making changes to the database that will improve headquarters’ ability to find and correct data errors that have been entered by staff at its field offices.

To address HUD’s concern that we used inaccurate, unverified data from its database to analyze PHMAP data on housing authorities’ scores by size and region, we recalculated the number of troubled housing authorities by size category for 1995 using data HUD verified with its field offices; we also modified this report to reflect a more accurate and lower number of troubled housing authorities in 1995. Recalculating the number of troubled authorities by size did not change our conclusion that a greater proportion of the authorities that HUD verified as being troubled are those with fewer than 100 units. In fact, while HUD’s database indicates that 44 percent of troubled authorities in 1995 were small, HUD’s verified list of troubled authorities indicates 49 percent were small. Furthermore, although HUD officials told us that a manually-verified list of troubled authorities for 1992 was not available, they agreed with our conclusion that the smallest housing authorities make up a greater proportion of troubled housing authorities in 1995 than in 1992.

Because our draft report presented no analysis of data on a regional basis (only data as drawn from HUD’s database) and because we draw no conclusions in that regard in this report, we have retained appendixes I-IV, which show average PHMAP scores and the number of troubled, standard, and high-performers in HUD’s regions. Where HUD provided us with manually verified data—particularly in appendix II showing troubled authorities—we have modified the appendixes to reflect the more accurate data.

The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness

Our review and those of others indicate that PHMAP scores are often inaccurate, imprecise, and must be changed when HUD verifies the data that public housing authorities have submitted to support their scores. Furthermore, professional property managers and others in the public housing industry question whether PHMAP can capture all aspects of management operations. Although HUD has taken some steps to help ensure that future scores are more accurate than they have been over the program’s first 4 years, these steps will be resource-intensive and do not address all of the program’s limitations. In the past, both HUD and the Congress have proposed additional uses for PHMAP, such as deregulating and awarding bonuses to PHAs with high PHMAP scores. However, until greater confidence exists that individual scores are accurate and HUD brings greater validity to PHMAP as a comprehensive measure of management operations, such additional uses for the program may not be appropriate.

Accuracy of Scores and Validity as a Management Assessment Tool Limit Uses for PHMAP

After performing on-site reviews of selected PHAs to confirm the accuracy of their PHMAP scores, HUD’s field offices changed half of the scores. In commenting on this report, HUD indicated that most confirmatory reviews involved high-risk PHAs, whose PHMAP data have been most susceptible to being found inaccurate. In similar reviews, HUD’s independent assessment contractors as well as HUD’s IG found that many scores or grades for specific indicators were inaccurate. To better identify PHAs that need oversight and technical assistance, HUD staff often supplement their decision-making with other measures of management problems to get a more complete picture of an authority’s performance. Professional property managers and industry representatives agreed that more information is needed than PHMAP provides to give a complete picture of how well a PHA’s management is performing.

After Confirmatory Reviews, PHMAP Scores Change Significantly

After performing confirmatory reviews of 200 PHAs in fiscal year 1995, HUD’s 49 field offices changed 98 PHMAP scores (see table 4.1).

In several cases, the changes HUD made to PHMAP scores also meant HUD would have to change the performance designation of those PHAs. For example, HUD lowered the scores of 14 PHAs enough to designate them as troubled, raised the scores of 4 troubled PHAs to 60 points or higher, and raised the scores of 10 standard-performing PHAs to 90 or higher.

Both of HUD’s independent assessment contractors as well as HUD’s IG have reviewed PHMAP data to confirm the accuracy of PHAs’ scores. For example, in 1993, the IG confirmed the scores of 12 housing authorities. As a result of this review, the IG concluded that the PHMAP scores for 9 of the 12 PHAs should be lowered because 3 of them fell below 60, a score which should have warranted the troubled designation. In a second report on PHMAP, the IG reported that six of HUD’s field offices reduced over half of the scores they reviewed. Similarly, one of HUD’s independent assessment contractors reported that for the 30 assessments it has performed at troubled housing authorities, it found 21 indicator grades and/or PHMAP scores that were inaccurate. Over 50 percent of the contractor’s assessments resulted in lowering the indicator grades to an “F.” The contractor most often lowered the indicators used to measure outstanding workorders and annual inspections of housing conditions and systems.

Several reasons explain why HUD and others changed so many PHMAP scores after performing a confirmatory review. Some field office staff said these scores changed because the PHAs did not understand all the requirements of PHMAP and therefore misreported their data. They also told us that PHMAP is particularly difficult for smaller housing authorities whose limited staff can find HUD’s paperwork requirements overwhelming.

HUD staff do not believe many PHAs intentionally try to deceive the Department by reporting false PHMAP information. Instead, they, as well as the contractor staff, said that the PHAs often have insufficient documentation to support the data they must submit to the field offices or do not understand how HUD wants them to report the information. For example, while a PHA may report the average number of days their housing units have been vacant, the PHA may not have the tenant files to document when the previous tenants moved out and when the new tenants’ leases took effect. Without supporting documentation or evidence of a system to track unit turnaround, HUD assigns an “F” to this indicator. Similarly, a PHA may be providing support programs for its residents, but fail to understand that its board of commissioners must approve those programs to receive a passing grade on PHMAP’s indicator for resident initiatives. Typically, when HUD’s field office staff find examples, such as these, during a confirmatory review, they use the correct data to recalculate the housing authority’s grade for each of the affected indicators.

HUD and Industry Professionals Supplement PHMAP With Additional Factors to Evaluate Management Performance

HUD’s field office staff did not use PHMAP alone to assess the management performance of its public housing authorities. Although they agreed that PHMAP accurately identifies troubled authorities, several staff said that they consider other factors besides PHMAP indicators to supplement their decision-making for the other authorities they oversee. They said that some PHAs with scores over 90 have management problems that the program’s indicators do not measure. Other factors used by some HUD staff to identify the potential for management problems at standard- and high-performing authorities include the failure of a PHA to implement consistent and effective operating policies and procedures, the frequency of changes in the executive leadership and the continued interference into a PHA’s daily operations by its board of commissioners, the number and the type of telephone calls received from a PHA’s residents and staff, and any adverse news stories about a PHA.

Staff at the five field offices we visited said that they believed some housing authorities with high PHMAP scores were not operating their housing programs efficiently or effectively. These field offices differed, however, in how they treated those PHAs. Staff at two field offices told us that although they use the scores to determine which PHAs need on-site reviews, they would not let a high score prevent them from visiting an authority they believed had serious management problems.

The HUD IG also questioned whether or not PHMAP scores accurately measure the management performance of public housing authorities. The IG’s reviews of high- and standard-performing PHAs found instances of fraud and program abuse. For example, the IG reported that the executive director of a high-performing PHA had charged over $62,000 in ineligible expenses, including excessive compensatory time, unsupported travel costs, and health and insurance benefits for his divorced spouse. Another PHA executive director falsified PHMAP data to obtain a high-performing designation. After reviewing the operations of a standard-performing PHA, the IG also cited numerous program abuses and mismanagement. The IG concluded that although PHMAP could be a useful tool to assess PHAs, the program was too unreliable for HUD to make oversight decisions.

Other public housing professionals—property managers and those representing industry associations—agreed that more information is needed than PHMAP provides to give a complete picture of how well a PHA is managed. For example, they noted that PHMAP does not automatically include an on-site observation and inspection of a PHA’s housing developments. One association noted that while a PHA could improve its PHMAP score by simply writing off more past due rents from former tenants as uncollectible to improve its grade on the indicator for rents uncollected, its PHMAP score would not measure how diligent an effort it had undertaken to collect the rent. Another industry association official knew of several examples of PHAs that were making good property management decisions, such as choosing to perform deferred maintenance when a unit became vacant rather than rent it immediately, that ironically led to lower PHMAP scores. Citing a similar situation, HUD has agreed that occasionally the best decision for a PHA is to take an action that yields a lower PHMAP score, and that the score should not be the sole driving force influencing a PHA’s decisions.

The Congress and HUD Have Proposed to Use PHMAP as a Basis for Deregulation and Funding Bonuses

While HUD’s primary use of PHMAP has been to identify troubled housing authorities and target technical assistance to them, the Congress and HUD have proposed to use this program for other purposes. In 1994, the Senate Committee on Banking, Housing, and Urban Affairs proposed some deregulation and additional flexibility for those authorities that had achieved PHMAP scores of 90 or above. In addition, in its fiscal year 1997 budget request, HUD proposed to give high-performing PHAs bonuses based in part on their PHMAP scores. Because PHMAP scores do not always measure the true management performance of the PHAs, the benefits of these proposals need to be weighed against the possibility of granting undeserved flexibility and awards.

To encourage individual PHAs to be more innovative, the Banking Committee proposed limited deregulation and additional flexibility for high-performing PHAs in two ways. First, it proposed permitting a PHA that generates income over a certain level to exclude that income from calculations of its need for a subsidy from HUD to operate and manage its properties. At that time, each dollar of extra income that a PHA generated reduced its subsidy by a dollar, thereby creating a disincentive to generate additional income from sources other than rent. Second, the Committee proposed to waive all but a few key regulations—such as nondiscrimination, equal opportunity, and tenant income eligibility—so high-performing PHAs could have more flexibility to bring innovative solutions to local problems and achieve more efficient operations.

In its fiscal year 1997 budget request, HUD proposed to award $500 million to high-performing PHAs as bonuses based, in part, on their PHMAP scores. As we reported in our testimony in June 1996 and as we found in the course of our work on this report, HUD does not confirm the scores of high performers and generally accepts them. In our June 1996 testimony, we recommended that the Congress consider not appropriating the bonus funding until HUD develops adequate performance measures and supporting information systems. The HUD appropriations bill which the Congress approved and the President signed did not contain funding for performance bonuses.

The three associations representing the public housing industry and the professional property managers that we interviewed all opposed or had strong reservations about using PHMAP scores for purposes other than identifying troubled housing authorities and targeting technical assistance to them. They also believed that other uses would be inappropriate because of the limited number of confirmatory reviews the field offices perform and the proportion of PHMAP scores that have been changed after a review. Two of the associations did not believe that PHMAP scores adequately measured the management performance of housing authorities because they thought some PHAs that received high scores did not provide their residents with decent, safe housing. The professional property management firm that independently verified some scores also agreed that the usefulness of these scores is limited. Because this firm has recommended lowering many scores after an independent assessment, the firm lacks confidence in the scores’ accuracy and does not believe that the program provides enough information about the management performance of PHAs for HUD to make effective funding decisions.

Conclusions

In recent years, both the Congress and HUD have proposed additional uses for PHMAP, such as bonuses to reward those housing authorities with the highest scores. While PHMAP has provided a quantifiable means to assess the management performance of housing authorities, the scores are not sufficiently accurate for detailed comparisons of performance. Although HUD is currently working to enhance the accuracy of these scores, they do not yet provide a comprehensive, generally accepted way to assess the performance of PHAs. To be useful for other purposes, not only would these scores have to be more accurate, but the program would have to be expanded to provide a more comprehensive measure of public housing authorities’ management operations.

Because HUD does not frequently confirm most scores—confirmatory reviews have focused on troubled PHAs—HUD does not know how many authorities are not receiving the proper designation. When HUD does confirm scores, it changes half of them—and more than half of these changes result in HUD’s lowering the score. We found that when HUD lowers a PHMAP score, it does so by an average of 14 points. If this average change held true for housing authorities in general, then HUD may not be properly designating as troubled those authorities currently scoring between 60 and the low 70s whose scores should be lower. As a result, those authorities are not receiving the oversight and technical assistance HUD should be providing to improve their performance.

Recommendations

We recommend that until it establishes a cost-effective means to ensure consistently accurate scores, HUD should not consider additional uses for PHMAP, including using its scores as criteria for funding bonuses, until it determines that PHMAP meets an acceptable level of accuracy and more comprehensively measures property management performance and require its field offices to confirm the PHMAP scores of housing authorities with scores low enough that they are at risk of being designated troubled.

Agency Comments

HUD agreed with our findings and recommendations. When we met with HUD officials, including the General Deputy Assistant Secretary for Public and Indian Housing, to discuss a draft of this report, they told us that the Department is no longer considering additional uses for PHMAP, such as using scores as criteria for funding bonuses. Even in the absence of using PHMAP for such purposes, we believe that it is important that HUD works to ensure scores are more consistently accurate and have, therefore, retained this recommendation. HUD has begun taking steps to address our recommendation that it confirm PHMAP scores of those housing authorities that are at risk of being designated troubled but expressed concern that it may not have sufficient resources to fully implement this recommendation.

HUD expressed three concerns relating to the information and conclusions presented in this chapter of our report. HUD believed that this chapter (1) assumes that PHMAP was intended to be an all-inclusive assessment system for property management, (2) does not place PHMAP in a historical perspective, and (3) reaches incorrect conclusions regarding the overall reliability of PHMAP scores.

We do not believe that we characterize PHMAP’s purpose as being an all-inclusive measure of property management. Our discussion of the program does not state that this is the purpose of PHMAP. Rather, the report discusses how the program’s limitations—including its intentional design not to be a complete performance measure—affect its suitability for additional purposes, such as those proposed in recent years by HUD and the Congress. HUD agreed that there is a perception that PHMAP is an all-encompassing system to assess the performance of PHAs and stated it is taking steps to address this misperception. Seeking to clarify the program’s purpose, HUD added language to its recently revised interim PHMAP rule (published in the December 30, 1996, Federal Register), that the program’s indicators reflect performance in only specific areas.

HUD correctly states that this report does not provide a historical perspective of PHMAP by discussing previous HUD systems for assessing and identifying troubled housing authorities. We believe that such information would not contribute substantially to our report’s three objectives to evaluate HUD’s use of the current program, provide trends in PHMAP scores from fiscal years 1992 through 1995, and discuss limitations in the program’s design and implementation that affect its usefulness for purposes other than identifying troubled housing authorities and targeting assistance to them. Therefore, we have not added the historical information HUD suggested to the report.

Finally, HUD is concerned that we have incorrectly reached conclusions about the reliability of all PHMAP scores based on the results of confirmatory reviews of high-risk authorities. HUD noted that the accuracy of the scores of these PHAs does not necessarily represent the accuracy of all PHMAP scores because the data provided by these PHAs are most susceptible to being inaccurate. Our report did not reach a conclusion about the reliability of all housing authorities’ scores because of the changes that resulted from confirmatory reviews. This report discusses the reliability of PHMAP scores for housing authorities whose scores are low enough that they may be at risk of being designated troubled. We have added language to the report to clarify this point.