Information Technology: HUD Needs to Address Significant Weaknesses in Its Cost Estimating Practices
Released: 2017-02-07T00:00:00

Letter

Background

HUD’s mission is to create strong, sustainable, inclusive communities and quality, affordable homes for all. To carry out this mission, the department administers community and housing programs that affect millions of households each year. These programs provide affordable rental housing opportunities and help homeless families and chronically homeless individuals and veterans. The department also administers mortgage insurance programs for single-family housing, multifamily housing, and health care facilities.

IT plays a critical role in HUD’s ability to perform its business functions, which involve the management of billions of dollars to carry out its mission. For example, the department’s IT environment consists of multiple systems that, among other things, are intended to help coordinate interactions with lending institutions to insure mortgages, collect and manage state and local housing data, process applications for community development, and process vouchers for different rental assistance programs. Its systems also support the processing of applications for, and the management of, more than 50 grant programs administered by the department.

However, according to HUD, its IT environment has not been sufficient to effectively support business functions because its systems are overlapping and duplicative, not integrated, necessitate manual workloads, and employ antiquated technologies that are costly to maintain. The department has reported that its environment consists of: over 200 information systems, many of which perform the same function and, thus, are overlapping and duplicative; stove-piped, nonintegrated systems that result in identical data existing in multiple systems; manual processing for business functions due to a lack of systems to support these processes; and antiquated technology (15 to 30 years old) and complex systems that are costly to maintain.

To address challenges with its IT environment, HUD has developed a number of plans in recent years to guide its modernization efforts. These include plans that outline how it intends to spend IT funds, an information resource management strategic plan, and an enterprise architecture roadmap. These plans contain information related to the department’s modernization efforts and actions aimed at improving its capacity to manage IT. Nevertheless, even with these plans and ongoing modernization efforts, the department reported in November 2016 that limited progress had been made in replacing legacy systems and manual processes with modern applications and enhanced capabilities.

HUD’s Information Technology Budget and Fiscal Year 2017 Investments

HUD’s IT budget covers two categories of spending: (1) operations and maintenance of existing systems and (2) new investments for modernization (often referred to as development, modernization, and enhancement). Operations and maintenance funds refer to the expenses required for general upkeep of the department’s existing systems. Funds for modernization support projects and activities that lead to new systems, or changes and modifications to existing systems that substantively improve capability or performance to better support HUD’s mission and business functions.

According to the Office of Management and Budget’s (OMB) IT Dashboard, over the past 5 years, the department spent between approximately 70 and 95 percent of its total IT budget on operations and maintenance; it dedicated a smaller portion—ranging from approximately 5 to 30 percent—to modernization efforts. Figure 1 illustrates the percentage of HUD’s IT spending during fiscal years 2012 through 2016, dedicated to operating and maintaining existing IT versus modernization efforts, as reported on the IT Dashboard.

Consistent with prior years, a majority of HUD’s fiscal year 2017 IT budget request is intended to support existing systems. Specifically, the department requested $286 million, of which approximately 87 percent ($250 million) is planned for operations and maintenance. According to the budget request, the department anticipates using operations and maintenance funds to support business administrative functions as well as its IT infrastructure, which includes servers, communications, equipment and support, desktops, mobile devices, and security.

Of the fiscal year 2017 IT budget request, approximately 13 percent ($36 million) is intended to support modernization investments aimed at improving the department’s IT environment. According to HUD’s budget request, these funds are to support new investments intended to deliver modernized enterprise capabilities that better support the department’s mission. Specifically, these investments are expected to, among other things, leverage enterprise-level technology, reduce the number of stand- alone systems, deliver cloud-based technologies, automate manual processes, and consolidate data.

Of the $36 million requested for modernization, approximately 81 percent ($29 million) was identified to support four modernization efforts, which were in various phases of planning and development. Table 1 provides a description of these investments and the amounts of their associated fiscal year 2017 budget requests.

Cost Estimating Best Practices

Our prior work has shown that implementing repeatable, disciplined processes that adhere to federal law and best practices can help agencies effectively plan, manage, and oversee modernization efforts. Disciplined processes include establishing guidance that can be used for developing reliable cost estimates that project realistic life-cycle costs. Such estimates are critical to a modernization effort’s success because they can be used to support key investment decisions that help to ensure finite resources are wisely spent. In addition, a reliable estimate is the foundation of a good budget and budget spending plan, which outlines how and at what rate an investment’s funding will be spent over time. Put another way, reliable cost estimates are essential for successful IT investments and modernization efforts because they help ensure that Congress and the department itself can have reliable information on which to base funding and budgetary decisions.

OMB calls for federal agencies to maintain current cost estimates that encompass the full life-cycle of an investment. Building on OMB’s requirements and drawing on practices promulgated by federal cost estimating organizations and private industry, GAO’s Cost Guide identifies cost estimating practices that, if followed correctly, have been found to be the basis for a reliable cost estimate.

An estimate created using these practices exhibits four broad characteristics: it is comprehensive, well-documented, accurate, and credible. Moreover, each characteristic is associated with a specific set of best practices. Table 2 summarizes, by characteristic, the best practices for developing reliable cost estimates identified in the Cost Guide.

Because specific and discrete best practices underlie each characteristic, an agency’s performance in each of the characteristics can vary. For example, an organization’s cost estimates could be found to be comprehensive and well-documented, but not accurate or credible. According to the Cost Guide, a cost estimate is considered reliable if each of the four characteristics is substantially or fully met; in contrast, if any of the characteristics are not met, minimally met, or partially met, the cost estimate cannot be considered reliable.

HUD’s Cost Estimates for Selected IT Investments Exhibited Significant Weaknesses That Made Them Unreliable

The cost estimates that HUD developed for each of the four selected investments exhibited significant weaknesses in that they did not meet or substantially meet best practices for each characteristic. As such, the estimates were unreliable and did not provide a sound basis for informing the department’s investment and budgetary decisions. Specifically, none of the estimates exhibited all of the characteristics of a reliable estimate, as they were not substantially or fully comprehensive, well-documented, accurate, and credible.

Only one estimate—for the Customer Relationship Management investment—more than minimally met best practices associated with any of the four characteristics because it partially met the practices for a comprehensive and accurate estimate. The remaining three investments minimally met or did not meet the best practices associated with the four characteristics. For example, the Enterprise Data Warehouse estimate minimally met all four characteristics; the Enterprise Voucher Management System estimate did not meet the characteristic for being accurate, and minimally met the other three characteristics; and the Federal Housing Administration Automation and Modernization estimate did not met the characteristic for being credible, while minimally meeting the rest of the characteristics.

Table 3 provides a summary of the extent to which the four investments’ cost estimates were comprehensive, well-documented, accurate, and credible.

Comprehensive. Of the cost estimates for the four selected investments, none were comprehensive. While one investment partially met associated best practices, the remaining three investments minimally met these practices. Specifically, although all of the estimates included costs for specific elements and phases of the investments, none of the estimates included both government and contractors costs of the investment over the life-cycle, from inception through design, development, deployment, and operations and maintenance, to retirement of the investment.

In addition, the Customer Relationship Management investment partially met the best practice related to defining the investment by, for example, explaining that the effort would result in a cloud-based solution that allowed the department to consolidate existing systems. However, none of the estimates fully met this practice, in part because investment documentation did not completely define system requirements, reflect current schedules, and demonstrate that efforts were technically reasonable. Further, work breakdown structures that had been developed were not sufficiently detailed for any of the investments to ensure that cost elements were neither omitted nor double counted, and allowed for traceability between the investment’s costs and schedule by deliverable, such as hardware or software components.

Moreover, although various assumptions were factored into the cost estimate for the Customer Relationship Management investment, the basis for the assumptions was not documented and, as a result, their reasonableness could not be determined. For the remaining three investments, where information was limited and judgments had to be made, the estimates did not contain cost-influencing ground rules and assumptions.

Well-documented. The four investments did not develop well- documented cost estimates because none were supported by detailed documentation that described how the estimate was derived and how the expected funding would be spent. This characteristic’s best practices were minimally met by all of the investments. In discussing the estimating methodologies used to develop the estimates, HUD officials reported using analogy, expert opinion, parametric, and other methods; however, the department did not document the specific methodologies used for any of the investments.

Specifically, HUD did not adequately document for each estimate the sources of data used, any assessments of data accuracy and reliability, and other circumstances affecting the data, such as the details related to calculations performed. The estimating information for the investments also was not captured in a way that the data used could be easily replicated and updated. Further, the documentation did not sufficiently discuss the technical baseline—which is intended to serve as the basis for developing an estimate by providing a common definition of the investment—or how the data were normalized.

In addition, while HUD officials reported briefing management on each of the investments and on the high-level estimate for the Enterprise Voucher Management System, the department did not brief management on the ground rules and assumptions underlying the estimate. A cost estimate is not considered valid until management has approved it, yet the department did not provide documentation that any of the four cost estimates had been reviewed and accepted by management.

Accurate. Overall, the four estimates were not accurate because only one estimate partially addressed the best practices associated with this characteristic, two estimates minimally addressed the best practices, and one estimate did not meet any of the associated practices. As such, we could not determine whether the cost estimates provided results that were unbiased and were not overly conservative or optimistic. More specifically, the estimate for the Customer Relationship Management investment partially addressed the best practices in that its calculations did not contain errors and actual costs from existing programs or historical data were used to develop the estimate. However, analyses had not been performed for any of the investments to ensure that the cost estimates were based on an assessment of most likely costs.

In addition, none of the investments’ estimates had been properly adjusted for inflation to ensure that cost data were expressed in consistent terms, which is important because doing so can help to prevent cost overruns. For example, officials for the Federal Housing Administration Automation and Modernization investment stated that the cost data had not been adjusted for inflation or normalized to constant year dollars to remove the effects of inflation. Further, the estimating techniques used to determine costs were not documented, which prevented an assessment of the accuracy of any calculations performed. Finally, while HUD officials responsible for the investments stated that the estimates were grounded in historical data, such as actual costs from comparable investments, they did not provide evidence to support this assertion for all four investments.

Credible. Three of the investments minimally met the best practices associated with developing a credible cost estimate and one investment did not meet the practices. Specifically, the estimates did not fully discuss limitations of the analyses because of uncertainty or biases surrounding the data or assumptions. For example, none of the investments conducted a sensitivity analysis, which is intended to reveal how the cost estimate is affected by a change in a single assumption and allows for the cost estimator to understand which variable most affects the cost estimate.

In addition, risk and uncertainty analyses were not conducted in a manner that conformed to the practices in the Cost Guide. For example, while HUD officials responsible for Enterprise Data Warehouse and Customer Relationship Management investments stated that risks were evaluated, evidence supporting these assertions was not provided. In addition, risk documentation was provided for the Federal Housing Administration Automation and Modernization investment, but the analysis was limited to a portion of the investment and, therefore, did not provide a comprehensive view of the level of risk and the degree of uncertainty associated with the estimate.

Moreover, department officials stated that cross checks were conducted for three of the investments’ estimates to determine whether applying other methods produced similar results; however, evidence was not provided to demonstrate how this was done. Lastly, an independent cost estimate developed by a group outside of the acquiring organization was not conducted to validate the reasonableness of the cost estimates developed for the four investments. (Additional details on our assessments of the four investments’ cost estimates can be found in appendix I.)

The significant weaknesses in the cost estimates can largely be attributed to the department’s lack of established guidance for developing reliable cost estimates, which is contrary to our prior work related to disciplined processes. HUD officials responsible for the selected investments’ estimates stated that department guidance had not yet been established and that IT investments were not required to develop estimates that exhibit the four characteristics of a reliable estimate. As a result, according to these officials, cost estimating practices are inconsistently implemented across the department and are decentralized because of the reliance on the efforts and experience of various subject matter experts and contractors.

With regard to improving its cost estimating practices, in January 2014, the department began efforts to develop cost estimating guidance by conducting an internal review of approaches used for developing estimates. Following this review, the department drafted guidance in June 2015 that was intended to conform to best practices in the Cost Guide. In August 2016, officials from the Offices of Strategic Planning and Management, the Chief Financial Officer, and the Chief Information Officer stated that HUD was in the process of further developing the guidance to reflect cost estimating best practices. However, as of December 2016, the guidance had not yet been established. According to the officials, finalizing the guidance has been a challenge due to competing priorities within the department, such as addressing weaknesses in its governance structure and management processes. Additionally, these officials stated that the department’s focus has been on establishing an infrastructure that is expected to enable implementation of better cost estimating practices. Moving forward, the officials stated that they expect to continue efforts to finalize and establish the guidance in the future, although time frames for doing so had not been determined.

Until HUD establishes guidance that calls for the implementation of best practices identified in the Cost Guide, the department is less likely to develop reliable cost estimates for its IT investments that can serve as the basis for informed investment decision making. In going forward without addressing the weaknesses identified in this report, the department risks being unable to effectively estimate funding needs for IT investments and using unreliable data to make budgetary decisions.

Conclusions

While it is critical that HUD’s IT investments develop cost estimates that provide Congress and the department reliable information on which to make decisions, the cost estimates for the four selected investments had significant weaknesses. Specifically, none of the cost estimates for these investments were reliable because they did not fully or substantially implement best practices that result in exhibiting characteristics of being comprehensive, well-documented, accurate, and credible. Many of the weaknesses found in the investments can be attributed to the lack of established cost-estimating guidance, which the department has not yet finalized because it has focused on addressing management weaknesses and taking action to establish an infrastructure to support improved cost estimation practices. Until HUD finalizes and ensures the implementation of guidance to improve its cost estimating practices, the department is at risk of continuing to make investment decisions based on unreliable information.

Recommendation for Executive Action

To increase the likelihood that its IT investments develop reliable cost estimates, we recommend that the Secretary of HUD finalize, and ensure the implementation of, guidance that incorporates the best practices called for in the GAO Cost Estimating and Assessment Guide.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from HUD, which are reprinted in appendix II. In its comments, the department agreed with our recommendation and indicated that it plans to take action in response.

HUD also provided technical comments, which we incorporated as appropriate. Among these comments, the department took issue with our conclusion that cost estimation had not been a priority and stated that HUD had been focused on establishing an infrastructure so that it could improve its cost estimating practices. We revised our conclusion to reflect the department’s actions in this regard.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Housing and Urban Development, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Assessments of Cost Estimates for Selected IT Investments at HUD

The following tables summarize our assessments, for each selected IT investment, regarding the extent to which each met the characteristics of a reliable cost estimate—comprehensive, well documented, accurate, and credible. Specifically, our assessments identified whether the investments estimates met, substantially met, partially met, minimally met, or did not meet each of the four characteristics and provides key examples of the rationale.

Customer Relationship Management

The Customer Relationship Management investment’s cost estimate partially met best practices for developing a comprehensive and accurate estimate and minimally met best practices for developing a well- documented and credible estimate.

Enterprise Data Warehouse

The Enterprise Data Warehouse investment’s cost estimate minimally met best practices for developing a comprehensive, well-documented, accurate, and credible estimate.

Enterprise Voucher Management System

The Enterprise Voucher Management System investment’s cost estimate minimally reflected best practices for developing a comprehensive, well- documented, and credible estimate and did not reflect best practices for developing an accurate cost estimate.

Federal Housing Administration Automation and Modernization

The Federal Housing Administration Automation and Modernization investment’s cost estimate minimally met best practices for developing a comprehensive, well documented, and accurate estimate and did not meet best practices for developing a credible cost estimate.

Appendix II: Agency Comments

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Teresa M. Yost (Assistant Director), Donald Baca (Analyst-in-Charge), Brian Bothwell, Kami Brown, Rebecca Eyler, Amanda Gill, and Karen Richey made key contributions to this report.