Information Technology: HUD Needs to Address Significant Weaknesses in Its Cost Estimating Practices
Published: 2017-02-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: HUD relies extensively on IT to deliver services and manage programs in support of its mission. For fiscal year 2017, HUD requested $36 million for IT investments intended to deliver modernized enterprise-level capabilities that better support the department's mission. Critical to the success of such efforts is the department's ability to develop reliable cost estimates that project life-cycle costs and provide the basis for, among other things, informed decision making and realistic budget formulation.
The joint explanatory statement that accompanied the Consolidated and Further Continuing Appropriations Act, 2015, included a provision for GAO to evaluate HUD's cost estimating practices. This review determined the extent to which HUD implemented cost estimating best practices for selected IT investments. GAO selected four IT modernization investments with the largest portion of requested funding for fiscal year 2017, interviewed relevant agency officials, and analyzed and compared each investment's cost estimate to best practices in the Cost Guide . This guide states that, when most or all of the practices are “fully” or “substantially” met, an estimate is considered reliable.
section_title: What GAO Found
paragraphs: The cost estimates that the Department of Housing and Urban Development (HUD) developed for the four selected information technology (IT) investments were unreliable and, thus, lacked a sound basis for informing the department's investment and budgetary decisions. GAO's Cost Estimating and Assessment Guide ( Cost Guide ) defines best practices that are associated with four characteristics of a reliable estimate—comprehensive, well documented, accurate, and credible. However, none of the cost estimates for the selected investments exhibited all of these characteristics. Only one estimate—for the Customer Relationship Management investment—more than minimally met best practices associated with any of the four characteristics because it partially met the practices for a comprehensive and accurate estimate. The remaining three investments minimally or did not meet the best practices associated with the four characteristics. For example, the Enterprise Data Warehouse estimate minimally met all four characteristics; the Enterprise Voucher Management System estimate did not meet the characteristic for being accurate and minimally met the other three characteristics; and the Federal Housing Administration Automation and Modernization estimate did not meet the characteristic for being credible, while minimally meeting the remaining characteristics (see table).
The significant weaknesses in the cost estimates for the selected investments can largely be attributed to the department's lack of guidance for developing reliable cost estimates. HUD officials responsible for the selected investments stated that the department had not required the development of estimates that exhibit the four characteristics of a reliable estimate. As a result, according to these officials, cost estimating practices have been decentralized and inconsistent across the department. While HUD drafted guidance in June 2015 that was intended to conform to the best practices in GAO's Cost Guide , the department has not yet finalized the guidance because it has focused on establishing the infrastructure needed to support improved cost estimation practices. Until HUD finalizes and ensures the implementation of guidance to improve its cost estimating practices, the department is at risk of continuing to make investment decisions based on unreliable information.
section_title: What GAO Recommends
paragraphs: To improve cost estimating practices, GAO recommends that HUD finalize and implement guidance that incorporates best practices called for in the Cost Guide . HUD concurred with this recommendation.

section_title: Letter
paragraphs: The Department of Housing and Urban Development (HUD) relies  extensively on information technology (IT) to deliver services and manage  programs that support its mission of creating strong, sustainable,  inclusive communities and quality, affordable homes for all. Among other  purposes, HUD relies on IT to support programs that manage  approximately $1 trillion portfolio of insured mortgages, provide roughly  $35 billion in rental housing assistance to needy families, and administer  community development grant programs with a combined value of about  $6.6 billion.
However, HUD has long experienced limitations in the systems  supporting its mission. Specifically, the department has reported since  2008 that its systems are overlapping, duplicative, and not integrated;  necessitate manual workarounds; and employ antiquated, costly-to- maintain technologies. For fiscal year 2017, HUD requested $36 million  for IT investments intended to deliver modernized enterprise-level  capabilities that better support the department’s mission.
Critical to the success of its modernization efforts is HUD’s ability to  effectively manage and apply disciplined processes for its IT investments,  including developing cost estimates that project realistic life-cycle costs.
Such estimates can provide the basis for informed decision making,  realistic budget formulation, and meaningful progress measurement.
The joint explanatory statement accompanying the Consolidated and  Further Continuing Appropriations Act, 2015, incorporated language from  Senate and House reports that included a provision for GAO to evaluate  HUD’s cost estimating practices. This review determined the extent to  which HUD implemented cost estimating best practices for selected IT  investments.
To address the objective, we obtained and analyzed available  documentation supporting HUD’s cost estimating practices for selected IT  investments. This documentation included projected cost estimates,  business cases, budget data, and management briefings and reports.
To select the specific investments for our review, we identified the  investments in HUD’s fiscal year 2017 IT budget request that had not  been fully implemented or recently approved for termination, and that  were intended to support the different lines of business at HUD. This  resulted in a list of nine potential modernization investments. From this  list, we then chose the four modernization investments that comprised the  largest portion of the funding requested, 81 percent, for fiscal year 2017.
For each of the four selections, we assessed the reliability of the  investments’ cost estimates by comparing the estimates and supporting  documentation to the best practices discussed in the GAO Cost  Estimating and Assessment Guide (Cost Guide). These practices have  been found to be the basis for reliable cost estimates.
Specifically, we assessed each investment against the four  characteristics of a reliable estimate—comprehensive, well-documented,  accurate, and credible. Each characteristic was defined by multiple  practices, which we assessed as being not met—the investment did not provide evidence that it implemented minimally met—the investment provided evidence that it implemented  a small portion (less than half) of the practices; partially met—the investment provided evidence that it implemented  about half of the practices;  substantially met—the investment provided evidence that it  implemented a large portion (more than half) of the practices; or  fully met—the investment provided evidence that it implemented all of  the practices.
We then summarized our assessments of the practices for each  characteristic. We considered a cost estimate to be reliable if each of the  four characteristics were substantially or fully met. If any of the  characteristics were not met, minimally met, or partially met, then we  determined that the cost estimate did not reflect the characteristics of a  reliable estimate.
We supplemented our analysis with interviews of cognizant HUD officials.  These officials included program and other officials responsible for each  of the four investments, as well as the Deputy Chief Information Officer,  the Director of the Office of the Strategic Planning and Management, and  the Assistant Chief Financial Officer for Budget. From these officials, we  sought additional information on each investment’s approach to  developing cost estimates and the steps taken to establish department  guidance related to cost estimation.
We conducted this performance audit from July 2016 to February 2017 in  accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objective. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective.
subsections: section_title: Background
paragraphs: HUD’s mission is to create strong, sustainable, inclusive communities and  quality, affordable homes for all. To carry out this mission, the department  administers community and housing programs that affect millions of  households each year. These programs provide affordable rental housing  opportunities and help homeless families and chronically homeless  individuals and veterans. The department also administers mortgage  insurance programs for single-family housing, multifamily housing, and  health care facilities.
IT plays a critical role in HUD’s ability to perform its business functions,  which involve the management of billions of dollars to carry out its  mission. For example, the department’s IT environment consists of  multiple systems that, among other things, are intended to help  coordinate interactions with lending institutions to insure mortgages,  collect and manage state and local housing data, process applications for  community development, and process vouchers for different rental  assistance programs. Its systems also support the processing of  applications for, and the management of, more than 50 grant programs  administered by the department.
However, according to HUD, its IT environment has not been sufficient to  effectively support business functions because its systems are  overlapping and duplicative, not integrated, necessitate manual  workloads, and employ antiquated technologies that are costly to  maintain. The department has reported that its environment consists of: over 200 information systems, many of which perform the same  function and, thus, are overlapping and duplicative;  stove-piped, nonintegrated systems that result in identical data  existing in multiple systems; manual processing for business functions due to a lack of systems to  support these processes; and antiquated technology (15 to 30 years old) and complex systems that  are costly to maintain.
To address challenges with its IT environment, HUD has developed a  number of plans in recent years to guide its modernization efforts. These  include plans that outline how it intends to spend IT funds, an information  resource management strategic plan, and an enterprise architecture  roadmap. These plans contain information related to the department’s  modernization efforts and actions aimed at improving its capacity to  manage IT. Nevertheless, even with these plans and ongoing  modernization efforts, the department reported in November 2016 that  limited progress had been made in replacing legacy systems and manual  processes with modern applications and enhanced capabilities.
subsections: section_title: HUD’s Information Technology Budget and Fiscal Year 2017 Investments
paragraphs: HUD’s IT budget covers two categories of spending: (1) operations and  maintenance of existing systems and (2) new investments for  modernization (often referred to as development, modernization, and  enhancement). Operations and maintenance funds refer to the expenses  required for general upkeep of the department’s existing systems. Funds  for modernization support projects and activities that lead to new  systems, or changes and modifications to existing systems that  substantively improve capability or performance to better support HUD’s  mission and business functions.
According to the Office of Management and Budget’s (OMB) IT  Dashboard, over the past 5 years, the department spent between  approximately 70 and 95 percent of its total IT budget on operations and  maintenance; it dedicated a smaller portion—ranging from approximately  5 to 30 percent—to modernization efforts. Figure 1 illustrates the  percentage of HUD’s IT spending during fiscal years 2012 through 2016,  dedicated to operating and maintaining existing IT versus modernization  efforts, as reported on the IT Dashboard.
Consistent with prior years, a majority of HUD’s fiscal year 2017 IT  budget request is intended to support existing systems. Specifically, the  department requested $286 million, of which approximately 87 percent  ($250 million) is planned for operations and maintenance. According to  the budget request, the department anticipates using operations and  maintenance funds to support business administrative functions as well  as its IT infrastructure, which includes servers, communications,  equipment and support, desktops, mobile devices, and security.
Of the fiscal year 2017 IT budget request, approximately 13 percent ($36  million) is intended to support modernization investments aimed at  improving the department’s IT environment. According to HUD’s budget  request, these funds are to support new investments intended to deliver  modernized enterprise capabilities that better support the department’s  mission. Specifically, these investments are expected to, among other  things, leverage enterprise-level technology, reduce the number of stand- alone systems, deliver cloud-based technologies, automate manual  processes, and consolidate data.
Of the $36 million requested for modernization, approximately 81 percent  ($29 million) was identified to support four modernization efforts, which  were in various phases of planning and development. Table 1 provides  a description of these investments and the amounts of their associated  fiscal year 2017 budget requests.
section_title: Cost Estimating Best Practices
paragraphs: Our prior work has shown that implementing repeatable, disciplined  processes that adhere to federal law and best practices can help  agencies effectively plan, manage, and oversee modernization efforts.  Disciplined processes include establishing guidance that can be used for  developing reliable cost estimates that project realistic life-cycle costs.  Such estimates are critical to a modernization effort’s success because they can be used to support key investment decisions that help to ensure  finite resources are wisely spent. In addition, a reliable estimate is the  foundation of a good budget and budget spending plan, which outlines  how and at what rate an investment’s funding will be spent over time. Put  another way, reliable cost estimates are essential for successful IT  investments and modernization efforts because they help ensure that  Congress and the department itself can have reliable information on  which to base funding and budgetary decisions.
OMB calls for federal agencies to maintain current cost estimates that  encompass the full life-cycle of an investment. Building on OMB’s  requirements and drawing on practices promulgated by federal cost  estimating organizations and private industry, GAO’s Cost Guide  identifies cost estimating practices that, if followed correctly, have been  found to be the basis for a reliable cost estimate.
An estimate created using these practices exhibits four broad  characteristics: it is comprehensive, well-documented, accurate, and  credible. Moreover, each characteristic is associated with a specific set of  best practices. Table 2 summarizes, by characteristic, the best practices  for developing reliable cost estimates identified in the Cost Guide.
Because specific and discrete best practices underlie each characteristic,  an agency’s performance in each of the characteristics can vary. For  example, an organization’s cost estimates could be found to be  comprehensive and well-documented, but not accurate or credible.  According to the Cost Guide, a cost estimate is considered reliable if each  of the four characteristics is substantially or fully met; in contrast, if any of  the characteristics are not met, minimally met, or partially met, the cost  estimate cannot be considered reliable.
section_title: HUD’s Cost Estimates for Selected IT Investments Exhibited Significant Weaknesses That Made Them Unreliable
paragraphs: The cost estimates that HUD developed for each of the four selected  investments exhibited significant weaknesses in that they did not meet or  substantially meet best practices for each characteristic. As such, the  estimates were unreliable and did not provide a sound basis for informing  the department’s investment and budgetary decisions. Specifically, none  of the estimates exhibited all of the characteristics of a reliable estimate,  as they were not substantially or fully comprehensive, well-documented,  accurate, and credible.
Only one estimate—for the Customer Relationship Management  investment—more than minimally met best practices associated with any  of the four characteristics because it partially met the practices for a  comprehensive and accurate estimate. The remaining three investments  minimally met or did not meet the best practices associated with the four  characteristics. For example, the Enterprise Data Warehouse estimate  minimally met all four characteristics; the Enterprise Voucher  Management System estimate did not meet the characteristic for being  accurate, and minimally met the other three characteristics; and the  Federal Housing Administration Automation and Modernization estimate  did not met the characteristic for being credible, while minimally meeting  the rest of the characteristics.
Table 3 provides a summary of the extent to which the four investments’  cost estimates were comprehensive, well-documented, accurate, and  credible.
Comprehensive. Of the cost estimates for the four selected investments,  none were comprehensive. While one investment partially met associated  best practices, the remaining three investments minimally met these  practices. Specifically, although all of the estimates included costs for  specific elements and phases of the investments, none of the estimates  included both government and contractors costs of the investment over  the life-cycle, from inception through design, development, deployment,  and operations and maintenance, to retirement of the investment.
In addition, the Customer Relationship Management investment partially  met the best practice related to defining the investment by, for example,  explaining that the effort would result in a cloud-based solution that  allowed the department to consolidate existing systems. However, none  of the estimates fully met this practice, in part because investment  documentation did not completely define system requirements, reflect  current schedules, and demonstrate that efforts were technically  reasonable. Further, work breakdown structures that had been developed  were not sufficiently detailed for any of the investments to ensure that  cost elements were neither omitted nor double counted, and allowed for  traceability between the investment’s costs and schedule by deliverable,  such as hardware or software components.
Moreover, although various assumptions were factored into the cost  estimate for the Customer Relationship Management investment, the  basis for the assumptions was not documented and, as a result, their  reasonableness could not be determined. For the remaining three  investments, where information was limited and judgments had to be  made, the estimates did not contain cost-influencing ground rules and  assumptions.
Well-documented. The four investments did not develop well- documented cost estimates because none were supported by detailed  documentation that described how the estimate was derived and how the  expected funding would be spent. This characteristic’s best practices  were minimally met by all of the investments. In discussing the estimating  methodologies used to develop the estimates, HUD officials reported  using analogy, expert opinion, parametric, and other methods; however,  the department did not document the specific methodologies used for any  of the investments.
Specifically, HUD did not adequately document for each estimate the  sources of data used, any assessments of data accuracy and reliability,  and other circumstances affecting the data, such as the details related to  calculations performed. The estimating information for the investments  also was not captured in a way that the data used could be easily  replicated and updated. Further, the documentation did not sufficiently  discuss the technical baseline—which is intended to serve as the basis  for developing an estimate by providing a common definition of the  investment—or how the data were normalized.
In addition, while HUD officials reported briefing management on each of  the investments and on the high-level estimate for the Enterprise Voucher  Management System, the department did not brief management on the  ground rules and assumptions underlying the estimate. A cost estimate is  not considered valid until management has approved it, yet the  department did not provide documentation that any of the four cost  estimates had been reviewed and accepted by management.
Accurate. Overall, the four estimates were not accurate because only  one estimate partially addressed the best practices associated with this  characteristic, two estimates minimally addressed the best practices, and  one estimate did not meet any of the associated practices. As such, we  could not determine whether the cost estimates provided results that were  unbiased and were not overly conservative or optimistic. More  specifically, the estimate for the Customer Relationship Management  investment partially addressed the best practices in that its calculations  did not contain errors and actual costs from existing programs or  historical data were used to develop the estimate. However, analyses had  not been performed for any of the investments to ensure that the cost  estimates were based on an assessment of most likely costs.
In addition, none of the investments’ estimates had been properly  adjusted for inflation to ensure that cost data were expressed in  consistent terms, which is important because doing so can help to  prevent cost overruns. For example, officials for the Federal Housing  Administration Automation and Modernization investment stated that the  cost data had not been adjusted for inflation or normalized to constant  year dollars to remove the effects of inflation. Further, the estimating  techniques used to determine costs were not documented, which  prevented an assessment of the accuracy of any calculations performed.  Finally, while HUD officials responsible for the investments stated that the  estimates were grounded in historical data, such as actual costs from  comparable investments, they did not provide evidence to support this  assertion for all four investments.
Credible. Three of the investments minimally met the best practices  associated with developing a credible cost estimate and one investment  did not meet the practices. Specifically, the estimates did not fully discuss  limitations of the analyses because of uncertainty or biases surrounding  the data or assumptions. For example, none of the investments  conducted a sensitivity analysis, which is intended to reveal how the cost  estimate is affected by a change in a single assumption and allows for the  cost estimator to understand which variable most affects the cost  estimate.
In addition, risk and uncertainty analyses were not conducted in a manner  that conformed to the practices in the Cost Guide. For example, while  HUD officials responsible for Enterprise Data Warehouse and Customer  Relationship Management investments stated that risks were evaluated,  evidence supporting these assertions was not provided. In addition, risk  documentation was provided for the Federal Housing Administration  Automation and Modernization investment, but the analysis was limited   to a portion of the investment and, therefore, did not provide a  comprehensive view of the level of risk and the degree of uncertainty  associated with the estimate.
Moreover, department officials stated that cross checks were conducted  for three of the investments’ estimates to determine whether applying  other methods produced similar results; however, evidence was not  provided to demonstrate how this was done. Lastly, an independent cost  estimate developed by a group outside of the acquiring organization was  not conducted to validate the reasonableness of the cost estimates  developed for the four investments.  (Additional details on our assessments of the four investments’ cost  estimates can be found in appendix I.)
The significant weaknesses in the cost estimates can largely be attributed  to the department’s lack of established guidance for developing reliable  cost estimates, which is contrary to our prior work related to disciplined  processes. HUD officials responsible for the selected investments’  estimates stated that department guidance had not yet been established  and that IT investments were not required to develop estimates that  exhibit the four characteristics of a reliable estimate. As a result,  according to these officials, cost estimating practices are inconsistently  implemented across the department and are decentralized because of the  reliance on the efforts and experience of various subject matter experts  and contractors.
With regard to improving its cost estimating practices, in January 2014,  the department began efforts to develop cost estimating guidance by  conducting an internal review of approaches used for developing  estimates. Following this review, the department drafted guidance in June  2015 that was intended to conform to best practices in the Cost Guide. In  August 2016, officials from the Offices of Strategic Planning and  Management, the Chief Financial Officer, and the Chief Information  Officer stated that HUD was in the process of further developing the  guidance to reflect cost estimating best practices. However, as of  December 2016, the guidance had not yet been established. According to  the officials, finalizing the guidance has been a challenge due to  competing priorities within the department, such as addressing  weaknesses in its governance structure and management processes.  Additionally, these officials stated that the department’s focus has been  on establishing an infrastructure that is expected to enable  implementation of better cost estimating practices. Moving forward, the  officials stated that they expect to continue efforts to finalize and establish  the guidance in the future, although time frames for doing so had not  been determined.
Until HUD establishes guidance that calls for the implementation of best  practices identified in the Cost Guide, the department is less likely to  develop reliable cost estimates for its IT investments that can serve as  the basis for informed investment decision making. In going forward  without addressing the weaknesses identified in this report, the  department risks being unable to effectively estimate funding needs for   IT investments and using unreliable data to make budgetary decisions.
section_title: Conclusions
paragraphs: While it is critical that HUD’s IT investments develop cost estimates that  provide Congress and the department reliable information on which to  make decisions, the cost estimates for the four selected investments had  significant weaknesses. Specifically, none of the cost estimates for these  investments were reliable because they did not fully or substantially  implement best practices that result in exhibiting characteristics of being  comprehensive, well-documented, accurate, and credible. Many of the  weaknesses found in the investments can be attributed to the lack of  established cost-estimating guidance, which the department has not yet  finalized because it has focused on addressing management weaknesses  and taking action to establish an infrastructure to support improved cost  estimation practices. Until HUD finalizes and ensures the implementation  of guidance to improve its cost estimating practices, the department is at  risk of continuing to make investment decisions based on unreliable  information.
section_title: Recommendation for Executive Action
paragraphs: To increase the likelihood that its IT investments develop reliable cost  estimates, we recommend that the Secretary of HUD finalize, and ensure  the implementation of, guidance that incorporates the best practices  called for in the GAO Cost Estimating and Assessment Guide.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from HUD, which  are reprinted in appendix II. In its comments, the department agreed with  our recommendation and indicated that it plans to take action in  response.
HUD also provided technical comments, which we incorporated as  appropriate. Among these comments, the department took issue with our  conclusion that cost estimation had not been a priority and stated that  HUD had been focused on establishing an infrastructure so that it could  improve its cost estimating practices. We revised our conclusion to reflect  the department’s actions in this regard.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Housing and Urban Development, and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.
section_title: Appendix I: Assessments of Cost Estimates for Selected IT Investments at HUD
paragraphs: The following tables summarize our assessments, for each selected IT  investment, regarding the extent to which each met the characteristics of  a reliable cost estimate—comprehensive, well documented, accurate, and  credible. Specifically, our assessments identified whether the investments  estimates met, substantially met, partially met, minimally met, or did not  meet each of the four characteristics and provides key examples of the  rationale.
subsections: section_title: Customer Relationship Management
paragraphs: The Customer Relationship Management investment’s cost estimate  partially met best practices for developing a comprehensive and accurate  estimate and minimally met best practices for developing a well- documented and credible estimate.
section_title: Enterprise Data Warehouse
paragraphs: The Enterprise Data Warehouse investment’s cost estimate minimally  met best practices for developing a comprehensive, well-documented,  accurate, and credible estimate.
section_title: Enterprise Voucher Management System
paragraphs: The Enterprise Voucher Management System investment’s cost estimate  minimally reflected best practices for developing a comprehensive, well- documented, and credible estimate and did not reflect best practices for  developing an accurate cost estimate.
section_title: Federal Housing Administration Automation and Modernization
paragraphs: The Federal Housing Administration Automation and Modernization  investment’s cost estimate minimally met best practices for developing a  comprehensive, well documented, and accurate estimate and did not  meet best practices for developing a credible cost estimate.
section_title: Appendix II: Agency Comments
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Teresa M. Yost (Assistant  Director), Donald Baca (Analyst-in-Charge), Brian Bothwell, Kami Brown,  Rebecca Eyler, Amanda Gill, and Karen Richey made key contributions to  this report.