Information Technology: Agencies Need to Improve Their Application Inventories to Achieve Additional Savings
Released: 2016-09-29T00:00:00

Letter

Background

In March 2012, OMB launched the PortfolioStat initiative which required agencies to conduct an annual review of their commodity IT portfolio to, among other things, achieve savings by identifying opportunities to consolidate investments or move to shared services. For PortfolioStat, OMB defined broad categories of commodity IT: enterprise IT systems, which include e-mail, identity and access management, IT security, web infrastructure, and collaboration tools; business systems, which include finance, human resources, and other IT infrastructure, which includes data centers, networks, desktop computers, and mobile devices.

Of those categories, the first two include software applications, which are software components and supporting software hosted on an operating system that create, use, modify, share, or store data in order to enable a business or mission function to be performed. This includes custom, commercial off-the-shelf, government off-the-shelf, or open-sourced software. The memorandum establishing the PortfolioStat initiative also required agencies to develop a commodity IT baseline including the number, types, and costs of investments for all commodity IT categories.

In a subsequent memorandum, OMB advocated the use of application rationalization to inform data center optimization efforts. Application rationalization is the process of streamlining the portfolio to improve efficiency, reduce complexity and redundancy, and lower the cost of ownership. It can be done by retiring aging and low-value applications, modernizing aging and high-value applications, eliminating redundant applications, standardizing on common technology platform and version (as is the case for moving to shared services), or consolidating applications. OMB stated in its memorandum that application rationalization would be a focus of PortfolioStat sessions and required agencies to describe their approach to maturing the IT portfolio, including rationalizing applications, in the information resource management plans and enterprise roadmaps that are required to be updated annually.

In December 2014, the law commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA) was enacted and required covered executive branch agencies (except for DOD) to ensure that Chief Information Officers (CIO) have a significant role in the decision making process for IT budgeting, as well as the management, governance, and oversight processes related to IT. The act also required that CIOs (in each covered agency except DOD) review and approve (1) all contracts for IT services prior to their execution and (2) the appointment of any other employee with the title of CIO, or who functions in the capacity of a CIO, for any component organization within the agency. OMB issued guidance in June 2015 that reinforces the importance of agency CIOs and describes how agencies are to implement the law.

In that same memorandum, OMB changed PortfolioStat from being an annual review session to quarterly reviews including a discussion of portfolio optimization efforts and focus on commodity IT. Specifically, the memorandum stated that agencies are to discuss how they use category management to consolidate commodity IT assets; eliminate duplication between assets; and improve procurement and management of hardware, software, network, and telecom services during the sessions. Furthermore, agencies are to share lessons-learned related to commodity IT procurement policies and efforts to establish enterprise-wide inventories of related information. The memorandum also specified key responsibilities for CIOs—including having increased visibility into all IT resources—and required agencies to develop plans to implement these responsibilities by December 2015.

Further, during the course of our review, in January 2016, OMB updated guidance to agencies requiring that they provide information regarding their IT asset inventories when making integrated data collection submissions. The guidance required agencies to provide a preliminary inventory by the end of February 2016 and a complete IT asset inventory, including information on systems, sub-systems, and applications by the end of May 2016 to OMB.

Finally, federal law and guidance specify requirements for protecting federal information and systems. Specifically, the Federal Information Security Management Act (FISMA) of 2002, among other things, requires agencies to maintain and update an inventory of major information systems at least annually, and the National Institute of Standards and Technology specifies that this should include an accurate inventory of software components, including the software applications which are the subject of our review. OMB plays a key role in monitoring and overseeing agencies’ security activities and their FISMA implementation. This includes tracking how well agencies are managing their inventories of hardware and software assets and protecting them.

GAO Has Reported on Efforts Related to Application Rationalization

In November 2013, we reported that agency commodity IT baselines were not all complete and recommended that 12 agencies complete their commodity IT baselines. As of March 2016, 6 of the 12 agencies—the Departments of Agriculture, Commerce, Housing and Urban Development, and Labor; the Social Security Administration; and the U.S. Agency for International Development—reported that they had completed their commodity IT baseline. The remaining 6 agencies reported making progress towards completion.

In May 2014, in a review examining federal agencies’ management of software licenses (which are types of enterprise IT applications), we determined, among other things, that only 2 of the 24 CFO Act agencies—the Department of Housing and Urban Development and the National Science Foundation—had comprehensive software license inventories. Twenty had partially complete inventories and two did not have any inventory. We recommended that agencies complete their inventories. We also recommended that OMB issue a directive to help guide agencies in managing licenses and that the 24 agencies improve their policies and practices for managing licenses. In June 2016, OMB issued a memorandum that is intended to improve agencies’ acquisition and management of enterprise software, consistent with our May 2014 recommendation. The memorandum contains elements related to having a comprehensive policy, such as developing and implementing a plan for centralizing the management of software licenses.

Most Agencies Fully Met at Least Three of the Four Practices for Establishing Complete Application Inventories

We identified four practices to determine whether agencies had a complete software application inventory. To do so, we primarily relied on best practices used in our recent report on federal software licenses which determined, among other things, whether agencies had a comprehensive software license inventory, and our guide for assessing the reliability of computer-processed data. We determined that to be considered complete agencies’ inventories should: include business systems and enterprise IT systems, as defined by OMB; include these systems from all organizational components; specify basic attributes, namely application name, description, owner, and function supported; and be regularly updated with quality controls in place to ensure the reliability of the information collected.

Most of the agencies fully met at least three of the four practices. Specifically,

4 agencies fully met all four practices;

9 agencies fully met three practices and 8 of these partially met the

6 agencies fully met two practices and 5 of these partially met the

2 agencies fully met one practice and partially met the three others,

3 agencies did not fully meet any practice.

U.S. Agency for International Development

The following are examples of how we assessed agencies against our practices. See appendix II for a detailed assessment of all the agencies.

The Environmental Protection Agency fully met three practices and partially met one. The agency fully met the first practice because its inventory includes enterprise IT and business systems, with the exception of very small systems. In addition, it included applications from all offices and regions in the organization. The agency partially met the practice for including application attributes in the inventory because, although it identifies the application name, and description, component managing the applications, and the business function associated with its applications, it does not identify the business function for every application. Officials stated that they are working to have this information populated for all applications. Lastly, the agency fully met the fourth practice of regularly updating the inventory because it has processes to update its inventory through the agency’s software life cycle management procedure and provided evidence of the annual data call issued by the CIO to ensure that the inventory is current.

The U.S. Agency for International Development fully met two practices and partially met two. Specifically, the agency’s inventory includes business and enterprise IT systems and the inventory includes basic application attributes. However, the agency’s inventory does not include systems from all organizational components because officials stated that coordination and communication in the geographically-widespread agency is difficult. In addition, the agency has processes for updating its inventory; however, it relies on manual processes to maintain it.

The Department of Transportation partially met all four practices.

While the department’s inventory for the common operating environment includes all business and enterprise IT systems and its inventory of applications includes business systems, the inventory of applications does not include all enterprise IT systems. Furthermore, both of its inventories do not include applications used by all of its components. Specifically, the inventory does not include applications used by the Federal Highway, Federal Railroad, and Federal Transit Administrations, among others, and the inventory for its common operating environment does not include applications used by the Federal Aviation Administration. The department also partially met the practice of including basic application attributes because, although the department’s inventory includes these attributes, its common operating environment does not provide the business function that the applications support. Further, while the Department of Transportation has a process for its partners to provide information on its individual inventories in order to update the inventory of applications, it does not have processes in place to ensure the reliability and accuracy of the reported information, and thus partially met this practice.

Regarding the four practices, the majority of the agencies fully met the practices of including business systems and enterprise IT system; including these systems from all organizational components; and specifying the application name, description, owner, and business function supported. Only five agencies fully met the practice of regularly updating the inventory and implementing quality controls for ensuring the reliability of the inventory data because they provided evidence of performing both of these activities. Table 3 shows the number of agencies who fully met, partially met, and did not meet the practices.

OMB’s requirement for agencies to complete an IT asset inventory by the end of May 2016 greatly contributed to most of the agencies including business systems and enterprise IT systems for all of their organizational components and specifying key attributes for them. Those agencies that did not fully address these practices provided various reasons for not doing so. For example, one agency stated that it has not made its software application inventory a priority because it has been focusing on major and high risk investments, while delegating applications to the component level. Others noted that the lack of automated processes make collecting complete inventory information difficult. Further, others noted that it is challenging to capture applications acquired by components in the department-wide inventory.

While it is reasonable to expect that priority be given to major and high risk investments, applications are nevertheless part of the portfolio and should be accounted for as such. Not accounting for them may result in missed opportunities to identify savings and efficiencies. It is also inconsistent with OMB guidance for implementing FITARA which requires that CIOs have increased visibility into all IT resources.

In addition, the lack of a comprehensive inventory presents a security risk. If agencies are not aware of all of their assets, they cannot secure them, resulting in a vulnerable posture. Given the importance of securing federal systems and data to ensuring public confidence and the nation’s safety, prosperity, and well-being, we designated federal information security as a government-wide high-risk area in 1997. In 2003, we expanded this area to include computerized systems supporting the nation’s critical infrastructure. In our high risk update in February 2015, we further expanded this area to include protecting the privacy of personal information that is collected, maintained, and shared by both federal and nonfederal entities.

Agencies Rationalize Some, but Not All Applications through Existing Investment Management Processes

As previously noted, application rationalization is the process of streamlining the portfolio to improve efficiency, reduce complexity and redundancy, and lower the cost of ownership. It can be done in many ways, including retiring aging and low-value applications, modernizing aging and high-value applications, eliminating redundant applications, standardizing on common technology platform and version (as is the case for moving to shared services), or consolidating applications. Based on common practices identified in technical papers from industry experts, to effectively perform rationalization, an agency should first establish a complete inventory of applications. It should then collect and review cost, technical, and business value information for each application, and use that information to make rationalization decisions. These practices are consistent with those used to manage investment portfolios. Therefore an agency can achieve application rationalization through established practices related to investment management, including budget formulation, security, or enterprise architecture.

Each of the six selected agencies relied on their investment management processes and, in some cases, supplemental processes to rationalize their applications to varying degrees. However, five of the six agencies acknowledged that their processes did not always allow for collecting or reviewing the information needed to effectively rationalize all their applications. The sixth agency, NSF, stated its processes allow it to effectively rationalize its applications, but we found supporting documentation to be incomplete. Only one agency, NASA, had plans to address shortcomings. The following describes the six selected agencies’ processes for rationalizing their applications, provides rationalization examples, identifies weaknesses and challenges, and addresses plans, if any, the agencies have for addressing them.

DOD: The department uses its investment management process for defense business systems to annually review its applications. Officials noted that the department’s enterprise architecture is also used to identify duplication and overlap among these applications. In addition, the department has identified eight enterprise common services for collaboration, content discovery, and content delivery it is requiring its components to use to, among other things, improve warfighting efficiency and reduce costs.

One example of rationalization that DOD provided resulting from its efforts with Executive Business Information System that was replaced by the Navy Enterprise Resource Planning system in a full migration in 2014. Estimated cost savings or avoidances were estimated at $268,000 in fiscal year 2012 and almost $200,000 per year in fiscal years 2013 through 2015. In addition, in an effort to improve its financial management systems, the department has efforts underway to reduce the number of financial management systems from 327 to 120 by fiscal year 2019.

However, officials acknowledged that its processes do not address all applications. Specifically, according to information provided by the department, about 1,200 enterprise IT and business systems which are associated with the Enterprise Information Environment Mission Area are not reviewed by the department—though they are reviewed by components—because they do not meet the definition of a defense business system.

Officials cited several challenges with implementing systematic rationalization efforts, including the department’s organizational structure and contractual agreements. As an example, they noted that the Navy’s Next Generation e-mail system is being procured through a contract with a particular vendor and as such would be difficult to consolidate with other department e-mail systems. They also noted that the cost of collecting additional cost, technical, and business value information, along with maintaining even more data at greater granularity, may outweigh the benefits.

The department does not have plans at this time to further enhance its processes to rationalize its applications. While we recognize the challenges and costs that may be associated with systematic rationalization efforts, the Enterprise Information Environment Mission Area could be considered as a near-term target for rationalization given the large number of enterprise IT and business systems associated with it. Modifying existing processes to allow for the collection, review, and evaluation of cost, technical, and business information of these systems at the department level could help identify opportunities for savings and efficiencies.

DHS: DHS has several processes for rationalizing applications. For example, through its investment management process, portfolios are regularly assessed against criteria which help identify duplication. In addition, the department uses its DHS Collaborative Architecture Methodology in conjunction with its segment architectures to help identify duplication and fragmentation, at different levels, including at the application level. The DHS IT Duplication Reduction Act of 2015 mandated the department to report on a strategy for reducing duplicative IT systems and the department used the DHS Collaborative Architecture Methodology process to address this mandate, including about 700 commodity IT and back-office applications in the scope of the effort.

Further, the department recently established an Application Services Council, chaired by its Enterprise Business Management Office. According to its charter, the council is a cross-component and cross- disciplined leadership team responsible for developing, maintaining, and overseeing the Enterprise Information Technology Services Portfolio, Lifecycle Governance Model, and Roadmap. It is expected to take a strategic approach to evaluating existing and future IT service offerings—including software, platform, and infrastructure services—and provide a forum to identify strategies, best practices, processes, and approaches for enterprise IT services, cloud computing, and shared service challenges. For example, officials reported the council is currently developing a standard service level agreement template and guidance, as well as a cloud adoption strategy. The department also reported other mechanisms related to rationalization include its Joint Requirements Council, strategic sourcing initiatives, IT acquisition reviews, and executive-level portfolio reviews.

In addition, it reported that it uses its DHS Enterprise Architecture Information Repository Technical Reference Model to track application products and software versions—mainly consisting of commercial off-the-shelf software. The product information is gathered through the use of continuous network discovery scans.

Examples of rationalization include the consolidation of learning management systems and the consolidation of site services, including help desk operations. The consolidation of learning management systems was identified through the segment architecture process and is expected to result in projected savings of 10 to 20 percent in fiscal year 2016 after transition costs are addressed. The modernization of the department’s help desk and on-site operations resulted in savings that cumulatively accrued to $202 million by fiscal year 2015 due to similar efforts among all department components.

However, DHS’s processes do not address all applications because, while the components may carry out their own rationalization efforts, the department does not always collect the application-level cost, technical, or business information for applications used by its components. Specifically, officials reported challenges tracking product level information for deployed applications and difficulty gaining visibility into all the supporting application products for large systems. Officials particularly noted they have been challenged to collect such information and cited a general lack of visibility into the components’ budget and their spending. They also noted it was not clear whether there was a good return on investment for the resources needed to collect additional technical, cost, and business value data for systematic application rationalization efforts. Officials reported the department had a financial systems modernization effort underway which would provide greater visibility into components’ spending but they did not have a plan to address the collection and review of technical and business value information.

While we recognize that collecting additional details on all applications may not be cost-beneficial, the department could consider taking a segmented approach and initially identify one high-cost function it is currently not collecting or reviewing detailed cost, technical, and business information for across the department. It could then modify existing processes to collect and review this information. These actions would assist the CIO in gaining visibility into all IT resources as specified in the OMB implementation guidance for FITARA and also help identify additional opportunities for savings and efficiencies.

NASA: NASA uses its current investment management process—the Capital Planning and Investment Control process—and its configuration management tools—to review its applications.

NASA reported examples of rationalization resulting in significant savings according to NASA officials. These included the NASA.gov Portal Cloud Transition which resulted in estimated savings of $4 million and the Enterprise Business Portal Transition/Consolidation which resulted in estimated savings of about $184,000 per year.

However, NASA officials acknowledged that their current processes do not provide the level of detail needed to effectively rationalize the agency’s applications.

In terms of challenges to rationalizing applications, officials stated that it is difficult to obtain transparency on all applications since each of the agency’s centers runs independently. In addition, officials stated that determining application business value is currently subjective to users because the agency’s process for obtaining this information is to ask the application owner the impact on the agency if the application did not exist, whereas application technical health information is more concrete. Furthermore, NASA officials stated that there is no systematic process to review applications facing end-of-life issues due to flat budgets and budget cuts.

NASA has developed a plan for a supplemental process (the annual capital investment review process) that is to allow the agency to, among other things, collect detailed data about its applications. The agency has begun to implement the plan and has completed the first milestone of the process, which included conducting a data call to gather and validate application information provided by the various centers and agency stakeholders. At the time of our review, NASA had also performed an initial review and analysis of the information collected and identified optimization opportunities, including developing a plan to consolidate, decommission, or invest to achieve maximum cost efficiencies and process effectiveness across the application program. Fully implementing the annual capital investment review process could better position the agency to identify additional opportunities for savings and efficiencies.

Interior: As part of its budget formulation process, Interior performs rationalization through annual reviews of its portfolio of investments (and supporting applications) against criteria which measure business value and technical fit.

Reported examples of application rationalization include Interior’s cloud e-mail and collaboration services initiative, which consolidated 14 disparate systems into a single enterprise system and achieved a cost savings/avoidance of $13.56 million, and the consolidation of the Enterprise eArchive System with the eMail Electronic Records and Document Management System which resulted in cost savings/avoidance of $6.1 million.

However, the department reported that its portfolio review process is not standardized because it has not been fully defined or established in policy. In addition, it has only been used at the department level, not at the bureaus or offices, and there is a lack of confidence in the data that is collected to support the analyses. In comments on a draft of this report, the department noted that it has also yet to document a plan to implement policy associated with these efforts which they believe would establish a standard analytical technique for rationalizing the investment portfolio. Such a plan would also help secure the commitment needed to carry out planned efforts.

The department reported several challenges to rationalizing its applications, including (1) ensuring the quality and accuracy of data collected since it relies largely on manual processes for collecting information and (2) the lack of standard portfolio evaluation techniques to support information resource management decision- making across the department.

The department has efforts underway which should help address these challenges. Specifically, it is making changes to its information resource management governance. According to the department, these changes, combined with efforts to implement the CIO responsibilities specified in FITARA, should help to address the challenges to rationalizing its applications and allow for rationalization of all applications. However, while the department has defined and begun to implement criteria to assess whether or not an investment and its underlying applications are wasteful, low-value, or duplicative, it has not documented its plan for improving its governance—which, according the department, would support application rationalization. Such a plan would help secure the commitment needed to carry out planned efforts.

Labor: Similar to the other agencies, the department uses its investment management process to review the majority of its business and enterprise IT applications. In addition, officials stated that the department initiated an enterprise-wide budget formulation and Information Technology Acquisition Review Board approval function beginning in fiscal year 2013 which has helped with rationalization.

Officials stated that their efforts have resulted in rationalization of commodity applications and on a case-by-case basis the rationalization of other applications, such as for a case management platform and an acquisition management system. Additional examples of application rationalization include the deployment of a web-based conferencing and collaboration shared service to employees which resulted in cost avoidance of travel costs of about $2.3 million. The department also noted benefits of moving to a cloud e-mail solution, such as saved time and increased user satisfaction.

However, officials identified weaknesses and challenges with rationalizing their applications. Specifically, they reported that, in most cases, IT investments are associated with a group of IT assets, including applications, and individual application information is therefore not reviewed, making it difficult to effectively rationalize. In addition, officials stated that the fact that each bureau-level agency has had authority and responsibility for managing its own applications and that the department has over 600 locations present challenges. Further, though senior officials including the CIO, agreed with the benefits of rationalization, they did not have any plans to rationalize. They questioned the value of developing such plans stating that (1) maintaining mission critical applications and the department’s aging infrastructure are current priorities and (2) funding may not be available to implement rationalization plans. While we agree that mission critical applications should be given priority, rationalizing mission support applications, including enterprise IT and business systems, could result in solutions which allow agencies to focus more on mission capabilities and at the same time generate savings which could be reinvested. As we noted for DHS, the department could consider taking a segmented approach to further rationalize and identify a function for which it could modify existing processes to collect and review detailed application cost, technical, and business value information.

NSF: NSF also uses its investment management processes and supporting budget formulation process—with key stakeholders such as the Executive IT Resources Board, Capital Planning and Investment Control Working Group, and Enterprise Architecture Working Group—to collect and review information for its investments. In addition, NSF’s Enterprise Modernization Roadmap—which is updated annually—identifies applications along with their associated business segment and modernization status and plans.

NSF identified its e-mail migration to a new platform, which was completed in July 2013, as an example of an application rationalization effort with the highest savings. According to the agency’s November 2015 integrated data collection submission to OMB, the migration effort resulted in cost avoidances of $60,000 in 2014. Other examples of application rationalization include modernization and consolidation of NSF’s grant systems, the 2014 retirement of the financial functions of a legacy system, and the implementation of its financial system modernization initiative.

However, while officials told us that evaluations for all applications meeting the scope of our review would be included in the roadmap, we only identified half of the applications (9 out of 18). In addition, cost information was only provided in the roadmap for three individual applications.

NSF officials told us that because they are a relatively small agency with a single mission in a single location, many of their processes are handled informally and not thoroughly documented but they are able to discuss all the applications with each other on a regular basis and as a result there is no duplication. Nevertheless, consistently documenting the evaluations and costs for all applications in the roadmap would improve transparency.

Conclusions

While it is encouraging that 13 of the 24 CFO Act agencies fully met at least three of the four practices for establishing a complete software application inventory, most could improve their software applications inventories—albeit to varying degrees—by taking steps to fully meet the practices we identified as being either partially met or not met. Doing so would better position them to identify opportunities to rationalize their applications, which could lead to savings and efficiencies. In addition, they would be better positioned to comply with OMB issued implementation guidance for the recent IT acquisition reform law which requires that CIOs have increased visibility into all IT resources and ensure they are effectively securing their IT assets.

Six selected agencies used their investment management processes and sometimes supplemental processes to rationalize their applications. Of the six agencies, one—NSF—had processes that allowed it to rationalize all applications, though the supporting documentation was not always complete. In addition, while the remaining five agencies’ processes did not allow for rationalizing all applications, only one—NASA—had plans to address identified weaknesses. While these agencies all had examples of rationalization resulting in savings and efficiencies, modifying their existing processes to more completely address their applications would help identify additional opportunities to achieve such savings and efficiencies, which even small, would add up across agencies and over time.

Recommendations for Executive Action

To improve federal agencies’ efforts to rationalize their portfolio of applications, we are recommending that: the heads of the Departments of Agriculture, Commerce, Education, Energy, Health and Human Services, Housing and Urban Development, the Interior, Labor, State, Transportation, the Treasury, and Veterans Affairs; and heads of the Environmental Protection Agency; National Aeronautics and Space Administration; National Science Foundation; Nuclear Regulatory Commission; Office of Personnel Management; Small Business Administration; Social Security Administration; and U.S. Agency for International Development direct their CIOs and other responsible officials to improve their inventories by taking steps to fully address the practices we identified as being partially met or not met; and the Secretaries of Defense, Homeland Security, the Interior, and Labor; and the Director of the National Science Foundation to direct the CIOs and other responsible officials to modify existing investment management processes to address applications more completely. Specifically, the Secretary of Defense should direct the responsible official to modify the department’s existing processes to collect and review cost, technical, and business information for the enterprise and business IT systems within the Enterprise Information Environment Mission Area applications which are currently not reviewed as part of the department’s process for business systems; the Secretary of Homeland Security should direct the department’s CIO to identify one high-cost function it could collect detailed cost, technical, and business information for and modify existing processes to collect and review this information; the Secretary of the Interior should direct the department’s CIO to document and implement a plan for establishing policy that would define a standard analytical technique for rationalizing the investment portfolio; the Secretary of Labor should direct the department’s CIO to consider a segmented approach to further rationalize and identify a function for which it would modify existing processes to collect and review application-specific cost, technical, and business value information; and the Director of the National Science Foundation should direct the CIO to consistently document evaluations for all applications and report cost information for them in the roadmap or other documentation.

Agency Comments and Our Evaluation

We provided a draft of this report to the 24 CFO Act agencies in our review for comment and received responses from all 24. Of the 24, 17 agreed with the recommendations directed to them; one (the Department of Defense) disagreed with the recommendations directed to it; five (the Department of the Treasury, the National Science Foundation, the Nuclear Regulatory Commission, the Small Business Administration, and the and U.S. Agency for International Development) stated that they had no comments; and one (the Department of Justice) agreed with the assessment and conclusion for three of the four practices associated with establishing a complete software application inventory and provided clarifying information on the two other practices. Several agencies also provided technical comments, which we incorporated as appropriate. The agencies’ comments and our responses are summarized below.

In e-mail comments, the Department of Agriculture’s Senior Advisor for Oversight and Compliance Enterprise Management stated that the department concurred with our recommendation. The department also provided technical comments which we incorporated as appropriate. As a result of these comments and additional documentation provided, we changed our evaluation of the practice associated with updating the software application inventory from not met to partially met.

In written comments, the Department of Commerce concurred with our recommendation and stated that the department is committed to implementing a more efficient process by regularly updating its application inventory to ensure the reliability of the data collected. The department also specified actions it plans to take to provide regular updates of its inventory. The department’s comments are reprinted in appendix III.

In written comments, the Department of Defense disagreed with both of our recommendations to the department. For the first recommendation, the department provided evidence showing that it updated its inventory subsequent to us sending the report for comment. As a result, we changed the rating for the related practice from partially met to fully met and removed the associated recommendation.

For the second recommendation, the department stated that 53 percent of the inventory records for the Enterprise Information Environment Mission Area we focused on were IT infrastructure assets (specifically network enclaves or circuits) and not applications subject to rationalization. The mission area nevertheless includes enterprise and business IT applications which could benefit from rationalization, as evidenced by the example of e-mail system consolidation provided in the comments. Given the number of systems involved (at least 1,200), collecting and reviewing cost, technical, and business information for them would help identify additional rationalization opportunities which could yield savings and efficiencies. We therefore believe a recommendation to address these systems is still warranted.

The department also stated that our draft implied that major IT infrastructure modernization efforts, many of which involve the Enterprise Information Environment Mission Area, were not reviewed or properly managed by the department. However, as noted in our report, we did not include IT infrastructure assets in the scope of our review and therefore made no comment on how these assets are being managed. We have restated our emphasis on enterprise and business IT systems as it relates to the mission area where appropriate.

Finally, in its comments the department stated that our report ignored significant Enterprise Information Environment Mission Area application rationalization efforts, such as the Pentagon IT consolidation under the Joint Service Provider, the Business Process and System Review, and ongoing efforts concerning public-facing websites and associated systems. While we were not informed of these efforts during our review, our intent was to highlight additional opportunities for rationalization, not discount any that might have already been implemented. The department also provided technical comments, which we incorporated into the report as appropriate. The department’s comments are reprinted in appendix IV.

In written comments, the Department of Education concurred with our recommendation and described actions it plans to take to address it. The department’s comments are reprinted in appendix V.

In written comments, the Department of Energy concurred with our recommendation. In addition, the department stated that it partially met the four practices associated with establishing a complete software application inventory and provided the IT Asset Inventory it submitted to OMB in May 2016 and other documentation supporting this claim. Our review of the documentation found that the inventory includes business and enterprise IT systems; however, it does not include those systems from all organizational components and it is missing the business function code for a large number of systems. Furthermore, while the department is updating the IT Asset inventory in response to OMB guidance for the fiscal year 2016 integrated data collection submission process, it has not implemented quality control processes to ensure the reliability of the data within the inventory. As a result, we changed the department’s rating for the practice associated with including business and enterprise IT systems from not met to fully met and from not met to partially met for the remaining three practices. We modified sections of the report specific to the department accordingly. The department’s comments are reprinted in appendix VI.

In written comments, the Department of Health and Human Services concurred with our recommendation and stated that that it would review the feasibility of fully addressing the practices it partially met. The department’s comments are reprinted in appendix VII.

In written comments, the Department of Homeland Security concurred with our recommendation and described actions it plans to take to implement it. The department’s comments are reprinted in appendix VIII.

In written comments, the Department of Housing and Urban Development concurred with our recommendation and stated that more definitive information with timelines will be provided once the final report has been issued. The department’s comments are reprinted in appendix IX.

In written comments, the Department of the Interior stated that it would agree with the recommendations if we made its requested changes. However, we disagreed with the request to change the rating for the practice associated with regularly updating the inventory from not met to partially met because, while the department provided evidence supporting its claim that it recently updated its inventory, the evidence was not sufficient. Specifically, the department provided an e-mail requesting the bureaus and offices to complete an inventory survey. However, the department did not show how the survey resulted in updates to the inventory. We incorporated the remaining requested changes in the report as appropriate. The department’s comments are reprinted in appendix X.

In written comments, the Department of Justice stated that it concurred with our assessment and conclusions. The department also provided clarifying information regarding its procedures to ensure application inventory accuracy and provided documentation showing that it updates the inventory and implements quality controls to ensure its reliability. As a result, we changed the rating for the related practice from partially met to fully met and removed the recommendation made to the department. The department’s comments are reprinted in appendix XI.

In written comments, the Department of Labor concurred with our recommendations to the department and stated that it would take the necessary steps to address the recommendations. The department’s comments are reprinted in appendix XII.

In written comments, the Department of State concurred with our recommendation to the department, and described current and planned actions to fully address it. The department’s comments are reprinted in appendix XIII.

In e-mail comments, the Department of Transportation’s Audit Liaison stated that the department concurred with our findings and recommendation.

In e-mail comments, the Department of the Treasury’s Audit Liaison stated that the department did not have any comments.

In written comments, the Department of Veterans Affairs concurred with our conclusions and recommendation. The department also provided information on the actions it plans to take to address the recommendation. The department’s comments are reprinted in appendix XIV.

In written comments, the Environmental Protection Agency generally agreed with our recommendation. The agency also asked that we include some of the language from the detailed evaluation in appendix II of the report to the example we have in the body to provide the full context of its practices. We added the language as requested. The agency’s comments are reprinted in appendix XV.

In e-mail comments, the General Services Administration’s Associate CIO of Enterprise Planning and Governance concurred with the report. The agency also provided evidence of its processes to update the inventory and ensure the reliability of the data in the inventory, including the coordination between its Enterprise Architecture Team and subject matter experts. As a result, we changed the agency’s rating for the related practice from partially met to fully met and removed our recommendation to the agency.

In written comments, the National Aeronautics and Space Administration concurred with our recommendation and stated that it would utilize the capital investment review process it is currently implementing to improve its inventory. The agency’s comments are reprinted in appendix XVI.

In e-mail comments, the National Science Foundation Office of Integrated Activities’ Program Analyst stated that it had no comments on the draft report.

In written comments, the Nuclear Regulatory Commission stated that it is in general agreement with the report. The agency’s comments are reprinted in appendix XVII.

In written comments, the Office of Personnel Management concurred with our recommendation and described plans to fully address it. The agency’s comments are reprinted in appendix XVIII.

In e-mail comments, the Small Business Administration Office of Congressional and Legislative Affairs’ Program Manager stated that the Office of the Chief Information Officer believes the report captures its current posture.

In written comments, the Social Security Administration agreed with our recommendation to the agency, but disagreed with the partially met rating for regularly updating the inventory, including implementing quality controls, stating that it had provided evidence supporting its implementation of the practice. However, as noted in the report, the Social Security Administration reported that its systems development lifecycle contains steps for maintaining the inventory but did not provide evidence showing that it is using this process to regularly update the inventory. Therefore we did not change our rating. The agency’s comments are reprinted in appendix XIX.

In an e-mail, the U.S. Agency for International Development Audit, Performance and Compliance Division’s Management Analyst stated that the agency did not have any comments.

We are sending copies of this report to interested congressional committees; the heads of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, the Interior, Justice, Labor, State, Transportation, the Treasury, and Veterans Affairs; the Environmental Protection Agency; the General Services Administration; the National Aeronautics and Space Administration; the National Science Foundation; the Nuclear Regulatory Commission; the Office of Personnel Management; the Small Business Administration; the Social Security Administration; the U.S. Agency for International Development; the Director of the Office of Management and Budget; and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov.

If you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine (1) whether agencies have established complete application inventories and (2) to what extent selected agencies have developed and implemented processes for rationalizing their portfolio of applications.

For the first objective, we reviewed the 24 major agencies covered by the Chief Financial Officers (CFO) Act of 1990. To ensure consistency, we decided to focus on the software applications associated with the business and enterprise information technology (IT) commodity IT categories defined in the Office of Management and Budget (OMB) guidance since they would be familiar to the agencies in our scope. OMB defines enterprise IT systems as e-mail, identity and access management, IT security, web infrastructure, and collaboration tools; and business systems as finance, human resources, and other administrative functions.

We then identified practices to assess whether agencies had a complete software application inventory. To identify these practices, we primarily relied on our guide for assessing the reliability of computer-processed data which addresses questions about the currency of the data and how often it is updated, procedures for ensuring the completeness of the data, and quality control processes in place to ensure the accuracy of the data; and on criteria used in our recent report on federal software licenses which determined whether agencies had a comprehensive software license inventory, among other things. To be considered complete, we determined an inventory should: include business systems and enterprise IT systems, as defined by OMB; include these systems from all organizational components; specify basic attributes, namely, application name, description, owner, and function supported; and be regularly updated with quality controls in place to ensure the reliability of the information collected.

Following the identification of these four practices, we asked the 24 CFO Act agencies for their software application inventories. We used a set of structured questions to determine whether the agencies implemented the practices and identify lessons learned and challenges faced in establishing a complete software application inventory. We analyzed supporting documentation, such as agency and departmental guidance, policies, and procedures for updating the inventories, and interviewed relevant agency officials, as needed. We compared the information received to the four practices. We determined a practice to be fully met if agencies provided evidence that they fully or largely implemented the practice for establishing a complete application inventory; partially met if agencies provided evidence that they addressed some, but not all, of the practice for establishing a complete application inventory; and not met if the agencies did not provide any evidence that they implemented the practice for establishing a complete application inventory.

To verify the inclusion of business and enterprise IT systems, we analyzed agencies’ inventories and looked for examples of each type of system identified by OMB in the business and enterprise IT commodity categories. We followed up with agencies when we were not able to identify a type of system to determine the reason for the omission. We considered the practice to be fully met if agencies’ inventories included all of the business and enterprise IT system types or if agencies provided valid reasons for excluding them. We considered the practice to be partially met if agencies acknowledged they were missing applications or if we determined system types to be missing and agencies did not provide a valid reason for this. Although we followed up with agencies to determine whether they maintained separate inventories of software licenses when they were not included in the inventories provided, we did not consider the inclusion of these applications in determining our rating because software licenses are expected to be tracked separately by OMB.

To verify the inclusion of systems from all organizational components, we analyzed agencies’ inventories against the list of organizational components to determine whether they were included. We followed up with agency officials to determine causes, if any, for missing components. We considered the practice to be fully met if inventories included applications from all organizational components or if agencies provided valid reasons for excluding them. We considered the practice to be partially met if agencies acknowledged they were missing organizational components or if we determined several components to be missing and agencies did not provide a valid reason for this.

Regarding application attributes, we determined that, at a minimum, agencies should have a name, a description, an owner, and function supported for each application. We considered the practice to be fully met if inventories included these attributes for all or most applications or the agencies provided evidence that attributes not included in the inventory provided were being tracked separately. We determined the practice to be partially met if agencies acknowledged that they were missing any of the attributes or if we determined them to be missing from the inventory and agencies did not provide alternate sources for them.

For the last practice, we determined whether agencies (1) used relevant methods to update and maintain the application inventory and (2) implemented controls to ensure the reliability of the information collected. Regarding these controls, we looked for the use of automated tools to collect and track information as their use increases reliability. We determined the practice to be fully met if agencies provided evidence that they regularly updated the inventory and had controls for ensuring the reliability of information collected, including the use of automated tools, or if agencies had mitigating factors when these processes were not in place. We determined the practice to be partially implemented when agencies provided policies and procedures but no evidence of actual inventory updates or quality controls. We also determined the practice to be partially implemented if agencies provided evidence of either regular updates or controls for ensuring reliability but not both or did not make use of automated tools for collecting or maintaining information and had no mitigation factors. Finally, we also determined the practice to be partially implemented if agencies provided draft policy and guidance of their processes.

For our second objective, we selected 6 of the 24 CFO Act agencies—the Departments of Defense, Homeland Security, the Interior, and Labor; and the National Aeronautics and Space Administration and National Science Foundation—to assess their application rationalization plans and efforts to implement them. We selected the agencies based on three factors: whether they had an application rationalization process; in our initial set of structured questions to agencies, we asked whether they had a plan or process for rationalizing applications and selected those that reported having one; the size of the agency based on fiscal year 2015 IT spending; we selected two large agencies (i.e., with spending equal to or greater than $3 billion), two medium agencies (i.e., with spending between $1 billion and $3 billion), and two small agencies (i.e., with spending of less than $1 billion) for a full range of IT spending; and if they were known for effectively rationalizing their applications based on OMB observations and our research on IT acquisition reform recognizing agencies for their application rationalization efforts.

We identified key practices for effectively rationalizing applications. To do so, we reviewed OMB guidance on federal IT management. We also reviewed technical reports on application rationalization from industry experts. We synthesized the information collected, looked for themes, and determined that, to effectively rationalize applications, agencies should have a process addressing the following four key practices: establish an application inventory; collect information on each application, such as total cost, technical details, and business value; evaluate the portfolio and make application rationalization decisions based on a review of collected information and determine what applications to retain, retire, replace, eliminate, modernize, or consolidate/move to shared services; and execute and manage the process by implementing decisions from the evaluation and evaluate process outcomes against defined metrics and adjust, as needed.

While our research identified specific processes for rationalizing applications, the principles of collecting application information and reviewing it to inform decision making are consistent with those used to manage investment portfolios. Therefore we considered established practices related to investment management, budget formulation, security, or enterprise architecture.

Since the first key practice was addressed in our first objective, we focused on the last three practices. To do so, we interviewed relevant officials using a structured set of questions that were developed in conjunction with internal experts. We also reviewed documentation to determine the extent to which agencies had processes addressing these practices. We also asked agencies to provide their two best examples of application rationalization in terms of savings or cost avoidance—to illustrate the results of rationalization. When agencies did not provide two examples meeting these conditions—the case for DOD, DHS, and NSF— we drew examples from other documentation they had provided.

Finally, we interviewed staff from OMB’s Office of the Federal Chief Information Officer to determine whether and how the office monitors agencies’ efforts to rationalize their portfolio of applications as recommended in OMB guidance. We also interviewed the staff to determine the impetus for the IT asset data inventory guidance and the planned used for the information collected.

We conducted this performance audit from May 2015 to September 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO’s Evaluation of Agencies’ Application Inventories

The following tables provide our evaluation of the 24 agencies’ application inventories.

Appendix III: Comments from the Department of Commerce

Appendix IV: Comments from the Department of Defense

Appendix V: Comments from the Department of Education

Appendix VI: Comments from the Department of Energy

Appendix VII: Comments from the Department of Health and Human Services

Appendix VIII: Comments from the Department of Homeland Security

Appendix IX: Comments from the Department of Housing and Urban Development

Appendix X: Comments from the Department of the Interior

Appendix XI: Department of Justice

Appendix XII: Comments from the Department of Labor

Appendix XIII: Comments from the Department of State

Appendix XIV: Comments from the Department of Veterans Affairs

Appendix XV: Comments from the Environmental Protection Agency

Appendix XVI: Comments from the National Aeronautics and Space Administration

Appendix XVII: Nuclear Regulatory Commission

Appendix XVIII: Comments from the Office of Personnel Management

Appendix XIX: Comments from the Social Security Administration

Appendix XX: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, the following staff made key contributions to this report: Sabine Paul (Assistant Director), Chris Businsky, Rebecca Eyler, Dan Gordon, James MacAulay, Lori Martinez, Paul Middleton, and Di’Mond Spencer.