# Recent Reports

Reports News from the GAO

- Broadband Deployment: Agencies Should Take Steps to Better Meet Deadline for Processing Permits What GAO FoundThe Bureau of Land Management (BLM) and Forest Service process the most applications from telecommunications providers to install communications use equipment or facilities—including for broadband internet—on federal property. However, GAO found that from fiscal years 2018 through 2022, BLM and Forest Service did not have sufficiently reliable—i.e., accurate and complete data—to determine the processing time for 42 percent and 7 percent, respectively, of their communications use applications. These agencies lacked the necessary controls to ensure staff entered key information, such as start and end dates, in their electronic systems. Without accurate, complete data to determine processing times, the agencies cannot track the extent to which they are complying with the statutory requirement that they grant or deny applications within 270 days.For those communications use applications with sufficient data, BLM and Forest Service reduced their average processing time by 57 percent from fiscal years 2018 through 2022. However, despite this overall improvement, about half of the applications either exceeded the 270-day deadline or did not have sufficiently accurate and complete information to determine if they met the deadline.Selected Agencies' Performance in Processing Communications Use Permit Applications Within the 270-Day Deadline, Fiscal Years 2018—2022Officials from BLM and Forest Service said that applicant non-responsiveness and staffing issues contributed to the delays in processing communications use applications. Forest Service has taken some steps to improve processing, including hiring additional staff, but neither agency has fully analyzed or addressed reasons why many applications processing time exceeded the deadline. Moreover, neither agency has a method to alert staff to applications that are at risk of exceeding the 270-day deadline. Until BLM and Forest Service address issues that contribute to delays and establish methods to flag at-risk applications, the agencies may continue to miss the deadline at similar rates, which could cause delays and increased costs in deploying needed broadband infrastructure.Why GAO Did This StudyMillions of Americans do not have broadband access, which has become essential for online activities like work, school, and health care. With the federal government investing billions of dollars in expanding access in the coming years, new broadband infrastructure will frequently cross federal lands or connect to federal buildings, which requires a federal permit. Legislation requires that executive agencies process applications for communications use permits—including for broadband infrastructure—within 270 days of receipt.GAO was asked to review issues related to processing federal broadband permits. This report examines the extent to which selected agencies (1) used reliable data to track application processing times for communications use permits; and (2) processed communications use applications by the required deadline.GAO analyzed federal permitting data; reviewed laws and reports on the application review process; and interviewed agency officials.
- Driver Assistance Technologies: NHTSA Should Take Action to Enhance Consumer Understanding of Capabilities and Limitations What GAO FoundNew vehicles are increasingly equipped with driver assistance technologies designed to prevent or mitigate crashes (crash avoidance technologies) and support the driving task (driver support systems). According to interviewed stakeholders and research GAO reviewed, when drivers have a realistic understanding of their vehicles' driver assistance technologies, they are more likely to use them as intended. There is some evidence, however, that consumers do not always have an accurate understanding of technologies' capabilities and limitations. One study found that between 27 and 79 percent of consumers surveyed had misperceptions about the limitations of different crash avoidance technologies in their vehicles. In addition, misuse is a safety concern particular to partial driving automation systems, a type of driver support system, which can take over some of the driving tasks in a vehicle but still requires the full attention of the driver.Vehicle Dashboard Displays Driver Assistance Technology ActivationWithin the Department of Transportation, the National Highway Traffic Safety Administration (NHTSA) provides consumers with information on crash avoidance technologies through its New Car Assessment Program (NCAP), additional information on its website, and other means. NHTSA uses checkmarks to indicate whether vehicles come equipped with the four crash avoidance technologies that it recommends and that meet NHTSA's performance criteria. In 2022, NHTSA published a draft roadmap with plans to upgrade NCAP, including recommending four more crash avoidance technologies and developing a rating system for them. These upgrades would provide more comprehensive and comparative information to consumers. However, NHTSA has not finalized its roadmap and has missed time frames even though work on these upgrades started years ago. Developing realistic time frames and publicly communicating its progress could help NHTSA provide consumers with more meaningful information. Aside from NCAP, NHTSA provides consumers with a description on partial driving automation systems, but there is little information about their intended use and operational limitations. Providing this information could assist consumers in developing a more accurate understanding of these systems.Why GAO Did This StudyAccording to NHTSA, almost 42,800 people died in vehicle crashes in 2022. New vehicles are increasingly equipped with driver assistance technologies that could help reduce crashes and fatalities. NHTSA administers NCAP to educate consumers about vehicle safety, including driver assistance technologies, and to assist in consumers' purchasing decisions.The Consolidated Appropriations Act, 2022, included a provision for GAO to review consumer education about driver assistance technologies. Among the issues this report examines are (1) consumers' use and understanding of driver assistance technologies; and (2) the extent to which NHTSA contributes to consumers' understanding and using the technologies as intended.GAO reviewed NHTSA's relevant rulemaking documents, website, and studies; analyzed a nongeneralizable sample of NHTSA consumer complaint data; and interviewed NHTSA and other relevant agency officials and a range of industry stakeholders, including automakers and safety organizations. GAO assessed NHTSA's efforts against key project schedule management practices.
- Information Technology: IRS Needs to Complete Planning and Improve Reporting for Its Modernization Programs What GAO FoundIn April 2023, the Internal Revenue Service (IRS) issued its agency-wide strategic operating plan outlining its vision to use billions of dollars contained in the Inflation Reduction Act of 2022 appropriation to transform the administration of the tax system and services provided to taxpayers. The plan identified five objectives, including a technology objective underpinning the other four.Figure: Inflation Reduction Act Strategic Operating Plan Transformation ObjectivesThe IRS plan, among other things, states that it will use the technology objective to retire and replace legacy systems and update programming languages, give taxpayers tools to access their data and use online services, and ensure continued security and privacy of taxpayer data. IRS identified eight initiatives and 38 key projects for the technology objective but noted the list of projects was not comprehensive. This is consistent with the Treasury Inspector General for Tax Administration 2023 report stating that the technology objective had a total of 142 projects. These 142 represent about 30 percent of the 475 projects for all five objectives.Projects under the technology objective include ongoing modernization programs that have been modified to account for the new appropriations. However, plans showing changes to the scope of future work, milestones, and efforts to retire legacy systems have not yet been updated. IRS officials stated that updating these plans is contingent on completing a roadmap to implement the strategic plan. These officials added that the first version of the roadmap was completed in December 2023. However, this roadmap did not include the technology objective. They said that this objective would be included in version two of the roadmap which they expected would be completed very soon. Completing the roadmap and then updating ongoing IT modernization plans to reflect revisions driven by the strategic plan are essential to the transformation's success.Regarding cost and schedule performance of IT modernization programs, IRS reported meeting most of its quarterly targets for fiscal years 2022 and 2023. These reports, however, could be improved by including programs' historical cost and schedule goals and how quarterly performance compares to overall program goals. For example, GAO previously reported that a key IT investment was within schedule estimates for 2019 and 2020 but that IRS had changed its overall program plans several times. These changes led to a 9-year milestone delay--from 2014 to 2023. However, the quarterly reports did not show this lengthy delay because they do not include programs' historical cost and schedule goals.Why GAO Did This StudyIRS relies extensively on IT to annually collect trillions of dollars in taxes, distribute hundreds of billions of dollars in refunds, and carry out its mission of providing service to America's taxpayers in meeting their tax obligations. In August 2022, Congress appropriated tens of billions to IRS through fiscal year 2031 to bolster taxpayer services and enforcement of the tax code, and modernize IT, among other things.The Joint Explanatory Statements accompanying the Financial Services and General Government Appropriations Bills for 2022 and 2023 include provisions for GAO to review the status of IRS's IT investments. This report, among other things, summarizes the technology objective of IRS's strategic plan, determines the extent to which the agency has updated plans for its IT modernization programs, and assesses IRS progress made on its modernization programs in fiscal years 2022 and 2023.GAO reviewed IRS's strategic operating plan and associated program documentation. In addition, GAO assessed IRS's progress on its reported modernization program cost and schedule performance for fiscal years 2022 and 2023. GAO also interviewed cognizant IRS officials.
- Artificial Intelligence: GAO's Work to Leverage Technology and Ensure Responsible Use What GAO FoundGAO is exploring internal use of artificial intelligence (AI) to make its work for Congress and taxpayers more efficient, in-depth, and effective. For example, GAO recently began deployment of a large language model to explore generative AI capabilities similar to those found in industry, supplemented with GAO-specific information and security. Among other capabilities, this prototype will also support other use cases to help efficiently synthesize past reports, assist with editorial review, and scan congressional documents for mandated work for GAO. By developing these tools, GAO is also gaining insight into the benefits and risks of AI, which will help GAO evaluate other agencies' use and better provide technical assistance to Congress. See the table for details on eight AI use cases GAO is currently exploring.List of Artificial Intelligence Use Cases GAO Is Exploring as of January 2024			Use case									Potential benefits								Organizes large volumes of text, such as public comments from Regulations.gov&nbsp;									Groups contents by similar themes			Prioritizes reviews based on relevant hierarchical topics								Summarizes draft GAO legislative mandates&nbsp;&nbsp;									Increases efficiency and reduces manual processes&nbsp;&nbsp;&nbsp;			Highlights potentially fragmented and duplicative mandates&nbsp;&nbsp;								Assists with copyediting according to GAO's style guide&nbsp;									Automates select copyediting tasks&nbsp;			Enables staff to focus on narrative structure and clarity&nbsp;								Provides automated responses to chat questions on published GAO work&nbsp;									Summarizes published GAO contents to enhance knowledge management and increase efficiency &nbsp;			Enhances specificity and accuracy of results&nbsp;								Summarizes qualitative responses from annual GAO Employee Experience Survey&nbsp;									Identifies trends, patterns, and sentiments quickly			Improves survey interpretation with less manual intervention			Assists with root-cause analyses								Monitors information about congressional committee calendars, press releases, and web contents&nbsp;&nbsp;									Matches congressional interests with relevant GAO work&nbsp;&nbsp;			Enhances timeliness of outreach and technical assistance&nbsp;&nbsp;								Enhances GAO auditing through use of extended reality glasses&nbsp;&nbsp;									Improves data collection and collaboration across locations&nbsp;			Reduces costs and risks&nbsp;			Enables new data capturing and analysis opportunities&nbsp;&nbsp;								Triages IT help desk requests and answers internal GAO policy questions									Provides 24/7 self-service assistance to GAO employees			Allows support staff to focus on more complex requests					Source: GAO. | GAO-24-107237Note: For the full list of use cases, updated periodically, see www.gao.gov/science-technology/artificial-intelligence-use-cases.AI also remains a top priority for many congressional committees and offices. GAO therefore has a robust body of work to meet Congress's need for information on AI's promise and risks. Since 2018, GAO has issued nearly 50 products on AI. It currently has 20 ongoing projects on AI with many more planned, including two technology assessments and one performance audit on generative AI. For example, in a landmark December 2023 report, GAO reviewed the implementation of AI at major federal agencies and made 35 recommendations to 19 agencies to fully implement federal AI requirements. In addition, in 2021, GAO published  Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities  , identifying 31 key practices to help ensure accountability and responsible use of AI. GAO's AI use cases are built upon this framework. The following is a selection of GAO AI-related products:Cybersecurity and Privacy Facial Recognition Services: Federal Law Enforcement Agencies Should Take Actions to Implement Training, and Policies for Civil Liberties.  GAO-23-105607. September 5, 2023.Facial Recognition: CBP and TSA are Taking Steps to Implement Programs, but CBP Should Address Privacy and System Performance Issues.  GAO-20-568. September 2, 2020.Facial Recognition Technology: Privacy and Accuracy Issues Related to Commercial Uses.  GAO-20-522. July 13, 2020.AI Opportunities and Challenges Artificial Intelligence in Natural Hazard Modeling: Severe Storms, Hurricanes, Floods, and Wildfires.  GAO-24-106213. December 14, 2023.Generative AI  (Science &amp; Tech Spotlight). GAO-23-106782. June 13, 2023.Digital Twins: Virtual Models of People and Objects.  (Science &amp; Tech Spotlight). GAO-23-106453. February 14, 2023.Federal Research and Management Artificial Intelligence: Agencies Have Begun Implementation but Need to Complete Key Requirements.  GAO-24-105980. December 12, 2023.Artificial Intelligence: DOD Needs Department-Wide Guidance to Inform Acquisitions.  GAO-23-105850. June 29, 2023.Artificial Intelligence: DOD Should Improve Strategies, Inventory Process, and Collaboration Guidance.  GAO-22-105834. March 30, 2022.Federal Workforce Artificial Intelligence: Actions Needed to Improve DOD's Workforce Management  GAO-24-105645. December 14, 2023.FAA Workforce: Better Assessing Employees' Skill Gaps Could Help FAA Prepare for Changes in Technology.  GAO-21-310. May 13, 2021.Automated Technologies: DOT Should Take Steps to Ensure Its Workforce Has Skills Needed to Oversee Safety.  GAO-21-197. December 18, 2020.Defense and Law Enforcement Information Environment: Opportunities and Threats to DOD's National Security Mission.  GAO-22-104714. September 21, 2022.Artificial Intelligence: Status of Developing and Acquiring Capabilities for Weapon Systems.  GAO-22-104765. February 17, 2022.Forensic Technology: Algorithms Strengthen Forensic Analysis, but Several Factors Can Affect Outcomes  . GAO-21-435SP. July 06, 2021.Why GAO Did This StudyAI, in general, refers to computer systems that can solve problems and perform tasks that have traditionally required human intelligence. Many federal agencies are using or planning to use AI, through applications such as facial recognition, automated email responses, and enhanced user experience on websites. At the same time, AI capabilities continue to expand in areas such as medical diagnostics, image generation, and emulating human writing styles.This testimony describes how GAO (1) plans to use AI internally to enhance effectiveness and efficiency across its mission and operations and (2) supports Congress through oversight of federal use of AI and foresight on its wider implementation.GAO performed the work on which this testimony is based in accordance with all applicable sections of GAO's Quality Assurance Framework.For more information, contact Taka Ariga at (202) 512-6888 or ArigaT@gao.gov.
- Federal Software Licenses: Agencies Need to Take Action to Achieve Additional Savings What GAO FoundTwenty-four federal agencies collectively identified 36 software vendors as those with the highest quantity of licenses installed, as of July 2022. Similarly, agencies reported 34 software vendors that were paid the highest amounts for fiscal year 2021 (see figure).Software Vendors with the Highest Amounts Paid Reported by Agencies for Fiscal Year 2021aThe 23 vendors shown as 0.87 percent are Broadcom, Computer Associates International, Entrust, ESCgov, FCN, Four, Intelligent Editing, LinkedIn, Mercom, MicroStrategy, NCS Technologies, Palantir Technologies, PKWARE, PTC, Quest Software, SAS Institute, Skillsoft, Splunk, Symantec, Thomson Reuters, Unison Software, Zoom Video Communications, and Zscaler.bThe two vendors shown as 1.74 percent are Environmental Systems Research Institute and Google.The most widely used and highest amounts paid for software products cannot be identified across the 24 agencies because agencies' license data for products were inconsistent and incomplete. For example, multiple software products within license agreements were not separately priced.Key activities for assessing the appropriate number of software licenses are (1) tracking licenses currently in use and (2) regularly comparing the inventory of software licenses currently in use to purchase records. Conducting such activities can help avoid purchasing too many licenses—referred to as over-purchasing—or purchasing too few licenses that may result in additional fees—referred to as under-purchasing. None of the nine agencies selected based on the size of their IT budgets—the Departments of Agriculture, Energy, Housing and Urban Development, Justice, State, Veterans Affairs; the Office of Personnel Management; Social Security Administration; and U.S. Agency for International Development—fully determined that their five most widely used software licenses were over- or under-purchased.Why GAO Did This StudyEach year, the federal government spends more than $100 billion on IT and cyber-related investments, including the purchase of software licenses. Federal agencies annually purchase thousands of software licenses from vendors.GAO was asked to review federal agency software licenses. Its objectives were to identify (1) the most widely used and the highest amounts paid for software licenses by vendor and product, and (2) the extent to which selected agencies determined whether they purchased too many or too few software licenses.GAO reviewed the 24 Chief Financial Officers Act of 1990 agencies' self-reported data on their five most widely used and highest amounts paid for software licenses. GAO then ranked these by the number of instances agencies cited specific vendors and products across the government. In addition, GAO selected nine of these agencies based on the size of their IT budgets for further review. GAO compared documentation for these agencies' five most widely used licenses to key activities identified in federal guidance. GAO also interviewed responsible officials at each of the 24 agencies.
- Artificial Intelligence in Natural Hazard Modeling: Severe Storms, Hurricanes, Floods, and Wildfires What GAO FoundGAO found that machine learning, a type of artificial intelligence (AI) that uses algorithms to identify patterns in information, is being applied to forecasting models for natural hazards such as severe storms, hurricanes, floods, and wildfires, which can lead to natural disasters. A few machine learning models are used operationally—in routine forecasting—such as one that may improve the warning time for severe storms. Some uses of machine learning are considered close to operational, while others require years of development and testing.GAO identified potential benefits of applying machine learning to this field, including:Reducing the time required to make forecasts by replacing components of models that are slow and that increase the cost of modeling.	Increasing model accuracy by more fully exploiting available data, using other data that traditional models cannot, and creating synthetic data to fill gaps.	Reducing the uncertainty of model output by improving ensemble modeling—the processes of generating combined predictions from numerous models—and making better use of historical data.Forecasting natural disasters using machine learningGAO also identified challenges to the use of machine learning. For example:Data limitations hamper the training of machine learning models and can reduce accuracy in some regions, such as rural areas where weather observations are sparse.	A lack of trust and understanding of the algorithms as well as concerns about bias can make forecasters and other users hesitant to use machine learning models.	Limited coordination and collaboration create challenges for fully developing some machine learning models. For example, some forecasters told us they lack opportunities to interact with researchers and convey their needs.	Workforce and resource gaps also create challenges. For example, the upfront costs to develop and run machine learning models are high, and some companies working on these models do not fully understand the data and phenomena they are modeling, according to academic researchers.GAO identified five policy options that could help address these challenges. These options are intended to inform policymakers, including Congress, federal and state agencies, academic and research institutions, and industry of potential policy implementations. The status quo option illustrates a scenario in which government policymakers take no additional actions beyond current ongoing efforts.Policy Options to Help Address Challenges to the Use of Machine Learning in Natural Hazard ModelingPolicy Option			Opportunities			Considerations		Facilitate improved data collection, sharing, and use (report p. 37).			Government policymakers could expand use of existing observational data and infrastructure to close gaps, expand access to certain data, and (in conjunction with other policymakers) establish guidelines for making data AI-ready.						Efforts to address gaps in data sets can improve machine learning model performance.				&nbsp;Expanded access to existing data would improve the ability of researchers and groups to develop and test machine learning technologies.				Adopting standards for AI-ready data could reduce resources needed to curate data and facilitate more efficient modeling.									Expanding observational infrastructure can be expensive and could divert limited resources from other efforts.				Agencies need to weigh the benefits of greater data sharing against any increase in risks related to data security and privacy.				Strict data standards may slow research and innovation if they burden or constrain machine learning researchers.					Expand education and training (report p. 38).			Government policymakers could update education requirements to include machine learning-related coursework and expand learning and support centers, while academic policymakers could adjust physical science curricula to include more machine learning coursework.						Updating education requirements would better prepare students to use machine learning in government.				More robust education can better prepare both researchers and end users in fields like meteorology and climatology to develop and use machine learning.									Education and training reforms may need to be repeatedly adjusted, as technological change in this space can be rapid and unpredictable.				Establishing and expanding professional development and training opportunities throughout government may require substantial investment.					Address hiring and retention barriers and certain resource shortfalls (report p. 39).			Government policymakers could address pay scale limitations for positions that include machine learning expertise and work with private sector policymakers to expand the use of public-private partnerships (PPP).						Providing workforce incentives to government employees for machine learning development could allow agencies to bolster workforce capacities.				Expanding PPPs might help agencies overcome computational resource shortfalls and help industry draw on government expertise.									Increasing salary limits for some employees would require agency budget increases or cuts to other budget items.				Expanding the use of PPPs could magnify resource disparities between government and private industry.				PPPs that entail hosting government data on collaborator systems may pose security risks that would need to be considered and addressed.					Take steps to mitigate bias and foster trust in data and machine learning models (report p. 40).			Policymakers could establish efforts to better understand and mitigate various forms of bias, support inclusion of diverse stakeholders for machine learning models, and develop guidelines or best practices for reporting methodological choices.						Sustained efforts to address bias in data sets can reduce the likelihood of models negatively affecting certain communities.				Acquiring diverse stakeholder perspectives throughout machine learning models’ life cycles can help reduce certain types of bias in data and models.				Fostering machine learning model transparency could improve end-user and decision-maker trust.									Embedding efforts to address bias throughout the model life cycle may increase model costs and slow model development.					Maintain status quo efforts (report p. 41).			Government policymakers could maintain existing policy efforts and organizational structures, along with existing strategic plans and agency commitments.						Some agency efforts are underway to address the challenges described.									The extent to which agencies will meet their commitments under status quo efforts is unclear. Status quo efforts may not address challenges specific to the use of AI in natural hazard modeling.					Source: GAO. | GAO-24-106213Why GAO Did This StudyNatural disasters cause on average hundreds of deaths and billions of dollars in damage in the U.S. each year. Forecasting natural disasters relies on computer modeling and is important for preparedness and response, which can in turn save lives and protect property. AI is a powerful tool that can automate processes, rapidly analyze massive data sets, enable modelers to gain new insights, and boost efficiency.This report on the use of machine learning in natural hazard modeling discusses (1) the emerging and current use of machine learning for modeling severe storms, hurricanes, floods, and wildfires, and the potential benefits of this use; (2) challenges surrounding the use of machine learning; and (3) policy options to address challenges or enhance benefits of the use of machine learning.GAO reviewed the use of machine learning to model severe storms, hurricanes, floods, and wildfires across development and operational stages; interviewed a range of stakeholder groups, including government, industry, academia, and professional organizations; convened a meeting of experts in conjunction with the National Academies; and reviewed key reports and scientific literature. GAO is identifying policy options in this report.For more information, contact Brian Bothwell at (202) 512-6888 or bothwellb@gao.gov.
- Artificial Intelligence: Actions Needed to Improve DOD's Workforce Management What GAO FoundThe Department of Defense (DOD) typically establishes standard definitions of its workforces to make decisions about which personnel are to be included in that workforce, and identifies its workforces by coding them in its data systems. DOD has taken steps to begin to identify its artificial intelligence (AI) workforce, but has not assigned responsibility and does not have a timeline for completing additional steps to fully define and identify this workforce. DOD developed AI work roles—the specialized sets of tasks and functions requiring specific knowledge, skills, and abilities. DOD also identified some military and civilian occupations, such as computer scientists, that conduct AI work. However, DOD has not assigned responsibility to the organizations necessary to complete the additional steps required to define and identify its AI workforce, such as coding the work roles in various workforce data systems, developing a qualification program, and updating workforce guidance. DOD also does not have a timeline for completing these additional steps. Assigning responsibility and establishing a timeline for completion of the additional steps would enable DOD to more effectively assess the state of its AI workforce and be better prepared to forecast future workforce requirements (see figure).Questions DOD Cannot Answer Until It Fully Defines and Identifies Its AI WorkforceDOD's plans and strategies address some AI workforce issues, but are not fully consistent with each other. Federal regulation and guidance state that an agency's Human Capital Operating Plan should support the execution of its Strategic Plan. However, DOD's Human Capital Operating Plan does not consistently address the human capital implementation actions for AI workforce issues described in DOD's Strategic Plan. DOD also uses inconsistent terms when addressing AI workforce issues, which could hinder a shared understanding within DOD. The military services are also developing component-level human capital plans that encompass AI and will cascade from the higher-level plans. Updating DOD's Human Capital Operating Plan to be consistent with other strategic documents would better guide DOD components' planning efforts and support actions necessary for achieving the department's strategic goals and objectives related to its AI workforce.Why GAO Did This StudyDOD has invested billions of dollars to integrate AI into its warfighting operations. This includes analyzing intelligence, surveillance, and reconnaissance data, and operating lethal autonomous weapon systems. DOD identified cultivating a workforce with AI expertise as a strategic focus area in 2018. However, in 2021 the National Security Commission on Artificial Intelligence concluded that DOD's AI talent deficit is one of the greatest impediments to the U.S. being AI-ready by the Commission's target date of 2025.House Report 117-118, accompanying a bill for the National Defense Authorization Act for Fiscal Year 2022, includes a provision for GAO to review DOD's AI workforce. This report evaluates the extent to which DOD has (1) defined and identified its AI workforce and (2) established plans and strategies to address AI workforce issues, among other objectives. GAO assessed DOD strategies and plans, reviewed laws and guidance that outline requirements for managing an AI workforce, and interviewed officials.
- Artificial Intelligence: Agencies Have Begun Implementation but Need to Complete Key Requirements What GAO FoundTwenty of 23 agencies reported about 1,200 current and planned artificial intelligence (AI) use cases—specific challenges or opportunities that AI may solve. Three agencies reported not having uses for AI. Agency reported uses included analyzing data from cameras and radar to identify border activities, analyzing photographs from drones, and targeting of scientific specimens for planetary rovers. The National Aeronautics and Space Administration (NASA) and the Department of Commerce (Commerce) reported the highest number of AI use cases (see figure).Agency Reported AI Use Cases in Fiscal Year 2022Most of the reported AI use cases were in the planning phase and not yet in production (i.e., currently used) (see figure). In about 200 instances, agencies reported that they were currently using AI.Agency Reported Artificial Intelligence (AI) Use Case Lifecycle Stage, as of Fiscal Year 2022a“Other” includes life cycle stage responses not included in the  2021 Guidance for Creating Agency Inventories of Artificial Intelligence Use Cases,  such as “research and design” and “exploratory.”GAO's analysis of agencies' inventories of use cases identified instances of incomplete and inaccurate data. Specifically, five agencies provided comprehensive information for each of their reported use cases while the other 15 had instances of incomplete and inaccurate data. For example, some inventories did not include required data elements, such as the AI life cycle stage or an indication of whether an AI use case was releasable or not. In addition, two inventories included AI uses that were later determined by the agencies to not be AI. Without accurate inventories, the government's management of its use of AI will be hindered by incomplete and inaccurate data.Federal agencies have taken initial steps to comply with AI requirements in executive orders and federal law; however, more work remains to fully implement these.Commerce and the General Services Administration (GSA) fully implemented selected requirements. Specifically, Commerce created a plan to develop AI technical standards and GSA established the AI Center of Excellence.	The Offices of Management and Budget (OMB) and Personnel Management (OPM) did not fully implement selected requirements. OMB has not yet developed guidance for the acquisition and use of AI. OPM has not yet established or updated an occupational category for those employees performing AI work.	The Office of Science and Technology Policy (OSTP) did not communicate its designation of which agencies were expected to fulfill specific AI requirements.	Ten of 23 agencies implemented all AI requirements specific to their agencies, 12 implemented some but not all, and one was exempt from the requirements. These requirements included preparing an inventory on the use of AI, planning for inventory updates, and planning for AI regulatory authorities.Addressing these requirements will improve agency identification, development, implementation, and oversight of AI.Why GAO Did This StudyWhile there are varying definitions of AI, they generally refer to computing systems that “learn” how to improve their performance. AI has the potential to rapidly change the world and holds substantial promise for improving government operations. However, AI poses risks that can negatively impact individuals, groups, organizations, communities, and society.The President's fiscal year 2023 budget request included $1.8 billion for nondefense research and development investment in AI. In addition, some agencies are using AI operationally to identify information security threats and facilitate the review of large datasets, among other uses. Given the rapid growth in capabilities and widespread adoption of AI, the federal government should have safeguards to manage AI's complexities, risks, and societal consequences.In this report, GAO reviewed the implementation of AI at major federal agencies. This report examines (1) federal agency reported current and planned uses of AI, (2) the extent to which federal agencies' AI reporting was comprehensive and accurate, and (3) the extent to which federal agencies have complied with selected federal policy and guidance on AI.This review focused on agencies with government-wide roles in AI implementation (including OMB and OPM) as well as agencies with individual responsibilities for AI implementation (including 23 of the 24 Chief Financial Officers (CFO) Act of 1990 agencies). The Department of Defense was excluded because GAO had issued recent AI reports on that department.GAO reviewed the 23 civilian CFO Act   agencies' AI inventories and analyzed reported use cases and supporting data. GAO also compared agency reported data to the requirements for developing an AI inventory outlined in federal guidance.In addition, GAO identified requirements from executive orders, OMB guidance, and a law regarding the implementation of AI. GAO then assessed agencies' implementation of these requirements. GAO also interviewed relevant officials.
- Information Technology: Federal Agencies Are Making Progress in Implementing GAO Recommendations What GAO FoundFor fiscal year 2024, 26 federal agencies plan to spend approximately $95 billion on about 6,700 IT investments, as reported on the IT Dashboard—a public website the Office of Management and Budget (OMB) launched in 2009. The $95 billion does not include about $37 billion of the Department of Defense's planned spending for unclassified national security systems, classified IT, or classified cyberspace. The IT investments on the dashboard were categorized by agencies in a variety of ways, such as by IT portfolio categories, investment types, and lifecycle costs. For example, according to agency-reported IT portfolio categories, agencies plan to spend about $45 billion on standard IT investments (i.e., defined by OMB as investments for technology goods and services common to all agencies such as IT infrastructure, security, and management), $34 billion on mission delivery, and $16 billion on mission support services. Additionally, of the approximately $95 billion reported on the dashboard, agencies plan to spend about $74 billion on operations and maintenance and $21 billion on development, modernization, and enhancement.In its December 2021 report, GAO highlighted 16 reports that made a total of 392 recommendations to 33 agencies to address issues related to duplicative IT and IT management roles and responsibilities. As of October 2023, agencies had implemented a total of 346 (88 percent) of the recommendations. Implementing the remaining 46 recommendations would better position agencies to oversee and effectively manage their IT development and acquisition efforts, reduce the risk of wasting federal funds and other resources on duplicative IT investments, and potentially save millions of dollars.In December 2021, GAO reported that although OMB's U.S. Digital Service (USDS) and the General Services Administration's (GSA) 18F programs had consistently coordinated on their IT projects and recruiting efforts, they had not consistently coordinated on the IT acquisition and development guidance they provided to agencies. GAO recommended that the two programs work with each other to establish and document an approach to coordinate on IT guidance provided to agencies.OMB and GSA implemented their recommendations by documenting an approach for coordinating with each other on the IT guidance that was being developed by their USDS and 18F programs. The process included monthly collaboration meetings between OMB and GSA officials to, among other things, share information on proposed USDS and 18F guidance and identify opportunities for addressing potentially duplicative guidance or collaborating jointly on the guidance. By taking these steps, USDS and 18F have reduced the risk of overlap or duplication of efforts, and the potential for presenting conflicting information in their guidance.Why GAO Did This StudyThe Modernizing Government Technology Act of 2017 included a provision for GAO to assess the number of IT procurement, development, and modernization programs, offices, and entities in the federal government, including USDS and 18F. The provision also called for GAO to assess the extent to which those entities duplicate work. GAO is to report on its work every 2 years.The specific objectives for this review were to (1) describe the procurement, development, and modernization investments identified on the IT Dashboard; (2) describe the implementation status of prior relevant GAO recommendations on IT duplication and management roles and responsibilities; and (3) describe the implementation status of prior GAO recommendations on USDS and 18F coordination on IT guidance to avoid duplication.To address the objectives, GAO analyzed investment data reported by all 26 agencies on the IT Dashboard, determined the implementation status of 392 GAO recommendations on duplicative IT and IT roles and responsibilities, and identified the actions OMB and GSA took to implement GAO recommendations made to the USDS and 18F programs.For more information, contact Kevin Walsh at (202) 512-6151 or walshk@gao.gov.
- Technology Modernization Fund: Although Planned Amounts Are Substantial, Projects Have Thus Far Achieved Minimal Savings What GAO FoundThe Technology Modernization Fund (TMF) awards funding to agencies to, among other things, modernize aging federal information systems. As of February 2023, TMF had received approximately $1.23 billion in appropriations, of which the Technology Modernization Board had awarded $636 million to 37 projects (see figure). Of the 37 awarded projects, eight have realized cost savings totaling $14.8 million; five of these anticipate further savings totaling $2.6 million. An additional 16 projects anticipate $738.6 million in combined savings. Thirteen projects neither anticipate nor have realized any cost savings.Technology Modernization Fund (TMF) Project Awards as of February 2023Of the seven projects that agencies had completed, two were on track with their savings (within the Office of Management and Budget's (OMB) 10 percent threshold), one was not on track, and one was anticipating cost avoidance rather than cost savings. For the remaining three projects, completion had not occurred within the prior 12 months. GAO determined that it would allow a full year of operations to occur before assessing savings.As required under the Competition in Contracting Act of 1984, all procurements, with certain exceptions, must use full and open competition so that any qualified entity can submit an offer. From September 2021 through February 2023, 45 of 52 (87 percent) contracts used competitive procedures that are considered as achieving full and open competition. The remaining seven contracts were not competed using full and open competition, but appropriately documented authorized exceptions.Why GAO Did This StudyThough agencies spend more than $100 billion annually on IT, the federal government has faced longstanding problems in its IT management. The statutory provisions commonly referred to as the Modernizing Government Technology Act established TMF to help address key challenges in modernizing federal information systems.The act includes a provision for GAO to report biennially on TMF. This third report (1) identifies the approved TMF projects, amounts of funds awarded, and estimated and actual cost savings; (2) determines the extent to which agencies' completed projects met their cost savings estimates; and (3) determines the extent to which agencies have used full and open competition for projects involving acquisitions of custom-developed IT.GAO analyzed and summarized TMF financial data and documentation associated with 37 projects awarded funds. In addition, GAO analyzed cost estimate documentation for these projects. For applicable completed projects, GAO compared the difference between estimated and realized cost savings to OMB's variance threshold of 10 percent.GAO also analyzed contract documentation for the 22 projects that issued contracts between September 2021 and February 2023. GAO determined whether each contract was awarded using full and open competition or exceptions were appropriately documented, in accordance with the Competition in Contracting Act of 1984 and the Federal Acquisition Regulation. GAO also interviewed relevant agency officials.For more information, contact Carol C. Harris at (202) 512-4456 or harriscc@gao.gov.
- Biometric Identity System: DHS Needs to Address Significant Shortcomings in Program Management and Privacy What GAO FoundSince rebaselining its original cost and schedule commitments in 2019, the Department of Homeland Security's (DHS) Homeland Advanced Recognition Technology (HART) program has further delayed its schedule. Specifically, in 2020 the program declared a second schedule breach and its first cost breach. Accordingly, DHS rebaselined the program again. This extended the schedule for delivering the initial capabilities to replace the legacy system by an additional 33 months beyond the 2019 plan. In addition, the 2022 rebaseline did not include an estimate for completing the program (see table).Changes in the Homeland Advanced Recognition Technology (HART) Program Schedule from 2019 to 2022			Milestone									Planned completion datea (as of May 2019)									Planned completion datea (as of May 2022)								Initial operational capability									December 31, 2020									September 30, 2023								Complete full program									June 30, 2024									Not yet planned					Source: GAO analysis of Department of Homeland Security data. | GAO-23-105959aThis represents the schedule threshold dates defined in the HART acquisition program baseline.Regarding costs, the program's 2022 rebaseline increased its estimated costs by $354 million. In April 2023, program officials stated that they needed to rebaseline HART's schedule a third time due to, among other things, higher than expected software defects and performance issues.The program's 2022 cost and schedule estimates did not fully follow GAO's identified cost and schedule best practices and were, therefore, unreliable. Specifically, the program's cost estimate did not substantially or fully meet the four characteristics of a reliable cost estimate. Moreover, the program's schedule estimate did not substantially or fully meet three of the four characteristics of a reliable schedule estimate. Until these weaknesses are addressed, the HART cost and schedule estimates will continue to be unreliable. In turn, this will impair the ability of senior leadership to make informed decisions regarding the program's future.DHS fully implemented five of 12 selected Office of Management and Budget privacy requirements. For example, the program addressed the requirement to appropriately encrypt information by demonstrating encryption settings for information at rest and in transit. However, DHS had gaps in the remaining seven requirements. For example, the program's privacy impact assessment, which is intended to analyze how personal information is collected, shared, and managed, was missing key information. Specifically, the assessment was missing information on (1) individuals whose data will be stored in the system and (2) the partners with whom the system will share information. In addition, the program did not have assurances that partners that provide information to the system will appropriately retain and dispose of personally identifiable information. Until DHS addresses these privacy weaknesses, the department lacks assurance that the hundreds of millions individuals' personally identifiable information that will be stored and shared by HART will be appropriately protected.Why GAO Did This StudyDHS currently uses an outdated system, implemented over 29 years ago, for providing biometric identity management services (e.g., fingerprint matching). The system stores over 290 million identities. In 2016, DHS initiated a multi-billion dollar program known as HART, which is intended to replace the legacy system. GAO previously reported that due to several challenges, in 2017 the program breached its schedule baseline. In 2019 the program established new cost and schedule commitments with DHS leadership (referred to as a rebaseline). This resulted in delaying the program by 3 years.GAO was asked to evaluate the HART program. This report's objectives were to (1) determine how the HART program has changed since the 2019 baseline, (2) assess the extent to which the program's cost and schedule estimates followed best practices, and (3) assess the extent to which DHS implemented selected privacy requirements for the program.GAO reviewed HART planning documentation, evaluated cost and schedule estimates against best practices identified by GAO, and compared privacy documentation to selected Office of Management and Budget privacy requirements. GAO also interviewed appropriate officials.
- DOD Software Licenses: Better Guidance and Plans Needed to Ensure Restrictive Practices Are Mitigated What GAO FoundOfficials from all three selected Department of Defense (DOD) components and two of the six selected investments described restrictive software license practices that impacted their cloud computing efforts. Officials from the selected components and investments stated that restrictive practices generally impacted the (1) cost of cloud computing, (2) choice of cloud service providers, and (3) other related impacts. The table provides examples of each of these types of impacts.Examples of Reported Restrictive Software License Practices by Selected Department of Defense (DOD) Components and Investments			Impact type									Impact description								Cost of cloud computing			&nbsp;									Infrastructure costs increased because vendors required additional fees to use their software with third party cloud service providers.								Licensing costs increased because a vendor bundled frequently used software with other software, making it available only at the bundled price.								Choice of cloud provider			&nbsp;									A vendor limited its use to only selected commercial cloud service providers.								A vendor required a specified cloud service provider.								Other			&nbsp;									Vendors required interoperability with a previous version of a different vendor's software, but that vendor does not allow customers to use the previous version unless they are using its cloud service platform.								A vendor may not help sustain a certain product if a customer is not using the specified cloud service provider.					Source: GAO analysis of data reported by selected Department of Defense components and investments. | GAO-23-106290Four of the six selected investments did not identify impacts from restrictive software licensing practices. According to officials, they may not have had impacts because these investments were configured to deploy software within the cloud instead of transferring software to the cloud.Key industry activities for managing the risk of impacts from restrictive practices include (1) identifying and analyzing impacts and (2) mitigating those impacts. However, the six selected investments GAO reviewed did not consistently address these key activities. Specifically, two investments identified an impact but did not analyze or develop plans for mitigating it, while four other investments did not address identifying, analyzing, or mitigating. The lack of relevant guidance allowed these shortfalls to occur. DOD's guidance and plans do not fully address identifying and analyzing the impacts of restrictive practices. Moreover, DOD's plans and guidance do not address mitigating impacts of restrictive practices. Until DOD updates and implements guidance and plans for managing the impacts of restrictive software licensing practices, the department will not be well-positioned to identify and analyze the impact of such practices or to mitigate the risks.Why GAO Did This StudyCloud computing enables on-demand access to shared computing resources. As DOD implements IT projects and migrates systems to the cloud, it may encounter restrictive software license practices. These practices include enterprise agreements or vendor processes that limit, impede, or prevent agencies' efforts to use software in cloud or multi-cloud computing.The House report accompanying the James M. Inhofe National Defense Authorization Act for Fiscal Year 2023 includes a provision for GAO to review the impact that restrictive software licensing practices could have on DOD cloud computing. The objectives of this review were to (1) describe how restrictive enterprise software licensing practices impact DOD cloud computing services and (2) evaluate the extent to which DOD is mitigating the potential impact of restrictive software licensing practices.GAO selected three DOD components (the Army, Air Force, and Navy) with the largest cloud budget requests for fiscal year 2023 and interviewed IT and acquisition officials from these components to describe the impacts of restrictive software licensing practices. GAO also selected six investments based on several factors, including IT budget size, and compared DOD documentation to key activities for mitigation identified by industry.
- Cloud Computing: DOD Needs to Improve Tracking of Data User Fees What GAO FoundData user fees (ingress and egress) are related to how users transfer and access data in a cloud environment. Data ingress is transferring data into the cloud and data egress is transferring data from the cloud. While data ingress is often free to users, cloud service providers generally charge data egress fees for transferring data out of storage (see figure).Figure: Transfer of Data into and out of the CloudThe Department of Defense (DOD) has begun to consider data egress fees when procuring and implementing cloud services. The department's recent contract negotiations with commercial providers resulted in discounts on data fees, including data egress fees. Vendor lock-in can happen in cloud computing when the cost of moving to a new provider is so high that a user stays with their incumbent provider. However, DOD officials stated that egress fees had not been a primary cause for vendor lock-in. These officials added that other factors could cause vendor lock-ins, including a lack of specific skills by government staff, or the reliance on cloud services unique to a specific cloud provider.DOD has mechanisms that could mitigate the impact data egress fees could have on DOD as it procures and implements cloud services across the department. DOD officials reported that data egress fees account for less than 1 percent of known cloud expenditures. However, the department does not have the capability to track and report on these fees. In addition, DOD's contract-specific tools do not track cloud expenditures, including data egress fees department-wide. DOD officials identified improved insight into cloud expenditures through recent department-wide contracts such as the Joint Warfighting Cloud Capability—a cloud contract with four commercial service providers—and other tools. However, DOD does not yet have a plan or time frame for adopting a tool that tracks data egress fees. Until DOD acquires and implements such a tool, it will continue to lack full insight into the impact of egress fees.Why GAO Did This StudyCloud computing enables agencies to have on-demand access to shared computing resources. The costs of doing so are often lower than if the agencies were maintaining the resources. In fiscal year 2022, major federal agencies obligated about $7 billion for cloud computing contracts, including approximately $3 billion by DOD. Cloud service providers charge users fees for transferring data out of the cloud—known as data egress fees. Committee reports from the Senate and House Armed Services Committees accompanying the James M. Inhofe National Defense Authorization Act for Fiscal Year 2023 include provisions for GAO to review cloud data egress fees at DOD, including their effects on vendor lock-in.This report determines the extent to which DOD (1) considered data egress fees when procuring and implementing cloud services and their potential for vendor lock-in and (2) mitigated the impact of data egress fees and tracked and reported on them. To assess DOD's cloud data egress fees, GAO analyzed relevant department guidance on cloud services and the tracking and reporting of cloud expenditures. It also reviewed supporting department documentation on cost reporting and tracking. In addition, GAO interviewed DOD officials.
- Leading Practices: Iterative Cycles Enable Rapid Delivery of Complex, Innovative Products Why This MattersAgencies are increasingly acquiring complex products, such as combined networks of hardware and software, which require new processes to design, produce, and deliver. GAO has found that to consistently deliver products with speed to users, acquisition programs for these networks—known as cyber-physical systems, such as aircraft and uncrewed vehicles—must adopt new approaches to evaluate performance and assess execution risks. Solutions, though, are unlikely to originate exclusively within government. Rather, identifying the practices that leading companies rely on to create cyber-physical products can provide crucial, cutting-edge information to acquisition leaders in government and, in turn, ultimately help frame changes to agencies’ acquisition processes.For over 20 years, GAO has made numerous recommendations to the Departments of Defense (DOD) and Homeland Security (DHS) and the National Aeronautics and Space Administration (NASA) to implement best practices for their major acquisition programs that underpin successful product development within leading companies. Over this time, agency implementation of these practices saved taxpayers tens of billions of dollars.Key TakeawaysLeading companies use iterative processes to design, validate, and deliver complex cyber-physical products with speed. Activities in these iterative cycles often overlap as the design undergoes continuous user engagement and testing. Knowledge about the product’s design is progressively refined and stored in a digital thread—a common source of information that helps stakeholders make decisions, like determining product requirements, throughout the product’s life.As they proceed, product teams refine the design to achieve a minimum viable product (MVP)—one with the initial set of capabilities needed for customers to recognize value. They use modern manufacturing tools and processes to produce and deliver the product in time to meet their customers’ needs.Leading Companies Progress through Iterative Cycles to Develop a Minimum Viable ProductSimilar to Agile software development, the iterative structure leading companies employ when developing complex, cyber-physical products revolves around companies rapidly designing, validating, and delivering products. These cycles are:Design modeling and simulation:&nbsp;During design modeling and simulation, product teams feed technical data from fast, iterative design cycles into the digital thread. Stakeholders—including users, engineers, and manufacturers and suppliers—use this information to confirm that the team has captured the right requirements and is on track to meet them.	&nbsp;Digital Thread Captures Information throughout the Product Life CycleFor example, one leading company establishes models based on high-quality data from physical engineering, and captures input from users and manufacturers to feed into the models. Another leading company collaborates with users to identify the one thing that will differentiate the product from others, which then functions as an indicator of product performance. The outcome of design modeling and simulation is a solution—in the form of an MVP—that can then be validated through testing.Validation: Leading companies validate the design using prototypes—including combinations of physical and entirely digital prototypes. This prototyping incorporates all hardware and software components to test the product’s integrated functionality in its operating environment. Sometimes companies do this by developing virtual representations of physical products—a process known as digital twinning—or by using 3D-printed parts to test performance.	For instance, one leading company uses a digital twin and virtual reality—immersion in a virtual environment using head-mounted displays—to enable users to step into a virtual vehicle and validate its design. Another leading company used a digital twin of an industrial motor drive to simulate its overload to the point of explosion. Compared with a physical test, which would have destroyed the prototype, developers could observe the specific point of explosion, locate defects, and fix them in the digital twin.	Production and Delivery: Leading companies develop the design to the point that it satisfies user requirements for a MVP. Product teams then stop designing hardware for the given MVP and prepare parts for production, recognizing that they can add functionality through software updates later. Companies use digital twinning to understand optimal factory design and manufacturing processes before the design enters production.	For example, one leading company uses digital twins to align parts assembly processes to ensure the robot handling the part can do so efficiently. The digital thread documents all the steps throughout production, from the design of the machinery and toolset, to the processes for manufacturing and assuring the product meets the company’s quality standards. Post-production user feedback informs further development of the next product.How GAO Did This StudyThree congressional mandates provide for GAO to annually assess selected DOD, DHS, and NASA acquisition programs, projects, and activities. In order to respond effectively to these mandates, GAO conducted this work to understand (1) how leading companies structure their development of complex, cyber-physical products; and (2) the specific practices that enable this structure to properly function.GAO identified 14 leading product development companies based on rankings in well-recognized lists; interviewed company representatives; reviewed supplementary documentation; and synthesized information to determine key product development structures and activities.
- Defense Software Acquisitions: Changes to Requirements, Oversight, and Tools Needed for Weapon Programs What GAO FoundThe Department of Defense's (DOD) ability to respond to evolving threats is increasingly determined by its ability to rapidly develop and deploy software. Efforts to modernize DOD's software acquisition include a transition to Agile development principles from a traditional waterfall approach. Agile is intended to deliver working software to users in less than a year and add capability iteratively based on user needs. In contrast, the waterfall approach could take over 10 years to deliver software and involves greater risk.Comparison of Waterfall and Agile Methods for Developing SoftwareIn 2020, DOD established six acquisition pathways—or sets of policy and guidance—that are tailored to the type of capabilities being acquired. DOD requires programs on its software pathway to use requirements processes tailored to support Agile development. For example, software pathway programs are to use streamlined requirements documents and develop user agreements, which help ensure programs iteratively develop software aligned with user needs. However, DOD requirements processes used by weapon programs developing software on a different pathway generally do not incorporate Agile principles. By not incorporating Agile principles into requirements processes, these programs risk developing capabilities that may not reflect changing user needs or threats.DOD developed policies and guidance for oversight of programs on the software pathway, including using metrics and value assessments that measure Agile development outcomes. However, DOD has not issued corresponding policies or guidance for weapon programs using Agile software development on other pathways. As a result, programs on other acquisition pathways, such as those developing new aircraft or ships, may not be positioned to conduct effective oversight of iteratively delivered software capabilities.The iterative and fast-paced nature of Agile relies on the use of modern engineering tools to help identify needs, develop capabilities, and understand outcomes. DOD, however, has not developed a plan or identified resources to enable the adoption of modern engineering tools across all programs. The absence of a plan prevents DOD from fully realizing the benefits of Agile.Why GAO Did This StudyCongress included a provision in statute for GAO to examine DOD's software modernization efforts. This report assesses the extent to which DOD has (1) policy and guidance that establish requirements processes to support the use of Agile software development in weapon programs; (2) policy and guidance to provide direction for overseeing Agile software development in weapon programs; and (3) enabled program adoption of modern engineering tools for Agile.GAO reviewed DOD's software acquisition and requirements policies and guidance, and GAO's leading practices in Agile and iterative development. GAO selected a nongeneralizable sample of five weapon programs based on their use of Agile, and other factors. GAO also interviewed relevant DOD officials, program officials, and user representatives.
- Nursing Homes: CMS Offers Useful Information on Website and Is Considering Additional Steps to Assess Underlying Data What GAO FoundGAO found that the nursing home quality information the Centers for Medicare &amp; Medicaid Services (CMS) provides on Care Compare—which was developed to assist consumers in selecting a nursing home—at least partially aligns with 11 of 15 characteristics of understandability and relevancy for an effective transparency tool (see figure). These characteristics include providing descriptions of key differences in clinical quality of care, enabling consumers to customize information, and comparing multiple nursing homes. However, GAO also found Care Compare did not align with four of the characteristics. GAO has made recommendations in the past that apply to some of these characteristics, and for others, CMS told GAO about ongoing efforts to address some of the characteristics GAO identified. For example, CMS officials told GAO they are exploring a way to include information on nursing home residents' experience, which Care Compare currently lacks.CMS takes various approaches to assess the completeness and accuracy of the sources for the staffing data, inspections, and quality measure information it provides on Care Compare. For example:Staffing data.  CMS has an automated process to check the completeness of staffing data nursing homes report.Inspection data.  CMS uses the Federal Monitoring Survey, which, among other things, replicates recent state inspections to check the completeness and accuracy of inspection results.Quality measures.  CMS has an automated process to check the completeness of the data nursing homes report that inform the measures.CMS also has processes to assess accuracy, and has recently taken additional steps for one of these measures related to antipsychotic drugs. CMS may audit the nursing home's documentation supporting this measure. CMS recently proposed a validation process for some additional quality measure data represented on Care Compare, though not the rating. According to CMS officials, the agency is exploring to what extent the agency will expand these audits to data that inform the quality measure rating.Why GAO Did This StudyCMS is responsible for ensuring that participating nursing homes meet federal standards, and provides information on nursing homes' quality on its web-based tool, Care Compare. CMS's Five-Star Rating System, which includes an overall and three component star ratings—health inspection, staffing, and quality measures—is a prominent source of information on Care Compare. CMS has worked to improve its usefulness to consumers, the quality of the underlying information, and aspects of the rating system.GAO was asked to examine the nursing home section of Care Compare and the Five-Star Rating System. Among other issues, this report (1) examines the understandability and relevance of nursing home quality information on Care Compare, and (2) describes CMS's assessment of underlying information.GAO reviewed nursing home information on Care Compare against 15 characteristics of understandability and relevancy for effective transparency tools that were identified in previous GAO work. GAO also reviewed CMS's assessment of three of the primary sources of nursing home information for Care Compare: staffing data, inspections, and quality measures.
- IT Systems Annual Assessment: DOD Needs to Improve Performance Reporting and Development Planning What GAO FoundAccording to the Department of Defense's (DOD) fiscal year (FY) 2023 submission to the Federal IT Dashboard, DOD planned to spend about $9 billion on its portfolio of 25 major IT business programs and about $31 billion on its 723 standard IT infrastructure investments from FY 2021 through FY 2023. These two areas accounted for 30 percent of total planned spending on the department's unclassified IT portfolio (see figure).The Department of Defense's Major IT Programs and IT Infrastructure Accounted for 30% of Total Planned Spending on Its Unclassified IT for Fiscal Years 2021–2023Sixteen of the 25 major IT business programs reported cost or schedule changes since January 2021, including 12 that had cost increases ranging from $43 thousand to $194 million (a median of $4.6 million); 12 had schedule delays ranging from 3 to 33 months (a median of 24 months). Program officials attributed the changes to factors such as new requirements and unanticipated technical complexities.Programs also reported performance data. As of January 2023, 22 of the 25 programs identified at least the minimum required number of operational performance metrics, consistent with Office of Management and Budget (OMB) guidance. However, the other three programs did not identify the minimum required metrics, including two that did not identify any metrics data. Additionally, 13 programs did not fully report on the extent to which they achieved their targets. By not ensuring that programs fully identify and report required performance metrics, DOD limits program accountability and its own ability to effectively oversee performance.As of February 2023, officials for the eight programs that we identified as actively developing software reported using recommended iterative development approaches and practices that can limit risks of adverse cost and schedule outcomes. In addition, five of the eight programs reported delivering software functionality every 6 months or less as called for in OMB guidance (see table).Department of Defense Major IT Business Programs Actively Developing Software Reported Using Iterative Development Approaches and Practices			Development approach or practice 									Number of programs that reported using each approach or practice 								Uses an iterative development approach									8 of 8								Uses Agile as an approach									6 of 8								Delivery of minimum viable product									7 of 8								Delivery of software at least every 6 months									5 of 8					Source: GAO analysis of Department of Defense program questionnaire responses, as of February 2023.| GAO-23-160117Moreover, recognizing the importance of user involvement throughout the software development process, officials for all eight programs in active development reported involving users through collecting feedback during requirements development and refinement. In addition, most of the 25 major IT business programs in various stages reported involving users through testing and surveying them about customer experience (see table).Department of Defense Major IT Business Programs in Various Stages of Development Reported Conducting Activities to Involve Users			User involvement activity 									Number of programs that reported conducting each activity 								Collecting user feedback during development									8 of 8								Involving users in testing									23 of 25								Surveying users about customer experience									20 of 25					Source: GAO analysis of Department of Defense program questionnaire responses, as of February 2023. | GAO-23-160117However, as of February 2023, 11 of the 25 programs did not demonstrate having approved plans for conducting user training and deployment as required by DOD. Program officials provided various reasons for not having the plans, including the system nearing retirement or predating the requirement. However, DOD officials acknowledged that programs should have user training and deployment plans and stated that they will follow up with the programs that did not have them. Without such plans, the department is at increased risk of programs not achieving required organizational changes and delivering business systems that do not meet their users' needs and are not widely adopted by users.Further, while program officials reported conducting cybersecurity assessments and tests, six programs did not demonstrate having an approved cybersecurity strategy as required. In June 2022, GAO reported that 10 of DOD's major IT business programs did not have approved strategies and recommended the DOD Chief Information Officer (CIO) ensure programs develop them. The department concurred with the recommendation and, as of March 2023, officials stated that they were following up with the programs that did not have one. Until the department ensures that all programs develop strategies, it lacks assurance that programs are positioned to effectively manage cybersecurity risks and mitigate threats. As a result, DOD programs are at increased risk of adverse cost, schedule, and performance impacts.Regarding legislative and policy changes, DOD has taken actions to implement the National Defense Authorization Act (NDAA) for FY 2021, which eliminated the DOD Chief Management Officer (CMO) position. This position previously had broad oversight responsibilities for the department's business systems. In September 2021, the Deputy Secretary of Defense directed an extensive realignment of the responsibilities previously assigned to the CMO. In March 2023, GAO reported on DOD's oversight of its business systems and recommended that DOD update guidance for addressing statutory requirements for initially approving and annually certifying business systems and maintain complete and accurate data for its systems, among other things. The department has efforts underway to implement changes, including plans to issue revised business systems investment management guidance. GAO will continue to monitor DOD's efforts to redistribute the roles and responsibilities formerly assigned to the CMO and to improve how the department manages its IT investments.Why GAO Did This StudyFor FY 2023, DOD requested approximately $45.2 billion for its unclassified IT investments, encompassing essential infrastructure, communications, and business systems. This includes the department's major IT programs, which are intended to help sustain key business operations such as contracting, logistics, human resources, and financial management.The NDAA for FY 2019, as amended, includes a provision for GAO to assess selected DOD IT programs annually through March 2026. GAO's objectives were to (1) examine how DOD's portfolio of major IT business programs has performed, (2) determine the extent to which DOD has implemented key software development and cybersecurity practices for selected programs, and (3) describe actions DOD has taken to implement legislative and policy changes that could affect its IT acquisitions.To address these objectives, GAO selected the 25 major IT business programs DOD reported in its FY 2023 submission to the Federal IT Dashboard (a public website with information on the performance of IT investments). GAO analyzed the Dashboard data to examine DOD's planned expenditures for these programs and for its standard IT infrastructure (the supporting hardware, software, and services that a business system requires to operate) from FY 2021 through FY 2023. GAO compared programs' operational performance data to OMB guidance. GAO also met with DOD OCIO officials to determine reasons for differences between how metrics data were reported and reporting guidance.In addition, GAO administered a questionnaire to the 25 program offices to obtain information about cost and schedule changes that the programs had experienced since January 2021. The questionnaire also sought information about software development and cybersecurity practices used by the programs, including whether users were involved during the development process. GAO compared the responses to relevant guidance and leading practices to identify gaps and risks. For programs that did not demonstrate having plans, strategies, or other comparable documents, GAO followed up with DOD officials for clarification.Further, GAO reviewed actions DOD has taken to implement its plans for addressing previously identified legislative and policy changes that could affect its IT acquisitions. This included reviewing policy, plans, and guidance associated with the department’s efforts to (1) reorganize former CMO responsibilities and (2) implement changes associated with its defense business systems investment management guidance and business enterprise architecture. GAO met with DOD officials to discuss each of the topics addressed in this report.
- Information Technology: DHS Needs to Continue Addressing Critical Legacy Systems What GAO FoundGAO reported in June 2019 that, of the 65 critical legacy IT systems identified by federal agencies as needing modernization, the Department of Homeland Security (DHS) had three such systems (see table). Further, GAO identified DHS's System 4 as one of the 10 most critical legacy systems across the federal government in need of modernization.Table: Critical Legacy Systems in Need of Modernization According to DHS, as of June 2019			System namea									Age of system, in years									Age of oldest hardware, in years									System criticality (according to DHS)									Security risk (according to DHS)								System 4									8-11									11									High									High								System L									9									2									High									Moderately low								System M									6									1									High									Low					Source: GAO analysis of Department of Homeland Security (DHS) data. | GAO-23-106853aDue to sensitivity concerns, GAO substituted alphanumeric identifiers for the names of the agencies' systems. GAO assigned a number to identify each of the 10 most critical legacy systems in need of modernization and assigned a letter to identify the remaining 55 systems. The identifiers in the table reflect how DHS's system names appeared in the 2019 report (GAO-19-471).In evaluating agencies' modernization plans for the 10 most critical legacy systems, GAO determined that DHS lacked a complete plan for modernizing System 4. Specifically, DHS's plan did not include milestones to complete the modernization and did not describe the planned disposition of the existing legacy system. In February 2022, DHS provided an updated modernization plan that included milestones for replacing the system and removing legacy hardware, which addressed our recommendation. By documenting its plan in sufficient detail, DHS increased the likelihood that the modernization will succeed.GAO has also previously reported on DHS's efforts to modernize and replace other legacy systems that support financial management, biometric identity management, and grants management. Specifically,In February 2023, GAO noted that, after attempting to modernize its financial management systems for decades, DHS implemented a governance structure to oversee component-level financial systems modernizations. However, the Coast Guard was unable to declare full operational capability as expected because it had not remediated issues from operational testing.	In June 2021, GAO reported that DHS's Homeland Advanced Recognition Technology program (intended to replace an outdated system for biometric identity management) was significantly behind schedule and had exceeded its estimated costs. GAO also found that DHS had not fully addressed key risk management and IT acquisition practices.	In April 2019, GAO reported that the Federal Emergency Management Agency's Grants Management Modernization program (intended to replace 10 legacy systems) had not fully addressed leading practices for business process reengineering, requirements, and cybersecurity risk management. The program also did not meet leading practices for a reliable schedule.DHS has now implemented 11 of the 19 recommendations GAO made in these reports. Implementing the remaining eight will help the department ensure these critical legacy systems are successfully replaced.Why GAO Did This StudyEach year, the federal government spends more than $100 billion on IT and cyber-related investments. Of this amount, agencies have typically reported spending about 80 percent on operations and maintenance of existing IT, including legacy systems. DHS's expected IT spending for fiscal year 2023 is about $10.1 billion; operations and maintenance is expected to consume about $8.8 billion of that total.Maintaining legacy systems (i.e., systems that are outdated or obsolete) can pose significant challenges. GAO reported in 2016 that agencies had system components that were at least 50 years old and vendors that were no longer providing support for hardware or software. In 2019, GAO reported that several critical federal legacy systems used outdated languages, had unsupported hardware and software, and were operating with known security vulnerabilities.GAO was asked to testify on its past legacy system reports and DHS's efforts to modernize. Specifically, GAO summarized (1) DHS's critical legacy IT systems and plan for modernizing and (2) progress and challenges with selected DHS IT modernizations. This statement is based on issued GAO reports and updated information on the department's implementation of GAO's recommendations.
- Electronic Health Records: VA Needs to Address Management Challenges with New System What GAO FoundThe Department of Veterans Affairs (VA) organizational change management activities for the Electronic Health Record Modernization (EHRM) program were partially consistent with seven leading practices and not consistent with one leading practice (see table).Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities Were Consistent with Organizational Change Management Leading Practices			Leading practice 									GAO assessment 								Developing a vision for change									Partially consistent								Identifying and managing stakeholders									Partially consistent								Communicating effectively									Partially consistent								Assessing the readiness for change									Partially consistent								Increasing workforce skills and competencies									Not consistent								Identifying and addressing potential barriers to change									Partially consistent								Establishing targets and metrics for change									Partially consistent								Assessing the results of change									Partially consistent					Source: GAO analysis of Department of Veterans Affairs (VA) data. | GAO-23-106731Until the program fully implements the leading practices for change management, future deployments risk continuing change management challenges that can hinder effective use of the new electronic health record (EHR) system.Most users have expressed dissatisfaction with the new system. VA's 2021 and 2022 surveys showed that users were not satisfied with the system's performance or training. About 79 percent (1,640 of 2,066) of users disagreed or strongly disagreed that the system enabled quality care. In addition, about 89 percent (1,852 of 2,074) of users disagreed or strongly disagreed that the system made them as efficient as possible. Further, VA has not established targets (i.e., goals) to assess user satisfaction. Until it does so, VA lacks a basis for determining when satisfaction has sufficiently improved for the system to be deployed at additional sites. Such a basis helps ensure that the system is not deployed prematurely, which could risk patients' safety.VA did not adequately identify and address system issues. Specifically, VA did not ensure that trouble tickets for the new EHR system were resolved within timeliness goals. It subsequently worked with the contractor to reduce the number of tickets that were over 45 days old. Nevertheless, the overall number of open tickets has steadily increased since 2020. Accordingly, it is critical that system issues be resolved in a timely manner. Additionally, although VA has assessed the system's performance at two sites, as of January 2023, it had not conducted an independent operational assessment, as originally planned and consistent with leading practices for software verification and validation. Without such an independent assessment, VA will be limited in its ability to (1) validate that the system is operationally suitable and effective, and (2) identify, track, and resolve key operational issues.In April 2023, VA announced that it planned to halt future deployments of the new EHR system to focus on making improvements at the five sites currently using the system.Why GAO Did This StudyVA uses the Veterans Health Information Systems and Technology Architecture (VistA), which includes the department's legacy EHR system, to manage health care for its patients. VistA is technically complex, costly to maintain, and does not fully support the need to exchange health data with other organizations. In June 2017, VA initiated the EHRM program to replace VistA.Congressional report language associated with the VA appropriations for fiscal years 2020 through 2022 contained provisions for GAO to review VA's EHR deployment. GAO's objectives were to determine the extent to which VA has (1) followed leading organizational change management practices for the EHRM program, (2) assessed satisfaction with the new system, and (3) identified and addressed EHR system issues. GAO identified leading change management practices and evaluated VA's activities against these practices. It also reviewed the results of surveys that VA conducted to determine users' satisfaction with the new EHR, conducted interviews with selected users, and interviewed officials on user satisfaction goals. Further, GAO analyzed VA's data on the contractor's performance meeting time frames for addressing system trouble tickets.
- Information and Communications Technology: DOD Needs to Fully Implement Foundational Practices to Manage Supply Chain Risks What GAO FoundThe Department of Defense (DOD) has fully implemented four and partially implemented three of seven selected foundational practices for managing information and communications technology (ICT) supply chain risks (see figure). These risks include threats posed by counterfeiters who may exploit vulnerabilities in the supply chain. Supply chain risk management is the process of identifying, assessing, and mitigating the risks associated with the global and distributed nature of ICT product and service supply chains.Assessment of the Department of Defense's (DOD) Implementation of Selected Foundational Information and Communications Technology (ICT) Supply Chain Risk Management PracticesBy fully implementing four of the foundational practices, DOD has taken steps to mitigate potential threats and secure its ICT supply chain. Regarding the three partially implemented practices, the department has begun several efforts that are not yet complete. For example, the department has developed a risk management strategy but has not approved guidance for implementing it. DOD has also piloted the use of several tools to review potential suppliers but the review of the results is ongoing. However, DOD did not specify time frames for when these actions would be completed. Fully implementing the three remaining practices would enhance the department's understanding and management of supply chain risks.DOD provided leadership and support for several government-wide efforts to protect the ICT supply chain. For example, the department offered a course and assisted small businesses in protecting their supply chains. Additionally, the department developed an action plan to facilitate cyber threat sharing and briefed a federal acquisition community of practice on performing cyber test and evaluations. DOD also shared ICT supply chain responsibilities as a member of the Federal Acquisition Security Council. Further, the council has the authority to issue exclusion orders to prevent purchasing from suppliers that may be compromised.Why GAO Did This StudyFederal agencies rely extensively on ICT products and services (e.g., computing systems, software, and networks) to carry out their operations. However, agencies face numerous ICT risks that can compromise the confidentiality, integrity, or availability of an organization's systems and the information they contain.Senate Report 117-39 accompanying the Fiscal Year 2022 National Defense Authorization Act included a provision for GAO to provide an assessment of DOD's efforts to address ICT supply chain risks. The specific objectives for GAO's report were to (1) assess the extent to which DOD is implementing foundational ICT supply chain risk management practices and (2) describe the extent to which DOD is leading or supporting government-wide efforts to protect the ICT supply chain.GAO compared the department's policies, procedures, and related documentation to seven foundational practices. These practices are based on National Institute of Standards and Technology guidance for ICT risk management. In addition, GAO analyzed documentation describing DOD's efforts to lead or support government-wide efforts to protect its supply chains. GAO also interviewed relevant agency officials.
- Information Technology: Agencies Need to Continue Addressing Critical Legacy Systems What GAO FoundIn June 2019, GAO identified 10 critical federal IT legacy systems (i.e., systems that are outdated or obsolete) that were most in need of modernization. These legacy systems provided vital support to agencies' missions. According to the agencies, these legacy systems ranged from about 8 to 51 years old and collectively cost about $337 million annually to operate and maintain. Several of the systems used older languages, such as Common Business Oriented Language (COBOL). GAO has previously reported that reliance on such languages has risks, such as a rise in procurement and operating costs, and a decrease in the availability of individuals with the proper skill sets. Further, several of the legacy systems were operating with known security vulnerabilities and unsupported hardware and software.Of the 10 agencies responsible for these legacy systems, GAO reported in June 2019 that eight agencies either did not have documented plans for modernizing their systems or had incomplete plans. Agency plans were incomplete if they were missing any of the key elements: (1) milestones to complete the modernization, (2) a description of the work necessary to modernize the legacy system, and (3) details regarding the disposition of the legacy system.Six of those eight agencies have implemented GAO's recommendations to identify and document modernization plans for their respective legacy systems. However, as of May 2023, two agencies (the Department of Transportation and the Office of Personnel Management) have not developed complete modernization plans (see table). Developing such plans is essential to addressing mission needs, dealing with security risks, and reducing operating costs.Table: Extent to Which Selected Agencies Had Documented Modernization Plans for Legacy Systems			Agency									Had modernization plan with key elements, as of June 2019?									Has addressed incomplete elements of modernization plan, as of May 2023?								Department of Transportation									No. Agency did not have a documented modernization plan.									No. In April 2022, agency officials informed GAO that they expected to go live with the modernized system in the fall of 2022; however, as of May 2023, GAO has not received documented plans for this modernization effort.								Office of Personnel Management									Partial. Agency had a modernization plan but it did not fully include milestones or work necessary, and it did not include the disposition of the legacy system.									No. As of May 2023, GAO has not received evidence that the agency has developed a comprehensive modernization plan for this system.					Source: GAO analysis of agency modernization plans. | GAO-23-106821Implementing GAO's prior recommendation to the Office of Management and Budget (OMB) on finalizing guidance directing agencies to identify systems needing modernization is essential. While OMB had drafted such guidance, it has not yet been issued. Doing so would provide greater assurance that the risks of continuing to operate legacy systems are being addressed government-wide.Why GAO Did This StudyEach year, the federal government spends more than $100 billion on IT and cyber-related investments. Of this amount, agencies have typically reported spending about 80 percent on operations and maintenance of existing IT, including legacy systems.Maintaining federal legacy systems can pose significant challenges. In May 2016, GAO reported instances where agencies had systems with components that were at least 50 years old or vendors that were no longer providing support for hardware or software. Similarly, in June 2019, GAO reported that several of the federal government's most critical legacy systems used outdated languages, had unsupported hardware and software, and were operating with known security vulnerabilities.This statement is based primarily on GAO's 2019 report on federal agencies' legacy systems. GAO summarized the (1) critical federal legacy systems identified as most in need of modernization and (2) status of agencies' plans for modernizing them. It also analyzed updated information on agencies' implementation of GAO's recommendations, and summarized other relevant legacy systems reports.
- Electronic Health Records: Challenges with VA's New System Call for Management Improvements What GAO FoundThe Department of Veterans Affairs (VA) organizational change management activities for the Electronic Health Record Modernization (EHRM) program were partially consistent with seven leading practices and not consistent with one leading practice (see table).Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities Were Consistent with Organizational Change Management Leading Practices			Leading practice 									GAO assessment 								Developing a vision for change									Partially consistent								Identifying and managing stakeholders									Partially consistent								Communicating effectively									Partially consistent								Assessing the readiness for change									Partially consistent								Increasing workforce skills and competencies									Not consistent								Identifying and addressing potential barriers to change									Partially consistent								Establishing targets and metrics for change									Partially consistent								Assessing the results of change									Partially consistent					Source: GAO analysis of Department of Veterans Affairs (VA) data. | GAO-23-106785Until the program fully implements the leading practices for change management, future deployments risk continuing change management challenges that can hinder effective use of the new electronic health record (EHR) system.Most users expressed dissatisfaction with the new system. VA's surveys showed that users were not satisfied with the system's performance and training. About 79 percent (1,640 of 2,066) of responding users disagreed or strongly disagreed that the system enabled quality care. In addition, users in GAO's structured interviews said the pharmacy-related system module presented challenges, such as incorrect medication orders and increased prescription processing times, which contributed to patient safety risks. Further, VA has not established targets (i.e., goals) to assess user satisfaction. Until it does so, VA lacks a basis for determining when satisfaction has sufficiently improved for the system to be deployed at additional sites. Such a basis helps ensure that the system is not deployed prematurely, which could risk patients' safety.VA did not adequately identify and address system issues. Specifically, VA did not ensure that trouble tickets for the new EHR system were resolved within timeliness goals. It subsequently worked with the contractor to reduce the number of tickets that were over 45 days old. Nevertheless, the overall number of open tickets has steadily increased since 2020. Accordingly, it is critical that system issues be resolved in a timely manner. Additionally, although VA has assessed the system's performance at two sites, as of January 2023, it had not conducted an independent operational assessment, as originally planned and consistent with leading practices for software verification and validation. Without such an independent assessment, VA will be limited in its ability to validate that the system is operationally suitable and effective.Subsequent to GAO's review, VA announced that it planned to halt future deployments of the new EHR system to focus on making improvements at the five sites currently using the system.Why GAO Did This StudyVA uses the Veterans Health Information Systems and Technology Architecture (VistA), which includes the department's legacy EHR system, to manage health care for its patients. VistA is technically complex, costly to maintain, and does not fully support the need to exchange health data with other organizations. In June 2017, VA initiated the EHRM program to replace VistA. VA has reported obligating about $9.42 billion on EHRM from fiscal year 2018 through the first quarter of fiscal year 2023.GAO was asked to testify on its recent review to determine the extent to which VA has (1) followed leading organizational change management practices for the EHRM program, (2) assessed satisfaction with the new system, and (3) identified and addressed EHR system issues. For its recent review GAO identified leading change management practices and evaluated VA's activities against these practices. It also reviewed results of surveys that VA conducted to determine users' satisfaction with the new EHR, conducted interviews with selected users, and interviewed officials on user satisfaction goals. Further, GAO analyzed system trouble ticket and related data regarding VA's service level agreement with its contractor.
- Electronic Health Information Exchange: Use Has Increased, but Is Lower for Small and Rural Providers What GAO FoundElectronic health information exchange is the ability to exchange medical records and other health information electronically among health care providers and between health care providers and patients. The Health Information Technology for Economic and Clinical Health (HITECH) Act provided federal enhanced Medicaid matching funds to states through 2021 to support certain efforts to advance electronic exchange. Nearly all states used these funds, and most have identified other sources to sustain those efforts.Survey data show that the use of various electronic exchange methods among hospitals and physicians has increased in recent years. However, GAO found that as of 2021, reported use among small and rural hospitals was lower than that of other hospitals. For example, see figure illustrating use by size of hospital.Exchange Methods Often Used among Acute Care Hospitals by Size, 2021Stakeholders GAO interviewed noted that small and rural providers were less likely to have the financial and technological resources to participate in or maintain electronic exchange capabilities.Federal efforts may address some impediments to electronic health information exchange. Specifically, the Trusted Exchange Framework and Common Agreement being implemented by the Office of the National Coordinator for Health Information Technology (ONC)—which aims to describe a common set of nonbinding principles to help facilitate exchange among health information networks—may mitigate costs providers face by providing a simpler approach to connecting with other providers. However, stakeholders noted that participation in this effort is voluntary and does not address issues like information technology staffing shortages and gaps in broadband access that pose particular challenges to electronic exchange for small and rural providers.Why GAO Did This StudyWidespread adoption of electronic health information exchange has the potential to improve health care quality and reduce costs. While these goals have been pursued for years, they have proved challenging to realize. Through the HITECH Act, the federal government provided $2.4 billion to states to improve and advance certain aspects of electronic health information exchange.In light of this federal investment, GAO was asked to review health information exchange. This report describes (1) states' use of the HITECH funding for health information exchange efforts and their plans to replace those funds, (2) the extent to which use of electronic health information exchange has changed since the enactment of the HITECH Act, and (3) federal efforts that aim to address challenges to electronic health information exchange.GAO reviewed information and interviewed officials from ONC and the Centers for Medicare &amp; Medicaid Services, the agency that administered the HITECH funding. GAO also conducted interviews with state agencies in eight states selected based on geographic variation and whether states accessed the HITECH funding for health information exchange, among other factors. In addition, GAO analyzed hospital and physician survey data, and interviewed a range of stakeholders, including physicians and organizations representing physicians, as well as organizations representing hospitals, payers, health information exchange organizations, and other industry representatives.For more information, contact Leslie V. Gordon at (202) 512-7114 or gordonlv@gao.gov.
- Software Acquisition: Additional Actions Needed to Help DOD Implement Future Modernization Efforts What GAO FoundThe Department of Defense's (DOD) response to evolving threats is increasingly determined by its ability to rapidly develop and deploy systems that heavily rely on software, such as weapons or information technology (IT) systems. DOD has taken many steps in the past few years to modernize its approach to developing and acquiring software. DOD's efforts at least partially implement all 17 Defense Science Board (DSB) and Defense Innovation Board (DIB) recommendations, some of which include multiple recommended actions. For example, DOD substantially implemented two DIB recommendations by streamlining software acquisition processes and piloting a new funding approach to deliver software faster.DOD Has at Least Partially Implemented All of the DIB and DSB RecommendationsHowever, for 13 of the 17 recommendations, DOD has yet to take certain actions outlined in the recommendations. For example, while DOD enhanced training for its software workforce, it has yet to establish a cadre of software developers. DOD officials stated that they have addressed the intent of the recommendations and do not plan to fully implement all recommended actions, in part, because certain actions may be impractical or outdated.DOD has outlined transformational plans to continue software modernization. According to DOD, its plans will require a cohesive department-wide effort that will take time to fully implement. However, DOD has yet to take certain steps recommended by GAO's past work to position itself to effectively implement its planned reforms. For example, DOD has yet to finalize implementation plans for these efforts or conduct strategic planning for its software workforce to ensure it has the needed skillsets to implement reforms. Taking such steps would better position DOD to implement its planned reforms, which are aimed at helping achieve its goal of more rapidly delivering software to its users.Why GAO Did This StudyDOD has made efforts to modernize its approaches to developing and acquiring software for its software-intensive systems—such as weapon and IT systems. However, it faces challenges executing approaches to rapidly deliver software. The DSB and DIB published reports in 2018 and 2019, respectively, which made recommendations to improve DOD's software practices.Congress included a provision in statute for GAO to examine DOD's implementation of DSB and DIB recommendations. This report assesses (1) the extent to which DOD addressed DSB and DIB recommendations; and (2) the extent to which DOD is positioned to implement its future software modernization plans.GAO reviewed DOD documents related to ongoing and future software reform initiatives and interviewed relevant officials. GAO then compared this information to DIB and DSB recommendations and key practices from past GAO work.
- IT Management: VA Needs to Improve CIO Oversight of Procurements What GAO FoundThe Department of Veterans Affairs (VA) procured IT and IT-related assets and activities that were often not approved by its Chief Information Officer (CIO). Such approval is required by the  Federal Information Technology Acquisition Reform Act  (FITARA). VA awarded 11,644 new contract actions categorized as IT between March 2018 and the end of fiscal year 2021. VA did not provide evidence of CIO approval for 4,513 (or 39 percent) of these contract actions.A more in-depth review of 26 selected IT contract actions from fiscal year 2021 confirmed that 12 had documentation showing approval by appropriate agency officials at the required level of authority. The remaining 14 contract actions lacked CIO approval documentation (see figure).The Department of Veterans Affairs' Documented Chief Information Officer Approvals for Selected Fiscal Year 2021 IT Contract ActionsOf the 14 contract actions lacking CIO approval, 13 were managed by non-IT contracting offices. According to VA officials, their contracting systems lack an automated control that would remind contracting officers of CIO review and approval requirements. Without an automated check or control to ensure contracting officer compliance, it is likely that there will continue to be IT procurements that will not be routed for CIO review, particularly for non-IT contracting offices. The lack of visibility into the procurement of much of VA's IT assets and activities constrained the CIO's opportunity to provide input on current and planned IT acquisitions. This, in turn, could result in awarding contracts that are duplicative or poorly conceived.Why GAO Did This StudyVA annually spends billions of dollars on IT each year to support the delivery of veterans' benefits and health care services. IT acquisition reform legislation, commonly referred to as FITARA, strengthens the authority of CIOs to provide needed direction and oversight.GAO was asked to review VA's IT management. The specific objective was to determine the extent to which VA's IT and IT-related assets and activities are being procured with CIO approval.GAO identified IT-categorized contract actions for new awards from March 2018 through the end of fiscal year 2021. GAO also selected 26 IT-categorized contract actions from fiscal year 2021 for an in-depth individual review of the approval documentation. The 26 actions selected represented a range of cost thresholds and different VA contracting offices.
- Global Cybercrime: Federal Agency Efforts to Address International Partners' Capacity to Combat Crime What GAO FoundThe Departments of State, Justice (DOJ), and Homeland Security (DHS) officials, and experts from international entities identified six mutual challenges in building global capacity to combat cybercrime. These included a lack of dedicated resources, difficulties in retaining highly trained staff, and inconsistent definitions of “cybercrime.” The expert panel also identified challenges in working with the U.S. government, including obstacles in obtaining information, lack of collaboration, and lack of dedicated funding streams.State, DOJ, and DHS have conducted a variety of activities to build foreign nations' capacity to combat cybercrime. These activities include engaging in information sharing with foreign partners and providing cyber training to foreign law enforcement officers. Agencies' activities can be grouped into four categories.Four Categories of Activities to Build Capacity to Combat CybercrimeThese agencies have documented accomplishments for many activities, such as nations joining international treaties aimed at combatting cybercrime. Further, State's plans include an evaluation of a regional forensics training center. This planned evaluation would meet the department's requirements. However, State has not conducted a comprehensive evaluation of the agencies' collective efforts. State is in the best position to conduct such an evaluation since it is authorized to provide foreign assistance funding to help build key allies' and partners' capacity to combat cybercrime. Until State conducts this comprehensive evaluation, the overall impact and results of federal assistance to global partners will likely remain unknown.Why GAO Did This StudyThe U.S. and its global partners are experiencing the effects of a massive cybercrime wave, which is growing in frequency and scale. In 2021, the Federal Bureau of Investigation received a record number of cybercrime complaints, over 840,000, with potential losses exceeding $6.9 billion. Further, in 2022, the intelligence community noted an increase in ransomware attacks by transnational criminals, which threaten to cause disruptions of critical services worldwide.GAO was asked to review federal efforts to build the capacity of allies and partner nations to combat cybercrime. This report's specific objectives were to (1) describe challenges in building global capacity to combat cybercrime, and (2) determine actions selected federal agencies are taking to build foreign nations' capacity to combat cybercrime and the extent to which they are evaluating the effectiveness of their efforts.GAO interviewed agency officials and convened a panel of experts representing entities focused on capacity building to combat global cybercrime. GAO also analyzed documentation from State, DOJ, and DHS, which provide the majority of U.S. capacity building assistance.
- DHS Financial Management: Actions Needed to Improve Systems Modernization and Address Coast Guard Audit Issues What GAO FoundThe Department of Homeland Security (DHS) has defined and implemented a tiered governance structure to provide oversight of its financial systems modernization programs. In 2018, DHS also established the Joint Program Management Office to lead all aspects of the modernization programs, in partnership with DHS components. DHS has both department-level and program-specific plans to modernize financial systems. Financial systems modernization plans at selected DHS components include U.S. Coast Guard, Federal Emergency Management Agency (FEMA), and U.S. Immigration and Customs Enforcement (ICE), among others.Coast Guard  deployed its new financial management system in December 2021 as part of a $510 million modernization program, and declared initial operational capability in June 2022. However, Coast Guard did not achieve expected full operational capability in December 2022. The program office is developing a remediation plan.	FEMA and ICE  are in the planning phases of their financial systems modernization efforts. In November 2022, DHS awarded contracts for software licenses and stated that it plans to award contracts for system integration services for these components.Additionally, DHS established a process and continues to document and consider lessons learned from current and past modernization attempts. These lessons are to be shared with upcoming modernization programs.Although DHS identified, documented, and tracked metrics to assess Coast Guard's system deployment, DHS found that the system was not achieving expected capabilities. This is because DHS did not address and remediate known issues identified in operational testing. DHS's subsequent operational testing and evaluation of the system found that it was not effective, responsive, or reliable. Therefore, DHS could not proceed to full operational capability of the system. It is now in the process of developing a remediation plan to address outstanding issues.DHS risks not fully achieving its goal of deploying systems that produce reliable data for management decision-making and financial reporting if it does not remediate serious issues identified by testing. Resolving deficiencies identified by testing before proceeding to the next phase in the acquisition process can help reduce the risk that future system modernization efforts at FEMA and ICE will not meet mission needs or expected capabilities.GAO also found that corrective action plans Coast Guard developed to address its fiscal year 2021 audit findings did not always contain all of the data attributes recommended in applicable guidance. For example, although DHS guidance emphasizes the importance of root cause analyses in resolving deficiencies, such analyses were often not done. Therefore, Coast Guard is at an increased risk that its corrective actions will not effectively address identified deficiencies.Why GAO Did This StudySince DHS's creation in 2003, it has faced significant internal control and financial management systems deficiencies. These issues contributed to GAO designating DHS financial management as high risk. To address its financial management issues, DHS is executing a multiyear plan, to include implementing modern financial management systems at its components, including Coast Guard, FEMA, and ICE.In this report, GAO (1) describes the oversight, program management, plans, and lessons learned from past and current financial systems modernization efforts; (2) examines the extent to which the Coast Guard is achieving expected capabilities with its newly deployed financial management system; and (3) examines the extent to which Coast Guard has addressed audit findings related to financial reporting and IT system weaknesses.GAO met with DHS officials, reviewed key documents and plans related to modernization efforts, and assessed Coast Guard corrective action plans to address fiscal year 2021 audit findings.
- Data Center Optimization: Agencies Continue to Report Progress What GAO FoundFederal agencies continue to close data centers and report related cost savings under the Data Center Optimization Initiative (DCOI). In fiscal year 2021, federal agencies closed 58 data centers. In fiscal year 2022, agencies closed 20 data centers by August 2022, and planned to close 58 additional data centers through the end of the fiscal year. Moreover, 22 of the 24 agencies met their related cost savings goals for fiscal year 2021, identifying a total of $612.326 million in cost savings. Agencies also planned to meet their cost savings goals for fiscal year 2022; as of August 2022, they had identified $334.324 million in cost savings. Agencies have a cumulative total of $6.6 billion in cost savings and avoidances from fiscal years 2012 through 2021.In fiscal year 2022, federal agencies reported mixed progress in meeting the Office of Management and Budget's (OMB) data center optimization targets. These targets consist of virtualization, data center availability, advance energy metering, and underutilized servers. As of September 2022, 13 agencies reported meeting the virtualization target, 17 met the availability target, and 14 met the metering and utilization targets. In addition, seven agencies do not have any agency-owned data centers or their remaining centers are exempted from optimization by OMB; therefore, reporting on optimization metrics for these agencies was not applicable. (See figure)24 Agencies Reported Progress towards Meeting Office of Management and Budget (OMB) Data Center Optimization Targets, as of September 2022GAO has made 126 recommendations since 2016 to help agencies meet their DCOI targets, and the agencies have implemented 110 of them. Once agencies fully address all previous GAO recommendations to meet their optimization performance targets, they should fully realize the expected benefits, including cost savings from DCOI.Why GAO Did This StudyIn 2010, OMB established a government-wide effort to reduce the number of duplicative or outdated federal data centers. In December 2014, Congress enacted federal IT acquisition reform legislation known as FITARA, which included provisions related to ongoing federal data center consolidation efforts. Less than two years later, OMB's Federal Chief Information Officer launched DCOI to build on prior efforts. Specifically, it required, among other things, that agencies consolidate and optimize existing facilities.FITARA included a provision for GAO to annually review agencies' data center inventories and strategies. This report addresses (1) agencies' progress on data center closures and the related savings and agencies' plans for future closures and savings; and (2) agencies' progress against OMB's data center optimization targets.GAO reviewed the 24 DCOI agencies' data center inventories as of March and August 2022, including their reported cost savings and data center optimization strategic plans, and interviewed relevant agency officials.
- Science & Tech Spotlight: Digital Twins—Virtual Models of People and Objects Why This MattersDigital twins are virtual representations that can mimic and predict the performance of their physical counterparts. Many industries use them to reduce costs, improve engineering design and production, and test supply chains. But some applications—like creating a digital twin of a person—raise technical, privacy, security, and ethical challenges.The TechnologyWhat is it? Digital twins are virtual representations of people or physical objects, processes, or systems, ranging from vehicles to industrial plants to clinical trial patients. These "living" computational models integrate with data from a physical twin, such that any changes made to the physical twin can automatically lead to changes in the digital twin (see fig. 1). Digital twins can be used to remotely maintain or monitor the physical twin, or predict how it will perform.Digital twins can be applied across many fields. For example, they can help more efficiently design aircraft parts, predict and avoid supply chain disruptions, and improve traffic safety. They could also monitor living systems or predict hypothetical effects on them for applications in health care, agricultural management, and climate forecasting.Figure 1. Example of the integration of physical and digital twins.How does it work? Digital twins rely on multiple existing technologies, including the Internet of Things, a network of interconnected "smart" devices that exchange information and collect large amounts of data. Sensors built in or placed on the physical twin or around its environment can collect data continuously. These sensors transmit data that update the digital twin through technologies such as network interfaces and cloud platforms, depending on the physical twin and its location. For example, sensors in a factory may transmit data to a digital twin via local wireless networks, but sensors in public spaces may require cellular networks for adequate connectivity.Digital twins use these data to dynamically change along with the physical object in real time. This allows users to interact with a 3D visualization, monitor changes, and predict the effect of future events. For example, digital twins of smart cities can be used to predict how extreme weather events will affect the power grid, and what cascading effects might follow. Digital twin data can also be analyzed using artificial intelligence and machine learning to test different scenarios or make predictions for the physical twin. For example, machine learning could automatically recognize an abnormal pattern in a patient's digital twin, enabling a quicker diagnosis and improved treatment (see fig. 2).Figure 2. Examples of how digital twins can be used in health care.How mature is it? Though the term “digital twin” was coined around 2010, NASA engineers originally applied the concept by building physical and computational models of the Apollo 13 mission vehicles on the ground that were adapted to match the vehicles’ changing conditions in space. Today, the use of digital twins is growing, owing to advances in high-performance computing, rapid data analysis, and smart sensors. Applications in industry include improving operations and predicting or mitigating the effects of manufacturing disruptions.For example, one company builds digital twins of wind turbines to help manufacturers identify potential problems before scaling up production. The U.S. Navy uses digital twins of shipyards to help inform officials of expected project costs. Digital twins are also being used to more efficiently rebuild areas of a Ukrainian city destroyed in the conflict with Russia, enabling officials to forecast reconstruction costs and improve the city’s infrastructure and resources. In the U.S., a digital twin of the Orlando region allows local government, companies, and nonprofits to simulate and visualize the effects of scenarios such as climate change and new transit routes.Digital twin technology is less mature in the biological sciences, primarily due to the complexities of living systems found in areas such as health care, agricultural management, and the environment. Researchers are studying whether digital twins of people could improve health care by predicting patient response to different therapies in clinical trials or by diagnosing health conditions using pattern recognition through machine learning. Moreover, the Food and Drug Administration’s strategic plan notes the technology’s cost-saving potential in evaluating medical devices prior to patient use. In addition, a European Union (EU)-funded project plans to use digital twins for personalized medicine by simulating the electromagnetic activity and physiology of patients’ brains to help treat diseases such as Alzheimer’s and epilepsy.Researchers are also exploring applications in agricultural management by using digital twins to study bee colonies to control honey production and examine the biology of fruit to improve transport and storage. The EU and the National Oceanic and Atmospheric Administration have initiated projects to build digital twins to monitor global environmental conditions.What are some concerns? Cloud computing services, software, and other products to implement digital twins have potential legal, ethical, and technical issues. According to experts, ethical concerns over data ownership and privacy could result in low public trust. This could&nbsp;occur, for example, if a pharmaceutical company sold health-related data from digital twins without consent. Technical issues such as data quality or accuracy could also reduce public confidence when digital twins are used for decision-making. For example, if data used to train machine learning do not accurately reflect the characteristics of a patient, the predictive analysis by the digital twin could be misleading or even discriminatory.Digital twins require many interrelated systems and networks for large-scale data collection and transmission, which poses interoperability and cybersecurity issues, according to researchers. Cybersecurity risks could include unauthorized access or modifications to the real data, compromising confidentiality and public trust. Additionally, relevant U.S. infrastructure may not be adequate for such high-performance computing. Researchers have highlighted that the high upfront costs for using digital twins may create access and affordability issues for less affluent city governments or patients whose health insurance does not cover its use.Researchers also emphasize that standards and regulations are not fully developed to address digital twin implementation, especially for more complex applications. Currently, users rely on a patchwork of standards for data management, security, and networking established by organizations such as the National Institute for Standards and Technology and the International Organization for Standardization.OpportunitiesManufacturing and engineering efficiency. Digital twins monitor in real time to help avoid potential problems, optimizing manufacturing and engineering projects and reducing costs.	Improved health care. Digital twins could help facilitate predictive and personalized medicine to improve patient outcomes and reduce some health care costs.ChallengesPrivacy and ethical issues. Security, bias, and data ownership issues, especially for digital twins applied in health care, can reduce public trust in the technology.	Technical and infrastructure barriers. Digital twins are difficult to scale up for more complex systems, and the current U.S. network infrastructure may not be adequate for expanded use.	Economic costs. Digital twins can be costly to develop, leading to potential inequities in their use.	Standards and regulations. Existing standards and regulations for digital twin implementation may not be adequate to address their application in complex systems, especially biological ones.Policy Context and QuestionsWhat options exist for dealing with ethical or equity issues that digital twins could raise, especially in areas such as health care?&nbsp;	What steps could help mitigate potential security and privacy risks associated with digital twin technology?	How could standards and regulations be further developed for digital twin technology, especially when applied to individuals?For more information, contact Brian Bothwell at (202) 512-6888 or bothwellb@gao.gov.
- Information Technology: IRS Needs to Complete Modernization Plans and Fully Address Cloud Computing Requirements What GAO FoundThe Internal Revenue Service's (IRS) legacy IT environment includes applications, software, and hardware, which are outdated but still critical to day-to-day operations. Specifically, GAO's analysis showed that about 33 percent of the applications, 23 percent of the software instances in use, and 8 percent of hardware assets were considered legacy. This includes applications ranging from 25 to 64 years in age, as well as software up to 15 versions behind the current version. As GAO has previously noted, and IRS has acknowledged, these legacy assets will continue to contribute to security risks, unmet mission needs, staffing issues, and increased costs.Modernization best practices call for documenting plans that include three key elements: milestones, work to be performed, and disposition of legacy systems. As of August 2022, IRS had documented plans for the 21 modernization initiatives that were underway, including nine associated with legacy systems. All 21 plans addressed two key elements. However, the plans for six of the nine initiatives did not address the disposition of legacy systems (see table). Officials stated they would address this key element at the appropriate time in the initiatives' lifecycle; however, they did not identify time frames for doing so.GAO Assessment of the Internal Revenue Service's (IRS) Modernization Plans as of August 2022			Modernization plan element 									Yes 									No 									Not applicable 								Includes milestones									21									0									0								Describes work to be performed									21									0									0								Includes disposition of legacy systems									3									6									12a					Source: GAO analysis of IRS documents. | GAO-23-104719aThese initiatives were not associated with legacy systems.IRS recently suspended operations of six initiatives, including two which are essential to replacing the 60-year old Individual Master File (IMF). The IMF is the authoritative data source for individual tax account data. GAO has reported that IRS has been working to replace IMF for well over a decade. According to officials, the suspensions were due to IRS's determination to shift resources to higher priorities; staff members working on these suspended initiatives were reassigned to other projects. As a result, the schedule for these initiatives is now undetermined. Accordingly, the 2030 target completion date for replacing the IMF that IRS announced last year is now unknown. This will lead to mounting challenges in continuing to rely on a critical system with software written in an archaic language requiring specialized skills.IRS's cloud computing efforts fully addressed 11, partially addressed one, and did not address two requirements identified in the Office of Management and Budget's June 2019 cloud computing strategy. Key shortfalls include IRS not conducting regular evaluations of customer experiences and user needs. Until IRS fully addresses all cloud computing requirements, it will have less assurance that it is adopting cloud solutions that efficiently and effectively help it meet its mission needs and protect sensitive taxpayer information.Why GAO Did This StudyIRS relies extensively on IT to annually collect trillions of dollars in taxes, distribute hundreds of billions of dollars in refunds, and carry out its mission of providing service to America's taxpayers in meeting their tax obligations. This reliance on IT includes a significant amount of outdated legacy systems that pose risks. At the same time, IRS has increasingly used internet-based computing services (i.e., cloud services) to address its IT modernization needs.GAO was asked to review IRS's legacy environment and cloud computing implementation. This report, among other things, 1) describes IRS's legacy IT environment; 2) assesses IRS's modernization plans; and 3) evaluates its cloud computing efforts.To do so, GAO analyzed legacy applications, software, and hardware data; assessed plans for 21 modernization initiatives against practices identified in prior GAO work; and compared IRS's cloud computing documentation to relevant federal requirements. GAO also interviewed knowledgeable IRS officials.