Veterans Affairs: Improved Management Processes Are Necessary for IT Systems That Better Support Health Care
Published: 2017-06-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: VHA, an administration within VA, provides a broad range of primary care, specialized care, and related medical and social support services to veterans. In doing so, VHA operates one of the nation's largest health care systems through 168 VA medical centers and more than 1,000 outpatient facilities. The administration managed total budget resources reported at nearly $91 billion in fiscal year 2016. Based on interest in VHA's ability to oversee its health care system and provide timely care, GAO reviewed IT management at VHA. Specifically, GAO determined the extent to which VA's (1) IT management processes are consistent with leading practices and (2) current IT systems support VHA's core business functions. To do so, GAO analyzed documentation and interviewed officials about VA's approach to IT management processes related to strategic planning, investment management, and enterprise architecture, and compared VA's processes to leading practices. In addition, GAO reviewed data related to VA's IT systems and VHA's IT business needs. GAO further reviewed IT needs from three key VHA program areas.
section_title: What GAO Found
paragraphs: The Department of Veterans Affairs (VA) has established information technology (IT) management processes that are partially consistent with leading practices. VA has issued strategic plans that identify goals and objectives related to health IT; established investment review boards at the department-level and within the Veterans Health Administration (VHA) that are responsible for selecting IT investments aligned to VHA priorities; and documented VHA's core business functions within an enterprise architecture. However, the IT strategic plans do not include performance measures and targets for their defined objectives, VA's department-level IT investment board has been inactive and its investment selection guidance lacks criteria, and the department has not fully identified metrics aligned to core business functions to inform investment decisions. Until VA can improve these processes, it risks having IT systems that may not fully support VHA's mission.
IT systems at VA are generally aligned to core business functions defined by VHA; however, among new service requests, which identify unmet needs of business owners, 817 out of a total of 2,772 IT needs identified for VHA since 1998 had not been met as of October 2016. About 39 percent of these open requests had been open for more than 5 years.
GAO's review of the business needs identified in three key program areas—Pharmacy Benefits Management, Veterans Access to Care, and Community Care—showed a number of long-standing needs. According to VA officials, their need to balance the resources for IT needs across the department is a reason that business needs have remained unresolved. Until VA prioritizes resources to address these needs, VHA's programs may not be well supported by IT systems capable of delivering health care services consistent with its objectives.
section_title: What GAO Recommends
paragraphs: GAO is recommending that VA address the deficiencies identified with IT strategic planning, investment management, and enterprise architecture; and ensure that the three programs' IT needs are addressed. VA agreed with GAO's recommendations and described actions planned to address them by the end of fiscal year 2018.

section_title: Letter
paragraphs: The Department of Veterans Affairs’ (VA) Veterans Health Administration  (VHA) provides a broad range of primary care, specialized care, and  related medical and social support services that are related to veterans’  health or special needs. VHA operates one of the nation’s largest health  care systems through its 168 VA medical centers and more than 1,000  outpatient facilities. The administration managed total budget resources  reported at about $91 billion in fiscal year 2016.
The demand for VHA’s services is expected to grow in the future. Due to  changes in the veteran population, including increases in the number of  women veterans, the health care needs of veterans may also change.  These and other changes, such as innovations in technology and health  care delivery, may affect VHA’s current strategies for accomplishing its  mission to provide care to the nation’s veterans. At the same time, VA  faces serious and long-standing problems with veterans’ access to health  care and management failures, including ongoing appointment scheduling  problems, unreliable appointment wait-time data, and inadequate  coordination of veteran care between VA and non-VA medical providers.
We and others have expressed significant concerns about VHA’s  management of its health care system, including VHA’s ability to  effectively provide and oversee timely access to health care for veterans.  Further, over many years, VA has experienced challenges in managing  its information technology (IT) resources, raising questions about the  effectiveness of its operations and its ability to deliver high-quality health  care to veterans and other outcomes needed to help advance the  department’s mission.
Based on interest in VHA’s ability to oversee its health care system and  provide timely care, you asked us to conduct a management review of  VHA that encompassed several key organizational components, including  IT management. This report presents the results of our review that  examined the extent to which VA’s (1) IT management processes are  consistent with leading practices and (2) current IT systems support  VHA’s core business functions.
To address the first objective, we identified VA’s management processes  that are intended to ensure that IT investments meet the business needs  of the organization (i.e., IT strategic planning, investment management,  and enterprise architecture). We then compared those processes to  leading practices that we and the Office of Management and Budget  (OMB) previously identified to assist organizations with improving their  management processes.
To accomplish our second objective, we reviewed data in the VA Systems  Inventory (VASI) to identify current IT systems and their relationships to  VHA’s core business functions, as described in VHA’s business  architecture. We then analyzed data from VHA’s new service request  (NSR) database, which serves as a repository for identified IT needs  submitted by VHA programs and business owners. This allowed us to  identify the total number of service requests, when requests were  entered, and the number of requests that remained unresolved.
We reviewed documentation related to both databases, such as the data  dictionary; examined data for unusual or missing entries; and interviewed  cognizant officials about data reliability and internal control procedures.  We determined the VASI and NSR data were reliable for the purposes of  our reporting objectives and used the data, among other sources of  evidence, to support our findings, conclusions, and recommendations.
We also conducted additional analyses of new service requests identified  by three VA programs related to health service delivery and on which we  have previously reported—Pharmacy Benefits Management Services,  Veterans Access to Care (scheduling and consults), and Community  Care—to determine the extent to which business needs have been  addressed over time. We interviewed VHA officials regarding the IT  systems used by the three programs, the process for identifying and  managing VHA business needs, and the current status of unmet needs,  to understand the extent to which current systems support VHA’s core  business functions. A full description of our objectives, scope, and  methodology can be found in appendix I.
We conducted this performance audit from January 2016 to June 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: VA’s mission is to promote the health, welfare, and dignity of all veterans  in recognition of their service to the nation by ensuring that they receive  medical care, benefits, social support, and lasting memorials. In addition  to its central office located in Washington, D.C., VA has field offices  located throughout the United States, as well as the U.S. territories and  the Philippines.
The department’s three major components—VHA, the Veterans Benefits  Administration (VBA), and the National Cemetery Administration (NCA)— are primarily responsible for carrying out its mission. More specifically,  VHA provides health care services, including primary care and  specialized care, and it performs research and development to improve  veterans’ needs. VBA provides a variety of benefits to veterans and their  families, including disability compensation, educational opportunities,  assistance with home ownership, and life insurance. Lastly, NCA provides  burial and memorial benefits to veterans and their families.
subsections: section_title: VA Relies Extensively on IT
paragraphs: The use of IT is critically important to VA’s efforts to provide benefits and  services to veterans. As such, the department relies extensively on IT to  meet the day-to-day operational needs of its medical centers, provide  veteran-facing systems, and otherwise support the department’s mission.  According to OI&T data as of October 2016, there were 576 active or in- development systems in VA’s inventory of IT systems. These systems  are intended to be used for the determination of benefits, benefits claims  processing, and access to health records, among other services.
VHA is the parent organization for 319 of these systems. Of the 319  systems, 244 were considered mission-related and provide capabilities  related to veteran health care delivery. VHA’s systems provide, for  example, capabilities to support electronic health records that health care  providers and other clinical staff use to view patient information in  inpatient, outpatient, and long-term care settings, as well as patient  admission to hospitals and clinics, and patient care through telehealth.  The remaining systems support corporate or non-mission related IT  functions.
For fiscal year 2017, the department’s budget request included nearly  $4.28 billion for IT. Specifically, VA requested approximately $2.53 billion  for sustainment, approximately $1.27 billion for payroll and  administration, and approximately $471 million for new systems  development or modernization efforts.
According to OI&T, of the $471 million requested for VA development and  modernization, approximately $166.6 million (about 35 percent) was  requested to support VHA development projects such as the Veterans  Health Information Systems and Technology Architecture, known as VistA  Evolution, and other clinical systems development. In addition, $276.7  million (about 11 percent) of the $2.53 billion in sustainment funding was  allocated to VHA-specific projects to support existing systems. The  remaining amounts of requested funds support the other VA  administrations as well as overall IT infrastructure that are not necessarily  aligned to any single administration. Figure 1 provides the breakdown of  VA’s proposed IT budget for fiscal year 2017.
Since 2007, VA has been operating a centralized organization in which  most key functions intended for effective management of IT are  performed by OI&T and led by the Assistant Secretary for Information and  Technology/Chief Information Officer (CIO). Figure 2 presents a simplified  organizational chart for VA.
OI&T has responsibility for managing the majority of VA’s IT-related  functions. The office provides strategy and technical direction, guidance,  and policy related to how IT resources are to be acquired and managed  for the department. According to VA, OI&T’s mission is to collaborate with  its business partners (such as VHA) and provide a seamless, unified  veteran experience through the delivery of state-of-the-art technology.
The CIO serves as the head of OI&T and is responsible for providing  leadership for the department’s IT activities. The CIO reports to the Office  of the Secretary of Veterans Affairs through the Deputy Secretary and  advises the Secretary regarding the execution of the IT appropriation. In  addition, the CIO is expected to serve as the principal advisor to top  management officials, such as the Under Secretaries of each of the three  administrations, on matters relating to IT management in the department.  This official is also tasked with reviewing and approving investments, as  well as overseeing the performance of IT programs and evaluating them  to determine whether to continue, modify, or terminate them.
Although VA centralized its key IT functions in order to maintain better  control over resources, we have previously reported that the office has  faced challenges in fully implementing and managing IT under its  centralized organizational structure. In addition, independent  assessments of the department’s efforts in 2013 and 2015 showed that  OI&T has had difficulty in preventing IT activities from occurring outside  its control. According to the assessments, it has also been challenged in  effectively collaborating with the department’s various business units and  in efficiently and cost-effectively delivering new IT capabilities.
Recognizing these challenges, the CIO initiated an effort in January 2016  to transform OI&T focus and functions. Among other things, the  transformation focused on reorganizing the units within OI&T. Beginning  in April 2016, VA established five organizational units within OI&T with  responsibility for performing and managing specific IT-related functions.
Enterprise Program Management Office. This office began initial  operations in April 2016, and is intended to serve as OI&T’s portfolio  management and project tracking organization. According to OI&T,  its goals are to align IT portfolios with the department’s strategic  objectives; enhance visibility and governance; analyze and report on  portfolio performance metrics; ensure the overall health of the IT  portfolio; and optimize resources for projects, people, and timelines.  The Enterprise Program Management Office includes the following six  functional areas: (1) Intake and Analysis of Alternatives is to work with  the VA administrations and other staff offices to develop requirements  to meet the needs of veterans, provide analysis of alternative  approaches to meeting those requirements, and integrate information  security; (2) IT Portfolios is to consolidate programs and projects  under five portfolios (Health, Benefits, Cemeteries, Corporate, and  Enterprise services); (3) Project Special Forces is to mitigate issues  that put projects at risk of failure; (4) Demand Management is  responsible for metrics gathering and analysis, development of  process tools, human resources, and training; (5) Transition Release  and Support is to manage OI&T’s integrated calendar supporting VA’s  Veteran-focused Integration Process; and (6) Application  Management is responsible for IT implementation efforts, including  testing, design, and data management.
Account Management. This function, led by four account  managers, is responsible for managing the IT needs of OI&T’s  business partners—VA’s administrations and staff offices, including  VHA. Account managers are to interface directly with their customers  to understand their needs, help identify and define the solutions to  meet those needs, and represent their interests by reporting directly to  the CIO. In this regard, account managers are to submit their  customers’ IT requirements to the Enterprise Program Management  Office, ensure that their business needs are understood by OI&T, and  ensure that business solutions are designed to meet their customers’  specifications. This function is also tasked with advocating for the  customers in the budget process. OI&T intends for this function to  address the challenge of effectively collaborating with business units.  As of December 2016, all four account managers were in place.
Quality and Compliance. This function is responsible for establishing  effective policy governance and standards and ensuring adherence to  the policies and standards. In addition, the quality and compliance  function is charged with identifying, monitoring, and measuring risks  across OI&T.
Data Management Organization. The organization is intended to  improve both service delivery and the veteran experience by engaging  with data stewards to ensure the accuracy and security of the  information collected by VA. The organization is to institute a data  governance strategy; engage with VA staff to ensure the accuracy  and security of collected data; analyze data sources to form an  enterprise data architecture; and establish metrics for data  efficiency, access, and value. OI&T also intends for the organization  to identify trends in the data collected on each veteran that could  improve their health care by providing predictive care and anticipating  needs.
Strategic Sourcing. This function is responsible for establishing an  approach to fulfilling the department’s requirements with vendors that  provide solutions to those requirements, managing vendor selection,  tracking vendor performance and contract deliverables, and sharing  insights on new technologies and capabilities to improve the  workforce knowledge base.
subsections: section_title: VHA Has Responsibility for the Identification of Its IT Needs
paragraphs: The VA Under Secretary for Health is the head of VHA and is supported  by the Principal Deputy Under Secretary for Health, four Deputy Under  Secretaries for Health, and nine Assistant Deputy Under Secretaries for  Health. Among these, the Deputy Under Secretary for Health for Policy  and Services oversees the work of the Assistant Deputy Under Secretary  for Health for the Office of Informatics and Information Governance within  VHA.
The Strategic Investment Management office, a division of the Office of  Informatics and Information Governance, was established to support the  IT needs of VHA by providing information on health-related information  systems that senior managers need to make sound decisions. There are  four organizational services within this office: Business Architecture,  Investment Governance Services, Open Source Management, and  Requirements Development and Management. Among other things, this  office advocates for VHA’s IT needs within the Planning, Programming,  Budgeting, and Execution process and coordinates with VHA business  owners and other VA organizations to support, document, analyze, and  evaluate clinical and business needs and requirements for IT  development.
The Strategic Investment Management office works closely with business  owners and program offices within VHA to assist with the IT governance  and budgeting processes, IT needs identification, requirements  development, and investment oversight. For example, the Strategic  Investment Management office works with program offices such as  Pharmacy Benefits Management Services, Veterans Access to Care  (scheduling and consults), and Community Care. These offices are  responsible for key functions and IT systems related to health service  delivery:
Pharmacy Benefits Management Services. This program office is  responsible for providing organizational guidance on a broad range of  pharmacy activities to the 260 pharmacies located in VA’s medical  centers and outpatient clinics. The office also has operational  responsibility for all aspects of the department’s seven consolidated  mail outpatient pharmacies, with the exception of IT. The Executive  Director of this office is responsible for identifying functional needs for  medical center pharmacies and consolidated mail outpatient  pharmacies and communicating those needs to OI&T for prioritization  and planning to acquire pharmacy IT capabilities.
Veterans Access to Care (scheduling and consults). This program  office is responsible for standardizing and coordinating system-wide  administrative clinic operations and management. Specifically, the  Executive Director serves as VHA’s business owner and manager in  collaboration with OI&T on matters regarding scheduling, including the  department’s electronic outpatient scheduling system.
Community Care. This program office is responsible for overseeing  all VHA community care programs and business processes, such as  determining veterans’ eligibility to receive health care benefits and  purchasing care from non-VA providers. Specifically, it is structured  around six functional areas: eligibility, referral and authorization, a  tiered network of community providers, care coordination, provider  payment, and customer service.
section_title: VA’s and VHA’s IT Environment Faces Challenges
paragraphs: As previously mentioned, an independent assessment recently noted that  VHA and OI&T faced a number of challenges in collaborating to execute  health IT improvements and developing new and modernized  capabilities. Specifically, in response to The Veterans Access, Choice,  and Accountability Act of 2014 (Choice Act), the assessment was  released in September 2015, stating that VHA and OI&T did not  collaborate effectively. The assessment found that VHA and OI&T often  did not agree on priorities for executing their strategic plans and have  struggled to identify, prioritize, and translate clinical goals and strategic  initiatives reflected in the department’s overarching planning documents  into buildable, testable health IT requirements that resulted in measurable  health care outcomes for the veteran. In addition, the report stated that  VA’s ability to deliver new capabilities for VistA had stalled and as a result  the VA health care system was in danger of becoming obsolete.
The Choice Act also established the Commission on Care (the  Commission). This independent entity evaluated veterans’ access to VA  health care and assessed how veterans’ care should be organized and  delivered during the next 20 years. In its final June 2016 report, the  commission acknowledged that, although VHA provided health care that  was, in many ways, comparable or better in clinical quality to that  generally available in the private sector, the care was inconsistent from  facility to facility. According to the commission, health care also could be  compromised by poorly functioning operational systems and processes.  The commission’s recommendations were intended to serve as a  foundation for organizational transformation at VA.
We have also issued numerous reports that highlighted challenges facing  VA’s efforts to improve IT management. For example, in May 2010, we  reported that, after spending an estimated $127 million over 9 years on its  outpatient scheduling system project, VA had not implemented any of the  planned system’s capabilities and was essentially starting over by  beginning a new initiative to build or purchase another scheduling  system. We also noted that VA had not developed a project plan or  schedule for the new initiative; department officials stated that VA  intended to do so after determining whether to build or purchase the new  application. We recommended that the department take six actions to  improve key systems development and acquisition processes essential to  the second outpatient scheduling system effort. The department generally  concurred with our recommendations, but has not provided information  about its actions to implement four of the six recommendations.
In May 2016, we reported that VA’s expenditures for its care in the  community programs, the number of veterans for whom VA has  purchased care, and the number of claims processed by VHA have all  grown considerably in recent years. Due to recent increases in  utilization of VA care in the community, the department has had difficulty  processing claims in a timely manner. We reported that VA officials and  claims processing staff had indicated that IT limitations, manual  processes, and staffing challenges delayed claims processing. The  department had implemented interim measures to address certain system  challenges, but did not expect to deploy solutions to address all  challenges, including those related to IT, until fiscal year 2018 or later.  Further, VA did not have a sound plan for modernizing its claims  processing system, which we recommended it develop. The department  concurred with this recommendation and stated that it intended to  address the recommendation through the planned consolidation of its  care.
We have also recently reported on VHA’s efforts to provide outpatient  pharmacy services to approximately 6.7 million veterans. Specifically, in  June 2017, we reported that pharmacists cannot always efficiently view  and share necessary patient data among VHA medical sites and cannot  transfer prescriptions to other VHA pharmacies or process prescription  refills received from other VHA medical sites through the system. As a  result, pharmacists do not have the necessary data to efficiently make  clinical decisions about prescriptions, which could negatively affect  patient safety. In addition, we noted that VA’s pharmacy system lacks  certain capabilities, such as the capability for exchanging prescriptions  with non-VHA providers; the system also does not maintain a perpetual  inventory capability. Among other actions, we recommended that VA  update its pharmacy system to view and receive complete medication  data, assess the impact of interoperability, and implement additional  industry practices. VA generally concurred with our recommendations.
section_title: Key IT Management Processes Are Partially Consistent with Leading Practices
paragraphs: VA has established IT management processes that are partially  consistent with leading practices. For example, the department has  developed multiple IT strategic plans and related documents that identify  its goals. However, these plans and documents do not include  performance metrics that the department could use to track progress  toward achieving its goals. Additionally, although VHA has an IT  investment management process that is consistent with leading practices,  VA’s department-level IT investment board has been inactive and  investment selection criteria have not been defined. Further, while VHA  has defined a business architecture that identifies its core business  functions, measurement of the extent to which those functions are  supported by IT investments is incomplete.
subsections: section_title: IT Strategic Plans Identify Goals, but Lack Performance Metrics That Are Needed to Track Progress
paragraphs: Strategic planning is essential to help an organization define what it seeks  to accomplish and identify the strategies it will use to achieve desired  results. Our research and experience at federal agencies has shown  that an agency must align IT goals with its strategic goals as part of an  institutionalized set of management capabilities. An IT strategic plan  outlines the agency’s goals and identifies performance metrics that permit  the agency to determine whether IT is making a difference in improving  performance. The resulting plan effectively guides modernization efforts  by serving as an agency’s vision, or road map, and helps align its  information resources with its business strategies and investment  decisions.
OMB has issued guidance for agencies to use in developing and  maintaining a strategic plan that describes the agency’s technology and  information resource goals, defines the level of performance to be  achieved, and demonstrates how the goals align with the agency’s  mission and organizational priorities. VA has also issued a directive that  requires IT strategic planning to include outcome-oriented performance  measures.
In accordance with leading practices, the department has produced  multiple strategic plans, road maps, and supplementary guidance that  describe the strategic direction for IT across the department. For  example, OI&T has issued the following documents and guidance, which  describe, among other things, the strategic goals and objectives,  transformation priorities, and the future vision for VA IT.
The Fiscal Year 2013 through 2015 Information Resources  Management (IRM) Strategic Plan and an associated Enterprise  Roadmap. Together, these documents describe the department’s IT  strategic goals and objectives. VA has taken steps to show alignment  between the IT strategic goals and objectives and the VA Strategic  Plan. For example, the objectives in the IRM Strategic Plan include,  among other things, managing the IT portfolio and utilizing  performance metrics for informed decision making. In addition, the  Enterprise Roadmap describes additional OI&T goals and priorities,  as well as select programs that are intended to support those priorities  between 2016 and 2018. For example, the roadmap identifies health  care modernization as one of VA’s key IT investments.
Enterprise Technology Strategic Plan, Fiscal Years 2017 through  2021. OI&T has issued a strategy to achieve VA’s IT vision, which is  to lead the department as “a world-class organization that provides a  seamless, unified veteran experience through the delivery of state-of- the-art technology.” It sets priorities that are intended to guide  decision making at the department. According to the plan, its priorities  are in alignment with the MyVA continuous improvement initiative.
Further, the plan describes the current technical environment. It also  details a vision for a future IT environment that plans to utilize new  and emerging technologies to improve information availability,  information security, reusable shared services, modern applications,  and scalable infrastructure.
Multi-Year Programming guidance. OI&T has issued annual  guidance for the IT Multi-Year Programming process, which is  intended to ensure that the IT appropriation is being directed to those  investments that satisfy the most pressing mission requirements of  the department. This guidance describes a number of strategic  challenges faced by OI&T. For example, the guidance from recent  years noted that the retirement of legacy systems and the increasing  cost of sustaining those systems were two challenges that should be  taken into consideration during the Multi-Year Programming cycle for  decisions on IT investments.
While OI&T produced these strategic plans, road maps, and  supplementary guidance related to IT, none of the documents includes  specific results-oriented performance metrics that are called for by VA’s  IT strategic planning directive and leading practices. For example, while  the IRM Strategic Plan includes a strategic objective related to aligning  investments with mission needs, it does not describe or point to a specific  target to be achieved, and related performance metrics for how progress  against this target will be measured.
In addition, VHA has taken steps to define a strategic direction for health  IT by issuing its Health Information Strategic Plan for Veterans Health  Administration Supporting VA Health Care Version 4.3 (HISP).  According to the HISP, this strategy is to inform OI&T’s IRM Strategic  Plan. The HISP identifies strategic goals and objectives related to health  IT within VHA. For example, one strategic goal included in the plan is to  enhance health information processes and practices to ensure that VA  health systems are efficient and cost effective, and have the capability  needed to deliver quality medical care to veterans. According to the plan,  two objectives for achieving this goal are to implement IT innovations that  support efficiency in business operations, such as digitalization of  business processes through the use of sensors or other monitoring and  automation systems, and to implement a performance measurement  capability to monitor and drive a culture of quality and safety.
However, VHA’s HISP does not identify corresponding performance  targets and metrics for strategic goals and objectives identified in the  plan. Further, this lack of performance targets and metrics has been a  longstanding issue. For example, a previous version of the plan stated  that a workgroup was established in October 2012 to identify performance  goals and to create an initial report by May 2013. According to VHA  officials, while VHA established a workgroup in October 2012 to identify  performance metrics, the workgroup’s recommendations were not  adopted.
OI&T officials acknowledged that the department’s strategic plans and  related documents do not contain performance targets and metrics, but  said that VA does report outcome-based operational performance metrics  for each major IT investment to OMB’s IT Dashboard. However, these  metrics are not specific to the IT goals and objectives outlined in the IRM  Strategic Plan and, thus, do not help report how VA is progressing toward  achieving its strategic goals and objectives.
Further, according to VHA officials, VHA offices are not staffed to identify,  track, and report on IT performance measures. Because VA’s IT strategic  plans do not identify performance metrics that could be used to track  progress toward strategic goals and objectives, VA and VHA lack the  ability to accurately track progress toward providing IT systems that  address VHA’s business needs and support the performance of its  mission.
section_title: VHA’s IT Investment Management Process Is Consistent with Leading Practices, but a Department-level Board Has Been Inactive and Clear Investment Selection Criteria Have Not Been Defined
paragraphs: According to leading practices for IT investment management,  establishing and following a systematic and organized approach to  investment management helps lay the foundation for successful,  predictable, and repeatable investment decisions. Critical elements  include instituting an IT investment board and ensuring that an  organization develops the process by which IT investments are selected,  reselected, and integrated with the process of identifying projects for  funding. Depending on its size, structure, and culture, an organization  may have more than one IT investment board and each investment board  may operate in accordance with its assigned authority and responsibility.  In addition, the investment selection process should include structured  reviews of IT proposals, the use of predetermined criteria for analyzing  and prioritizing proposals, and analysis and documentation of decisions  made to fund some proposals and not others.
VA has taken steps to establish a systematic and organized approach to  IT investment management. Specifically, the department has integrated  its investment management approach with its IT Multi-Year Programming  cycle, which is the process used by OI&T to identify and prioritize  business needs over a 5-year programming horizon. With the VA budget  submission and data collected from the prior Multi-Year Programming  cycle as the starting point for the annual process, OI&T uses the list of  priorities from VHA, VBA, and NCA to develop an initial IT Program. VA  has also instituted multiple levels of investment management, including  establishing IT governance and a selection approach in VHA, in addition  to department-level IT investment review boards.
Within VHA, the administration has developed a governance structure for  prioritizing business needs and selecting its IT investments based on  those needs. This structure, formally established in November 2015,  includes the following components.
Capability management boards: These four boards generally meet  monthly to engage with program offices and assess and rank the  priority of various business needs by scoring them with weighted  criteria related to, for example, how the proposal aligns with VHA  mission priorities and the expected benefits as well as the impact of  risk to VHA, the maturity of requirements, the complexity of the issue,  and the dependencies between individual investments.
Integration Board: The co-chairs of each of the capability  management boards generally meet monthly as the Integration Board  to ensure that the prioritized lists submitted by each capability  management board are consistent and that dependencies between  the proposals are assessed. The Integration Board begins to  incorporate cost estimates into the process, develops alternative  scenarios for prioritization that anticipate OI&T budget allocations, and  recommends a consolidated list of investment priorities to the IT  Committee.
IT Committee: This committee is charged with setting VHA’s IT  strategic direction, overseeing its IT governance and needs  prioritization process, and advocating for VHA’s IT funding. Further,  this committee is part of the National Leadership Council and is  responsible for coordinating with the Council’s other committees to  ensure that IT needs are appropriately supported with funding that is  consistent with VHA goals, and resolves issues in the execution of the  budget to include reprogramming, as appropriate. The committee  provides a final list of prioritized investments to the National  Leadership Council as part of the Multi-Year Programming process.
National Leadership Council and Under Secretary for Health: The  National Leadership Council is VHA’s advisory body for decision  making and is comprised of senior VHA leaders, including those  within the Office of the Under Secretary for Health. This body is  responsible for endorsing the VHA-related IT investment decisions  that are submitted to OI&T. According to VHA officials, the  administration negotiates with senior executives such as the Deputy  Secretary of VA and the CIO in building the budget request that goes  to OMB.
VHA’s Architecture and Requirements Investment Work Group supports  these governance boards by, for example, normalizing and analyzing the  submitted IT needs and providing data and cost estimates to help the  governance bodies make informed decisions. (See figure 3 for a depiction  of VHA’s IT governance structure.)
For its part, VA’s department-level IT governance is comprised of two  boards that are assigned the responsibility of combining the business  needs from VHA and the other business partners and formulating a final  IT budget according to department-wide priorities. According to OI&T’s IT  Multi-Year Programming guidance, the initial list of programs and their  associated funding levels proceeds through these boards for additional  review, adjustment, and approval.
IT Leadership Board: According to its charter, VA’s highest level IT  investment board is responsible for, among other things, aligning IT  resources with business needs, managing the projects, and  developing and approving the IT budget.
IT Planning, Programming, Budgeting and Execution Board: The  charter for this board states that it is to help facilitate the Multi-Year  Programming process, monitor budget execution, and make  recommendations to the IT Leadership Board regarding overall long- term plans. According to VA officials, this board also is to make  determinations on what projects are eligible for funding with the IT  appropriation.
However, the IT Leadership Board has not met since July 2015 and is not  currently functioning as the department-level IT investment board.  Further, VA has not documented criteria that the board could have used  to weigh tradeoffs between investments, determine whether one  investment is funded over another, or identify how investments are  reselected once they are operational.
Because the board has not met, OI&T officials stated that an ad hoc  group of senior executives was delegated responsibility for making IT  investment decisions for the fiscal years 2017-2021 Multi-Year  Programming cycle. However, VA did not document the criteria that these  groups used to make decisions, nor did the groups document their  decisions. For example, there was no documentation of the department’s  decision to not approve VHA’s high-priority request for $45.8 million in  proposed development funding to improve pharmacy IT capabilities in the  fiscal year 2017 cycle.
According to OI&T officials, VA has been working to change its approach  to department-level IT governance and investment selection as part of the  ongoing transformation that has been occurring since January 2016.  Among these changes, OI&T chartered 11 new governance boards by October 2016 that are to focus on various aspects of IT strategy,  solutions, and standards. One of these boards—the Portfolio Investment  Management Board—has been identified by its charter as the  department-level IT investment review board to be responsible for  integrating IT investment decisions with VA’s mission, strategic plan,  budget, and enterprise architecture.
While the Portfolio Investment Management Board has been defined as  the department-level decision-making body, officials said more time is  needed to determine how the board’s responsibilities will be carried out in  relationship to the other 10 boards, which also are responsible for various  aspects of IT projects, planning, and budgets. In addition, the Portfolio  Investment Review Board and OI&T have not issued additional guidance  or other documentation related to how the new IT governance structure  will work to oversee management of IT across the department.
While the transformation of OI&T has the potential to improve the  selection of IT investments going forward, the department has not yet  documented criteria related to how decisions and tradeoffs will be made  or fully demonstrated how the new structure will work. According to OI&T  officials, the transformation of IT governance is an evolving process and  they plan to continue to improve the process for selecting IT investments  and the budgeting process as the department builds the upcoming fiscal  year 2019 through fiscal year 2023 budget submissions. However,  without using a department-level board to govern IT investments and  criteria for selecting them, the department risks wasting limited resources  and funding investments that may not fully support VHA’s most important  business functions and priorities.
section_title: VHA’s Business Architecture Defines Its Core Business Functions, but Measurement of the Extent to Which Functions Are Supported Is Incomplete
paragraphs: Leading enterprise architecture and investment management practices  maintain that enterprise architecture can be used to link the organization’s  strategic mission value (performance results and outcomes) to its  technical resources. As such, organizations should implement a  methodology for ensuring that IT investments meet business needs and  comply with the architecture. In addition, the extent to which mission  value is actually realized indicates progress toward the desired state  defined in the architecture and should be periodically measured and  reported.
VHA has employed a methodology to identify its core business functions  in its enterprise architecture and has documented guidance for aligning or  mapping IT needs and investments to those functions. These activities  are performed to ensure that there is a link between what is reviewed  during the investment-selection process and the business needs of the  organization. Specifically, according to administration officials, the VHA  Business Function Framework is the architectural model that describes  the core business functions that are necessary to the mission of  delivering health care services and supporting the needs of veterans,  health care providers, and resource partners. This framework defines a  total of 262 core business functions as part of the VHA business  architecture. For example, one line of business described by the  framework is “Deliver Health Care.” Under this line of business, there are  86 supporting functions, such as “Provide Clinical Decision Support” and  “Provide Nursing Services,” which identify at a high level the core  business functions necessary to deliver health care at VA.
According to VHA, the Business Function Framework is primarily used to  show how business functions map to new service requests, requirements,  and IT systems, the results of which are input into the NSR database and  VASI. For example, VHA maps IT needs and investments (which can  include multiple systems) to the Business Function Framework. VHA  officials noted that every IT need and system is intended to be mapped to  one or more of the defined business functions.
VHA has also taken steps toward measuring and reporting the extent to  which mission value is actually realized. Specifically, the administration  has mapped core business functions to existing clinical, operational, and  outcome measures. According to the VHA Business Architecture team,  available performance metrics were aligned to a number of core business  functions for the fiscal years 2017 and 2018 reviews and the results were  provided to VHA capability management boards and could be viewed by  board members. In instances where a metric indicated poor performance,  proposed investments were assessed for their potential to help VHA  improve its performance.
Nevertheless, measurement of the extent to which business functions are  supported is incomplete. Specifically, VHA has aligned existing metrics  with 65 of the 262 core business functions for the fiscal year 2017 Multi- Year Programming cycle. For fiscal year 2018, the team reported that it  aligned metrics to 64 of the core business functions. According to VHA  officials, the Business Architecture team would like to identify additional  operational metrics used by VHA. However, the officials stated that the  Business Architecture team is not staffed to identify, track, trace, and  report on IT performance metrics and that IT metrics are OI&T’s  responsibility. Without aligning additional metrics to all core business  functions, VHA is not positioned to effectively gauge the extent to which  IT systems address its business needs and support the performance of its  mission.
section_title: VHA’s Core Business Functions Are Not Fully Supported by Current IT Systems
paragraphs: VA’s IT systems are generally aligned to VHA core business functions,  but the administration has unaddressed needs that indicate current IT  systems do not fully support the functions. To have an effective internal  control system, an organization should design its information systems to  achieve its objectives. The management processes discussed in this  report (i.e., strategic planning, investment management, and enterprise  architecture) are intended to help ensure that the department’s  investment decisions for IT systems address VHA’s strategic and  functional needs. VASI shows that the vast majority of VHA’s 262 core  business functions are supported by the department’s current IT systems  or, according to department officials, do not have a need for system  support.
However, our review of new service requests, which are requests in the  NSR database for identified IT needs submitted by VHA programs and  business owners, determined that VHA’s core business functions are not  fully supported by systems. The NSR database contains needs that have  been submitted over time that have not been addressed by an IT system  and provide an indication of functions that are not fully supported by  systems. In this regard, as of October 2016, VHA had 2,772 requests for  IT needs documented in the NSR database since 1998. Of these,  approximately 817 were open requests—IT needs identified throughout  VHA that had not been met. Further, 316, or about 39 percent, of these  open needs are long-standing—they have been open for more than 5  years. Figure 4 provides a breakdown of new service requests as of  October 2016.
According to department officials, requests are not of equal weight and  vary in level of impact and work effort required. For example, the NSR  database consists of requests ranging from the creation or modification of  reports to the development of new systems. Nonetheless, these requests  represent business needs that have not been met, which means there is  functionality that is not being provided.
The fact that business functions are not fully supported is further  illustrated when reviewing needs associated with three program areas— pharmacy benefits management, scheduling, and community care— which all have open requests that represent long-standing, unmet IT  needs. These programs are responsible for key functions and IT systems  related to health service delivery.
Pharmacy Benefits Management Services. As of November 2016, the  program office tracked more than 280 open requests to meet IT needs,  approximately 38 percent of which were identified 5 or more years ago.  For example, the office had a request from 2000 for the development of  an inpatient pharmacy order interface to share pharmacy order  information with external and commercial systems. In addition, the office  had a 2013 request related to a project intended to enhance and  modernize VistA Evolution Pharmacy. It also had two requests from 2014  related to a project intended to develop the ability to receive inbound  electronic prescriptions and a project intended to address known patient  safety issues.
Veterans Access to Care (scheduling and consults). As of late  September 2016, the program office had more than 20 open requests.  Approximately 32 percent of these requests were entered into the NSR  database more than 5 years ago. For example, the program office tracked  two requests from 2006 related to recommendations made by a VHA  Consult Task Force group. The group was created in August 2004 to  address disconnects among the consult package, the scheduling  package, and the electronic wait list. In addition, the office continues to  track a request made in 2007 for the development of a scheduling  application to address deficiencies including wait times, resource  management, and user satisfaction in order to improve coordination of  patient care. This significant long-standing request remains open after a  decade without plans for when and how an IT solution will be developed  to address this business need.
Community Care. The program office, which has been established more  recently than pharmacy benefits management or scheduling, was tracking  more than 50 open requests as of late September 2016. Approximately  30 percent of these requests were entered into the NSR database more  than 5 years ago but were still considered relevant to the community care  program. For example, the office tracked a request from 2006, related to  the IT solution for flagging emergency care claims. This request had been  unaddressed for more than 10 years and the absence of such a system  resulted in labor-intensive and error-prone manual processes. Program  officials stated that an IT solution to address this is scheduled for release  by the end of 2017.
Multiple factors have contributed to VHA’s core business functions not  being entirely supported by the department’s IT systems. VA spends a  significant amount of money on sustaining existing systems, which  department officials said has limited the funds available for enhancing or  modernizing those systems or acquiring new systems to address VHA’s  unmet needs. Furthermore, according to department officials, VHA is  challenged because the administration has more business needs than  available resources and funding.
Additionally, weaknesses in IT strategic planning, investment  management, and enterprise architecture processes previously discussed  in this report have contributed to a lack of understanding of the extent to  which VHA’s business functions require additional IT system support to  meet the needs and strategic goals of the administration. As a result, the  department risks continuing to make investment decisions and tradeoffs  that may fail to address gaps in IT support within its resource limits and  may hinder the progress VHA is able to make in improving delivery of  health care services to veterans.
section_title: Conclusions
paragraphs: To VA’s credit, the department’s IT strategic plans describe a vision and  identify goals and objectives related to IT in general and to health IT  within VHA. VHA and OI&T also have established a governance structure  responsible for prioritizing the administration’s business needs and  reviewing IT investments for inclusion in the budget. In addition, VHA’s  core business functions are documented in its enterprise architecture and  used to align business needs to IT investments as part of selecting  investments.
However, VA’s partial implementation of effective IT strategic planning,  investment management, and enterprise architecture has put the  department at risk of being unable to fully support VHA with the  information systems it needs to perform its mission of providing high- quality health care to veterans. Weaknesses in key processes leave VA  unable to gauge the extent to which it is providing information systems  that meet VHA’s needs. Specifically, the department has not assigned  targets or established metrics for measuring performance toward  achieving its strategic planning objectives. In addition, VA’s department- level IT investment management activities have lacked implementation of  governance boards, application of selection criteria, and documentation of  investment decisions. Also, VHA has only aligned metrics with about one  quarter of the core business functions identified in its enterprise  architecture. According to OI&T officials, ongoing transformation of IT  governance is intended to improve the process by which investments are  made. However, the results of this transformation have yet to be fully  documented and demonstrated. Thus, VA is not well positioned to meet  VHA’s information system needs.
Not surprisingly, VHA’s IT systems fall short of meeting the needs of  clinicians and the veterans they are to serve. While the administration’s  core business functions have been aligned to at least one of the  department’s current information systems, unaddressed business needs  remain and indicate that the functions are not fully supported. Further,  within three VHA program areas—pharmacy benefits management,  scheduling, and community care—many identified IT needs have been  unresolved or unfunded for 5 or more years. Thus despite identifying and  prioritizing needs, VHA’s core business functions have not been fully  supported by the department’s current information systems and may  remain unaddressed for a considerable amount of time. Until the  department fully implements IT management processes in accordance  with leading practices, it will lack assurance that its information systems  fully support VHA’s core business functions and delivery of health care  services to veterans.
section_title: Recommendations for Executive Action
paragraphs: To assist VA in improving key IT management processes to ensure that  investments support the delivery of health care services, we recommend  that the Secretary of Veterans Affairs direct the Under Secretary for  Health and the Chief Information Officer to take the following four actions:  Identify performance metrics and associated targets for the goals and  objectives in the department’s IT strategic plans, including the  Information Resources Management strategic plan and the Health  Information Strategic Plan, as they relate to the delivery of health IT  and the VHA mission.
Ensure that the department-level investment review structure is  implemented as planned and that guidance on the IT governance  process is documented and identifies criteria for selecting new  investments, and reselecting investments currently operational at  VHA.
Identify additional performance metrics to align with VHA’s core  business functions, and then use these metrics to determine the  extent to which the department’s IT systems support performance of  VHA’s mission.
Ensure that unmet IT needs identified by key program areas— pharmacy benefits management, scheduling, and community care— are addressed appropriately and that related business functions are  supported by IT systems to the extent required.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report (reprinted in appendix II), VA  agreed with our four recommendations. The department also provided  information on actions it has taken or planned to implement our  recommendations, including target completion dates for those actions.
For example, in its comments, VA asserted that it has taken steps that  fully addressed our recommendation to ensure that its department-level  investment review structure is implemented as planned, and that  guidance on the IT governance process is documented and identifies  criteria for selecting new investments and reselecting investments that  are currently operational at VHA. Specifically, the department noted that it  had established a new governance process in October 2016 and  implemented it as planned. Further, the department provided, as an  attachment to its comments, an updated charter for the Portfolio  Investment Management Board (dated March 28, 2017) as additional  evidence of the board’s process for evaluating IT investments. In our  follow up on the department’s implementation of our recommendations,  we will assess whether the actions noted are fully responsive to this  recommendation.
The department also discussed planned actions for addressing our  recommendation related to identifying performance metrics and targets  for the goals and objectives in VA’s IT strategic plans. Specifically, the  department described its intention to develop or revise and maintain  performance metrics that align with strategic and health IT goals and  objectives.
VA also outlined steps the department intends to take in response to our  recommendation that it identify additional metrics to align with VHA’s core  business functions and then use these metrics to determine the extent to  which the department’s IT systems support VHA’s mission. These steps  include developing a set of core metrics to provide continuous input into  investment portfolio decisions and establishing a methodology for  ensuring that IT investments are aligned to business needs and that  expected outcomes are defined prior to making the investments.
Further, in response to our recommendation that it ensure that unmet IT  needs for the pharmacy benefits management, scheduling, and  community care program areas are addressed appropriately, the  department stated that VHA leadership has recently reviewed all  outstanding requests from these program areas to confirm their validity. In  addition, the department stated that it plans to include the outstanding  needs of these key program areas in its VHA IT Requirements  Governance Process during fiscal year 2018 to ensure the needs are  addressed in this multi-year planning review.
According to VA, its actions in response to our recommendations are  expected to be completed by the end of fiscal year 2018. If the  department ensures that these and other activities it identified are  appropriately documented and effectively implemented, then VA should  be better informed to make IT investment decisions that improve the  delivery of health care services to veterans.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, the Under Secretary for  Health, the Chief Information Officer, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov  If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this study were to determine the extent to which the  Department of Veterans Affairs’ (VA) (1) information technology (IT)  management processes are consistent with leading practices and (2)  current IT systems support the Veterans Health Administration’s (VHA)  core business functions.
To address the first objective, we compared VA’s IT management  processes for IT strategic planning, investment management, and  enterprise architecture to leading practices that federal statutes, prior  GAO reports, and the Office of Management and Budget (OMB) have  identified to assist organizations with improving the management  processes. This comparison focused on the specific aspects of the  processes that are intended to ensure that IT investments meet the  business needs of the VHA organization. For example:  IT strategic planning: We identified the strategic plans and related  planning guidance issued by the Office of Information and Technology  (OI&T) and VHA that focused on IT systems and health care IT at VA.  We reviewed the department’s assertions in these plans for how IT  strategic goals align to the goals of the VA Strategic Plan. Then, we  compared the contents of the plans to leading practices identified from  federal statutes, prior GAO reports, guidance from OMB related to IT  strategic planning, and a relevant VA directive. In particular, we  determined whether VA had taken steps to include strategic goals and  objectives that define the levels of performance to be achieved as  they relate to ensuring that IT supports the mission needs of the  department and VHA; and established related metrics that are  specific, verifiable, and measurable.
IT investment management: We analyzed charters and meeting  minutes establishing and demonstrating the implementation of  governance structures responsible for IT investments at VHA and the  department level. We then compared the existing governance  structure to critical processes and activities related to governance  described in GAO’s IT Investment Management framework. We also  analyzed department documentation and guidance related to how  business needs are identified and prioritized by VHA and selected to  be part of the budget for IT investments by OI&T. We examined  results of this process for the fiscal year 2017 budget formulation  process. We compared our analysis to critical processes related to  investment selection described in GAO’s framework. In addition, we  interviewed officials familiar with the VHA prioritization process and  OI&T investment management and budget formulation processes to  clarify department policies and guidance.
Enterprise architecture: We analyzed department documentation  and interviewed cognizant officials about the steps taken to ensure  that IT investments support the department’s business needs and  compared our findings to key elements described in GAO’s Enterprise  Architecture Management Maturity Framework and IT Investment  Management. Further, we compared the number of metrics that were  aligned to business functions by VHA to the list of all business  functions identified in the enterprise architecture to determine the  extent to which the functions have associated metrics available to  inform the investment management process.
To address the second objective, we examined department data to  understand how VA might demonstrate that its IT systems are designed  to meet its objectives. First, we analyzed the VHA Business Function  Framework (Version 2.11), which documents the VHA functional  operations within the business architecture, to compile a list of all core  business functions that VHA has determined are necessary to deliver  health care. This framework provides the basis by which the department  shows relationships between various components of the enterprise  architecture and is used to help view, organize, and prioritize VHA’s  business activities.
We then compared this list of core business functions to data in the VA  Systems Inventory (VASI) database, which identifies VA’s current  inventory of IT systems and how they are mapped to the VHA core  business functions. VHA officials noted that VASI is the authoritative  source for business function mapping. We assessed the reliability of data  from VASI and determined that the data were reliable for the purposes of  our reporting objectives.
For any core functions initially not aligned to a current IT system, we  reconciled the differences with cognizant VA officials. There were 5  functions (from a total of 262 functions) that could not be reconciled. We  determined that this number, which represented less than 2 percent of the  total number of functions, was not significant to our findings. While the  results of this alignment showed a relationship between many current IT  systems and VHA’s core business functions can be demonstrated, the  results did not provide insight into how well the functions are being  supported by those IT systems.
We then analyzed data from VHA’s new service request (NSR) database,  which captures information related to business needs such as IT  enhancements submitted throughout the department. We analyzed data  from the NSR database to identify the number of requests in the  database, when requests were entered, and the number of requests that  remain open. Our analysis allowed us to describe the number of open  requests, but could not provide insight into the depth of work required for  requests themselves, nor the weight the business owners assigned to  each open need, because the NSR database does not include data on  the importance, level of impact, and work effort required to address each  request.
We found the VA data from VASI and the NSR database to be sufficiently  reliable for the purposes of our reporting objectives and used the data as  evidence to support our findings, conclusions, and recommendations. For  each data set, we reviewed documentation related to the databases, such  as the data dictionary, tested the data sets to look for duplicate records  and missing data in key fields, and examined the relationship between  data elements. We also interviewed department officials about data  reliability and internal controls procedures for the database and  interviewed knowledgeable officials on the results of our findings.
We conducted additional analyses of three programs related to health  service delivery on which we have previously reported—Pharmacy  Benefits Management Services, Veterans Access to Care (scheduling  and consults), and Community Care. Our review of NSRs for the  aforementioned program offices included verifying the open NSRs  assigned to each program office and interviewing cognizant officials from  VHA regarding the IT systems used by the three programs, the needs  identification and management process to understand the extent to which  VHA business needs are being addressed, and about the extent to which  current systems supporting VHA core business functions in their  respective areas. The results of this analysis are not generalizable to all  functional areas, but provide insight into the extent of IT support for the  three specific programs.
We conducted this performance audit from December 2015 to June 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: David A. Powner, (202) 512-9286 or pownerd@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Mark Bird (Assistant Director),  Jennifer Stavros-Turner (Analyst in Charge), Chris Businsky, Rebecca  Eyler, Jacqueline Mai, Dwayne Staten, and Charles Youman made key  contributions to this report.