VA IT Management: Organization Is Largely Centralized; Additional Actions Could Improve Human Capital Practices and Systems Development Processes
Released: 2016-09-16T00:00:00

Letter

Background

VA’s mission is to promote the health, welfare, and dignity of all veterans in recognition of their service to the nation by ensuring they receive medical care, benefits, social support, and lasting memorials. It is the second largest federal department and, in addition to its central office located in Washington, D.C., has field offices throughout the United States, as well as the U.S. territories and the Philippines.

The department’s three major components—the Veterans Benefits Administration (VBA), the Veterans Health Administration (VHA), and the National Cemetery Administration (NCA)—are primarily responsible for carrying out its mission. More specifically, VBA provides a variety of benefits to veterans and their families, including disability compensation, educational opportunities, assistance with home ownership, and life insurance. VHA provides health care services, including primary care and specialized care, and it performs research and development to improve veterans’ needs. Lastly, NCA provides burial and memorial benefits to veterans and their families.

Collectively, the three components rely on approximately 340,000 employees to provide services and benefits. These employees work in 167 VA medical centers, approximately 1,200 community-based outpatient clinics, 300 veterans centers, 56 regional offices, and 131 national and 90 state or tribal cemeteries situated throughout the nation.

VA Relies Extensively on Information Technology

VA operates and maintains an IT infrastructure that is intended to provide the backbone necessary to meet the day-to-day operational needs of its medical centers, veteran-facing systems, benefits delivery systems, memorial services, and all other IT systems supporting the department’s mission. The infrastructure is to provide for data storage, transmission, and communications requirements necessary to ensure delivery of reliable, available, and responsive support to all VA staff offices and administration customers, as well as veterans.

In this regard, the department operates approximately 240 information systems, manages 314,000 desktop computers and 30,000 laptops, and administers nearly 460,000 network user accounts for employees and contractors to facilitate providing benefits and health care to veterans. These systems are used for the determination of benefits, benefits claims processing, patient admission to hospitals and clinics, patient care through telehealth, and access to health records, among other services.

For example, VBA relies on its disability benefits claims processing system—the Veterans Benefits Management System—to collect and store information such as military service records, medical examinations, and treatment records from VA, the Department of Defense, and private medical service providers. Information technology is widely used and critically important to supporting the department in delivering health care to veterans. VHA’s systems provide capabilities to establish and maintain electronic health records that health care providers and other clinical staff use to view patient information in inpatient, outpatient, and long-term care settings. Specifically, the Veterans Health Information Systems and Technology Architecture, known as VistA, consists of many computer applications and modules that collect, among other things, information about a veteran’s demographics, allergies, procedures, immunizations, and medical diagnoses.

In 2014, VA issued its 6-year strategic plan, which emphasizes the agency’s goal of increasing veterans’ access to benefits and services, eliminating the disability claims backlog, and ending veteran homelessness. According to the plan, the department intends to improve access to benefits and services through the use of improved technology to provide veterans with access to more effective care management. The plan also calls for VA to eliminate the disability claims backlog by fully implementing an electronic claims process that is intended to reduce processing time and increase accuracy. Further, the department has an initiative under way that provides services, such as health care, housing assistance, and job training, to end veteran homelessness. To this end, VA is working with other agencies, such as the Department of Health and Human Services, to implement more coordinated data entry systems to streamline and facilitate access to appropriate housing and services.

VA reported spending approximately $3.9 billion to improve and maintain its IT resources in fiscal year 2015. Specifically, the department reported spending approximately $548 million on new systems development efforts, approximately $2.3 billion on maintaining existing systems, and approximately $1 billion on payroll and administration.

For fiscal year 2016, the department received appropriations of approximately $4.1 billion for IT. The department had requested approximately $505 million for new systems development efforts, approximately $2.5 billion for maintaining existing systems, and approximately $1.1 billion for payroll and administration. In addition, in its 2016 budget submission the department requested appropriations for the following purposes, among others: improving veteran access to benefits and services ($378.1 million); reducing the time it takes to process disability claims and increase accuracy ($295.1 million); expanding information security ($180.3 million); implementing the Veterans Access, Choice, and Accountability Act of 2014 (the Veterans Choice Act) ($440.6 million); maintaining the IT infrastructure ($1.828 billion); and improving VistA ($182.6 million).

Further, for fiscal year 2017, the department’s budget request included nearly $4.3 billion for IT. The department requested approximately $471 million for new systems development efforts, approximately $2.5 billion for maintaining existing systems, and approximately $1.3 billion for payroll and administration. In addition, in its 2017 budget submission, the department requested appropriations to make improvements in a number of areas, including the following: veterans’ access to health care, to include enhancing health care- related systems, standardizing immunization data, and expanding telehealth services ($186.7 million); veterans’ access to benefits by modernizing systems supporting benefits delivery, such as the Veterans Benefits Management System and the Veterans Services Network ($236.3 million); veterans’ experiences with VA by focusing on integrated service delivery and streamlined identification processes ($171.3 million);

VA employees’ experiences by enhancing internal IT systems ($13 information security, including implementing strong authentication, ensuring repeatable processes and procedures, adopting modern technology, and enhancing the detection of cyber vulnerabilities and protection from cyber threats ($370.1 million).

VA Has Faced Long- Standing Challenges in IT Management

Even as the department has engaged in various attempts to improve its IT management capabilities, we have issued numerous reports that highlighted challenges facing its efforts. In February 2014, we reported that about 2 years after taking actions toward developing a single interoperable electronic health record system with the Department of Defense (DOD), VA and DOD announced that they would instead work toward modernizing their separate existing health information systems and work to ensure those systems could share information. However, VA proceeded with this plan without developing a cost and schedule analysis to support the assertion that pursuing a separate modernized system while enabling interoperability with DOD’s system would be less expensive and could be achieved faster than developing a single system, which is contrary to best practice. We recommended that VA work with DOD to develop a cost and schedule estimate for their modernization approach, as well as their joint effort, and compare the estimates to determine which approach would cost less and take less time. VA concurred with our recommendation, but had not completed actions to address it.

In addition, we reported in May 2010 that after spending an estimated $127 million over 9 years on its outpatient scheduling system project, VA had not implemented any of the planned system’s capabilities and was essentially starting over by beginning a new initiative to build or purchase another scheduling system. We also noted that VA had not developed a project plan or schedule for the new initiative, stating that it intended to do so after determining whether to build or purchase the new application. We recommended that the department take six actions to improve key systems development and acquisition processes essential to the second outpatient scheduling system effort. The department generally concurred with our recommendations, but as of May 2016, had not completed actions to implement four of the six recommendations.

Key IT-Related Functions Provide a Foundation for IT Management

Effectively managing IT needs depends on federal departments and agencies, including VA, having key functions in place. Toward this end, we have identified and reported on a set of essential and complementary functions that serve as a sound foundation for IT management. These include the following:

Leadership: Effective leadership, such as that of a CIO, can drive change, provide oversight, and ensure accountability for results. Congress has also recognized the importance of having a strong agency CIO. For example, as part of the Clinger-Cohen Act, Congress required executive branch agencies to establish the position of agency CIO. The act also gave these officials responsibility and accountability for IT investments, including IT acquisitions, monitoring the performance of IT programs, and advising the agency head on whether to continue, modify, or terminate such programs. More recently, in December 2014, Congress passed federal information technology acquisition reform legislation (commonly referred to as FITARA), which strengthened the role that agency CIOs are to play in managing IT. For instance, the law requires the head of covered agencies to ensure that the CIO has a significant role in the decision process for IT budgeting, as well as the management, governance, and oversight processes related to IT.

Strategic planning: Strategic planning defines what an organization seeks to accomplish and identifies the strategies it will use to achieve desired results. A defined strategic planning process allows an agency to clearly articulate its strategic direction and to establish linkages among planning elements such as goals, objectives, and strategies. A well-defined IT strategic planning process helps ensure that an agency’s IT goals are aligned with its strategic goals. Also, as part of its strategic planning efforts, an organization should develop an enterprise architecture, which is an important tool to help guide the organization toward achieving the goals and objectives in its IT strategic plan. In addition, the organization should implement human capital management practices to sustain a workforce with the skills necessary to execute its strategic plan.

Systems development and acquisition: Agencies should follow disciplined processes for developing or acquiring IT systems. These include defining the requirements that address the needs of the system users, managing project risk to identify potential problems before they occur, reliably estimating cost to help managers evaluate affordability and performance against a project’s plans, and developing an integrated and reliable master schedule that defines when and how long work will occur and how each activity is related to the others, among others. Best practices in these areas have been identified by organizations such as SEI and GAO.

Quality assurance: Effective quality assurance supports the delivery of high-quality products by providing staff and management with appropriate visibility into, and feedback on, processes and associated work products throughout the life of a systems development or acquisition project. Quality assurance can also help ensure that planned processes are implemented as intended.

Systems operations and maintenance: Given the size and magnitude of the investments that executive branch agencies make in this area (nearly 60 percent of VA’s reported IT spending in 2015), it is important that operations and maintenance be managed effectively to ensure that existing investments (1) continue to meet agency needs, (2) deliver value, and (3) do not duplicate or overlap with other investments. To accomplish this, agencies are required by the Office of Management and Budget to perform annual operational analyses of these investments, which are intended to serve as periodic examinations of the investments’ performance against, among other things, established costs, schedules, and performance goals.

Service management: Agencies should develop and implement a process for ensuring that IT services, such as server management and desktop support, are aligned with and actively support the business needs of the organization. The Information Technology Infrastructure Library identifies key practices for successful service management. These include developing a service catalog that identifies all IT services delivered by the service provider, as well as establishing service-level agreements between the IT service provider and its customer on the expected service-level targets.

IT investment management: IT projects can significantly improve an organization’s performance, but they can also become costly, risky, and unproductive. Agencies can maximize the value of these investments and minimize the risks of acquisitions by having an effective and efficient IT investment management and governance process, as described in GAO’s guide to effective IT investment management. Emphasizing the importance of investment management, the Clinger-Cohen Act requires executive branch agencies to establish a process for selecting, managing, and evaluating IT investments in order to maximize the value and assess and manage the risks of the acquisitions.

Information security and privacy: Effective security for federal IT systems and data is essential to prevent data tampering, disruptions in critical operations, fraud, and inappropriate disclosure of sensitive information, including personal information entrusted to the government by members of the American public. Recognizing the importance of information security and privacy, Congress enacted the Federal Information Security Modernization Act of 2014 (FISMA 2014), which requires executive branch agencies to develop, document, and implement an agency-wide information security program. Additionally, in order to help agencies develop such a program, the National Institute of Standards and Technology has developed guidance for information security and privacy.

VA Manages and Performs Key IT- Related Functions through a Largely Centralized Organization, Which Is Taking Steps to Address Challenges

Since 2007, VA has been operating a centralized IT organization in which most key functions intended for effective management of IT are performed by the department’s Office of Information & Technology (OI&T) and led by the department-level CIO. Two department-level IT governance boards also have responsibility to assist in performing certain functions. However, the department has faced challenges to effectively managing IT using a centralized approach. These include ensuring that (1) all projects are managed and controlled by OI&T, (2) effective communication occurs between OI&T and the VA business units, and (3) new IT capabilities are delivered efficiently and cost-effectively. To address these and other challenges, the CIO recently announced a transformation initiative aimed at improving OI&T’s accountability for, and focus on, achieving a more transparent and veteran-centric organization.

VA Has Largely Centralized Its Management of IT-Related Functions within OI&T

OI&T has responsibility for managing the majority of VA’s IT-related functions, under a largely centralized structure that the department attempted for nearly two decades to establish. The department’s efforts toward achieving a centralized management structure were strengthened when the Military Quality of Life and Veterans Affairs Appropriations Act, 2006 established a new appropriation account for the department’s IT systems. In addition, in order to obligate or expend amounts for IT systems development, modernization, and enhancement, the Consolidated Appropriations Act of 2016 required that the Secretary or CIO submit to the Committees on Appropriations of both Houses of Congress a certification of the amounts to be obligated and expended for each development project.

Subsequent to the 2006 appropriations act, the Secretary of Veterans Affairs assigned control over the IT appropriation to a department-level CIO. In addition, the Secretary approved a centralized organization structure for IT-related functions in February 2007. By April 2007, all of the department’s personnel that had been involved in IT operations, maintenance, and development, with the exception of those in the Office of Inspector General, were permanently assigned to OI&T. (See appendix II for a summary of key events in the history of VA’s IT centralization efforts.)

The CIO serves as the head of OI&T and, accordingly, is responsible for providing effective leadership over the department’s IT activities. The CIO reports directly to the Office of the Secretary of Veterans Affairs through the Deputy Secretary and advises the Secretary regarding the execution of the IT appropriation. In addition, the CIO is expected to serve as the principle advisor to top management officials, such as the Under Secretaries of each of the three administrations, on matters relating to information and technology management in the department. This official also is tasked with reviewing and approving IT investments, as well as overseeing the performance of IT programs and evaluating them to determine whether to continue, modify, or terminate a program or project.

Under the CIO’s leadership, seven organizational units within OI&T had responsibility for performing and managing the other specific IT-related functions through March 2016.

Architecture, Strategy, and Design: This office was responsible for the department’s IT strategic planning. Specifically, it was tasked with providing a framework of strategies and processes to ensure IT programs and projects are designed and executed to satisfy current and future department needs.

Further, the office was tasked with performing other functions, such as the development and maintenance of an integrated technical, business, systems, and data architecture; development of systems design, engineering, and integration standards; and the examination of existing IT requirements and solutions for efficiency and potential redundancy.

Product Development: Systems development was managed by this office. Among other things, the office was responsible for developing and testing software products, including new products and upgrades to existing legacy systems, based on defined requirements from its customers (which are the three administrations and other staff offices).

In addition to systems development, the office had responsibility for monitoring the progress of all IT projects and reporting that progress to the department’s CIO and other senior leadership. This office was also tasked with developing and maintaining policies and guidance for implementing, and ensuring that responsible staff are trained in using, the department’s project management and accountability system, referred to as PMAS. (This management tool is discussed in greater detail later in the report.)

IT Resource Management: Acquisition processes and strategies were facilitated by the IT Resource Management office. Toward this end, the office was tasked with providing acquisition program management oversight and disseminating acquisition policy and procedures, among other things.

In addition, the IT Budget and Finance office within IT Resource Management was tasked with planning, executing, and overseeing the department’s technology budget, as well as ensuring that the appropriation funds are used appropriately. According to a director within the IT Budget and Finance office, this office was responsible for performing these functions for the fiscal year 2015 and 2016 IT budgets. The director also stated that the CIO, through this office, oversaw 100 percent of the IT appropriation. Further, in fiscal year 2016 the IT Budget and Finance office was charged with creating certification letters signed by the CIO and submitted to the Committees on Appropriations detailing the systems development, modernization, and enhancement projects VA had planned for the IT appropriation. To date, the CIO has certified projects for fiscal year 2016 in October 2015, February 2016, and April 2016.

Office of Quality, Performance, and Oversight: This office was tasked with managing quality assurance activities. It was intended to act as an independent party to determine if OI&T’s governance processes are adequate and functioning as anticipated. The office was also responsible for facilitating the reporting and understanding of performance measures and metrics related to IT program activities and strategic objectives. Further, it was charged with providing independent systems integration testing and other services to help ensure the integrity of the department’s systems and compliance with applicable guidance, such that provided through PMAS.

In addition, the office was responsible for providing organizational development and training for IT, to include coverage of information security, privacy, and rules of behavior. It also was responsible for establishing IT human capital policies and conducting workforce planning.

Office of Service Delivery and Engineering: All operations and maintenance activities associated with the department’s IT environment, including those in medical centers and regional offices, were directed by the Office of Service Delivery and Engineering. This office also was tasked with overseeing and managing VA’s data centers, network, and telecommunications; monitoring production for all information systems; delivering operations services (including deployment, maintenance, monitoring, and support) to all of the department’s geographic locations; and engineering and designing all of the IT system platforms and infrastructure.

Office of Information Security: The department’s information security and privacy infrastructure was managed within this office. In this regard, the office was responsible for, among other things, ensuring the confidentiality, integrity, and availability of the department’s information and information systems, as well as cybersecurity, risk management, records management, incident response, critical infrastructure protection, and business continuity. The office also was charged with developing, implementing, and overseeing the policies, procedures, training, communication, and operations related to improving how the department and its partners safeguard the personally identifiable information of veterans and VA employees.

Office of Customer Advocacy: This office was in charge of service management. In this regard, it was responsible for working with VHA, VBA, NCA, and the department’s staff offices to ensure that OI&T provides the services they need. The office also had responsibility for collecting and analyzing customer satisfaction metrics, and for responding to IT support requests at the department, business office, and employee levels.

Figure 1 depicts the seven organizational units within OI&T that had responsibility for performing and managing IT-related functions through March 2016.

OI&T reported that its organizational units performed key IT-related functions with the support of nearly 7,300 federal employees and approximately 7,800 contractor staff. Nearly 600 of the federal employees were located in Washington, D.C., at the VA Headquarters, while the vast majority of the employees were assigned to other facilities, such as medical centers and regional offices.

Table 1 identifies each OI&T organizational unit and the number of employees and approximate number of contractors that supported VA’s IT program.

In addition to these organizational resources, OI&T developed PMAS to serve as a mechanism for overseeing the performance of IT programs and ensuring accountability for results. PMAS was intended to provide information to facilitate oversight through a dashboard that captures project data, such as descriptions of the projects, baseline milestone dates, actual milestone dates, contract information, and project status information. The CIO’s responsibilities with regard to PMAS include authorizing new projects and project increments in the system; approving the funding needed for the projects; monitoring a project’s status via reporting, review, and assessment; and addressing any significant issues with the projects.

Further, according to the CIO and OI&T documented procedures, TechStat reviews are required for every project that fails to deliver on its committed delivery date. TechStat reviews are the mechanism that the CIO, along with other senior leaders, used to determine whether a project should continue, be modified, or be terminated.

Governance Boards Are to Assist the CIO with IT- Related Functions

In VA’s organization structure, the IT investment governance boards are intended to play a role in strategic planning and IT investment management, as they are responsible for providing the CIO with recommendations on the IT investments they feel would best meet the strategic and business objectives of the department. These two boards are intended to provide executive oversight for IT initiative planning and management.

IT Planning, Programming, Budgeting, and Execution Board: The board is tasked with recommending (1) the prioritization of all IT funding requests, including what should and should not be funded during the year; (2) execution of the IT appropriation; and (3) decisions as to whether to fund IT-related projects outside of the IT appropriation. The IT Chief Financial Officer, who also serves as the Deputy Assistant Secretary for the IT Resource Management office within OI&T, serves as the board’s chairperson. The board includes voting members from VA’s three administrations, as well as non- voting members from the OI&T organizational units, among others. In March 2016, the CIO stated in congressional testimony that the roles of this board will be assumed by a newly established Enterprise Program Management Office (EPMO).

IT Leadership Board: This board is tasked with reviewing and validating portfolio recommendations from the Planning, Programming, Budgeting, and Execution Board, as well as making decisions on any issues that cannot be resolved by that board. According to the department’s FITARA implementation plan, the IT Leadership Board has been tasked with approving the IT budget and then submitting it to the VA Executive Board—the department’s highest-level decision-making board chaired by the Secretary—for approval. The CIO serves as the IT Leadership Board’s chairperson and its members include the Under Secretaries for VA’s three administrations and all VA staff offices.

VA Is Taking Action to Address Challenges to Effectively Managing IT

Although VA centralized its key functions in order to maintain better control over resources, OI&T has faced challenges to fully implementing and managing IT under a centralized organizational structure. In particular, independent assessments of the department’s efforts in 2013 and 2015 showed that the office has had difficulty in preventing IT activities from occurring outside the control of OI&T. It has also been challenged in effectively collaborating with the department’s various business units and in efficiently and cost-effectively delivering new IT capabilities.

IT activities have occurred outside the control of OI&T. In September 2013, the VA Inspector General reported that the department’s Office of Acquisition, Logistics, and Construction had used $13 million in supply funds, rather than approved IT funds, to develop a suite of electronic contracting applications to support the office’s procurement process. Further, the report noted that the Office of Acquisition, Logistics, and Construction had not used the department’s IT project management system, PMAS, to develop the contracting system, although doing so was mandated by the Secretary in June 2009 for all IT development projects department- wide.

According to the report, the office followed its own project management process that was similar to PMAS. However, in doing so, the system development effort was not subjected to OI&T’s risk management, monitoring, oversight and control, and reporting processes within PMAS. According to the report, officials in the Office of Acquisition, Logistics, and Construction incorrectly believed that VA allowed for a PMAS exception because they had used supply funds, and not IT funds, for the development. The Inspector General recommended that the Office of Acquisition, Logistics, and Construction implement controls to ensure the systems development project and all future IT development efforts fall within the control and oversight of the PMAS process. The office concurred with the recommendation.

More recently, in March 2015, the Inspector General reported that VHA had improperly obligated $92.5 million and spent $73.8 million in medical support and compliance funds, rather than IT funds, to develop its health care claims processing system through the VHA Financial Services Center. According to the report, the former VHA Deputy Chief Business Officer stated that funds and development services provided through the Financial Services Center were used in hopes of achieving a faster delivery of the claims processing system. The Inspector General reported that, by doing so, VHA had avoided competing with other VA projects for IT systems appropriations. Thus, the Inspector General recommended that the Under Secretary for Health obtain the appropriate funding to support the development of the claims processing system. The interim Under Secretary for Health concurred with the recommendation and agreed to establish oversight mechanisms and issue guidance to ensure that VHA uses the appropriate funds.

Collaboration between OI&T and business units was not always effective. In September 2015, challenges with collaboration between OI&T and VA business units were highlighted by an independent assessment of VA’s health care delivery systems and management processes, including health IT processes. Several clinicians interviewed for that assessment stated that collaboration between systems developers and clinicians seemed to have disappeared after the centralization, resulting in uncoordinated execution of health IT strategies and limited development of new and improved capabilities for existing IT systems. The assessment noted that although the goals of OI&T and VHA did not conflict at the strategic level, the organizations often did not agree on priorities for executing the strategic plans. The assessment team recommended that VA and VHA transform IT strategy, planning, and execution by, among other things, establishing service-level agreements and refining the planning and budgeting process to ensure that business needs are effectively identified, prioritized, funded, and used to drive health IT investments. The team recommended that VA also develop a governance policy to ensure the strategic plans are executed well and in a timely manner.

Similarly, staff within the Office of Finance informed us of their dissatisfaction because OI&T controls the budget and the priorities for systems development efforts and IT change management. These staff stated that OI&T’s priorities do not always align with the needs of the administrations or other staff offices, thus potentially delaying the delivery of needed services.

PMAS limited VA’s efforts to efficiently and cost-effectively deliver new IT capabilities. The September 2015 independent assessment also stated that overly demanding processes for system development, as defined by OI&T’s PMAS, impeded cost-effective delivery of new health IT capabilities and limited VA’s ability to measure the value of its investments. The assessment report further stated that the PMAS process was schedule driven and risk averse, leading many project managers to limit the amount of functionality in each release, thereby increasing the total time for any useful capability to be released. In addition, assessment interviewees indicated that documentation required by PMAS consumed a significant percentage of development time. Accordingly, the assessment team recommended that VA establish product-focused teams to ensure delivery of needed capabilities to users and that the department’s system development process be modified from focusing on documentation and schedule to focusing more on functionality delivered.

CIO Transformation Strategy Aims to Address Challenges

Recognizing challenges in IT management, the CIO initiated an effort in January 2016 to transform the focus and functions of OI&T. In doing so, the CIO stressed the Secretary’s goal of achieving a more veteran- focused organization that places greater emphasis on transparency, accountability, innovation, and teamwork. The CIO’s transformation strategy calls for OI&T to focus on stabilizing and streamlining core processes and platforms, mitigating weaknesses highlighted in information security and GAO assessments, and improving outcomes by institutionalizing a new set of IT management capabilities.

According to the CIO, one of the most significant challenges the department faces is securing its assets, networks, systems, and data. To address this challenge, in September 2015, as part of the CIO’s effort to transform OI&T, VA documented a long-term cybersecurity strategy. This strategy identified five goals for successfully building a comprehensive cybersecurity capability: (1) protecting veterans’ information and VA data by ensuring privacy concerns are addressed and ensuring secure technology and data systems; (2) providing secure and resilient information systems technology, business applications, publicly accessible platforms, and shared data networks; (3) protecting VA infrastructure and assets, including the boundary environments that provide potential access and entry into VA; (4) enabling effective operations by improving governance and organizational alignment at enterprise, operational, and tactical levels (points of service interactions); and (5) recruiting and retaining a talented cybersecurity workforce covering multiple disciplines within cybersecurity.

As another part of this transformation, the CIO began transitioning the oversight and accountability of IT projects from PMAS to a new project management process called the Veteran-focused Integration Process (VIP) in January 2016, in an effort to better streamline systems development and speed up the delivery of new IT capabilities. According to the VIP guide, all systems development projects that touch the VA network or use money from the IT appropriation will be required to follow VIP. The CIO intends for VIP to facilitate an expanded use of the Agile system development methodology resulting in, for example, the delivery of system functionality on a more frequent, 3-month delivery cycle, whereas PMAS delivered functionality in 6-month cycles. In addition, VIP is intended to streamline IT project management by reducing the number of artifacts (i.e., documentation) that must be developed (from 58 to 8) and the number of critical decision points (from 5 to 2). According to the CIO’s transformation strategy, under the new VIP process, OI&T will also be required to consider security and architecture standards early in project planning phases, whereas PMAS required this later in the development process. According to the CIO, the department intends to have all IT projects managed under VIP by September 30, 2016.

In addition to implementing VIP, the CIO also intends to establish five new functions within OI&T as part of the transformation:

Enterprise Program Management Office. This office began initial operations in April 2016 and is intended to serve as OI&T’s portfolio management and project tracking organization. According to the strategy, the goals of the new organization are to align IT portfolios with agency strategic objectives; enhance visibility and governance; analyze and report on portfolio performance metrics; ensure the overall health of the IT portfolio; and optimize resources for projects, people, and timelines. EPMO includes six functional areas: (1) Intake and Analysis of Alternatives, responsible for working with the VA administrations and other staff offices to develop requirements to meet the needs of veterans, provide analysis of alternative approaches to meeting those requirements, and integrate information security; (2) IT Portfolios, responsible for consolidating programs and projects under five IT portfolios (Health, Benefits, Cemeteries, Corporate, and Enterprise services); (3) Project Special Forces, responsible for mitigating issues that put projects at risk of failure; (4) Lean Systems Engineering, responsible for metrics gathering and analysis, development of process tools, human resources, and training; (5) Transition Release and Support, responsible for managing OI&T’s integrated calendar supporting VIP; and (6) Application Management, responsible for IT implementation efforts, including testing, design, and data management.

The EPMO is expected to replace the Product Development office and the Planning, Programming, Budgeting, and Execution board. As of July 2016, VA had not provided information about whether and how the transformation will affect the IT Leadership Board.

Account Management. The account management function, led by three account managers, is to be responsible for managing the IT needs of OI&T’s business partners—the VA’s major components. Account managers are to interface directly with the VA administrations and staff offices to understand their needs, help identify and define the solutions to meet those needs, and represent their customers’ interests by reporting directly to the CIO. In this regard, account managers are to submit their customers’ IT requirements to the EPMO, ensure that their business needs are understood by OI&T, and ensure that business solutions are designed to meet their customers’ specifications. This function is also tasked with advocating for their customers in the budget process. OI&T intends this new function to address the challenge of effectively collaborating with business units. As of May 2016, the three account managers were in place.

Quality and Compliance. This function is to be responsible for establishing effective policy governance and standards and ensuring adherence to the policies and standards. In addition, the quality and compliance function is expected to be charged with identifying, monitoring, and measuring risks across the OI&T organization.

Data Management Organization. The organization is being established to improve both service delivery and the veteran experience by engaging with data stewards to ensure the accuracy and security of the information collected by VA. The organization is intended to institute a data governance strategy; engage with business owners to ensure accuracy and security of collected data; analyze data sources to form an enterprise data architecture; and establish metrics for data efficiency, access, and value. OI&T also intends for the organization to identify trends in the data collected on each veteran that could improve their health care by providing predictive care and anticipating needs.

Strategic Sourcing. This function is expected to be responsible for establishing an approach to fulfilling the agency’s requirements with vendors that provide solutions to those requirements, managing vendor selection, tracking vendor performance and contract deliverables, and sharing insights on new technologies and capabilities to improve the workforce knowledge base.

Figure 2 depicts the OI&T organization as of April 2016, which includes previously existing components and functions, as well as the new functions described above. OI&T was still in the process of fully defining the roles and responsibilities of the organizational units as of July 2016.

In addition to these new functions, the strategy calls for a transformation of the services that OI&T provides in field offices, to include requiring service-level agreements for all VA organizations and using those agreements to define the support that is needed, migrating data and applications to the cloud, and developing a strategy for consolidating data centers. According to the CIO, the transformation strategy is expected to be completed by the first quarter of fiscal year 2017, although the vast majority of the plan, including establishing the five new functions, is to be executed by the end of fiscal year 2016.

VA Generally Implemented IT Human Capital Management Best Practices but Has Not Projected Leadership Retirements or Fully Identified Skill Gaps

Key to an agency’s success in effectively managing its IT systems is sustaining a workforce with the necessary knowledge, skills, and abilities to execute a range of management functions that support the agency’s mission and goals. Achieving such a workforce depends on effective human capital management, which includes developing a human capital strategic plan; analyzing the workforce; analyzing the gaps between current skills and future needs, and developing strategies for filling the gaps; and training and developing the workforce. Taking such steps is consistent with activities outlined in human capital management practices that we and the Office of Personnel Management have developed.

OI&T took steps toward implementing effective IT human capital practices by documenting an IT human capital strategic plan and initiating an update based on changed priorities, regularly analyzing workforce data, identifying skill gaps for the current fiscal year, and implementing an IT training program. However, OI&T did not implement other human capital practices that are essential for analyzing the workforce and identifying IT skill gaps. Specifically, the office had not (1) tracked and reviewed historical and projected leadership retirements, and (2) identified the gaps in future skill areas.

OI&T Developed a Human Capital Strategic Plan

Strategic human capital planning that is linked to an agency’s strategic goals can be used as a tool to identify the workforce needed for the future and to develop strategies for shaping the workforce. Since 2001, we have identified strategic human capital management as a government-wide high-risk area.

OPM regulations require agencies to develop a human capital strategic plan that identifies goals and objectives that are consistent with the agency’s strategic plans and annual performance goals. These goals and objectives are to address each of the five systems for comprehensive human capital management that are outlined in OPM’s Human Capital Assessment and Accountability Framework (HCAAF). In addition, OPM’s regulations require agencies to include in their human capital strategic plans performance measures and milestones that assess the agency’s progress in meeting the identified goals and objectives.

In October 2012, the VA Inspector General reported that OI&T had not developed a human capital strategic plan and recommended that the office do so. In response to the recommendation, OI&T developed a 6- year human capital strategic plan in September 2013. This plan outlined four human capital strategic goals: maximize employee talent through recruitment, outreach, hiring and sustain a productive, diverse workforce and achieve results by valuing and recognizing performance in an environment in which all employees are encouraged to contribute; ensure OI&T supports a culture of leadership and continuous learning; ensure the human capital strategic plan is aligned with other VA strategic plans and integrated into workforce planning.

In addition to outlining the human capital strategic goals, this plan also identified objectives and strategies for achieving them. For example, the objectives included improving the processes for hiring new employees; building a diverse OI&T workforce; eliminating competency gaps in key leadership positions; and developing an office-wide integrated workforce analysis capability. Strategies that are linked to the goals and objectives include implementing a plan to streamline and improve employee orientation, increasing the percentage of minorities and veterans in the OI&T workforce, identifying and defining the competencies that OI&T leaders must develop to ensure staff in leadership positions have the right skills needed to meet organizational goals, and implementing a workforce planning process. See appendix III for a summary of the goals, objectives, and strategies outlined in OI&T’s strategic human capital plan.

The human capital strategic plan also was aligned with specific strategies that were identified in VA’s 6-year Strategic Plan, as well as the strategic goals within the department’s Office of Human Resources and Administration Strategic Plan for fiscal year 2014-2020. For example, the department’s strategic plan includes strategies for the development of leadership, and OI&T’s plan also includes the same. Specifically, as stated earlier, one of the goals within OI&T’s plan is to ensure the office supports a culture of leadership and continuous learning. Strategies for this IT human capital strategic goal include identifying and defining the competencies that OI&T leaders must develop and retain.

The plan also was aligned with a Human Resources and Administration strategic goal—to advocate for veteran employment within the department. In addition, the plan identified strategies to recruit, hire, and promote qualified veterans.

Further, OI&T’s human capital strategic goals are linked to each of the systems for comprehensive human capital management outlined in OPM’s framework. For example, one of the goals is to maximize employee talent through recruitment, outreach, hiring, and retention, which links to the talent management system in the framework. In addition, the plan identifies performance measures that the office is to use to monitor its progress in achieving each of the human capital goals. These performance measures include, among others, improved leadership abilities, and improvements in the number of hires and percentages for minorities, employees with disabilities, and veterans.

As of March 2016, OI&T was in the process of updating the human capital strategic plan as part of the CIO’s transformation strategy for OI&T. According to the office’s human capital management director, the revised plan is expected to be in place by the end of December 2016. By developing a human capital strategic plan and ensuring that the plan is updated to reflect the changing mission requirements of the office, OI&T should be better positioned to provide the necessary strategic direction for meeting its workforce needs and goals.

OI&T Regularly Analyzed Workforce Data but Was Not Tracking Leadership Retirements

Performing workforce analysis is a key component in strategic human capital planning because it assists agencies in developing strategies for acquiring, training, and retaining staff. According to OPM, workforce analysis allows agencies to identify trends impacting their workforce. The analysis also provides a basis for developing actions to address workforce trends that may impact an agency’s future mission capabilities. OPM also stresses the importance of ensuring the continuity of leadership within an agency, and suggests that agencies perform ongoing workforce analysis to identify current and future workforce and leadership needs. Among other things, for relevant agency mission requirements, OPM requires a workforce analysis that includes a forecast of future leadership requirements and changes due to retirements. Similarly, VA’s workforce planning policy requires each staff office, including OI&T, to annually analyze historical workforce data and develop workforce projections that, at a minimum, include employee counts, retirements and other losses, new hires, workforce diversity, and leadership retirements. The workforce analysis also is to include a determination of gaps in the workforce and strategies to close those gaps.

OI&T conducted a workforce analysis in May 2013 that identified for the office the number of employees, retirements and projected eligible retirements through fiscal year 2018, employee turnover and hiring rates, and workforce diversity percentages. Further, OI&T has analyzed most of the required workforce information on a monthly basis. Specifically, the analyses we reviewed for October 2015 to January 2016 included workforce data and projections related to employee counts, losses, new hires, and workforce diversity. According to the OI&T Director of Human Capital Management, the office used these data to examine trends, increase awareness of staffing related issues or opportunities, and identify workforce gaps, among other things.

Further, the department’s Integrated Human Resources Management Council was tasked with analyzing workforce data, such as retirement rates, on a quarterly basis as part of an overall VA effort to review performance metrics related to human resource strategic goals. The Integrated Human Resources Management Council’s review results included workforce gaps and strategies to close them. For example, in fiscal year 2015 VA had not met its target for hiring veterans. One of the strategies the council identified for increasing the number of veterans hired was to collaborate with stakeholders to identify and implement innovative and targeted recruitment opportunities.

Nevertheless, even as it has taken these actions, OI&T was not analyzing historical leadership retirement data; nor was it developing leadership retirement projections as required by the department’s workforce and succession policy. Leadership positions include senior executives as well as other staff at the General Schedule levels 13 through 15. According to the OI&T Director of Human Capital Management, the office collects data on all employee losses, including retirements. Further the OI&T human capital strategic plan included projected eligible retirements. However, historic data on leadership retirements were not analyzed and a forecast of future leadership retirements was not developed, as called for by VA’s workforce planning policy.

The OI&T Director of Human Capital Management stated that, although the office has the capability to analyze leadership retirements, there is no specific reason why it has not done so. Without tracking and forecasting leadership retirements, OI&T faces a risk of being unprepared to identify and effectively respond to vacancies in key leadership positions, which in turn can contribute to ineffective IT management.

OI&T Identified Skill Gaps but Has Not Determined Its Needs beyond the Current Year

Determining core skills (e.g., problem solving), leadership skills (e.g., developing others), and technical skills (e.g., configuration management) is essential for an organization such as OI&T to successfully achieve its missions and goals. This is especially important as changes in national security, technology, budget constraints, and other factors change the environment within which organizations operate. The identified skills should be linked to the agency’s mission and long-term goals in order to identify the workforce needed for the future and develop strategies for shaping this workforce. Agencies should then identify the gaps in current and future skills needed to achieve results, and develop strategies for filling the gaps.

OI&T had conducted annual gap analyses to identify its skill needs, and it had developed strategies for filling the gaps. Specifically, the IT Workforce Development group within the Office of Quality, Performance, and Oversight began conducting the analyses in June 2013, and the analyses were conducted annually thereafter. On an annual basis, the analyses identified each skill area where a gap existed, along with the percentage of staff that was determined to be below the targeted proficiency level for a particular role and career level (e.g., entry, intermediate, and senior).

In addition, the analyses identified the top 10 technical skill gaps within OI&T for each year and compared the gaps from the previous year to those in the current year. This comparison showed either an increase, decrease, or no change in the percentage of staff below the targeted proficiency level for a particular skill area. For example, in March 2016, IT Workforce Development reported that the gap in staff proficiency in 9 of the top 10 technical skill areas within OI&T had decreased from the previous year and that the gap in staff proficiency in 1 skill area had remained the same. Table 2 shows the comparison of the skills gaps for the 2015 and 2016 analyses.

After the initial analysis was conducted in June 2013, the office identified training that was either available or being developed to address each skill gap. For example, in its March 2016 analysis report, the IT Workforce Development group noted that 48 new courses were being developed based on gaps identified in the March 2015 analysis. This training covered the areas of emerging technologies, project management, communications, interpersonal skills, analytical reasoning, and conflict management. As of July 2015, OI&T had provided approximately 470 on- the-job training opportunities. According to the IT Workforce Development group, on-the-job training is used to increase options for staff to demonstrate advanced-level proficiency, including in many of the skill gap areas identified. For example, the group developed an opportunity for senior staff to guide junior staff in the area of project management. The group also provided staff the opportunity to learn from subject matter experts in the area of infrastructure design. As of April 2016, employees had taken advantage of approximately 160 of these opportunities.

The IT Workforce Development group also recommended in its 2016 gap analysis that several other actions be taken by OI&T to address the identified skill gaps. For example, the group recommended that OI&T continue to develop learning plans that focus on advance levels of proficiency; develop training on VIP, including training on Agile program management; and prioritize training for leadership-related competencies, among others.

Nevertheless, while OI&T conducted annual skill gap analyses and developed training courses and recommended other actions for addressing skill gaps, the office has not identified the gaps in skill areas that could be needed in future years. Specifically, the analyses focused on gaps in the time periods when they were conducted; however, the analyses did not identify the skills or the gaps in the skills that may have been needed in future years in order to implement the goals and objectives identified in the department’s long-term IT strategic plan or for planned IT initiatives.

According to a program manager in the IT Workforce Development group, the annual skill gap analyses were based on current staffing levels and on what each organization within OI&T wanted to achieve through the current fiscal year, but not on what skills would be needed for future years. However, by only focusing on the current year and not including future years in its skill gaps analysis, OI&T may not have been aware of gaps in skills that would be needed to successfully accomplish other goals, such as the completion of a multi-year IT development project, or longer-term goals, such as the department’s future IT operating environment as described in the office’s 5-year IT strategic plan. By including the IT skills needed for future-years in its gap analyses, OI&T could have increased assurance that its staff has the capabilities to deliver long-term IT support that contributes to improved services for veterans.

OI&T Took Steps to Implement a Training Program

Training programs are an integral part of a learning environment that can enhance an agency’s ability to attract and retain employees with the core, leadership, and technical skills needed to achieve results. The essential aim of these programs is to assist the agency in achieving its mission and goals by improving individual and, ultimately, organizational performance. Training programs involve establishing learning priorities; identifying training initiatives; ensuring delivery of learning opportunities; and evaluating the program and demonstrating how training efforts contribute to improved performance and results.

OI&T took several steps to implement a training program. These included establishing an advisory board, developing competency models and individual development plans, and providing training to its employees. Specifically, the office established a Training Advisory Board which was tasked with identifying and validating training needs in OI&T. The board, which was established in November 2014, is headed by the OI&T Chief Learning Officer and has senior management representatives from each organizational unit. According to its charter, the advisory board discusses and determines the priority of the training needs of each of the organizational units. In addition, the charter states that the board is intended to ensure that available training is aligned with VA’s mission, strategic planning, and OI&T priorities. The board’s most recent meetings were held in October 2015 and April 2016. During the meetings, the board discussed the training needs for OI&T’s organizations, the development of training courses, and upcoming learning opportunities.

The IT Workforce Development group used competency models and individual development plans to identify training initiatives that could address the learning needs of OI&T’s employees. According to an official in the IT Workforce Development group, employees were aligned with competency models based on their job responsibilities. These competency models identified training needs for professional development. Additionally, the IT Workforce Development official stated that OI&T used individual development plans to reinforce the required knowledge, skills, and abilities necessary for employees to progress into new positions at higher skill levels. The individual development plan is essentially a to-do list that is generated based on identified gaps in the employee’s skills. The plan identifies the employee’s competency model, training needs, and the due dates for the learning opportunities identified to meet their training needs.

Further, OI&T has taken steps to ensure the delivery of learning opportunities. Specifically, in fiscal year 2015, the office provided over 700 training courses. For example, it provided training on information security awareness, which over 15,000 employees and contractors completed, and on risk assessment, which was completed by 284 employees and contractors. The courses also covered a wide range of other IT-related subjects, such as project management, enterprise architecture, and cybersecurity. The training courses were delivered through several methods, such as webinars, on-the-job training, classrooms, and virtual classrooms. The IT Workforce Development group used event announcements and the delivery of course schedules to each organizational unit to help ensure employees were made aware of the provided opportunities.

Lastly, OI&T took steps to evaluate its training program. Specifically, in April 2015, the office performed an assessment that identified weaknesses in its training program, such as the need to supplement competency modeling efforts with learning opportunities that focus on IT strategic business priorities for overall organizational development. The assessment also identified the need to provide learning opportunities that address emerging trends and technologies at all leadership levels.

Further, the assessment included strategies to improve employee, leadership, and organizational development. For example, one strategy to improve employee development was to survey employees to identify challenges in job responsibilities and the least understood emerging technologies, and then to use that information to provide theme-based development opportunities to address the identified needs. Additionally, OI&T gauged the satisfaction of the training attendees by providing course evaluations, which assessed the relevance of the training to job responsibilities and the effectiveness of the instructor. For example, in February 2016, 90 percent of course attendees indicated that they were satisfied with the courses; 83 percent indicated that the content of the courses were relevant to their jobs; and 95 percent indicated that the instructor was effective in conveying practical knowledge about the subject matter.

By taking these steps to implement a training program, OI&T is better positioned to ensure that it has the ability to train and develop an IT workforce to effectively support the department’s mission and goals.

VA Documented Processes to Manage IT System Development and Acquisitions, but Gaps Existed

Instituting disciplined, repeatable practices for IT development and acquisition is key to ensuring that investments in IT cost-effectively deliver the capabilities needed to support an organization’s mission. SEI and GAO have identified best practices that are essential to the development and acquisition of products and services for IT projects. These include practices related to project planning, requirements management, risk management, project monitoring and control, product validation, and process and product quality assurance, as well as developing and maintaining a reliable, high-quality project schedule. According to SEI, when an organization’s processes are documented, it allows the organization to more consistently apply these best practices. Further, documented processes are important for sustaining institutionalized best practices regardless of future leadership changes. Of 123 best practices that we selected to evaluate, VA’s documented processes reflected 97 practices, partially reflected 12 practices, and did not reflect 14 practices.

Most Project Planning Best Practices Were Addressed in OI&T’s Documented Processes

Project planning involves establishing and maintaining estimates of project parameters, developing a project plan, and obtaining commitment to the plan from those who are responsible for implementing and supporting it. For example, establishing project estimates involves developing a work breakdown structure that details project tasks, responsibilities, and schedules; specifying estimates of work products and task attributes for the project; and estimating the project’s effort and costs for work products and tasks based on estimation rationale.

In addition, developing a project plan includes identifying major milestones, schedule assumptions, constraints, and task dependencies; and creating and maintaining a project plan that ties together the budget, schedule, milestones, and stakeholder identification and interaction.

Further, obtaining commitment to the plan includes identifying agreements regarding interfaces between project elements and other projects; documenting commitments with an appropriate level of signatories; and making adjustments to the project plan to reconcile variances in estimated and available resources.

OI&T’s project planning processes reflected 13 of 15 selected best practices. Of the 15 practices, the processes we examined included criteria that addressed 4 of 5 practices for establishing project estimates. These included developing a work breakdown structure that specifies project tasks, responsibilities, and schedules; identifying products to be acquired externally; providing estimates of work products and task attributes; and planning project lifecycle phases.

One practice—to estimate the project’s effort and costs for work products and tasks based on estimation rationale—was partially reflected in the documented processes. In this regard, VA’s project scheduling tool contained processes that called for project managers to estimate project effort. However, the processes did not address how the managers were to calculate the estimated effort to be expended to complete a project.

Further, the processes included criteria related to all 7 selected practices for developing a project plan. These included identifying major milestones, defining and quantifying resources needed for the project, and creating and maintaining a project plan that ties together the budget, schedule, milestones, and stakeholder identification and interaction.

Finally, the processes reflected 2 of 3 selected practices for obtaining commitment to the plan. In particular, they included identifying commitments regarding interfaces between project elements and other projects, and for documenting commitments with an appropriate level of signatories. However, the processes did not address making adjustments to the project plan to reconcile variances in estimated and available resources.

Until OI&T incorporates all the recommended project planning best practices in its process documentation, the office increases the risk that its project managers could inconsistently calculate estimates of the effort needed across projects and fail to adjust project plans based on resource availability, thus potentially affecting the timely completion of projects. Table 3 summarizes the extent to which these best practices were documented in OI&T’s project planning processes.

OI&T’s Documented Processes Addressed Almost All Requirements Management Best Practices

Best practices recommend that project teams manage requirements of the project’s products and components, and ensure alignment between those requirements and the project plans and work products. Managing requirements can be accomplished by understanding and obtaining commitment to requirements, and ensuring that project work and associated requirements align with each other. Examples of these best practices include analyzing requirements to ensure that established criteria for managing requirements are met; documenting all requirements and requirements changes; maintaining bidirectional requirements traceability from a requirement to its derived (i.e., lower-level) requirements and allocation to work products and back; and identifying changes to be made to plans and work products based on modifications to the requirements baseline.

OI&T had documented processes that reflected 10 of 11 selected best practices for managing project requirements. For example, the processes included assessing the impact of requirements on existing commitments, negotiating and documenting changes to commitments, and maintaining a requirements change history with the rationale for the changes. However, the documentation did not address one of the practices. Specifically, it did not include identifying changes that should be made to plans and work products resulting from changes to the requirements baseline.

Until the office incorporates in its documented processes the best practice for identifying changes to be made based on modifications to a project’s requirements baseline, OI&T’s staff will not be best positioned to implement the associated changes and maintain consistent processes across IT projects. Table 4 summarizes the extent to which these best practices were reflected in OI&T’s requirements management processes.

Majority of Risk Management Best Practices Were Documented in OI&T’s Processes

Risk management best practices call for the identification of potential problems before they occur so that risk-handling activities can be planned throughout the life of the project to mitigate adverse impacts on achieving objectives. These practices involve preparing for risk management, identifying and analyzing risks, and mitigating identified risks. For example, preparing for risk management involves determining risk sources and categories, and developing risk mitigation techniques. In addition, identifying and analyzing risks includes determining those that are associated with cost, schedule, and performance; and reviewing the work breakdown structure and project plan to help ensure that all aspects of the work effort have been considered. Further, mitigating risks includes determining the levels and thresholds in which a risk becomes unacceptable and triggers the execution of a risk mitigation plan or contingency plan; establishing a schedule for each risk handling activity, with a start date and anticipated completion date; determining the costs and benefits of implementing the risk mitigation plan for each risk; and collecting performance measures on risk handling activities.

OI&T established processes that reflected 26 of 28 selected best practices for risk management. Of the 28 best practices, the office’s processes addressed all 9 practices related to preparing for risk management. For example, they defined criteria for evaluating and quantifying risk likelihood and severity, established thresholds for risk categories, and defined a time period for risk monitoring or reassessment.

In addition, the documented processes included criteria related to all 9 practices for identifying and analyzing risks. For example, the processes reflected best practices for documenting the context, conditions, and potential consequences of each risk; categorizing and grouping risks according to defined risk categories; and prioritizing risks for mitigation.

With regard to mitigating risks, the documentation reflected 8 of 10 selected best practices. For example, it included identifying the person or group responsible for addressing risks, developing contingency plans for critical risks, and monitoring risk status. However, it did not address one practice for determining the costs and benefits of implementing the risk mitigation plan for each risk. Further, one practice—to collect performance measures on risk handling activities—was partially reflected in the processes. In this regard, VA’s processes called for the creation and distribution of monthly performance reports, but they did not address risk mitigation activities.

Although VA included 26 of 28 best practices related to risk management in its processes, it is not positioned to fully ensure that the costs and benefits of executing risk mitigation plans will be consistently determined and that performance measures for risk mitigation efforts are collected. Table 5 summarizes the extent to which these best practices were documented in OI&T’s risk management processes.

OI&T’s Documented Processes Addressed Some but Not All Project Monitoring and Control Best Practices

The purpose of project monitoring and control is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Toward this end, a project’s documented plan is considered the basis for monitoring activities and corrective actions taken when performance deviates significantly from the plan. Examples of monitoring the project against the project plan include tracking progress against the schedule, monitoring project costs and the effort expended for project completion, documenting the results of data management activity reviews, and identifying and documenting significant issues and deviations from the project plan. In addition, managing corrective actions to address identified issues involves determining and documenting the appropriate actions needed to address the issues and monitoring corrective actions for completion.

OI&T’s documented processes reflected 18 of 28 selected best practices related to project monitoring and control. Of the 28 practices, the processes reflected 12 of 22 selected practices for monitoring the project against the project plan and all 6 selected practices related to managing correction actions. For monitoring the project against the project plan, the processes included tracking change requests and problem reports to closure, documenting the results of milestone reviews, and reviewing the project plan, project status, and risks at selected milestones. In addition, for managing corrective actions, the processes included gathering issues that require corrective action for analysis, analyzing the results of corrective actions to determine effectiveness, and determining and documenting lessons learned as a result of taking appropriate corrective actions.

However, three of the selected practices were partially reflected in the processes for monitoring the project against the project plan. Specifically, documentation on monitoring project costs and the effort expended for project completion included criteria for monitoring project costs; however, it did not call for tracking expended effort. With regard to monitoring resources provided and used, the processes call for VA to monitor projects to ensure that staff and resources are provided, but they did not clearly require monitoring for the utilization of those staff and resources.

Similarly, OI&T partially documented its processes for reviewing the results of collecting and analyzing project performance measures, such as customer service satisfaction metrics. In particular, the processes had been developed to ensure the agreed-upon functionality was delivered to customers. However, the customers’ level of satisfaction based on the functionality requested and received for the project was not addressed.

In addition, the processes did not address seven of the selected practices. These practices were associated with monitoring the knowledge and skills of project staff, periodically reviewing data management activities against the project plan, identifying and documenting significant data management issues and their impacts, documenting results of data management activity reviews, periodically reviewing stakeholder involvement in projects, documenting the results of stakeholder involvement status reviews, and tracking action items to closure.

Until OI&T completely documents its processes for monitoring and controlling projects, staff responsible for executing these steps may not be fully aware of a project’s progress toward achieving the milestones defined in the project plan. Table 6 summarizes the extent to which these best practices were addressed in OI&T’s project monitoring and control processes.

Validation Best Practices Were Reflected in OI&T’s Documented Processes

Best practices for product validation call for project teams to demonstrate that a product or component fulfills its intended use when placed in its intended environment. The methods employed to accomplish validation can be applied to work products as well as to the product components. Key practices, as identified by SEI, include preparing for validation by identifying and selecting products, and validating the product or product components. For example, preparing for validation includes identifying the product, product component, and/or features to be validated throughout the life of the project and identifying requirements for the validation environment and passing them to the requirements development process. Further, validating the product or its components involves ensuring that activities (i.e., those that validate IT hardware and services) are performed throughout the project lifecycle and that deviations from validation procedures are documented.

Of the 17 selected best practices for product validation, OI&T’s documented processes reflected all 10 practices related to preparing for validation, and all 7 practices for carrying out the validation of the product or its components. For example, the office’s processes addressed best practices for identifying test equipment and tools, ensuring the validation method was selected early in the development process, identifying and documenting validation failure causes as needed, comparing actual validation test results to expected results, and analyzing validation data for defects.

By incorporating the selected validation best practices into its processes, OI&T has increased assurance that staff responsible for performing validation of the department’s IT products or their components will be positioned to consistently apply key processes to ensure that products perform as expected when placed in the intended environment. Table 7 summarizes the extent to which these best practices were addressed in OI&T’s validation processes.

Process and Product Quality Assurance Best Practices Were Mostly Reflected in OI&T’s Documentation

Best practices for process and product quality assurance are intended to support the delivery of high-quality products by providing project staff and management with appropriate visibility and feedback on processes and associated work products throughout the life of the project. Toward this end, process and product quality assurance involves objectively evaluating processes and work products, and providing objective insight to staff and managers. Examples of objectively evaluating processes and work products include establishing and maintaining clearly stated criteria, based on business needs, for assessing processes and work products; identifying lessons learned that could improve processes; and documenting a description of the quality assurance reporting chain and defining how it will ensure objectivity. In addition, examples of providing objective insight include escalating noncompliance issues that cannot be resolved in the project to the appropriate level of management; tracking noncompliance issues to resolution; recording process and product quality assurance activities in sufficient detail so that the status and results are known; and periodically reviewing open noncompliance issues and trends with management designated to receive and act on them.

OI&T had documented processes that addressed 12 of 14 selected best practices related to process and product quality assurance. Specifically, the processes reflected 4 of 5 practices for objectively evaluating processes and work products. These included practices related to establishing and maintaining clearly stated criteria, based on business needs, for evaluating processes and work products; identifying each noncompliance found during the evaluation; identifying lessons learned; and evaluating selected work products at selected times.

One practice—to document a description of the quality assurance reporting chain and define how it will ensure objectivity— was partially addressed in the processes. In this regard, while a quality assurance standard described how each review would ensure objectivity, the description of the reporting chain was not included.

In addition, the office’s processes reflected 8 of 9 selected practices for providing objective insight. These included documenting noncompliance issues that cannot be resolved in the project, analyzing noncompliance issues to determine if quality trends can be identified and addressed; and ensuring that relevant stakeholders are aware of results of evaluations and quality trends in a timely manner. However, they did not address the periodic review of open noncompliance issues and trends with appropriate management.

By not fully documenting best practices related to quality assurance in its processes, OI&T has less assurance that staff will appropriately report quality assurance issues and periodically review open issues and trends so that the relevant manager can act on them. Table 8 summarizes the extent to which these best practices were reflected in OI&T’s process and product quality assurance processes.

Most Project Schedule Best Practices Were Partially Addressed or Not Addressed in OI&T’s Process Documentation

The success of a project depends, in part, on having an integrated and reliable master schedule that defines when and how long work will occur, and how each activity is related to the others. A project’s schedule provides not only a road map for systematic project execution, but also the means by which to gauge progress, identify and resolve potential problems, and promote accountability at all levels of the program. VA’s guide for IT project management deems project schedules as essential to a project’s success and notes that failure to meet project schedule dates may trigger additional project reviews by senior leaders, which could lead to a project’s shutdown.

Further, our Schedule Assessment Guide defines 10 best practices related to 4 characteristics that are important to developing high-quality, reliable schedule estimates—comprehensive, controlled, well- constructed, and credible. Table 9 describes the characteristics of high- quality, reliable schedule estimates and their associated best practices that guided our analysis.

Almost all of OI&T’s processes for developing project schedules either partially addressed or did not address the 10 best practices. Specifically, the documentation for developing comprehensive, controlled, well- constructed, credible project schedules reflected 1 best practice, partially reflected 6 best practices, and did not reflect 3 best practices.

Without documented processes for developing schedules that reflect best practices, OI&T is at increased risk that schedules created for its projects will not be reliable and, therefore, will not be useful tools for measuring program performance against approved project plans. Table 10 summarizes the results of our assessment of whether OI&T’s documented processes reflected the 4 characteristics of a high-quality, reliable schedule estimate, their associated best practices, and examples of our rationale for the assessment.

VA Officials Asserted That Inclusion of Missing Best Practices in Documented Processes Was Not Necessary

In discussing OI&T process documentation, officials in the Product Development and Service Delivery and Engineering offices, including an Assistant Deputy CIO within Product Development, stated that the department’s documented IT processes did not include all of the selected best practices because they were considered basic, fundamental skills that a properly trained project planner should possess. The officials added that their inclusion would be overly burdensome and counterproductive when working in an Agile environment.

However, the selected best practices do not apply to a particular development approach; rather, they were designed to provide value across different approaches, including Agile. In addition, documented processes are key to the consistent implementation of best practices that can withstand future leadership changes. Without incorporating these best practices into its processes, VA is at risk of missing critical steps that could help ensure that its processes are implemented as intended and are consistently applied across IT projects.

Conclusions

While VA has largely centralized the performance of key IT-related functions in OI&T, led by the CIO, it has faced challenges in fully implementing and managing IT under this organizational structure. In particular, the office has had difficulty in preventing IT activities from occurring outside the control of OI&T and PMAS. It has also been challenged in effectively collaborating with the department’s various business units and in efficiently and cost-effectively delivering new IT capabilities. The OI&T transformation effort recently initiated by the CIO, and projected to be completed by the first quarter of 2017, is intended to address these and other challenges.

OI&T has also taken steps to implement human capital management best practices by developing and documenting and beginning to update a human capital strategic plan that aligns with VA’s strategic plans, creating action plans for implementing OI&T human capital goals and objectives, reviewing its workforce quarterly, identifying skill gaps, and implementing a training program. However, without annually tracking and reviewing data related to leadership retirements or identifying skills needed in future years, VA will be hindered in ensuring that it has the leadership and staff with skills needed to support the successful delivery of IT capabilities.

Further, opportunities exist for VA to strengthen its processes for system development and acquisition by including key system development best practices. VA’s documented processes for managing IT system development and acquisition generally reflected best practices in the key areas we reviewed. However, there were gaps in most of these areas. Ensuring that these processes address all key practices will assist the department in effectively managing its IT system development and acquisitions.

Recommendations for Executive Action

To assist VA in sustaining an IT workforce with the necessary knowledge, skills, and abilities to execute its mission and goals, we recommend that the Secretary of Veterans Affairs direct the Chief Information Officer to track and review OI&T historical workforce data and projections related to leadership retirements, and identify IT skills needed beyond the current fiscal year to assist in identifying future skills gaps.

To assist VA in establishing comprehensive and documented processes that reflect system development and acquisition best practices, we recommend that the Secretary of Veterans Affairs direct the Chief Information Officer to revise OI&T’s documented processes related to project planning, to include (1) estimating the level of effort that will need to be expended for work products and tasks, and (2) making adjustments to the project plan to reconcile differences between estimated and available resources; requirements management, to include identifying changes to be made to plans and work products as a result of requirements baseline changes; risk management, to include (1) determining costs and benefits of implementing the risk mitigation plan for each risk and (2) collecting performance measures on risk handling activities; project monitoring and control, to include the 10 best practices that were missing from the guidance; process and product quality assurance, to include (1) documenting a description of the quality assurance reporting chain and defining how objectivity will be ensured, and (2) periodically reviewing open noncompliance issues and trends with management that is designated to receive and act on them; and project scheduling, to include the 9 best practices that were missing from the guidance and revise the documented processes where the guidance was contrary to best practices.

Agency Comments and Our Evaluation

In written comments on a draft of our report (reprinted in appendix IV), VA generally agreed with our conclusions and concurred with our eight recommendations. Further, the department provided, and requested that we include in our report, information describing OI&T’s new organizational structure and its cyber security initiative. We have incorporated this information in relevant sections of the report.

VA’s comments described steps that it had taken or planned to implement our recommendations. For example, the department asserted that it had taken steps that fully addressed our recommendation that it track and review OI&T historical workforce data and projections related to leadership retirements. In our follow up on the department’s implementation of our recommendations, we will evaluate whether the actions noted are fully responsive to this recommendation.

The department also discussed planned actions for addressing our recommendation related to identifying future IT skills. Specifically, VA stated that it plans to, among other actions, include in the next skills gap analysis, long-term recommendations at the organizational level that show the types of skills each organization needs to increase and which proficiency level targets need the most emphasis.

Further, in response to our recommendations that it revise OI&T’s documented processes related to project planning and requirements management, VA stated that it plans to document changes to these processes as the department transitions from PMAS to VIP. In addition, with respect to the majority of the remaining deficiencies in VA’s documented processes—related to risk management, project monitoring and control, quality assurance, and project scheduling—the department stated that our recommendations are to be addressed through the implementation of VIP, Agile processes, and other systems development process tools.

According to VA, its actions in response to our recommendations are expected to be completed by the end of fiscal year 2017. If the department ensures that these and other activities it identified are appropriately documented and effectively implemented, then VA should be better positioned to sustain an IT workforce with the necessary knowledge, skills, and abilities to execute its mission and goals, and to ensure critical systems development and acquisition processes are implemented as intended and are consistently applied across IT projects.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Veterans Affairs and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

Should you or your staffs have questions on matters discussed in this report, please contact me at (202) 512-6304. I can also be reached by e- mail at melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report.

GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of this study were to determine (1) how the Department of Veterans Affairs (VA) is organized to manage and perform key information technology (IT)-related functions and the extent to which it has centralized the management of IT resources, (2) the extent to which VA has implemented effective IT human capital management, and (3) the extent to which VA has established key processes to effectively manage major system development and acquisition efforts.

To address the first objective, we obtained and reviewed the department’s documentation identifying and describing its IT organizational structure and functions. These included the IT strategic plan, governance plan, organizational descriptions contained within the VA’s Functional Organization Manual, the IT organization chart, and VA’s project management framework user guides. In addition, we reviewed memoranda, testimony from VA officials, and assessment reports that described the department’s efforts to centralize the management of its IT resources and the challenges it faces in doing so. In addition, we obtained and reviewed the Chief Information Officer’s (CIO) IT transformation strategy intended to address the challenges in maintaining a centralized program. We also interviewed responsible program officials regarding the organizational structure and performance of key IT-related functions.

To address the second objective, we reviewed information describing the department’s management of its IT human capital, such as human capital strategic planning documentation, skills gap analyses, and human capital performance measures. We compared the department’s actions to agency-level human capital management best practices that are also applicable to major agency components such as the Office of Information & Technology (OI&T). These best practices are those that we and the Office of Personnel Management (OPM) have identified. We also compared VA’s actions against those required by VA’s Office of Human Resources and Administration. These included practices for effective strategic human capital planning, workforce planning, and strategic training. The practices are identified in our Key Principles for Effective Strategic Workforce Planning; OPM’s final regulations to implement certain provisions of the Chief Human Capital Officers Act of 2002; and VA’s Workforce and Succession Planning directive. We also interviewed officials responsible for the department’s IT human capital management.

For the third objective, we reviewed policies, procedures, and supporting documentation describing the department’s key processes for managing major IT system development and acquisition efforts. We assessed the processes against key best practices that the Software Engineering Institute (SEI) and we have identified. The practices we selected are fundamental to effective IT system development and acquisition. These included recognized practices for project planning, requirements management, risk management, project monitoring and control, project validation, quality assurance, and project schedules. These practices are identified in SEI’s Capability Maturity Model® Integration (CMMI®) for Development, Version 1.3; SEI’s CMMI® for Acquisition, Version 1.3; and our Schedule Assessment Guide. Further, we reviewed systems development lifecycle policies, procedures, and other supporting documentation such as templates for required project artifacts within the department’s ProPath system and Project Management Accountability System (PMAS). The scope of our review did not include assessment of VA’s implementation of these processes.

Our methodology to determine the extent to which VA included systems development and scheduling best practices developed by SEI and us in its procedures and guidance included three levels of compliance: (1) the program office provided evidence that satisfied the elements of the best practice, (2) the program office provided evidence that satisfied some but not all of the elements of the best practice, and (3) the program office provided no evidence that satisfied the elements of the best practice.

We conducted this performance audit from February 2015 to August 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Timeline of VA’s Efforts to Organize the Management of IT Functions

The Department of Veterans Affairs (VA) has had a long history of organizing its information technology (IT) functions. The information contained in table 11 describes key events in the department’s efforts.

Appendix III: Summary of VA’s Office of Information & Technology Human Capital Goals, Objectives, and Strategies

The Department of Veterans Affairs (VA) Office of Information & Technology (OI&T) developed a 6-year human capital strategic plan in September 2013. The information contained in table 12 describes the goals, objectives, and strategies outlined in OI&T’s strategic human capital plan.

Appendix IV: Comments from the Department of Veterans Affairs

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Mark Bird (Assistant Director), Nicole Jarvis (Analyst in Charge), Christopher Businsky, Juana Collymore, Sharhonda Deloach, Jason Lee, Lee McCracken, and Christy Tyson made key contributions to this report.