prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.

Report 1: The defense lab enterprise consists of 63 labs, warfare centers, and engineering centers across the Departments of the Army, Navy, and Air Force, as shown in Figure 1 below  About 50,000 federally employed scientists and engineers work at these defense labs to support warfighter needs and develop transformative capabilities  Defense labs are managed and operated within the military service chain of command  DOD budgets for technology and product development activities under its research, development, test, and evaluation budget, which DOD groups into seven budget activity categories for its annual budget estimates  Air Force and Army labs rely on appropriated funding provided from the service—often referred to as mission funding—or from customers (or some combination thereof)  Customers, such as program offices, provide funding to defense labs for technology development activities and related research  The Air Force and Army funding structure is in contrast to Navy research and development activities, which operate under the Navy Working Capital Fund—a revolving fund that finances Department of the Navy activities on a reimbursable basis  Under this funding model, the Navy employs a Capital Investment Program to obtain capital assets, including minor military construction projects for labs  The program provides the framework for planning, coordinating, and controlling Navy working capital funds and expenditures to obtain capital assets  Figure 2 illustrates the varying funding models used by the military service labs  In addition to its labs, DOD sponsors other entities to provide for its technology development needs  Specifically, these include: FFRDCs are operated by universities, other not-for-profit or nonprofit organizations, or private firms under long-term contracts and provide special research and development services that generally cannot be readily satisfied by government personnel or private contractors  For example, the Massachusetts Institute of Technology Lincoln Laboratory develops key radar and electronic warfare technologies for integrated air and missile defense systems  In addition, the Software Engineering Institute operated by Carnegie Mellon University provides cybersecurity solutions for defense entities  While DOD sponsors 10 FFRDCs in total, it designates 3 FFRDCs as research and development labs, which maintain long-term competencies in key technology areas  In addition to these, DOD sponsors 2 systems engineering and integration FFRDCs and 5 studies and analysis FFRDCs  UARCs provide specialized research and development services similar to FFRDCs and also operate under long-term contracts  However, unlike FFRDCs, DOD requires that UARCs be affiliated with a university  Generally, UARCs may not compete against industry in response to a competitive Request for Proposals for development or production that involves engineering expertise  DOD currently sponsors 13 UARCs  Key DOD offices provide oversight to the defense labs: The Under Secretary of Defense for Research and Engineering (USD(R&E))—the principal advisor to the Secretary of Defense for research, engineering, and technology development activities and programs—serves as DOD’s chief technology officer  The powers and duties of this office include establishing policies and providing oversight for DOD’s research, engineering, and technology development activities  The Defense Laboratories Office—within the Office of the USD(R&E)—supports DOD’s research and engineering mission by helping to ensure comprehensive department-level insight into the activities and capabilities of the defense labs  This office carries out a range of core functions related to the defense labs, including analysis of capabilities, alignment of activities, and advocacy  Congress has granted authorities that address hiring, infrastructure, and technology transition challenges to defense labs since 1995  These authorities provide defense lab directors with certain flexibilities within the established legal framework to manage their operations  While Congress has provided a number of authorities, in this report we focus on four authorities that our prior work on best practices in science and technology management and expedited lab hiring has shown are, or have the potential to be, the most crucial for supporting innovation within DOD labs  Laboratory Initiated Research Authority  This authority provides lab directors with the means to fund some of the research projects that the lab will pursue  The authority provided in Section 219 of the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009, as implemented, provides lab directors with a means to fund projects they consider to be a priority in four allowable categories: (1) basic and applied research, (2) technology transition, (3) workforce development, and (4) revitalization, recapitalization, or repair or minor construction of lab infrastructure  These projects include those not specifically tied to defined requirements, outside of the normal 2-year budget planning process  The authority directs the Secretary of Defense to establish mechanisms under which lab directors may use an amount of funds equal to not less than 2 percent and not more than 4 percent of all funds available to the defense lab for projects under the four allowable categories  Further, lab directors are permitted to obtain additional funding by charging customers a fixed percentage fee that may not exceed 4 percent of costs  Direct Hire Authorities  These authorities provide lab directors with a streamlined and accelerated hiring process  Congress has enacted four types of direct hire authorities since 2008, which help labs compete with private industry and academia for high-quality scientific, engineering, and technical talent  Specific types of direct hire authorities include hiring: (1) candidates with advanced degrees; (2) candidates with bachelor’s degrees; (3) veterans; and (4) students currently enrolled in graduate or undergraduate science, technology, engineering, and mathematics (STEM) programs , Laboratory Enhancement Pilot Program  This authority provides methods for effective lab management operations  Section 233 of the National Defense Authorization Act for Fiscal Year 2017 established a pilot program for lab directors to propose alternative and innovative methods that might lead to more effectively managing labs, and authorized lab directors to waive any regulation, restriction, requirement, guidance, policy, procedure, or departmental instruction that would affect implementation of these methods, unless such implementation would be prohibited by a provision of an existing statute or common law  Micro-purchase Authority  This authority facilitates the purchasing process for labs  The FAR states a preference for government agencies to purchase and pay for micro-purchases of supplies or services using the government-wide commercial purchase card up to and at the micro-purchase threshold, but micro-purchases may be conducted using any of the simplified acquisition methods  This facilitates the ability of lab officials to quickly and easily acquire needed items for their activities and reduce the administrative costs associated with such small purchases  While the FAR micro-purchase was generally $3,500 during our review, Congress increased it to $10,000 for activities of the science and technology reinvention labs in Section 217 of the National Defense Authorization Act for Fiscal Year 2017  As we found in June 2018, the federal government spends approximately $137 billion annually government-wide on research and development (R&D) to help further agencies’ missions, including at federal labs  From fiscal years 2015 to 2017, DOD, Energy, and NASA represented three of the top four federal agencies with the highest annual federal R&D spending, accounting for about 66 percent of total federal R&D spending on average, as shown in Figure 3  While the labs primarily support the agencies that directly fund them, DOD, Energy, and NASA research entities also collaborate extensively to support activities of shared interest  For example, DOD and NASA research centers have collaborated to develop hypersonic vehicle capabilities  Further, Energy’s national labs help provide critical national security capabilities for DOD and support NASA’s deep space mission radioisotope requirements  In 2017, Energy reported performing about $2 6 billion of work per year from fiscal years 2011 through 2015 for other federal agencies and other customers, including DOD  Most defense labs have used the selected authorities since 2008, but their use has sometimes been limited for a variety of reasons  According to lab directors, this is because of DOD legal and policy restrictions and stakeholder concerns  For example: Use of the laboratory initiated research authority was limited by DOD’s military construction funding and financial management policies  Use of the direct hire authority was limited, in part, by personnel- related delays, security clearance challenges, and military hiring restrictions  Use of the laboratory enhancement pilot program was limited by stakeholder uncertainty about how to use this authority effectively  Use of the increased micro-purchase authority was limited by stakeholder concerns about the authority’s potential effect on small businesses  We found that most defense labs have used the laboratory initiated research authority  Twenty-three of 31 of respondents to our survey— about 74 percent—reported obligating funds under this authority  However, we found that most labs are not using the full 4 percent of all funds available to each lab, or charging customers the full fixed percentage fee of 4 percent of costs, as allowed by law  Specifically, we found that, as of September 2018: Navy labs reported charging customers a percentage fee of about 2 percent of costs as of fiscal year 2018  Prior to this, Navy labs only charged a 1 percent fixed fee on these costs  Because Navy labs are working capital funded organizations, they can use payments from customers for goods delivered or services performed  Army labs reported using between 2 and 3 percent of all funds available to the lab for projects under the four allowable categories and charging customers a fixed fee of between zero and 3 percent of costs to fund such activities  Only the Air Force Research Laboratory reported using the full 4 percent of all funds available to the lab  According to agency officials, the lab is using 3 percent of all funds available to the lab and is allowing individual technology directorates the option to use the additional 1 percent of funds available  In fiscal year 2018, three of the lab’s nine technology directorates chose to use this additional 1 percent  However, the lab has not charged customers a fixed percentage fee on their costs at all  As figure 4 shows, in fiscal year 2017, the aggregate fixed percentage fee charged by labs in each of the military departments totaled under the full 4 percent allowed by law for each funding source  Decisions to charge lower percentages are decisions to forego additional potential funding, although agencies have various reasons why this can happen, as we will discuss later  In total, DOD reported that this authority provided almost $300 million to labs in fiscal year 2017 and funded more than 1,750 projects across the four allowable categories, as Figure 5 illustrates  We previously found, in June 2017, that the laboratory initiated research authority provides defense lab directors with limited flexibility to initiate science and technology projects  These projects include those that are not road mapped or tied to defined requirements outside of the normal 2- year budget planning process, and are focused on both near- and long- term needs  For this review, defense lab officials we interviewed stated that the laboratory initiated research authority enables their scientists and researchers to pursue projects not necessarily tied to requirements and provides necessary funds for workforce development and lab infrastructure projects  Further, as shown in Figure 6, lab directors we surveyed generally view the authority as both fostering innovation and increasing efficiency across the four allowable categories on which funds can be used  In accordance with the one of the statutory purposes for the use of the funds, lab directors have developed new, innovative technologies using this authority  For example, DOD reported that: In fiscal year 2017, the Naval Surface Warfare Center, Crane Division, developed and fielded a solution to an urgent requirement for defeating small unmanned aerial vehicles that attack Navy assets or surveil naval activities  The center delivered this technology to the warfighter in May 2017 just 7 weeks after the Navy submitted the requirement  The Army Research Laboratory used the authority to fund a project that eventually developed a material that could increase the speed and lower the power needs of future generations of computer chips, thereby supporting Army networks  The Navy invested more than $700 thousand in laboratory initiated research authority funds to commission a Ballast Water Research Lab at Naval Surface Warfare Center, Carderock Division  Through the use of this new facility, engineers will be able to study ways to treat ballast water to prevent introduction of non- native aquatic species into a new environment that can be disastrous for the marine life that already inhabit that environment, and ensure that the Navy is able to meet various port regulations around the world for its ships  The Air Force Research Laboratory invested funds in fiscal year 2017 to renovate an existing facility to provide high performance computing capability to aid the rapid development of “game-changing” technologies and weapon systems  Officials at the Army’s Space and Missile Defense Command Technical Center noted they used the laboratory initiated research authority for the first time in fiscal year 2018 because the current executive director, who assumed the position in 2017, prioritized implementing this authority  Most of the Center’s planned investments are focused on workforce development and laboratory infrastructure projects; officials cited a high energy laser technology lab as one of the projects being supported by the revitalization, recapitalization, or minor military construction portion of this authority  Although the majority of defense labs reported using the laboratory initiated research authority, interviews we conducted throughout our review, along with other DOD reports, identified certain obstacles that have, at times, impeded wider usage  DOD-wide military construction funding restrictions  DOD restrictions limit the amount of laboratory initiated research authority funds that labs can spend on lab infrastructure  DOD’s limit is $6 million for the revitalization and recapitalization projects that can be funded under the laboratory initiated research authority  Lab officials stated that this amount is often insufficient to construct advanced lab facilities  Air Force Research Laboratory officials indicated that it is nearly impossible to construct lab facilities for less than $6 million  Officials at the Army’s Aviation and Missile Research, Development and Engineering Center echoed this sentiment and noted that they have primarily used funds to renovate existing buildings rather than fund new lab facility construction  In January 2017, the Defense Science Board identified lab infrastructure challenges, including that the average age of research and development facilities was nearly 50 years  Further, the Board reported that the labs are usually not successful in competing against broader service needs for military construction funds  Air Force does not charge customers a fixed percentage fee of costs  The Air Force Research Laboratory reported that it is not charging customers the allowable fixed percentage fee of costs to fund science and technology activities because it does not have a mechanism in place to do so  Air Force Research Laboratory officials estimated the lab would collect approximately $3 million a year if the lab charged customer activities the maximum allowable fee (4 percent)  Air Force financial management officials stated that the service’s accounting system does not currently have an automated capability to transfer the allowable percentage fee of costs to a central account at the Air Force Research Laboratory  This lack of capability, officials noted, creates a significant administrative burden for charging these fees  The officials stated that they have not yet estimated the cost to add an automated capability  Although it is possible for the Air Force Research Laboratory to charge customer work orders manually—outside of the Air Force’s accounting system—officials with the Office of the Assistant Secretary of the Air Force for Financial Management and Comptroller perceive that the resources (time and people) required to manage such a process would be cost prohibitive  However, according to these officials, the Air Force has not assessed the costs required to improve the accounting system to do so, nor has it identified the potential benefits any improvements would provide  Federal internal control standards state that changes in condition affecting an entity and its environment often require changes to the entity’s internal control system, as existing controls may not be effective for meeting objectives (or addressing risks) under changed conditions  Further, these standards state that any internal control deficiencies require further evaluation and remediation by management  By not assessing the potential costs and benefits related to the options for collecting these allowable fees, the Air Force could be missing out on a potential source of funding to support its needs  DOD lacks clear guidance on how the Navy should use the initiated research authority for some infrastructure investments within the Capital Investment Program  In our review of DOD documentation, we found that, among the military departments, Navy labs funded recapitalization and revitalization projects using the laboratory initiated research authority the least  As recently as early 2017, a DOD-commissioned study found that defense labs face substantial infrastructure deficiencies that it has not yet identified funding to address  In fiscal year 2017, Navy labs invested $7 3 million in lab recapitalization projects, compared to $32 9 million and $53 7 million at the Air Force and Army, respectively  Navy lab officials told us that their ability to fund lab recapitalization and revitalization projects using funds available under the laboratory initiated research authority is limited because they have not been provided with clear guidance as to whether and how to use the laboratory initiated research authority within the Capital Investment Program of the Navy Working Capital Fund  Some Navy lab officials stated that they have found ways to use the initiated research authority for certain infrastructure investments  These officials stated that they used authority outside of the Capital Investment Program of the Navy Working Capital Fund, for instance, for projects below applicable thresholds because using the authority within the Program creates a bureaucratic and financial burden for them  For example, officials at two separate warfare centers—Naval Surface Warfare Center, Crane Division, and the Naval Air Warfare Center, Aircraft Division, noted that they did not expend funds in either fiscal year 2016 or fiscal year 2017 for recapitalization and revitalization projects  Both cited the Capital Investment Program as a significant barrier to their desired use of the laboratory initiated research authority  Officials from the Office of Budget, within the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller agreed that, to date, clarifying guidance on the use of the laboratory initiated research authority within the Capital Investment Program has not been issued, effectively limiting the extent to which the labs can use it for infrastructure needs  According to these officials, the Office of the Secretary of Defense (OSD) Comptroller—in coordination with the Office of Financial Policy and Systems within the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller—is responsible for developing the clarifying guidance their office has sought  This persistent lack of guidance on whether or how Navy labs should use the laboratory initiated research authority within the context of the Capital Investment Program presents an opportunity cost  Namely, the Navy’s labs have missed out on, and continue to miss, opportunities to invest in needed improvements to its aging lab infrastructure  The Army requires its laboratories to apply similar percentages to what is refers to as “Army direct appropriations” and “customer funds ” The Army requires that the percentage fee applied to direct appropriations not vary from the percentage fee applied to customer funds by more than 1 percent  The Army implemented this policy to maximize the laboratory initiated research authority’s effect on its 17 laboratories  However, the Office of the USD(R&E) reported in March 2018 that the policy was having a significant limiting effect on the breadth and scope of activities executed under this authority  Similarly, we found that the policy may, in practice, create a disincentive for Army lab directors to use the authority  In their responses to our survey, Army lab directors, representing key capability areas, acknowledged their concern about the percentage fee they assessed on customer funds affecting their ability to increase or maintain their customer bases  Further, some Army lab directors reported assessing a lower percentage fee on customer funds than allowed, which could help retain customers that might otherwise be driven away with higher assessed fees to carry out activities  As a result, these labs generally are setting a lower percentage fee on their directly appropriated funds, thereby lowering the overall laboratory initiated research funding available to them  Nonetheless, the Army has not assessed its policy to determine whether changes are needed to eliminate these disincentives  Continuing to operate without such an assessment could result in Army labs using the laboratory initiated research authority to fund fewer self-initiated projects—with the downstream effect that fewer new technologies for warfighters are available  The Navy applies a consistent fixed percentage fee of costs across its labs  Within the Navy, senior leadership has set the fixed percentage fee of costs the labs charge on customer funds at 2 percent  A senior Navy science and technology official stated that Navy leadership set a uniform fixed percentage fee to charge to customer activities across the Navy lab enterprise, in part, to ensure the labs were not inadvertently competing against one another for customer funds  For example, without a uniform rate, a Navy warfare center could offer a lower fee to entice a customer to use it rather than another center  The use of a fixed percentage fee facilitates program offices selecting warfare centers on the basis of best available match in capabilities  On the other hand, the Navy’s fixed 2 percentage fee of costs does limit—by half, as compared to the maximum 4 percent allowable—the amount of fees that Navy labs can collect  Consequently, several Navy lab directors told us that they would like to have the ability to increase the fixed percentage fee of costs above the Navy’s 2 percent to provide their labs with additional resources they said they need for innovation-related investments  Among the lab directors that responded to our survey, 30 of 31 replied that their lab had used at least one of the four types of direct hire authorities previously discussed since fiscal year 2014  Officials view direct hire authority as allowing the labs to compete with private industry for qualified applicants  Lab directors reported they generally believe that each type of direct hire is extremely or very useful for fostering innovation and increasing efficiency, as shown in Figure 7  Selected Officials’ Testimony on the Value of Direct Hire Authority: The U S  Army Engineer Research and Development Center “was able to meet this important goal [of annually hiring more than 160 new researchers] in large part because of its direct hiring authorities, which save time, effort, and costs, and allow the organization to more effectively hire the best and brightest minds available ” – Dr  Jeffrey P  Holland, Past Director, U S  Army Engineer Research and Development Center, in testimony before the Senate Committee on Armed Services (Emerging Threats and Capabilities Subcommittee), May 3, 2017  “The Air Force’s ability to recruit, retain, and develop the STEM workforce is vital toward building the future Air Force; Congress has been greatly supportive of these efforts…the addition of direct hire for candidates has been extremely useful in hiring qualified scientists and engineers in less than half the time of traditional hiring methods ” – Jeffrey Stanley, Air Force Deputy Assistant Secretary— Science, Technology and Engineering in testimony before the House Committee on Armed Services (Emerging Threats and Capabilities Subcommittee), March 14, 2018  Although participation in the laboratory enhancement pilot program is open to the DOD labs—and 19 of the 31 lab directors, or 61 percent, that responded to our survey reported they were participating—to date, only the Navy has formally established a pilot program for its labs  The Army and Air Force have not yet used this relatively new authority  A senior Navy science and technology official told us the Navy took important steps to facilitate the implementation of that service’s pilot program  According to the Navy official: The Office of the Deputy Assistant Secretary of the Navy for Research, Development, Test and Evaluation led the effort across the Navy labs, compiling—from each lab’s submission—a single list of proposals to forward to Navy leadership that would apply to all participating Navy labs  The Navy pursued a three-phased approach with its pilot program, with Phase 1 primarily focused on contracting and acquisition policy- related matters  Senior Navy research and development officials perceived these matters as being the easiest from which to obtain buy-in from Navy policy officials and attorneys, as well as Navy leadership  Phase 2 will include proposals related to Information Technology systems for research and development networks, while Phase 3 will most likely address personnel issues  Navy research and development officials deferred proposals— including information technology network enhancements—that might require extensive discussions with policy officials and attorneys stakeholders across the Navy  These proposals were pushed back to allow time for those stakeholders to see how the pilot program was being implemented and executed by the labs  None of the Army and Air Force labs has yet established a laboratory enhancement pilot program  Consistent with Army policy, the Medical Research and Materiel Command and the Space and Missile Defense Command Technical Center submitted proposals; however, they have yet to establish a pilot program  The Army’s Research, Development and Engineering Command, with input from its subordinate labs and engineering centers, developed a list of lab enhancement proposals but, as of September 2018, had yet to formally submit these final proposals to Army leadership for approval  These include initiatives in business operations, contracting, finance, information technology, and personnel management  A senior Army science and technology acknowledged that organizations across the military department have concerns about providing the labs with too much autonomy to use this new authority  Air Force Research Laboratory officials said they previously submitted a list of approximately 30 proposals to the Defense Laboratories Office in September 2017, but ultimately pulled back those requests because of stakeholder concerns within the Air Force  Specifically, officials with the Office of the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering stated that the Air Force Materiel Command, to which the lab is a subordinate organization, had not seen the proposals before they were submitted  In addition, these officials identified concerns about how various stakeholders throughout the Air Force—such as those from financial management and personnel—would react to these proposals  These proposals could potentially sidestep the stakeholders’ oversight function of related lab activities  A senior Air Force Research Laboratory official stated that the lab re-submitted its proposals to the Air Force Materiel Command and that Air Force leadership was still reviewing them at the time of this report  Twenty-six of 31 labs directors—84 percent—reported having used the $10,000 micro-purchase threshold authority granted by Congress in 2016  However, we found that contracting and small business management officials’ concerns with this authority have created implementation challenges at some defense labs  For instance, a senior Navy official indicated that multiple stakeholders from across the Navy—including its Office of Small Business Programs—raised concerns about the authority’s potential impact on small businesses as micro-purchasing allows defense labs to bypass small business set asides  Several labs reported similar stakeholder concerns that prevented implementation of the micro-purchase threshold increase  At the same time, however, lab officials we interviewed expressed the view that the increased threshold will be beneficial, consistent with their opinions about the laboratory enhancement pilot program  For example, officials at the Naval Research Laboratory stated that increasing the threshold to $10,000 allows their scientists and engineers to directly purchase necessary equipment and materials through simplified procedures  They identified examples of projects that had been delayed by as much as several months because scientists and engineers used other than simplified acquisition procedures to purchase a relatively inexpensive piece of equipment, such as a specialized microscope, because the cost was above the previous threshold of $3,500  Similarly, the Army’s Armament Research, Development and Engineering Center reported that the micro-purchase threshold increase enables the lab to use simplified acquisition procedures for more items  As a result, they noted that the new authority increases efficiency by reducing contracting time and cost for those additional items  The Navy’s Space and Naval Warfare Systems Center Atlantic similarly reported that requirements, which were previously procured using other than simplified acquisition procedures, took up to 60 to 90 days to procure, while it took as little as 3 to 4 days under this new authority, which enabled its scientists and engineers to purchase materials needed for critical, time sensitive projects  However, lab officials acknowledged that the $10,000 micro-purchase threshold authority—like the laboratory enhancement pilot program—is too new to fully understand how it will increase efficiency and foster innovation over the long term  DOD sponsors several research centers, which are governed through noncompetitive agreements, including contracts  These centers provide the department with access to scientific experts employed by universities and other non-profit organizations  Scientists employed by these external to DOD research centers—specifically, three lab FFRDCs and 13 UARCs—execute DOD-funded science and technology development projects in emerging technical areas  DOD staff oversee these centers using routine oversight of funded research tasks and comprehensive reviews, which help DOD determine whether the centers’ funding should continue  DOD and research center officials told us that their ability to authorize work at the FFRDCs that DOD sponsors is limited by legislative restrictions on the staffing levels at these centers, as well as by infrastructure modernization challenges they face  DOD sponsors three research and development FFRDC labs that were established under noncompetitive procedures  Two of the three lab FFRDCs are operated by universities and one is operated by a nonprofit company  DOD also has contracts with 13 UARCs that fulfill a similar scientific role as the lab FFRDCs, while also differing from them in other respects  These differences are described in more detail in table 1  DOD’s contractor-operated research centers received about $1 3 billion annually in DOD funding in fiscal year 2016 and fiscal year 2017, according to DOD data  The two largest research and development FFRDCs, the Lincoln Laboratory and the Software Engineering Institute, received about 67 percent of total research center funding from DOD in 2017  UARCs received an average of $27 million in DOD funding, which was a 15 percent decrease from 2016  Research centers may also receive work and funding from other federal departments and private companies after obtaining DOD sponsor approval  Appendix II provides an overview of DOD FFRDC and UARC funding in fiscal years 2016 and 2017  DOD Sponsorship and Contract Awards: We reported in 2014 that FFRDCs in the federal government are defined through the sponsoring agreement between the agency and the contractor retained to operate the FFRDC  A written agreement of sponsorship between the government and the FFRDC must be prepared when the FFRDC is established, which may be included in a contract between the government and the FFRDC, or in another legal instrument under which an FFRDC accomplishes effort, or it may be in a separate written agreement  Historically, DOD sponsors retain contractors for many years or decades as FFRDC operators  We found that research centers undertake DOD-sponsored projects and, in some limited instances, scientific projects initiated by centers that are overseen by DOD staff  Individual sponsors enter into noncompetitive contracts with FFRDCs and UARCs  DOD uses noncompetitive contracts to establish or maintain an essential engineering, research, or development capability to be provided by an educational or other nonprofit institution or a federally funded research and development center  Scientific Project Funding: We found that project sponsors provide funding to existing contracts  For example, the government issues orders for requirements under Lincoln Laboratory’s indefinite delivery indefinite quantity base contract as funding sponsors approve new projects  Individual project sponsors, along with the primary sponsor, oversee how project funds are spent by the centers  Project sponsors decide whether they will continue to work with these entities based on perceived performance success  This effectively provides an incentive for FFRDCs and UARCS to perform successfully  This work and review cycle is described in Figure 9 below  FFRDCs and UARCs also partner with DOD government-operated labs to plan and execute technology development projects  For example, according to Navy officials, Naval Surface Warfare Center, Carderock Division collaborated with Navy-sponsored UARCs, such as Penn State’s Applied Research Laboratory, to help develop Navy submarine propeller and propulsion designs  Self-initiated Projects: Research center officials said that DOD provides some research centers with limited funds to self-initiate innovative projects  This funding helps the centers ensure that development projects are not limited to just satisfying near-term DOD requirements  Instead, future generations of DOD technologies can be funded  For example, officials at Johns Hopkins University Applied Physics Laboratory proactively conducted work on advanced naval defense technologies in response to similar technology development in adversary countries  Although Navy sponsors did not fund this initial work, they subsequently provided funding in this area after Hopkins’ research identified a risk reduction strategy for the Navy, according to the Johns Hopkins officials  This allowed the UARC to move relatively quickly on a new science and technology project idea  DOD uses 13 UARCs and three lab FFRDCs to obtain direct access to scientific expertise in emerging technical areas, supplementing research conducted at DOD’s government-owned and operated labs  These research centers provide DOD with additional scientific capabilities and the ability to expand quickly into new technical fields  Hiring Scientific Personnel: Although FFRDCs are largely federally funded, they are generally operated, managed, and administered by either a university or consortium of universities, other not-for-profit or nonprofit organization, or an industrial firm, as an autonomous organization or as an identifiable separate operating unit of a parent organization  The contractor operating the FFRDC exercises primary control over its FFRDC’s business concerns, such as personnel policies and compensation  DOD-funded research centers have flexibility in hiring scientists that leverage a parent institution’s expertise in emerging scientific fields  For example, leadership officials at the Army Institute for Soldier Nanotechnologies UARC at MIT and the Software Engineering Institute FFRDC at Carnegie-Melon University noted that projects they have conducted for DOD have benefitted from university experts in fields such as dark matter physics and artificial intelligence  Personnel Compensation: Research center officials we spoke with noted that their workforce policies permit them to flexibly hire, fire, and compensate staff as needed  Although employee salaries are established separately from the government schedule, they are approved by the government  Further, officials noted that university centers typically offer salaries in line with the labor market, but do not attempt to compete on a salary basis with relatively high, unaffordable private sector company salaries  Instead, they compete on the basis of other factors, such as offering scientists the opportunity to work for a prestigious university conducting science and technology research  Research Center Infrastructure: As with personnel matters, research centers have discretion to manage infrastructure in accordance with the policies and procedures of their parent institutions  While one center, Lincoln Laboratory, is located on government property, others primarily reside on property owned or leased by their parent institutions  According to agency officials, DOD contributes funding for the use and repair of these facilities through their contracts with research centers  Officials noted that Lincoln Laboratory uses military construction funding to pay for new buildings as it is located on government property  Trusted Advisor Role: FFRDCs and UARCs function as trusted advisors for the government and operate in the public interest with objectivity and independence  FFRDCs are independent, private-sector, non-profit organization units required to be free from personal or organizational conflicts of interest, as the FFRDCs answer to the government customer  As a result, DOD’s lab FFRDCs perform tasks that are closely associated with the performance of inherently governmental functions and have access to sensitive and proprietary data  Research center officials noted challenges limiting their work providing scientific expertise to DOD  FFRDCs are also limited in executing infrastructure investments  Limitation on Available Work Hours: DOD FFRDCs are limited by an annual ceiling set by Congress on the amount of staff years of technical effort (STE) that may be funded for defense FFRDCs  We previously found in October 2008 these limits were imposed in response to concerns that DOD was inefficiently using its FFRDCs  We found that the STE workload limitation aimed to ensure that FFRDC work was appropriate and limited resources were being used for DOD’s highest priorities  As a result, Software Engineering Institute officials said they decline many DOD programs’ requests for assistance due to the annual work hour limitation  Further, officials at the Office of the Secretary of Defense’s Studies and FFRDC Management Office reported that this limit significantly constrains the use of DOD’s FFRDCs and that DOD customer demand for their services is significantly greater than the annual STE limit  OSD officials indicated that FFRDC related work must be deferred to later years when these limits are reached, since there are no other legally compliant alternatives capable of fulfilling these requirements  Infrastructure: FFRDC officials we interviewed identified infrastructure challenges—including aging facilities and equipment—as hindering their research and development efforts  For example, many buildings at the Massachusetts Institute of Technology (MIT) Lincoln Laboratory are over 60 years old; MIT considers over half of them to be in substandard condition  According to an MIT official, these facilities, located on government property, were not structurally designed for modern research and have relatively poor vibration isolation, resulting in inefficient workarounds or work that could not be performed  Officials from the Defense Laboratories Office noted that the MIT Lincoln Laboratory is unique among DOD’s FFRDCs in that it is operated on government- owned property  A 2013 study, conducted on behalf of the White House Office of Science and Technology Policy, found that lab infrastructure project funding proposals must compete with hospitals, barracks, runways, and roads and, therefore, tend to be lower on the priority list for military construction funding  A 2017 Defense Science Board report and DOD officials we spoke with indicated this continues to be true  While contract research centers have significant flexibility to execute infrastructure work, they are still affected by limited availability of military construction funding  Officials at another center noted that in some instances, DOD sponsors have been unable or slow to provide required secure facilities and equipment within needed time frames  Delays of this nature can affect the research centers’ ability to deliver the technologies or related services needed by DOD  The Department of Energy (Energy) primarily relies on contractor- operated FFRDCs to operate its labs, while the majority of NASA labs and centers are government-operated  Energy’s national labs form the core of the agency’s scientific work and mission  This is in contrast to DOD-funded labs, which constitute a relatively small aspect of DOD’s overall mission  We have previously found that Energy’s labs can use funding for minor infrastructure improvements  NASA centers can also approve and fund certain facility projects, in accordance with NASA policies, and they have encountered significant challenges with aging infrastructure  Also, in some cases, energy and space research centers have significant challenges with hiring replacement staff and competing with private sector employers for staff  Energy’s labs can hire scientific personnel with the flexibility of private companies, while NASA centers were previously provided hiring flexibilities by Congress in 2004 to facilitate staff hiring  While Energy and NASA’s research entities follow their specific governance models, there are broad characteristics common across these agencies as well as DOD  Table 2 illustrates that while research centers are largely government-owned, the government is not always the operator  As we have reported, Department of Energy national labs are primarily operated by for-profit, non-profit and university FFRDC contractors using management and operating contracts, which are competed on a limited basis  Energy’s funding sponsors and headquarters officials are required to reevaluate FFRDC performance in increments not to exceed 5 years by federal acquisition regulations, which inform future decisions to renew the agreement  In 1990, we designated Energy’s contract management—including both contract administration and project management—a high-risk area because of Energy’s inadequate management and oversight of contractors, leaving the department vulnerable to fraud, waste, abuse, and mismanagement  In 2009, we subsequently narrowed the focus of Energy’s high-risk designation to the National Nuclear Security Administration and Office of Environmental Management, which together oversee four national labs  Further, in our 2017 High Risk report, we found that these two agencies had made progress in addressing our contract management concerns, but we identified continued problems with the agencies having sufficient capacity to mitigate contract and project management risks  Also, we found that they had demonstrated little progress in addressing contract management challenges, particularly in the area of financial management  The Department of Energy uses performance-based management and operating contracts, which have been subject to limited competition, with universities, non-profit companies and for-profit companies to operate the national labs on government-owned property  These contractor-operated FFRDCs provide the vast majority of Energy’s science and technology capacity, rather than supplementing the work of government-operated labs like DOD’s FFRDCs  Energy has depended on the expertise of private organizations to execute its science and technology work since the Manhattan Project produced the first atomic bomb during World War II  The Spallation Neutron Source is an experimental research facility at Oak Ridge National Laboratory—a government-owned contractor-operated laboratory  The Spallation Neutron Source includes the world’s most powerful pulsed-neutron sources and provides information about the structure and properties of materials that cannot be obtained by other means  The Spallation Neutron Source is a user facility whereby researchers from universities, national laboratories, and industry submit proposals, which are peer- reviewed and must compete for time at the user facility  The primary focus of each lab varies based on its expertise and facilities  Energy largely oversees its lab contractors through its headquarters program offices, which include the National Nuclear Security Administration, Office of Science, the Office of Fossil Energy, as well as co-located government field offices  Office of Science-sponsored labs primarily support scientific research for energy and physical sciences, while the National Nuclear Security Administration-sponsored (NNSA) labs primarily focus on nuclear weapons and related science and technologies  Energy also oversees its lab contractors’ activities through on-site Energy oversight offices that work alongside lab management at each FFRDC  Some labs specialize in earlier-phase science, while other labs work on later-phase nuclear weapons technologies in addition to earlier-phase science  As Figure 10 shows, these labs are spread across the United States  Energy has only one government-operated and government-owned lab, the National Energy Technology Laboratory  Key differences between Energy’s contractor-operated and government-operated governance models are described in table 3  Energy’s FFRDCs use their own personnel systems, which Energy officials stated provide more flexibility for hiring and retaining qualified staff  Management within these FFRDCs can move staff in or out of scientific areas more quickly than government labs can, thereby providing greater agility to meet Energy’s needs in emerging science areas  For example, Energy’s lab oversight staff at Oak Ridge National Laboratory told us that use of lab contractors’ human resources management systems allows for workforce flexibilities to meet Energy’s needs  While these contractors have leeway in managing their human resources systems, Energy’s headquarters maintains oversight—through its contracting officers—over employee compensation  Energy’s FFRDC contractors manage and operate nearly all of the department’s government-owned national lab facilities—including day-to- day management of government-controlled facilities and real property  Lab operators used funding to complete minor construction projects, which cost $10 million or less  This funding comes from a percentage of science and technology projects’ funding, requires local Energy oversight office approval, and has streamlined project management requirements  In contrast, major infrastructure upgrades are funded through relatively long and complex line-item funding processes, and projects over $50 million are subject to more rigorous project management requirements  Energy’s labs use a small portion of their funding to initiate discretionary projects for science and technologies that will benefit sponsors in the long-term by maintaining the scientific and technical vitality of the laboratories  To maintain and enhance lab expertise, the National Defense Authorization Act for Fiscal Year 1991 authorized Energy’s contractor-operated labs receiving funding for national security programs to use a percentage of lab funds to perform lab-directed R&D of a creative and innovative nature  The actual percentages allowed to be used for lab-directed R&D are subject to Energy’s approval  Energy’s entities sponsor most national lab projects based on their needs and lab expertise  Typically, earlier foundational science projects are funded through a process whereby funding sponsors issue calls for proposals to Energy’s national labs  Interested scientific teams at labs provide proposals to conduct these projects for sponsor consideration  Sponsors then assess proposals for scientific merit and decide which teams receive funding to execute their projects  NNSA provides funding for later-phase nuclear technology development projects to its labs after agreement is made regarding objectives and deliverables for specific projects, according to Lawrence Livermore National Laboratory officials  Despite their flexibilities with regard to hiring and infrastructure decisions compared to government operated labs, Energy’s lab leadership and government oversight officials noted human resource and facilities related challenges, such as: Sufficiently compensating staff located in high-cost of living areas  For example, the labor market of the San Francisco area, where several Department of Energy national labs are located, is highly competitive for employers  Commercial firms offer salaries and compensation that typically exceed those of government-funded, contractor-operated labs, although Energy’s contractors have more pay flexibility than is allowed for Energy’s government employees  Obtaining government clearances in a timely manner  Energy’s NNSA oversight officials and lab management staff, in particular, cited this challenge, which they stated has led to a backlog of people needing clearances  Government hiring freeze constraining overall hiring  Officials at Energy’s government-operated National Energy Technology Laboratory reported that as a result of a government hiring freeze, the lab has increasingly hired private contractor staff to the point that more than half of the total lab staff is now comprised of contractor employees  Major infrastructure challenges at Energy labs  Energy reported in July 2018 that over half of all national lab buildings are in either substandard or inadequate condition  The Energy Inspector General also identified infrastructure modernization as one of Energy’s top management challenges  This finding followed a mandated commission’s report in 2015 that facilities and infrastructure across Energy’s national lab network were hampered by high levels of deferred maintenance and excess facilities  The majority of NASA’s science and technology facilities are operated within the governance framework of government-operated research centers, similarly to most DOD labs  While government-operated, they have been granted additional legislative flexibilities for hiring employees beyond those normally available to government entities  NASA locates its science and technology staff at four government- operated research centers, one contractor-operated FFRDC, and at five NASA centers assisting space and space flight development  These centers and the Jet Propulsion Laboratory—NASA’s sole sponsored FFRDC—execute NASA’s research missions including technology development in exploration and aeronautics  The differences between these two governance approaches are described in Table 4  NASA also works with Johns Hopkins University Applied Physics Lab, a UARC, to develop major space flight missions  The NASA Glenn Research Center—a government- operated laboratory—is currently developing solar electric propulsion technologies intended to allow manned and unmanned spacecraft to be propelled far beyond earth orbit using solar power  This project is developing large, flexible, radiation-resistant solar arrays that can be unfurled to capture solar energy powering fuel-efficient electrostatic thrusters  Scientists expect a system-level flight test within the next decade to demonstrate key technologies supporting NASA’s Lunar Orbital Platform-Gateway project, a platform to mature necessary short- and long-duration deep space exploration capabilities  Headquarters, including funding sponsors providing oversight for their individual projects  Title 51, Chapters 201 and 203 of U S  Code and technology efforts with DOD and other organizations, including use of NASA lab and test facilities  Operated by university contractor having sole source contract  Not permitted to compete against industry, except for operation of an FFRDC  property (originally part of DOD)  5-year contract renewable to 10 years total  Headquarters, including funding sponsors providing oversight for their individual projects  Title 51, Chapters 201 and 203 of U S  Code; 10 U S C  § 2304 (c) (3)(B) ; Federal Acquisition Regulation § 35 017 based NASA oversight staff  sponsors seeking FFRDC assistance with NASA approval  “Lab” as used in this context refers to science and technology organizations equivalent to NASA Research Centers, DOD UARCs and FFRDCs  Mission leadership officials at NASA Headquarters—including the Associate Administrators for Aeronautics Research, Human Exploration, Science and Space Technology—oversee NASA’s research centers as well as the Jet Propulsion Laboratory  These officials are responsible for technology programs providing funds to research centers and the Jet Propulsion Laboratory to support their specific mission areas  NASA’s science and technology project portfolios are based on the requirements and priorities established by NASA’s leaders in collaboration with key stakeholders in academia and industry among others  In planning their science and technology work, NASA’s Glenn Research Center officials noted that NASA research center directors consider the capabilities and resources—including staff and facilities—of other research centers to minimize redundant work  NASA depends on a highly skilled civil servant and contractor workforce to plan and execute its missions  Congress provided NASA with additional human resource authorities beyond those otherwise allowed for federal government personnel through the NASA Flexibility Act of 2004  We found in September 2008 that NASA sought this flexibility to ensure that it could hire and retain the workforce it desired  This law consisted of multiple provisions to address a range of human capital challenges and to strengthen all levels of the workforce  The provisions included incentives—including compensation—to allow NASA to compete successfully in the labor market with the private sector and reshape its workforce more effectively to support the Agency’s mission  NASA also employs a significant contractor workforce across its different centers  Glenn Research Center officials we interviewed stated their portfolio of science and technology projects—and funding—mostly aligns with NASA’s top requirements and priorities  They, along with NASA sponsors, create technology roadmaps and investment plans to determine their future projects  NASA policy requires that NASA’s scientific teams offer proposals for potential research and science and technology projects  This is similar in some ways to how many DOD and Energy centers must find sponsors willing to fund specific technology development projects, rather than receiving technology development funding for a given year  These proposals are reviewed by peer review teams, who identify for selecting officials those proposals they believe have the most scientific merit  Ames Research Center officials said they believe this process can foster innovation, encourage employees to keep skills honed, and mitigate complacency  Glenn Research Center officials said that while most of the work they conduct is for sponsored applied research or advanced technology development, about 2 percent of their science and technology budget is spent on early-stage scientific innovation  Recommended projects of this nature proposed by the research center are typically approved by headquarters officials, according to these Glenn officials  NASA provides technical grants for basic research and applied science to university scientists nationwide on a competitive basis, and also funds similar research done internally at research centers  As with DOD and Energy’s research centers, NASA officials have identified some key operating challenges, including: Aging infrastructure and facilities  The NASA Inspector General listed infrastructure area as one of the top five management and performance challenges facing NASA  Further, the Inspector General identified deficiencies with facilities planning and reported that about 80 percent of facilities at three of four NASA research centers are over 50 years old, while about half of the facilities at the Jet Propulsion Laboratory and the fourth research center are that old  Infrastructure projects and upgrades of $1 million or less are undertaken by research center management instead of at the NASA headquarters level  Construction above this threshold has significantly more requirements and is approved by NASA headquarters  Glenn Research Center officials indicated it is difficult to obtain funding for projects that exceed the minor infrastructure threshold, in part, due competition with major construction of facilities proposals from across the agency for limited funds  As a result, they put most of their efforts into sustaining existing infrastructure  Workforce shortages in key technical areas  As we found in May 2018, NASA has experienced workforce challenges on several major projects such as the Mars 2020 and Europa Clipper projects  Also, over 40 percent of NASA’s workforce is either eligible to retire now or will be eligible in the next 5 years  NASA headquarters officials noted that NASA’s workforce is aging because NASA has a low attrition rate—about 4 percent annually—and high numbers of staff stay several years beyond retirement  Further, in 2017, the NASA Inspector General found gaps in NASA’s workforce planning for specific capability areas and how workforce plans would meet future needs, and recommended that NASA establish standardized guidance defining the data and analyses for these planning efforts  NASA concurred with and identified its plan to implement this recommendation  However, NASA has not implemented this recommendation, according to the NASA Inspector General’s latest semiannual report to Congress  Congress provided DOD lab directors with key authorities to foster targeted, timely investments in the most pressing technology areas  Lab directors have used these authorities—such as laboratory initiated research and direct hire authorities—to varying degrees, but more needs to be done to facilitate innovation and efficiency  Specifically, service specific obstacles in the Air Force, Navy, and Army impede lab directors from capitalizing on laboratory initiated research authority to a greater extent  Service leadership can take actions to better understand and potentially remove barriers to more fully use laboratory initiated research tools  We are making the following three recommendations to DOD: The Secretary of the Air Force should assess the potential costs and benefits of implementing accounting system improvements that would allow the Air Force Research Laboratory to charge customers a fixed percentage fee on provided science and technology activities to the extent allowed under the laboratory initiated research authority  (Recommendation 1) The Secretary of the Navy should clarify whether and how to use the laboratory initiated research authority within the Capital Investment Program  (Recommendation 2) The Secretary of the Army should assess existing Army policy for laboratory initiated research authority and determine whether to implement changes to eliminate disincentives for lab usage of the authority  (Recommendation 3) We provided a draft of this report to DOD, Energy, and NASA for review and comment  Energy and NASA did not provide any comments on the draft report  In DOD’s written comments, reproduced in appendix III, DOD concurred with our three recommendations  Further, in its response to our third recommendation, DOD stated that the Army plans to initiate a study by January 2, 2019, regarding its use of the laboratory initiated research authority  According to DOD, the Army’s study will identify potential opportunities for policy improvements  We are sending copies of this report to the appropriate congressional committees and offices; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; the Secretary of Energy; and the NASA Administrator  In addition, the report will be made available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions concerning this report, please contact me at (202) 512-4841  Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report  Staff members making key contributions to the report are listed in appendix IV  This report examines (1) how the Department of Defense (DOD) labs have used selected legislative authorities to foster innovation and efficiency and identify what barriers impede their use; (2) identifies and describes governance models used by selected DOD-sponsored federally funded research centers and university affiliated research centers; and (3) identifies and describes governance models used non-defense labs, specifically at the Department of Energy (Energy) and National Aeronautics and Space Administration (NASA)  To address the first objective, we selected four specific authorities for our review based on previous work identifying science and technology best practices and expedited lab hiring: Laboratory Initiated Research Authority  The authority provided in Section 219 of the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009, as implemented, provides lab directors with flexibility to fund projects in four allowable categories: basic and applied research; technology transition; workforce development; and revitalization, recapitalization, or repair or minor military construction of lab infrastructure  Laboratory Enhancement Pilot Program  Section 233 of the National Defense Authorization Act for Fiscal Year 2017 established a pilot program for lab directors to propose alternative and innovative methods that might lead to more effectively managing and operating labs and authorized lab directors to waive any regulation, restriction, requirement, guidance, policy, procedure, or departmental instruction that would affect implementation of these methods unless such implementation would be prohibited by a provision of an existing statute or common law  Direct Hire Authority  Four types of direct hire authorities authorized by Congress since 2008 are intended to provide a streamlined and accelerated hiring process to allow the labs to successfully compete with private industry and academia for high-quality scientific, engineering, and technician talent  Micro-purchase Authority  The Federal Acquisition Regulation states a preference for government agencies, to purchase and pay for micro-purchases of supplies or services using the government-wide commercial purchase card up to and at the micro-purchase threshold, but micro-purchases may be conducted using any of the simplified acquisition methods  While the FAR micro-purchase threshold was generally $3,500 at the time of our review, Congress increased this threshold to $10,000 for activities of DOD science and technology reinvention laboratories in Section 217 the National Defense Authorization Act for Fiscal Year 2017  Although Congress has provided additional legislative authorities to defense lab directors to address hiring, infrastructure, and technology transition challenges, the authorities that we covered in our review are the ones that our prior and current work have shown are currently, or have the potential to be, the most critical for supporting science and technology reinvention laboratories’ innovation mission within DOD labs  DOD lab leaders use these authorities to flexibly fund projects intended to facilitate research and development; propose alternative and innovative methods that might lead to more effective lab management; directly hire personnel at DOD labs including students currently enrolled in science, technology, engineering, and mathematics (STEM) programs; and expand critical science and technology purchases using simplified acquisition methods  To identify the extent to which DOD laboratories have used these authorities as well as to identify what potential barriers existed to using these authorities, we administered a survey to 44 STRL directors (or their equivalent) to collect information on the use of these specific authorities, their perceptions about the effectiveness of those authorities, and their perceptions about any barriers to using these authorities  The members of the population surveyed were the 44 defense laboratories defined as science and technology reinvention laboratories  For the purposes of our review, we defined laboratories as inclusive of Air Force technical directorates (10), Army warfare centers (17), and Navy warfare centers (17)  We emailed questionnaires to the laboratories beginning in late March 2018, and survey data collection ended in early May 2018, with 31 labs returning completed questionnaires, for an overall response rate of 71 percent at the laboratory level  We took steps to minimize the potential errors that the practical difficulties of conducting any survey may introduce  Nonresponse error can result when a survey fails to capture information from all population members selected into a survey sample  Of the 13 questionnaires not returned, 4 were Army warfare centers, and 9 were Air Force research directorates  Throughout the data collection period, we made multiple follow-up attempts by email and phone to those labs not yet responding  The Air Force Research Laboratory (AFRL) provided a single survey response for the entire laboratory enterprise  Not all returned questionnaires may have answers to every question applicable to a respondent  However, this question-level nonresponse did not exceed one for any of the questions applicable to all 31 labs  Because we selected the entire population of laboratories for our survey, our estimates are not subject to sampling error  We developed our list of the 44 labs in our population in consultation with DOD, and are confident that none were left out, so our or survey has no known sources of coverage error  We conducted pretests of the draft questionnaire with 3 laboratories in the population and made revisions to reduce the possibility of measurement error from differences in how questions were interpreted and the sources of information available to respondents  After reviewing the answers received, we also followed up as necessary with respondents to clarify apparent inconsistencies or other possible misreports, and made changes to responses where corrections were needed  A second, independent analyst checked the accuracy of all computer analyses to minimize the likelihood of errors in data processing  To obtain additional information on this objective, we reviewed relevant legislation which established or amended these authorities and reviewed applicable DOD and service policy documentation  Further, we collected military service related information on the usage of two authorities, such as: Spending data on the use of the laboratory initiated research authority  We gathered this information from DOD-mandated reports to Congress on the use of this authority and military service officials  We determined these data to be reliable based on reviews of agency documentation collected and interviews with agency officials  Data on the usage of direct hire authorities by the service laboratories  We collected direct hire data from each of the military services including the number of direct hire authority candidates hired as well as the number of direct hire positions the laboratories were authorized to hire  We determined these data to be reliable based on reviews of agency documentation collected and interviews with agency officials  We also used select findings from our May 2018 report where we evaluated DOD’s use of hiring authorities, including direct hire authority  More information about the scope and methodology of our prior work can be found in that report  In addition, we also collected information on military service proposals to utilize the laboratory enhancement pilot program authority  To obtain further information on department- and service-level involvement in and perspectives of defense laboratory authorities and challenges, we interviewed officials responsible for the management, execution, and oversight of DOD’s science and technology enterprise, including military service labs  At the Office of the Secretary of Defense and military department headquarters level, those responsible for the management and oversight of science and technology activities, we met with officials from the: Office of the Assistant Secretary of Defense for Research and DOD Defense Laboratories Office; Office of the Deputy Assistant Secretary of the Army for Research and Office of the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering; Office of the Assistant Secretary of the Air Force for Financial Office of the Deputy Assistant Secretary of the Navy for Research, Development, Test, and Evaluation; and Office of the Budget, within the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller We also met with military department lab officials responsible for the management and execution of science and technology activities from the: Army Research, Development and Engineering Command; Army Research Laboratory; Army Aviation and Missile Research, Development, and Engineering Air Force Research Laboratory; Naval Research Laboratory; Naval Surface Warfare Center, Headquarters; and Naval Surface Warfare Center, Carderock Division To identify and describe governance models used by selected DOD- sponsored federally funded research centers (FFRDCs) and university affiliated research centers (UARCs), we focused our review on the 3 FFRDCs designated as research and development labs as well as all 13 UARCS sponsored by DOD entities  We reviewed appropriate sections of the FAR language related to FFRDCs and UARCs, DOD guidance for working with FFRDCs and UARCs, relevant contracts, and performance assessments  Further, we met with officials from the office of the Deputy Director, OSD Studies and Federally Funded Research & Development Centers Management and Office to discuss overall FFRDC and UARC management, policies, and challenges facing FFRDCs and UARCs  We interviewed officials at selected research and development FFRDCs and UARCS to discuss their experience conducting DOD research and interactions with their customers, such as defense program executive offices  We met with officials at the two major research and development lab FFRDCs—The Lincoln Laboratory at the Massachusetts Institute of Technology (MIT) and the Software Engineering Institute at Carnegie Mellon University  We also selected a university affiliated research center sponsored by the Army and Navy: The Applied Physics Laboratory at Johns Hopkins University and the Institute for Soldier Nanotechnologies also at the MIT  To identify and describe governance models by non-defense labs, we selected Energy and NASA to focus our efforts  We identified 17 Energy national labs and 4 NASA research centers conducting basic and applied research similar to DOD labs  These agencies, along with DOD, represent 3 of the top 4 agencies in terms of average federal research and development spending from fiscal years 2015 to 2017  In our August 2016 GAO Technology Readiness Assessment Guide, we drew heavily from DOD, NASA, and Energy for best practices, terminology, and examples  This contributed to our decision to focus on Energy and NASA’s research entities in this laboratory governance review  We did not include the fourth agency—the National Institutes of Health—in our review because it is not as similar to DOD  We also reviewed relevant Energy and NASA guidance as well as relevant FAR sections  At Energy, we met with officials from the National Nuclear Security Administration which is semi-autonomous entity within Energy responsible for managing the nation’s nuclear weapons and nuclear security  We also met with Officials from the Office of Science, a program office responsible for supporting energy related fundamental science and research  To gain further insights on operating structures, funding arrangements, and their overall experience we met with lab leadership at selected Energy labs which were chosen based on initial discussions with agency officials and our review of past GAO work: Oak Ridge National Laboratory, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, and National Energy Technology Laboratory (the sole Energy government owned and operated laboratory) We also met with leadership from Battelle Memorial Institute, which is the sole or joint contract manager for five Energy national labs including Oak Ridge National Laboratory  In addition, Battelle is an integrated subcontractor at Lawrence Livermore National Laboratory  At NASA, we met with officials with NASA’s Science Mission Directorate and Mission Support Directorate to discuss overall research center management and operations  We also leveraged ongoing and recently completed work at GAO to gain additional insight on NASA’s operations such as human capital management  Almost all of NASA’s research, space, and space flight centers conduct research and development activities  However, we focused our review on four research centers where NASA primarily conducts its aeronautics research, which has substantial overlap with DOD activities  To gain additional insight into the experience of lab leaders at NASA research centers, we met with officials at NASA’s Glenn Research Center and Ames Research Center  In addition, we also met with officials at the NASA Jet Propulsion Center, which is the only NASA-sponsored FFRDC  We conducted this performance audit from July 2017 to December 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  Office of the Secretary of Defense (OSD) In addition to the contact named above, Christopher R  Durbin (Assistant Director); Charlie Shivers, III (Analyst-in-Charge); Emily Bond; Lorraine Ettaro; Carl Ramirez; Sylvia Schatz; Sean Seales; Brian Smith; and Robin Wilson made significant contributions to this report 


Report 2: The national pipeline system consists of more than 2 7 million miles of networked pipelines transporting oil, natural gas, and other hazardous liquids  Hazardous liquid and natural gas pipelines—primarily buried underground in the continental United States—run under remote and open terrain, as well as densely populated areas  These pipelines are of three main types: Hazardous liquid: About 216,000 miles of hazardous liquid pipeline transport crude oil, diesel fuel, gasoline, jet fuel, anhydrous ammonia, and carbon dioxide  Natural gas transmission and storage: About 319,000 miles of pipeline—mostly interstate—transport natural gas from sources to communities  Natural gas distribution: About 2 2 million miles of pipeline—mostly intrastate—transport natural gas from transmission sites to consumers  Figure 1 depicts the network of hazardous liquid and natural gas transmission pipelines in the United States  More than 3,000 pipeline companies operate the nation’s pipeline systems, which can traverse multiple states and the U S  borders with Canada and Mexico  Many pipeline systems are comprised of the pipelines themselves, as well as a variety of facilities, such as storage tanks, compressor stations, and control centers  Most pipeline systems are monitored and moderated through automated ICS or Supervisory Control and Data Acquisition (SCADA) systems using remote sensors, signals, and preprogramed parameters to activate and deactivate valves and pumps to maintain flows within tolerances  Federal agencies and pipeline operators determine the criticality of pipeline systems and their facilities based on their importance to the nation’s energy infrastructure; service to installations critical to national defense; or, if attacked, have the potential to cause mass casualties and significant impact on public drinking water affecting major population centers  Accordingly, those determined to be critical merit increased attention to security  However, as we previously reported, the inherent design and operation of U S  pipeline systems may reduce some potential impacts of lost service  The pipeline sector is generally considered to be resilient and versatile  Historically, pipeline operators have been able to quickly respond to the adverse consequences of an incident—whether it is damage from a major hurricane or a backhoe—and quickly restore pipeline service  Pipeline infrastructure also includes redundancies such as parallel pipelines or interconnections that enable operators to reroute material through the network  Figure 2 depicts the U S  pipeline system, its basic components, examples of vulnerabilities, and the entities to which it supplies energy and raw materials  These entities include utility companies, airports, military sites, and industrial and manufacturing facilities  According to TSA, pipelines are vulnerable to physical attacks—including the use of firearms or explosives—largely due to their stationary nature, the volatility of transported products, and the dispersed nature of pipeline networks spanning urban and outlying areas  The nature of the transported commodity and the potential effect of an attack on national security, commerce, and public health make some pipelines and their assets more attractive targets for attack  Oil and gas pipelines have been and continue to be targeted by terrorists and other malicious groups globally  Terrorists have also targeted U S  pipelines, but have not succeeded in attacking them  Further, environmental activists and lone actors seeking to halt the construction of new pipelines through sabotage have recently emerged as a new threat to pipelines  For example, in March 2017, activists used blowtorches to cut holes in empty portions of the Dakota Access Pipeline in two states  In February 2017, local law enforcement officers fatally shot a man who used an assault rifle to damage the Sabal Trail Pipeline, a natural gas pipeline under construction in Florida  The sophisticated computer systems that pipeline operations rely on are also vulnerable to various cyber threats  According to DOE, the frequency, scale, and sophistication of cyber threats have increased, and attacks have become easier to launch  NCCIC reported that the energy sector, which includes pipelines, experienced more cyber incidents than any sector from 2013 to 2015, accounting for 35 percent of the 796 incidents reported by all critical infrastructure sectors  In 2016, NCCIC reported that the energy sector was the third most frequently attacked sector  Further, according to DOE, the cost of preventing and responding to cyber incidents in the energy sector is straining the ability of companies to adequately protect their critical cyber systems  For example, a 2015 study by the Ponemon Institute estimated the annualized cost of cyber crime for an average energy company to be about $28 million  Ineffective protection of cyber assets from these threats can increase the likelihood of security incidents and cyber attacks that disrupt critical operations; lead to inappropriate access to and disclosure, modification, or destruction of sensitive information; and threaten national security, economic well-being, and public health and safety  Unintentional or nonadversarial threat sources may include failures in equipment or software due to aging, resource depletion, and errors made by end users  They also include natural disasters and failures of critical infrastructure on which the organization depends, but that are outside of the control of the organization  Intentional or adversarial threats may include corrupt employees, criminal groups, terrorists, and nations that seek to leverage the organization’s dependence on cyber resources (i e , information in electronic form, information and communications technologies, and the communications and information-handling capabilities provided by those technologies)  These threat adversaries vary in terms of their capabilities, their willingness to act, and their motives, which can include seeking monetary gain or seeking an economic, political, or military advantage  Cyber threat adversaries make use of various techniques, tactics, practices, and exploits to adversely affect an organization’s computers, software, or networks, or to intercept or steal valuable or sensitive information  For example, an attacker could infiltrate a pipeline’s operational systems via the internet or other communication pathways to potentially disrupt its service and cause spills, releases, explosions, or fires  Moreover, ICS, which were once largely isolated from the Internet and the company’s information technology systems, are increasingly connected in modern energy systems, allowing cyber attacks to originate in business systems and migrate to operational systems  For example, malicious nation-state actors used spear-phishing and other similar approaches in 2018 against energy sector organizations to gain access to their business systems, conduct reconnaissance, and collect information about their ICS  Similarly, in April 2012, the Industrial Control Systems Cyber Emergency Response Team reported that an unidentified cyber attacker had conducted a series of cyber intrusions into U S  natural gas pipeline systems beginning in December 2011  Federal policy and public-private plans establish roles and responsibilities for the protection of critical infrastructure, including pipelines  These include Presidential Policy Directive 21 (PPD-21), the NIPP, and Executive Order 13636  PPD-21, issued in February 2013, reflects an all- hazards approach to protecting critical infrastructure, including natural disasters, terrorism, and cyber incidents  The directive also identifies the 16 critical infrastructure sectors and assigns roles and responsibilities for each critical infrastructure sector among nine designated federal sector-specific agencies  While PPD-21 identified the critical infrastructure sectors and assigned responsibility for each sector’s sector-specific agency, the NIPP outlines critical infrastructure stakeholder roles and responsibilities regarding critical security and resilience  It describes a voluntary partnership model as the primary means of coordinating government and private sector efforts to protect critical infrastructure  As part of the partnership structure, the designated sector-specific agencies serve as the lead coordinators for security programs of their respective sector  As sector-specific agencies, federal departments or agencies lead, facilitate, or support the security and resilience programs and associated activities of their designated critical infrastructure sector  For example, DHS and DOT are both designated as sector-specific agencies for the transportation systems sector, which includes pipelines  Each sector also has a government coordinating council, consisting of representatives from various levels of government, and many have a sector coordinating council (SCC) consisting of owner-operators of these critical assets or members of their respective trade associations  For example, the Transportation Government Coordinating Council has been established, and the Pipeline Modal SCC has been established to represent pipeline operators  The NIPP also outlines a risk management framework for critical infrastructure protection  As shown in Figure 3, the NIPP uses a risk management framework as a planning methodology intended to inform how decision makers take actions to manage risk  The risk management framework calls for public and private critical infrastructure partners to conduct risk assessments to understand the most likely and severe incidents that could affect their operations and communities, and use this information to support planning and resource allocation  According to DHS, the risk management framework is influenced by the nature and magnitude of a threat, the vulnerabilities to that threat, and the consequences that could result, as shown in Figure 4  Federal policy has encouraged voluntary information-sharing mechanisms between the federal government and critical infrastructure owners and operators  For example, Information Sharing and Analysis Centers (ISAC) are formed by critical infrastructure owners and operators to gather, analyze, appropriately sanitize, and disseminate intelligence and information related to critical infrastructure  They typically collect, analyze and disseminate actionable threat information to their members and provide members with tools to mitigate risks and enhance resiliency  ISACs in which pipeline operators may participate have been formed including the Oil and Natural Gas ISAC, Downstream Natural Gas ISAC, and Electricity ISAC  Finally, in February 2013, the president issued Executive Order 13636, Improving Critical Infrastructure Cybersecurity, which cited repeated cyber intrusions into critical infrastructure as demonstrating the need for improved cybersecurity  Executive Order 13636 outlined actions for improving critical infrastructure cybersecurity, including direction for the National Institute of Standards and Technology (NIST) to lead the development of a voluntary risk-based cybersecurity framework that would comprise a set of industry standards and best practices to help organizations manage cybersecurity risks  NIST issued the framework in 2014 and updated it in April 2018  The order also addressed the need to improve cybersecurity information sharing and collaboratively develop risk-based standards and stated that U S  policy was to increase the volume, timeliness, and quality of cyber threat information shared with private sector entities so that these entities may better protect and defend themselves against cyber threats  Protecting the nation’s pipeline systems is a responsibility shared by both the federal government and private industry  As a result, several federal departments, agencies, and the private sector have significant roles in pipeline physical and cyber-related security  These entities include the following: Transportation Security Administration (TSA)  TSA, within DHS, has primary oversight responsibility for the physical security and cybersecurity of transmission and distribution pipeline systems  Within TSA, the Security Policy and Industry Engagement’s Pipeline Security Branch is charged with overseeing its pipeline security program  Pursuant to its authority, TSA’s Pipeline Security Branch first issued its voluntary Pipeline Security Guidelines in 2011, and released revised guidelines in March 2018  In accordance with the 9/11 Commission Act, TSA’s Pipeline Security Branch identifies the top 100 critical pipeline systems in the nation  To do so, it uses system annual throughput, which is based on the amount of hazardous liquid or natural gas product transported through a pipeline in 1 year (i e , annual throughput)  TSA also ranks the relative risk among the top 100 critical pipeline systems, discussed later in the report  Additionally, TSA’s Pipeline Security Branch is responsible for conducting voluntary Corporate Security Reviews (CSR) and Critical Facility Security Reviews (CFSR), which assess the extent to which the 100 most critical pipeline systems are following the intent of TSA’s Pipeline Security Guidelines  See figure 5 below for an overview of the CSR and CFSR processes  In addition, TSA Intelligence and Analysis is responsible for collecting and analyzing threat information related to the transportation network, and sharing relevant threat information to pipeline stakeholders  National Cybersecurity and Communications Integration Center (NCCIC)  Within DHS, NCCIC assists critical infrastructure owners in addressing cyber incidents and attacks, including those targeting industrial control systems  The NCCIC’s mission is to reduce the likelihood and severity of incidents that may significantly compromise the security and resilience of the nation’s critical information technology and communications networks  NCCIC’s role is to serve as the federal civilian interface for sharing information related to cybersecurity risks, incidents, analysis, and warnings with federal and nonfederal entities, and to provide shared situational awareness to enable real-time actions to address cybersecurity risks and incidents to federal and nonfederal entities  Pipeline and Hazardous Materials Safety Administration (PHMSA)  PHMSA, within DOT, is responsible for regulating the safety of hazardous materials transportation and the safety of pipeline systems, some aspects of which can be related to pipeline security  In 2004, PHMSA and TSA entered into a memorandum of understanding regarding their respective roles in all modes of transportation  In 2006, they signed an annex to the memorandum of understanding that further delineates lines of authority and responsibility between TSA and PHMSA on pipeline and hazardous materials transportation security  The annex identifies TSA as the lead federal entity for transportation security, including hazardous materials and pipeline security, and PHMSA as responsible for administering a national program of safety in natural gas and hazardous liquid pipeline transportation, including identifying pipeline safety concerns and developing uniform safety standards  Department of Energy (DOE)  DOE is responsible for protecting electric power, oil, and natural gas delivery infrastructure and, in December 2015, was identified in statute as the sector-specific agency for cybersecurity for the energy sector  The Office of Cybersecurity, Energy Security, and Emergency Response is the lead for DOE’s cybersecurity efforts  In addition, DOE operates the National SCADA Test Bed Program, a partnership with Idaho National Laboratory, Sandia National Laboratories, and other national laboratories which addresses control system security challenges in the energy sector  Among its key functions, the program performs control systems testing, research, and development; control systems requirements development; and industry outreach  Federal Energy Regulatory Commission (FERC)  FERC regulates the U S  bulk electric power system, which is increasingly powered by natural gas pipeline systems  FERC has regulatory authority over interstate natural gas pipelines under the Natural Gas Act  However, its role is limited to natural gas pipeline siting and rate regulation  The North American Electric Reliability Corporation is the federally designated U S  Electric Reliability Organization, and is overseen by FERC  The North American Electric Reliability Corporation, with approval from FERC, has developed mandatory critical infrastructure protection standards for protecting electric utility–critical and cyber-critical assets  Private sector  Although TSA has primary federal responsibility for overseeing interstate pipeline security, private sector pipeline operators are responsible for implementing asset-specific protective security measures  As we previously reported, operators have increased their attention on security by incorporating security practices and programs into their overall business operations  Pipeline operators’ interests and concerns are primarily represented by five major trade associations with ties to the pipeline industry—the Interstate Natural Gas Association of America (INGAA), American Gas Association (AGA), American Public Gas Association, American Petroleum Institute (API), and Association of Oil Pipe Lines  According to TSA officials, pipeline operators, and association representatives, these associations have worked closely with the federal government on a variety of pipeline security-related issues, including collaborating on TSA’s voluntary standards and information sharing  All of the pipeline operators and pipeline association representatives we interviewed reported receiving security information from federal and nonfederal entities  Pipeline operators also reported providing security- related information to federal agencies, including TSA, as incidents occur  Multiple federal entities exchange alerts of physical and cybersecurity incidents and other risk-related information with critical infrastructure partners, including pipeline operators  For example, DHS components including TSA’s Intelligence and Analysis and NCCIC share security- related information on physical and cyber threats and incidents with sector stakeholders  Specifically, Intelligence and Analysis provides quarterly intelligence briefings to pipeline operators  NCCIC also issues indicator bulletins, which can contain information related to cyber threat indicators, defensive measures, and cybersecurity risks and incidents  In addition, TSA and other federal entities have coordinated to address specific pipeline-related security incidents  For example, TSA officials coordinated with DOT, DOE, the Department of Justice, and FERC through the Oil and Natural Gas subsector SCC to address ongoing incidents of vandalism and sabotage of critical pipeline assets by environmental activists in 2016  In July 2017, according to DOT officials, PHMSA and TSA collaborated on a web-based portal to facilitate sharing sensitive but unclassified incident information among federal agencies with pipeline-related responsibilities  See table 1 for the key federal information sharing entities and programs that exchange security-related or incident information with critical infrastructure stakeholders, including the pipeline sector  Pipeline operators also share security-related information with TSA and the NCCIC  In its Pipeline Security Guidelines, TSA requests that pipeline operators report by telephone or email to its Transportation Security Operations Center (TSOC) any physical security incidents that are indicative of a deliberate attempt to disrupt pipeline operations or activities that could be considered precursors to such an attempt  TSA’s Pipeline Security Guidelines also request that operators report any actual or suspected cyber attacks that could impact pipeline industrial control systems or other information technology-based systems to the NCCIC  According to the TSOC’s operating procedures, if a reported incident meets certain criteria, such as the incident was intended to or resulted in damage or requires a general evacuation of a facility, the TSOC watch officer is then to contact Office of Security and Industry Engagement officials  According to TSA officials, the TSOC does not conduct investigations of the specific security incidents that pipeline operators report  However, TSOC staff do analyze the incident information they receive for national trends and common threats  TSA officials stated that they share their observations with pipeline operators and other critical infrastructure asset owners during monthly and quarterly conference calls that TSA holds with pipeline operators  All the pipeline operators and association representatives we interviewed identified other nonfederal information sharing entities, including ISACs, fusion centers, industry associations, and SCCs, which provide forums for exchanging information about physical and cyber incidents throughout the pipeline sector  See table 2 for nonfederal information sharing entities identified as available to pipeline operators  Operators and TSA officials reported that the current backlog in granting security clearances for some key pipeline operator employees was a significant factor affecting information sharing between TSA and pipeline operators  TSA officials acknowledged that some pipeline operators have had difficulty obtaining security clearances for key employees due to ongoing backlogs in processing requests by the Office of Personnel Management National Background Investigation Bureau, and that TSA’s ability to share timely information with operators whose staff do not have a clearance may be hindered  Three of the 10 pipeline operators we interviewed identified receiving timely classified security information as a specific challenge due, in part, to difficulties staff have had obtaining security clearances  Further, 7 of the 10 pipeline operators that we interviewed reported experiencing delays in obtaining a security clearance or were aware of others who had experienced this issue  However, according to three operators we interviewed, TSA was helpful in facilitating approval of security clearances for the operators’ personnel to access classified information when necessary  This security clearance challenge is not faced by pipeline operators alone  In January 2018, we designated the backlog of investigations for the clearance process and the government-wide personnel security clearance process as a high-risk area  We will continue to monitor agencies’ progress in reducing the backlog and improving the security clearance process  Pipeline operators that we interviewed reported using a range of guidelines and standards to address their physical and cybersecurity risks, and all of them reported implementing TSA’s voluntary Pipeline Security Guidelines that were applicable to their operations  TSA revised and issued its Pipeline Security Guidelines in March 2018, but the revised guidelines lack a defined process to consider updates to supporting guidance such as to the NIST Framework for Improving Critical Infrastructure Cybersecurity (Cybersecurity Framework)  Furthermore, TSA has not clearly defined the terms within the criteria that pipeline operators are to use to determine the criticality of their facilities  Pipeline operators that we interviewed reported using a range of guidelines and standards to address their physical and cybersecurity risks  For example, all 10 of the pipeline operators we interviewed stated they had implemented the voluntary 2011 TSA Pipeline Security Guidelines the operators determined to be applicable to their operations  The guidelines provide TSA’s recommendations for pipeline industry security practices such as establishing a corporate security program and identifying critical facilities among others (see sidebar)  Five of the 10 pipeline operators we interviewed characterized the guidelines as generally or somewhat effective in helping to secure their operations, 1 was neutral on their effectiveness, and 4 did not provide an assessment of the guidelines’ effectiveness  However, one operator pointed out that they had not adopted the guidelines’ recommended interval of 36 months or less for conducting security vulnerability assessments due to staffing limitations  Also, another pipeline operator noted that they were working to implement the guidelines in the operations of a newly acquired asset that they determined was not using the guidelines in the same manner as their company  All of the pipeline operators we interviewed reported using other guidelines or standards to address pipeline systems’ security risks  For example, pipeline operators reported using and industry association representatives reported that their members use INGAA’s Control Systems Cyber Security Guidelines for the Natural Gas Pipeline Industry, API’s Pipeline SCADA Security standard, and the NIST Cybersecurity Framework as sources of cybersecurity standards, guidelines, and practices that may be scaled and applied to address a pipeline operator’s cybersecurity risks  Further, pipeline operators are required to adhere to regulations related to pipeline safety and, depending upon their assets, operations, and location, may be required to adhere to regulations for electrical utilities, chemical storage facilities, and locations near waterways  For example, all pipeline operators must adhere to DOT’s PHMSA safety regulations  In addition, pipeline operators whose systems include chemical facilities may be required to comply with DHS’s Chemical Facility Anti-Terrorism Standards (CFATS)  Pipeline operators whose systems include a terminal located on a U S  port may be required to comply with Maritime Transportation Security Act regulations  For a listing of federal and industry guidelines identified as applicable to security by the pipeline operators, see appendix I  TSA’s Pipeline Security Branch issued its revised Pipeline Security Guidelines in March 2018, but TSA has not established a documented process to ensure that revisions occur and fully capture updates to supporting standards and guidance  The guidelines were revised to, among other things, reflect the dynamic threat environment and to incorporate cybersecurity principles and practices from the NIST Cybersecurity Framework, which were initially issued in February 2014  To revise the guidelines and incorporate feedback, according to Pipeline Security Branch officials, they incorporated outcomes from pipeline modal threat assessments and best practices from security reviews, and collaborated with pipeline sector stakeholders—including industry associations and other federal agencies with a role in pipeline security  Officials from the industry associations we interviewed confirmed that they provided input to the revised pipeline guidelines, including meeting with and consolidating comments from member pipeline operators  See figure 6 for a timeline of events pertinent to federal pipeline security guidelines  TSA’s Pipeline Security Smart Practice Observations for pipeline operators states that security plans should have a documented process to include security plan reviews and updates on a periodic and an as- needed basis  Standards for Internal Control in the Federal Government states that periodic review of policies, procedures, and related control activities should occur to determine their continued relevance and effectiveness in achieving identified objectives or addressing related risks  The NIPP and NIST also emphasize the need to provide updates on incident response guidance and security procedures, respectively  Moreover, other pipeline industry guidance cited by TSA’s guidelines also has a prescribed interval for review and revision  For example, API reviews its standards at least every 5 years  However, TSA has not instituted a documented process to consider the need to update the Pipeline Security Guidelines on a regular basis  Pipeline Security Branch officials acknowledged the value of having a defined process for reviewing and, if necessary, revising TSA’s Pipeline Security Guidelines at regular defined intervals to ensure it includes, among other things, newly identified best practices and updated industry guidance that are relevant to pipeline operators, such as the elements of the latest version of NIST’s Cybersecurity Framework  For example, TSA’s revisions to its guidelines incorporated some, but not all of the elements of the NIST Cybersecurity Framework version 1  Specifically, to improve incident response, the NIST framework recommends implementing an incident response analysis and feedback function to a security program  However, TSA’s Pipeline Security Guidelines do not include similar steps for pipelines operators to include in their pipeline security programs  Further, because NIST released version 1 1 of the Cybersecurity Framework in April 2018, the guidelines that TSA released in March 2018 do not incorporate cybersecurity elements that NIST added to the latest Cybersecurity Framework such as the Supply Chain Risk Management category  Pipeline Security Branch officials said that they have not instituted a review process on a regular basis because they intended to review and revise TSA’s guidelines on an as-needed basis in response to updated supporting guidance, but could provide no timeline for doing so  Without a documented process defining how frequently Pipeline Security Branch staff are to review and revise its guidelines, TSA cannot ensure that its guidelines reflect the latest known standards and best practices for physical and cybersecurity, or address the persistent and dynamic security threat environment currently facing the nation’s pipeline system  Under TSA’s Pipeline Security Guidelines, pipeline operators are to self- identify the critical facilities within their system and report their critical facilities to TSA  TSA’s Pipeline Security Branch conducts CFSRs at the critical facilities that pipeline operators have identified  However, our analysis of TSA’s data found that at least 34 of the top 100 critical pipeline systems deemed highest risk indicated that they had no critical facilities  Accordingly, TSA would not conduct a CFSR at any of these systems’ facilities because their operators identified none of them as critical  The fact that pipeline operators of about one third of the highest risk systems identified no critical facilities may be due, in part, to the Pipeline Security Branch not clearly defining the criteria outlined in the Pipeline Security Guidelines that pipeline operators are to use to determine the criticality of their facilities  Three of the 10 operators we interviewed stated that some companies reported to TSA that they had no critical facilities, and may possibly be taking advantage of the guidelines’ lack of clarity  Accordingly, operators that report no critical facilities would avoid TSA’s reviews of their facilities  service or deliverability resulting in a state or local government's inability to provide essential public services and emergency response for an extended period of time; Significantly damage or destroy national intended use of major rivers, lakes, or waterways (e g , public drinking water for large populations or disruption of major commerce or public transportation routes); service or deliverability to a significant number of customers or individuals for an extended period of time; operations for an extended period of time (i e , business critical facilities)  Our review of the eight criteria included in TSA’s Pipeline Security Guidelines (see sidebar) found that no additional examples or clarification are provided to help operators determine criticality  Although we previously noted that 5 of the 10 operators we interviewed generally found TSA’s Guidelines as a whole helpful in addressing pipeline security, more than half of the operators we interviewed identified TSA’s criticality criteria as a specific area for improvement  Specifically, 3 of the 10 pipeline operators that we interviewed stated that TSA had not clearly defined certain terms within the criteria, and 3 additional operators of the 10 reported that additional consultation with TSA was necessary to appropriately apply the criteria and determine their facilities’ criticality  For example, 2 operators told us that individual operators may interpret TSA’s criterion, “cause mass casualties or significant health effect,” differently  One of these operators that we interviewed stated that this criterion could be interpreted either as a specific number of people affected or a sufficient volume to overwhelm a local health department, which could vary depending on the locality  Another operator reported that because TSA’s criteria were not clear, they created their own criteria which helped the operator identify two additional critical facilities  Pipeline Security Branch officials acknowledged there are companies that report having no critical facilities in their pipeline systems  According to Pipeline Security Branch officials, pipeline operators are in the best position to determine which of their facilities are critical, and the companies that have determined that their pipeline systems have no critical facilities also have reported sufficient redundancies to make none of their facilities critical to the continuity of their operations  According to these officials, they have had extensive discussions with pipeline company officials to assess the validity of their criticality determinations, and have closely questioned companies to ensure they have properly applied TSA’s criteria  However, according to TSA’s Pipeline Security Guidelines, operators should use a consistent set of criteria for determining the criticality of their facilities  In addition, Standards for Internal Control in the Federal Government states that management should define objectives clearly to enable the identification of risks  To achieve this, management generally defines objectives in specific and measurable terms and ensures the terms are fully and clearly set forth so they can be easily understood  Pipeline Security Branch officials acknowledged that the criticality definitions in the Pipeline Security Guidelines could be clarified to be more specific  Additionally, an industry association representative reported that the association, in consultation with TSA, has been developing supplementary guidance for its members to clarify certain terms in TSA’s critical facility criteria  As of October 2018 this guidance is still under review at the association and has not been made available to the association’s members  Pipeline Security Branch officials confirmed they worked with the industry association on its supplementary guidance, but also acknowledged that the supplementary guidance may only be distributed to the association’s membership  Without clearly defined criteria for determining pipeline facilities’ criticality, TSA cannot ensure that pipeline operators are applying its guidance uniformly  Further, because TSA selects the pipeline facilities on which to conduct CFSRs based on operators’ determinations, TSA cannot fully ensure that all of the critical facilities across the pipeline sector have been identified using the same criteria, or that their vulnerabilities have been identified and addressed  TSA’s Intelligence and Analysis identifies security risks to pipeline systems through various assessments  Additionally, TSA’s Pipeline Security Branch conducts security reviews to assess pipeline operators’ implementation of TSA’s Pipeline Security Guidelines, but gaps in staffing and lack of a workforce plan may affect its ability to carry out effective reviews  The Pipeline Security Branch also developed a pipeline risk assessment to rank relative risk of the top 100 critical pipeline systems and to prioritize its security reviews of pipeline companies, but shortfalls in its calculations of threat, vulnerability, and consequence may limit its ability to accurately identify pipeline systems with the highest risk  Finally, the pipeline risk assessment has not been peer reviewed to validate the assessment’s data and methodology, which we previously reported as a best practice in risk management  TSA’s Intelligence and Analysis produces assessments related to pipeline security risks, including Pipeline Modal and Cyber Modal Threat Assessments and the Transportation Sector Security Risk Assessment  The Pipeline and Cyber Modal Threat Assessments are issued on a semiannual basis; TSA Intelligence and Analysis may also issue additional situation-based products on emerging threats  The Pipeline Modal and Cyber Modal Threat Assessments evaluate, respectively, physical and cyber threats to pipelines  The pipeline modal threat assessment evaluates terrorist threats to hazardous liquid and natural gas pipelines, and the cyber modal threat assessment evaluates cyber threats to transportation, including pipelines  Both assessments specifically analyze the primary threat actors, their capabilities, and activities—including attacks occurring internationally—as well as other characteristics of threat  The Transportation Sector Security Risk Assessment assesses threat, vulnerability, and consequence for various attack scenarios across the five transportation modes for which TSA is responsible  The scenarios define a type of threat actor—including homegrown violent extremists and transnational extremists, such as al Qaeda and its affiliates—a target, and an attack mode  For example, a scenario might assess the risk of attacks using varying sizes of improvised explosive devices on pipeline system assets  As part of the assessment process, TSA engages with subject matter experts from TSA and industry stakeholder representatives to compile vulnerabilities for each mode, and TSA analyzes both direct and indirect consequences of the various attack scenarios  According to Pipeline Security Branch officials, the assessments produced by TSA’s Intelligence and Analysis provide key information to inform the pipeline security program’s efforts  According to TSA officials, TSA conducts pipeline security reviews— Corporate Security Reviews (CSRs) and Critical Facility Security Reviews (CFSRs)—to assess pipeline vulnerabilities and industry implementation of TSA’s Pipeline Security Guidelines  However, as shown by Figure 7 below, the number of CSRs and CFSRs completed by TSA has varied during the last five fiscal years, ranging from zero CSRs conducted in fiscal year 2014 to 23 CSRs conducted in fiscal year 2018, as of July 31, 2018  TSA officials reported that staffing limitations have prevented TSA from conducting more reviews  As shown in table 3, TSA Pipeline Security Branch staffing levels (excluding contractor support) have varied significantly over the past 9 years ranging from 14 full-time equivalents (FTEs) in fiscal years 2012 and 2013 to one FTE in fiscal year 2014  They stated that, while contractor support has assisted with conducting CFSRs, there were no contractor personnel providing CSR support from fiscal years 2010 through 2017, but that has now increased to two personnel in fiscal year 2018  TSA prioritizes reviewing and collecting information on the nation’s top 100 critical pipeline systems  According to TSA officials, they would need to conduct 46 CSRs in order to review the top 100 critical pipeline systems  In July 2018, TSA officials stated that TSA’s current target was to assess each pipeline company every 2 to 3 years; this would equate to about 15 to 23 CSRs per year  TSA officials stated that they expect to complete 20 CSRs and 60 CFSRs per fiscal year with Pipeline Security Branch employees and contract support, and have completed 23 CSRs through July 2018 for fiscal year 2018  Given the ever-increasing cybersecurity risks to pipeline systems, ensuring that the Pipeline Security Branch has the required cybersecurity skills to effectively evaluate pipeline systems’ cybersecurity is essential  Pipeline operators we interviewed emphasized the importance of cybersecurity skills among TSA staff  Specifically, 6 of the 10 pipeline operators and 3 of the 5 industry representatives we interviewed reported that the level of cybersecurity expertise among TSA staff and contractors may challenge the Pipeline Security Branch’s ability to fully assess the cybersecurity portions of its security reviews  TSA officials stated that Security Policy and Industry Engagement staff are working with DHS’s National Protection and Programs Directorate to help address cyber- related needs, including identifying specific cybersecurity skills and competencies required for the pipeline security program  The officials were uncertain, however, whether TSA would use contractor support or support from the National Protection and Programs Directorate to provide identified skills and competencies  TSA officials also stated that Security Policy and Industry Engagement staff work with TSA’s human resource professionals to identify critical skills and competencies needed for Pipeline Security Branch personnel, and helps its workforce maintain professional expertise by providing training and education for any identified skill or competency gaps  Our previous work has identified principles that a strategic workforce planning process should follow including developing strategies tailored to address gaps in number, deployment, and alignment of human capital approaches for enabling and sustaining the contributions of all critical skills and competencies  Workforce planning efforts, linked to an agency’s strategic goals and objectives, can enable it to remain aware of and be prepared for its needs, including the size of its workforce, its deployment across the organization, and the knowledge, skills, and abilities needed for it to pursue its mission  Agencies should consider how hiring, training, staff development, performance management, and other human capital strategies can be aligned to eliminate gaps and improve the long-term contribution of skills and competencies identified as important for mission success  TSA has not established a workforce plan for its Security Policy and Industry Engagement or its Pipeline Security Branch that identifies staffing needs and skill sets such as the required level of cybersecurity expertise among TSA staff and contractors  When asked for TSA strategic workforce planning documents used to inform staffing allocations related to the pipeline security program, TSA officials acknowledged they do not have a strategic workforce plan  Rather, according to these officials, TSA determines agency-level staffing allocations through the Planning, Programming, Budgeting and Execution process, which is used to decide policy, strategy, and the development of personnel and capabilities to accomplish anticipated missions  According to TSA officials, when they use this process they look at existing resources and then set priorities based on the TSA Administrator’s needs  However, a strategic workforce plan allows an agency to identify and prepare for its needs, such as the size of its workforce, its deployment across the organization, and the knowledge, skills, and abilities needed to pursue its mission  TSA officials stated that the agency has a detailed allocation plan for strategically aligning resources to screen passengers at TSA-regulated airports, but not for the entire agency  By establishing a strategic workforce plan, TSA can help ensure it has identified the knowledge, skills, and abilities that the future workforce of TSA’s Pipeline Security Branch may need in order to meet its mission of reducing pipeline systems’ vulnerabilities to physical and cybersecurity risks, especially in a dynamic and evolving threat environment  Further, as greater emphasis is placed on cybersecurity, determining the long- term staffing needs of the Pipeline Security Branch will be essential  Furthermore, a workforce plan could enable TSA to determine the number of personnel it needs to meet its stated goals for conducting CSRs and CFSRs  After TSA identifies the top 100 critical pipeline systems based on throughput, the Pipeline Security Branch uses the Pipeline Relative Risk Ranking Tool (risk assessment), which it developed in 2007, to assess various security risks of those systems  We previously reported, in 2010, that the Pipeline Security Branch was the first of TSA’s surface transportation modes to develop a risk assessment model that combined all three components of risk—threat, vulnerability, and consequence—to generate a risk score  The risk assessment generates a risk score for each of the 100 most critical pipeline systems and ranks them according to risk  The risk assessment calculates threat, vulnerability, and consequence for each pipeline system on variables such as the amount of throughput in the pipeline system and the number critical facilities  The risk assessment combines data collected from pipeline operators, as well as other federal agencies, such as the Departments of Transportation and Defense, to generate the risk score  However, the last time the Pipeline Security Branch calculated relative risk among the top 100 critical pipeline systems using the risk assessment was in 2014  Pipeline Security Branch officials told us that they use the pipeline risk assessment to rank relative risk of the top 100 critical pipeline systems, and the standard operating procedures for conducting CSRs state the results of the risk ranking are the primary factor considered when prioritizing corporate security reviews of pipeline companies  According to Pipeline Security Branch officials, the risk assessment has not changed since 2014 because the Pipeline Security Branch is still conducting CSRs based on the 2014 ranking of pipeline systems  As outlined in table 4 below, we identified several factors that likely limit the usefulness of the current risk assessment in calculating threat, vulnerability, and consequence to allow the Pipeline Security Branch to effectively prioritize reviews of pipeline systems  For example, because the risk assessment has not changed since 2014, information on threat may be outdated  Additionally, sources of data and underlying assumption and judgments regarding certain threat and vulnerability inputs to the assessment are not fully documented  For example, threats to cybersecurity are not specifically accounted for in the description of the risk assessment methodology, making it unclear if cybersecurity is part of the assessment’s threat factor  Further, the risk assessment does not include information that is consistent with the NIPP and other DHS priorities for critical infrastructure risk mitigation, such as information on natural hazards and the ability to measure risk reduction (feedback data)  According to Pipeline Security Branch officials, the risk ranking assessment is not intended to be a fully developed risk model detailing all pipeline factors influencing risk  Rather, officials said they are primarily interested in assessing risk data that impacts security  However, because TSA’s Pipeline Security Program is designed to enhance the security preparedness of the pipeline systems, incorporating additional factors that enhance security into their risk calculation would better align their efforts with PPD-21  For example, PPD-21 calls for agencies to integrate and analyze information to prioritize assets and manage risks to critical infrastructure, as well as anticipate interdependencies and cascading impacts  For a more detailed discussion of the shortfalls we identified, refer to appendix II  In addition to the shortfalls identified above, the risk assessment has not been peer reviewed since its conception in 2007  In our past work, we reported that independent, external peer reviews are a best practice in risk management and that independent expert review panels can provide objective reviews of complex issues  According to the National Research Council of the National Academies, external peer reviews should, among other things, address the structure of the assessment, the types and certainty of the data, and how the assessment is intended to be used  The National Research Council has also recommended that DHS improve its risk analyses for infrastructure protection by validating the assessments and submitting them to independent, external peer review  Other DHS components have implemented our prior recommendations to conduct peer reviews of their risk assessments  For example, in April 2013, we reported on DHS’s management of its Chemical Facility Anti- Terrorism Standards (CFATS) program and found that the approach used to assess risk did not consider all of the elements of consequence, threat, and vulnerability associated with a terrorist attack involving certain chemicals  The Infrastructure Security Compliance Division, which manages the CFATS program conducted a multiyear effort to improve their risk assessment methodology and included commissioning a peer review by the Homeland Security Studies and Analysis Institute, which resulted in multiple recommendations  As part of the implementation of some of the peer review’s recommendations, DHS conducted peer reviews and technical reviews with government organizations and facility owners and operators, and worked with Sandia National Laboratories to verify and validate the CFATS program’s revised risk assessment methodology, which was completed in January 2017  According to Pipeline Security Branch officials, they are considering updates to the risk assessment methodology including changes to the vulnerability and consequence factors  These officials said the risk assessment was previously reviewed within the past 18 months by industry experts and they consider input from several federal partners including DHS, DOT, and the Department of Defense  Officials also said they will consider input from industry experts and federal partners while working on updating the risk assessment  However, most of the proposed changes to the risk assessment methodology officials described are ones that have been deliberated since our last review in 2010  Therefore, an independent, external peer review would provide the opportunity for integration and analysis of additional outside expertise across the critical infrastructure community  While independent, external peer reviews cannot ensure the success of a risk assessment approach, they can increase the probability of success by improving the technical quality of projects and the credibility of the decision-making process  According to the National Research Council of the National Academies, independent, external peer reviews should include validation and verification to ensure that the structure of the risk assessment is both accurate and reliable  Thus, an independent, external peer review would provide better assurance that the Pipeline Security Branch can rank relative risk among pipeline systems using the most comprehensive and accurate threat, vulnerability, and consequence information  TSA has established performance measures, as well as databases to monitor pipeline security reviews and analyze their results  However, weaknesses in its performance measures and its efforts to record pipeline security review recommendations limit its ability to determine the extent that its pipeline security program has reduced pipeline sector risks  Furthermore, we identified data reliability issues in the information that TSA collects to track the status of pipeline security review recommendations, such as missing data, inconsistent data entry formats, and data entry errors  TSA has three sets of performance measures for its pipeline efforts: the Pipeline Security Plan in the 2018 Biennial National Strategy for Transportation Security (NSTS), a management measure in the DHS fiscal year 2019 congressional budget justification, and summary figures in their CSR and CFSR databases  As a result of our 2010 work, TSA established performance measures and linked them to Pipeline Security Plan goals within the Surface Security Plan of the 2018 NSTS  See table 5 below for the 2018 NSTS Pipeline Security Plan performance measures  As shown in table 6 below, DHS also included a management measure in its fiscal year 2019 congressional budget justification to track the annual number of completed pipeline security reviews  Finally, TSA Pipeline Security Branch officials said they use summary figures in the CFSR status database and the CSR goals and priorities database as performance measures  For example, these include the percentage of CFSR recommendations implemented and the average percentage compliance with the guidelines by fiscal year  We previously found that results-oriented organizations set performance goals to clearly define desired program outcomes and develop performance measures that are clearly linked to the performance goals  Performance measures should focus on whether a program has achieved measurable standards toward achieving program goals, and allow agencies to monitor and report program accomplishments on an ongoing basis  Our previous work on performance metrics identified 10 attributes of effective performance  Table 7 identifies each key attribute of effective performance measures along with its definition  We evaluated the current performance measures included in the 2018 NSTS, the DHS fiscal year 2019 congressional budget justification, the CSR goals and priorities database, and the CFSR status database related to TSA’s Pipeline Security Branch  We primarily focused on key attributes which could be applied to individual measures  These include clarity, linkage, measurable targets, objectivity, reliability, and baseline and trend data  Our prior work on performance measurement found that all performance measure attributes are not equal and failure to have a particular attribute does not necessarily indicate that there is a weakness in that area or that the measure is not useful; rather, it may indicate an opportunity for further refinement  Based on our evaluation, the TSA-identified measures do not possess attributes that we have identified as being key to successful performance measures  As a result, TSA cannot fully determine the extent to which the Pipeline Security Branch has achieved desired outcomes, including the effectiveness of its efforts to reduce risks to pipelines  Specifically, many of TSA’s measures cover agency goals and mission, but they generally lack clarity and measurable targets, provide significantly overlapping information, and do not include baseline and trend data  Clarity  The pipeline-related measures in the 2018 NSTS are not clear because they do not describe the methodology used to calculate them, and the names and definitions are not clearly described  For example, NSTS goal 1 includes an objective to conduct training of employees responding to terrorist attacks  The desired outcome is to improve the capability of industry employees to respond and recover from terrorist attacks  However, the performance measure is the percentage of critical pipeline systems implementing the TSA Pipeline Security Guidelines  It is not clear if this measure is specific to the sections of the guidelines related to employee training or overall implementation of the guidelines  The CFSR status database measures include the percentage of recommendations implemented by topic, such as “Site Specific Security Measures,” “Signage,” or “Miscellaneous ” However, the database does not specifically define these topics or explain the methodology for calculating the measures  Unclear measures could be confusing and misleading to users  Core program activities  The pipeline-related measures in the 2018 NSTS cover some of the agency’s core program activities, such as conducting security exercises with the pipeline industry and providing intelligence and information products to the industry  However, the NSTS Pipeline Security Plan measures do not specifically include some core program activities, such as updating the TSA Pipeline Security Guidelines or the results of conducting CSRs and CFSRs in order to collect the information necessary for the existing performance measures  The CSR goals and priorities database and the CFSR status database include measures intended to track some of the results of pipeline security reviews, such as the average percentage compliance with the guidelines by fiscal year and the percentage of CFSR recommendations implemented  If core program activities are not covered, there may not be enough information available in those areas to managers and stakeholders  Limited overlap  The pipeline-related measures in the 2018 NSTS do not have limited overlap  As discussed previously, four of the five NSTS measures are based on the percentage of critical pipeline systems implementing TSA’s Pipeline Security Guidelines  The management measure is based on the number of complete pipeline security reviews  The CFSR status database measures are based on the percentage of recommendations implemented overall and by groups  Finally, the CSR goals and priorities database measures are based on the average compliance percentage of companies that had CSRs conducted in fiscal years 2016 and 2017  This is similar to four of the five NSTS measures  Significantly overlapping measures may lead to redundant, costly information that does not add value for TSA management  Linkage  The pipeline-related measures in the 2018 NSTS generally exhibited this key attribute  For example, all of the NSTS measures were arranged by agency strategic goals and risk-based priorities  However, the management measure in DHS’s fiscal year 2019 congressional budget justification and the CFSR status database measures did not specify the TSA goals and priorities to which they were aligned  If measures are not aligned with division and agency- wide goals and mission, the behaviors and incentives created by these measures do not support achieving those goals or mission  Measurable target  TSA’s measures generally did not include measurable targets in the form of a numerical goal and none of the NSTS measures had measurable targets  For example, the NSTS measure under the Security Planning priority, which tracks the percentage of critical pipeline systems implementing TSA’s Pipeline Security Guidelines, does not state what specific percentages would be considered an improvement in industry security plans  However, the management measure did include target numbers of pipeline security reviews by fiscal year  Both the CFSR status database measures and CSR goals and priorities database measures did not include measurable targets  Without measurable targets, TSA cannot tell if performance is meeting expectations  Objectivity  Because the pipeline-related measures in the 2018 NSTS, the CFSR status database, and the CSR goals and priorities database generally lack clarity and measurable targets, TSA cannot ensure its measures are free from bias or manipulation, and therefore, are not objective  If measures are not objective, the results of performance assessments may be systematically overstated or understated  Reliability  Because the pipeline-related measures in the 2018 NSTS, the CFSR status database, and the CSR goals and priorities database generally lack clarity, measurable targets, and baseline and trend data, it is not clear if TSA’s measures produce the same result under similar conditions; therefore, the pipeline-related measures are unreliable  If measures are not reliable, reported performance data may be inconsistent and add uncertainty  Baseline and trend data  TSA’s measures generally did not include baseline and trend data  For example, none of the NSTS measures included past results and compared them to measurable targets  TSA officials were unable to identify measures or goals to assess the extent to which pipeline operators have fully implemented the guidelines or increased pipeline security, but did say developing a feedback mechanism to measure progress in closing vulnerability gaps was important  However, the management measure did include the number of completed pipeline security reviews for each fiscal year from 2014 through 2017, as well as numerical goals  The CFSR status database includes information on CFSRs conducted from May 22, 2012, through June 29, 2017, but the measures are calculated for the entire time period rather than year-by-year  The CSR goals and priorities database measures include percentage compliance with the guidelines for CSRs conducted in fiscal years 2016 and 2017, as well as a combined measure  However, baseline and trend data are not tracked or reported in either database  Collecting, tracking, developing, and reporting baseline and trend data allows agencies to better evaluate progress being made and whether or not goals are being achieved  Pipeline Security Branch officials explained that in addition to the measures reported in the 2018 NSTS Pipeline Security Plan, they primarily rely on measures assessing CSR and CFSR implementation for assessing the value of its pipeline security program  TSA officials reported that they collect and analyze data and information collected from CSRs and CFSRs to, among other things, determine strengths and weaknesses at critical pipeline facilities, areas to target for risk reduction strategies, and pipeline industry implementation of the voluntary Pipeline Security Guidelines  For example, TSA officials reported that they analyzed information from approximately 734 CFSR recommendations that were made during fiscal years 2012 through 2016  They found that pipeline operators had made the strongest improvements in security training, public awareness outreach and law enforcement coordination, and site specific security measures  The most common areas in need of improvement were 24x7 monitoring, frequency of security vulnerability assessments, and proper signage  However, as described above, we found those measures also did not comport with key attributes for successful measures and we report below on reliability concerns for underlying data supporting those measures  In addition, while the Pipeline Security Branch may not rely on the measures included in the 2018 NSTS Pipeline Security Plan and the fiscal year 2019 congressional budget justification, they are important for reporting the status of pipeline security efforts to TSA as a whole and to external stakeholders such as Congress  Taking steps to ensure that the pipeline security program performance measures exhibit key attributes of successful performance measures could allow TSA to better assess the program’s effectiveness at reducing pipeline physical and cybersecurity risks  This could include steps such as modifying its suite of measures so they are clear, have measurable targets, and add baseline and trend data  Further examples include the following: Adding measurable targets consisting of numerical goals could allow TSA to better determine if the pipeline security program is meeting expectations  For example, measurable targets could be added to TSA’s existing measures by developing annual goals for the percentage of recommendations implemented to the CFSR status database and then reporting annual results  To make measures clearer, TSA could verify that each measure has a clearly stated name, definition, and methodology for how the measure is calculated  For example, the NSTS objective for security training mentioned above could have more specific language explaining how the measure is calculated and whether it applies to pipeline operators’ implementation of the training-related portions of the TSA Pipeline Security Guidelines or overall implementation  Finally, adding baseline and trend data could allow TSA to identify, monitor, and report changes in performance and help ensure that performance is viewed in context  For example, the NSTS measures, CFSR status database measures, and CSR goals and priorities database measures could have annual results from prior years  This could help TSA and external stakeholders evaluate the effectiveness of the pipeline security program and whether it is making progress toward its goals  According to TSA officials, the primary means for assessing the effectiveness of the agency’s efforts to reduce pipeline security risks is through conducting pipeline security reviews— Corporate Security Reviews (CSRs) and Critical Facility Security Reviews (CFSRs)  However, TSA has not tracked the status of CSR recommendations for over 5 years and related security review data are not sufficiently reliable  When conducting CSRs and CFSRs, TSA staff makes recommendations to operators, if appropriate  For example, a CSR recommendation might include a suggestion to conduct annual security-related drills and exercises, and a CFSR recommendation might include a suggestion to install barbed wire on the main gate of a pipeline facility  In response to recommendations that we made in our 2010 report, TSA developed three databases to track CSR and CFSR recommendations and their implementation status by pipeline facility, system, operator, and product type  In addition, the agency recently developed a fourth database to collect and analyze information gathered from pipeline operators’ responses to CSR questions  TSA officials reported that they use this database to assess the extent that TSA’s pipeline security program has met NSTS goals and Pipeline Security Branch priorities  TSA officials stated that they use the CSR goals and priorities database for follow-up on recommendations, indications of improvement in pipeline security, and as an input into TSA performance goals and measures, including the performance measures for the 2018 NSTS Pipeline Security Plan  We found several problems with the databases that indicate that the pipeline security program data are not sufficiently reliable and do not provide quality information that is current, complete, and accurate  First, the CSR recommendations database only included information for reviews conducted from November 2010 through February 2013  TSA officials stated that the agency stopped capturing CSR recommendations and status information in 2014  A TSA official stated that one factor was that the pipeline staffing level was one FTE in fiscal year 2014  However, the Pipeline Security Branch did not resume entering CSR recommendation-related information when staffing levels rose to 6 FTEs in the following year and beyond  As a result, TSA is missing over 5 years of data for the recommendations it made to pipeline operators when conducting CSRs  The agency collected some information from CSRs conducted in fiscal years 2016 and 2017 in the separate CSR goals and priorities database  However, this database does not include all of the information that TSA collects when conducting CSRs  Specifically, the CSR goals and priorities database does not state which companies were reviewed, what specific recommendations were made, or the current status of those recommendations, and only records operators’ responses to 79 of the 222 CSR questions  Second, our review identified instances of missing data, inconsistent data entry formats, and data entry errors in the four databases  For example: The CSR recommendations database had missing data in all 13 of the included fields and a data entry error shifted 50 observations into the wrong fields, impacting both the Status Date and Completion Code fields  The CSR goals and priorities database had seven entries with inconsistent data formatting and five of those entries were not taken into account when calculating summary figures  The CFSR recommendations database had missing data in 3 of 9 fields  There was also inconsistent data entry formats in 3 fields  The CFSR status database had missing data in 7 of 29 fields and inconsistent data entry formats in 4 fields  Finally, TSA has not documented its data entry and verification procedures, such as in a data dictionary or user manual, and does not have electronic safeguards for out-of-range or inconsistent entries for any of the databases it uses to track the status of CSR or CFSR recommendations and analyze operator responses to the CSR  TSA Pipeline Security Branch officials told us that they had not documented data entry and verification procedures and did not have electronic safeguards  This was for two reasons  First, the officials stated that the databases are small and maintained in a commercial spreadsheet program that does not allow for electronic safeguards  However, based on our review of the databases, the spreadsheet program does allow for a variety of electronic safeguards  For example, entries can be restricted to only allow selections from a drop-down list or only allow dates to be entered  Second, only a small number of TSA employees enter information into these databases  TSA officials explained that typically one TSA employee is responsible for entering information from pipeline security reviews, and another individual, usually whoever conducted the review, is tasked to verify the accuracy of the data entered  As a result, according to the officials, any errors would be self-evident and caught during these TSA employees’ reviews  Our work has emphasized the importance of quality information for management to make informed decisions and evaluate agencies’ performance in achieving key objectives and addressing risks  The Standards for Internal Control in the Federal Government states that management should use quality information to achieve agency objectives, where “quality” means, among other characteristics, current, complete, and accurate  In addition, DHS’s Information Quality Guidelines state that all DHS component agencies should treat information quality as integral to every step of the development of information, including creation, collection, maintenance, and dissemination  The guidelines also state that agencies should substantiate the quality of the information disseminated through documentation or other appropriate means  Without current, complete, and accurate information, it is difficult for TSA to evaluate the performance of the pipeline security program  Until TSA monitors and records the status of these reviews’ recommendations, it will be hindered in its efforts to determine whether its recommendations are leading to significant reduction in risk  By entering information on CSR recommendations and monitoring and recording their status, developing written documentation of its data entry and verification procedures and electronic safeguards, and improving the quality of its pipeline security program data, TSA could better ensure it has the information necessary to effectively monitor pipeline operators’ progress in improving their security posture, and evaluate its pipeline security program’s effectiveness in reducing security risks to pipelines  A successful pipeline attack could have dire consequences on public health and safety, as well as the U S  economy  Recent coordinated campaigns by environmental activists to disrupt pipeline operations, and the successful attempts by nation-state actors to infiltrate and obtain sensitive information from pipeline operators’ business and operating systems, demonstrate the dynamic and continuous threat to the security of our nation’s pipeline network  To help ensure the safety of our pipelines throughout the nation, it is important for TSA to address weaknesses in the management of its pipeline security program  TSA’s Pipeline Security Branch revised its security guidelines in March 2018 to, among other things, reflect the dynamic threat environment and incorporate NIST’s Cybersecurity Framework cybersecurity principles and practices  However, without a documented process defining how frequently TSA is to review and, if deemed necessary, revise its guidelines, TSA cannot ensure that its guidelines reflect the latest known standards and best practices for physical and cybersecurity, or address the persistent and dynamic security threat environment currently facing the nation’s pipeline system  Further, without clearly defined criteria for determining pipeline facilities’ criticality, TSA cannot ensure that pipeline operators are applying guidance uniformly and that all of the critical facilities across the pipeline sector have been identified; or that their vulnerabilities have been identified and addressed  TSA could improve its ability to conduct pipeline security reviews and the means that it uses to prioritize which pipeline systems to review based on their relative risk ranking  Establishing a strategic workforce plan could help TSA ensure that it has identified the necessary skills, competencies, and staffing allocations that the Pipeline Security Branch needs to carry out its responsibilities, including conducting security reviews of critical pipeline companies and facilities, as well as their cybersecurity posture  Better considering threat, vulnerability, and consequence elements in its risk assessment and incorporating an independent, external peer review in its process would provide more assurance that the Pipeline Security Branch ranks relative risk among pipeline systems using comprehensive and accurate data and methods  We are making 10 recommendations to TSA: The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to implement a documented process for reviewing, and if deemed necessary, for revising TSA’s Pipeline Security Guidelines at regular defined intervals  (Recommendation 1) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to clarify TSA’s Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities  (Recommendation 2) The TSA Administrator should develop a strategic workforce plan for its Security Policy and Industry Engagement’s Surface Division, which could include determining the number of personnel necessary to meet the goals set for its Pipeline Security Branch, as well as the knowledge, skills, and abilities, including cybersecurity, that are needed to effectively conduct CSRs and CFSRs  (Recommendation 3) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to update the Pipeline Relative Risk Ranking Tool to include up-to-date data to ensure it reflects industry conditions, including throughput and threat data  (Recommendation 4) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to fully document the data sources, underlying assumptions and judgments that form the basis of the Pipeline Relative Risk Ranking Tool, including sources of uncertainty and any implications for interpreting the results from the assessment  (Recommendation 5) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to identify or develop other data sources relevant to threat, vulnerability, and consequence consistent with the NIPP and DHS critical infrastructure risk mitigation priorities and incorporate that data into the Pipeline Relative Risk Ranking Tool to assess relative risk of critical pipeline systems, which could include data on prior attacks, natural hazards, feedback data on pipeline system performance, physical pipeline condition, and cross-sector interdependencies  (Recommendation 6) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to take steps to coordinate an independent, external peer review of its Pipeline Relative Risk Ranking Tool, after the Pipeline Security Branch completes enhancements to its risk assessment approach  (Recommendation 7) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to ensure that it has a suite of performance measures which exhibit key attributes of successful performance measures, including measurable targets, clarity, and baseline and trend data  (Recommendation 8) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to take steps to enter information on CSR recommendations and monitor and record their status  (Recommendation 9) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to improve the quality of its pipeline security program data by developing written documentation of its data entry and verification procedures, implementing standardized data entry formats, and correcting existing data entry errors  (Recommendation 10) We provided a draft of this report to DHS, DOE, DOT, and FERC  DHS provided written comments which are reproduced in appendix III  In its comments, DHS concurred with our recommendations and described actions planned to address them  DHS, DOE, DOT, FERC, also provided technical comments, which we incorporated as appropriate  We also provided draft excerpts of this product to the American Petroleum Institute (API), the Association of Oil Pipe Lines, the American Gas Association (AGA), the Interstate Natural Gas Association of America (INGAA), the American Public Gas Association, and the selected pipeline operators that we interviewed  For those who provided technical comments, we incorporated them as appropriate  With regard to our first recommendation, that TSA implement a documented process for reviewing, and if deemed necessary, for revising its Pipeline Security Guidelines at regular defined intervals, DHS stated that TSA will implement a documented process for reviewing and revising its Pipeline Security Guidelines at regular defined intervals, as appropriate  DHS estimated that this effort would be completed by March 31, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our second recommendation, that TSA clarify its Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities, DHS stated that TSA will clarify its Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities  DHS estimated that this effort would be completed by May 31, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our third recommendation, that TSA develop a strategic workforce plan for its Security Policy and Industry Engagement's Surface Division, DHS stated that TSA will develop a strategic workforce plan for the division, which includes determining the number of personnel necessary to meet the goals set for the Pipeline Security Branch, as well as the knowledge, skills, and abilities, including cybersecurity, that are needed to effectively conduct CSRs and CFSRs  DHS estimated that this effort would be completed by June 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our fourth recommendation, that TSA update the Pipeline Relative Risk Ranking Tool to include up-to-date data in order to ensure it reflects industry conditions, including throughput and threat data, DHS stated that TSA will update the Pipeline Relative Risk Ranking Tool to include up-to-date data in order to ensure it reflects industry conditions, including throughput and threat data  DHS estimated that this effort would be completed by February 28, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our fifth recommendation, that TSA fully document the data sources, underlying assumptions, and judgements that form the basis of the Pipeline Relative Risk Ranking Tool, including sources of uncertainty and any implications for interpreting the results from the assessment, DHS stated that TSA will fully document the data sources, underlying assumptions, and judgements that form the basis of the Pipeline Relative Risk Ranking Tool  According to DHS, this will include sources of uncertainty and any implications for interpreting the results from the assessment  DHS estimated that this effort would be completed by February 28, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our sixth recommendation, that TSA identify or develop other data sources relevant to threat, vulnerability, and consequence consistent with the NIPP and DHS critical infrastructure risk mitigation priorities and incorporate that data into the Pipeline Relative Risk Ranking Tool to assess relative risk of critical pipeline systems, DHS stated that TSA will identify and/or develop other sources relevant to threat, vulnerability, and consequence consistent with the NIPP and DHS critical infrastructure risk mitigation priorities  DHS also stated that TSA will incorporate that data into the Pipeline Risk Ranking Tool to assess relative risk of critical pipeline systems, which could include data on prior attacks, natural hazards, feedback data on pipeline system performance, physical pipeline condition, and cross-sector interdependencies  DHS estimated that this effort would be completed by June 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our seventh recommendation, that TSA take steps to coordinate an independent, external peer review of its Pipeline Relative Risk Ranking Tool, after the Pipeline Security Branch completes enhancements to its risk assessment approach, DHS stated that, after completing enhancements to its risk assessment approach, TSA will take steps to coordinate an independent, external peer review of its Pipeline Relative Risk Ranking Tool  DHS estimated that this effort would be completed by November 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our eighth recommendation, that TSA ensure that the Security Policy and Industry Engagement's Surface Division has a suite of performance measures which exhibit key attributes of successful performance measures, including measurable targets, clarity, baseline, and trend data, DHS stated that TSA’s Surface Division’s Pipeline Section will develop both physical and cyber security performance measures, in consultation with pipeline stakeholders, to ensure that it has a suite of performance measures which exhibit key attributes of successful performance measures, including measurable targets, clarity, baseline, and trend data  DHS estimated that this effort would be completed by November 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our ninth recommendation, that TSA take steps to enter information on CSR recommendations and monitor and record their status, DHS stated that TSA will enter information on CSR recommendations and monitor and record their status  DHS estimated that this effort would be completed by October 31, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our tenth recommendation, that TSA take steps to improve the quality of its pipeline security program data by developing written documentation of its data entry and verification procedures, implementing standardized data entry formats, and correcting existing data entry errors, DHS stated that TSA will develop written documentation of its data entry and verification procedures, implementing standardized data entry formats, and correcting existing data entry errors  DHS estimated that this effort would be completed by July 31, 2019  This action, if fully implemented, should address the intent of the recommendation  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until one day from the report date  At that time, we will send copies to the appropriate congressional committees; the Secretaries of Energy, Homeland Security, and Transportation; the Executive Director of the Federal Energy Regulatory Committee; and other interested parties  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact Chris Currie at (404) 679-1875 or curriec@gao gov, and Nick Marinos at (202) 512-9342 or marinosn@gao gov  Key contributors to this report are listed in appendix IV  This appendix lists security guidance and guidance-related tools that the pipeline operators and industry association officials we interviewed identified as adopted or available in order to secure their physical and cyber operations  This list should not be considered to include all physical and cybersecurity guidance that may be available or used by all pipeline operators nor do all operators use all guidance listed  The Transportation Security Administration’s (TSA) Pipeline Security Branch developed the Pipeline Relative Risk Ranking Tool (risk assessment) in 2007  The risk assessment calculates threat, vulnerability, and consequence on variables such as the amount of throughput in the pipeline system (consequence input)  Pipeline Security Branch officials told us that they use the pipeline risk assessment to rank relative risk of the top 100 critical pipeline systems, and the standard operating procedures for conducting Corporate Security Reviews (CSR) state the results of the risk ranking are the primary factor considered when prioritizing CSRs of pipeline companies  However, we identified several factors that likely limit the usefulness of the current assessment in calculating threat, vulnerability, and consequence to allow the Pipeline Security Branch to effectively prioritize reviews of pipeline systems  For example, because the risk assessment has not changed since 2014, information on threat may be outdated  Additionally, sources of data and underlying assumption and judgments regarding certain threat and vulnerability inputs to the assessment are not fully documented  For example, threats to cybersecurity are not specifically accounted for in the description of the risk assessment methodology, making it unclear if cybersecurity is part of the assessment’s threat factor  Further, the risk assessment does not include information that is consistent with the National Infrastructure Protection Plan (NIPP) and other Department of Homeland Security (DHS) priorities for critical infrastructure risk mitigation, such as information on natural hazards and the ability to measure risk reduction (feedback data)  According to Pipeline Security Branch officials, the risk ranking assessment is not intended to be a fully developed risk model detailing all pipeline factors influencing risk  Rather, officials said they are primarily interested in assessing risk data that impacts security  However, because TSA’s Pipeline Security Program is designed to enhance the security preparedness of the pipeline systems, incorporating additional factors that enhance security into their risk calculation of the most critical pipeline systems would better align their efforts with Presidential Policy Directive 21 (PPD-21)  For example, PPD-21 calls for agencies to integrate and analyze information to prioritize assets and manage risks to critical infrastructure, as well as anticipate interdependencies and cascading impacts  Below we present the various shortfalls in the risk assessment—outdated data, limited description of sources and methodology, and opportunities to better align with the NIPP and other DHS priorities for critical infrastructure risk mitigation—in the context of the components that comprise a risk assessment: threat, vulnerability, and consequence  Whereas in 2010 we made recommendations to improve the consequence component in the pipeline relative risk ranking tool, we have currently identified shortfalls that cut across all risk components: threat, vulnerability, and consequence  We identified several shortfalls in the pipeline risk assessment’s calculation of threat  First, while the risk assessment assesses consequence and vulnerability by pipeline system through use of multiple variables, it currently ranks threat for pipeline systems equally  Second, the evolving nature of threats to pipelines may not be reflected, since the risk assessment was last updated in 2014  Third, the threat calculation does not take into account natural hazards  Pipeline Security Branch officials said they currently rank threat equally across pipeline systems because they do not have granular enough threat information to distinguish threat by pipeline  However, ranking threat equally effectively has no effect on the risk calculation for pipeline systems  Further, this judgment is not documented in the risk assessment’s methodology  According to the NIPP, a risk assessment’s methodology must clearly document what information is used and how it is synthesized to generate a risk estimate, including any assumptions and judgments  Additionally, our analysis of the pipeline risk assessment found that it includes at least one field that TSA could use to differentiate threat by pipeline  Specifically, the risk assessment includes a field that accounts for whether a pipeline experienced a previous security threat (including failed attacks), and information provided by Pipeline Security Branch suggests some pipeline systems have experienced such threats  However, the Pipeline Security Branch did not capture these events in the risk assessment’s calculation, which Pipeline Security Branch officials said should be part of the threat calculation, but could not account for why they were not calculated for the systems in the risk assessment  These officials also clarified that incidents such as suspicious photography or vandalism do not constitute an attack to be accounted for in the threat calculation  Documenting such assumptions, judgments, or decisions to exclude information could provide increased transparency to those expected to interpret or use the results  Pipeline Security Branch officials also said that they ranked threat equally because TSA Intelligence and Analysis data show that threats to the oil and natural gas sector have been historically low, and Intelligence and Analysis does not conduct specific threat analysis against individual pipeline systems  However, the Pipeline Security Branch has not updated the risk assessment since June 2014; therefore, the threat information it used to determine threat calculations—and decide to rank threat equally—may be outdated and not reflect the threats to the industry that have emerged in recent years  In fact, pipeline operators we interviewed indicated that the types of threats that concern pipeline operators have evolved  For example, 5 of the 10 operators we interviewed indicated that environmental activists were an increased threat to the pipeline industry because they use sabotage techniques, such as valve turning and cutting in service pipelines with blow torches, against pipelines  Additionally, 6 of 10 pipeline operators we interviewed said cyber attacks from nation-state actors were a primary threat to their industry  Further, when TSA issued its revised Pipeline Security Guidelines in March 2018, it stated that its revisions to the guidelines were made to reflect the ever-changing threat environment in both the physical and cybersecurity realms  However, threats to cybersecurity are not specifically accounted for in the description of the risk assessment methodology  Recent Pipeline Modal and Cyber Modal Threat Assessments include cyber threats to the pipeline industry, but the description of the pipeline risk assessment’s methodology does not specify what types of threat assessments (sources) are used to calculate its threat score  To better align with the guidance in the NIPP for documenting sources of information when conducting risk assessments, the Pipeline Security Branch should document the information used  Keeping the risk assessment updated with current information, as well as documenting those data sources, could help the Pipeline Security Branch ensure it is using its limited resources to review the pipeline systems with greater risk  Natural Hazard Threats to Pipelines The Transportation Systems Sector, of which pipelines are a part, is critical to the Pacific Northwest, but also at risk from natural hazards, like earthquakes  For example, according to the Department of Homeland Security, an earthquake in the Puget Sound region—which relies on the transportation of crude oil from Alaska—could cripple the ports of Seattle and Tacoma, as well as the Olympic and Williams Pipelines greatly impacting the Pacific Northwest Economic Region  Hurricanes are the most frequent disruptive natural hazard for the oil and natural gas subsector and can cause the shutdown of facilities in an area, even when the facilities themselves are not directly affected by the storms  For example, according to the U S  Energy Information Administration, the flow of petroleum into the New York area via pipeline from the Gulf Coast relies on the ability to move it through major terminals  In August 2017, Hurricane Harvey caused major disruptions to crude oil and petroleum product supply chains, including those to New York Harbor from Houston, Texas via the Colonial Pipeline  Due to the hurricane, decreased supplies of petroleum products available for the pipeline in Houston forced Colonial Pipeline to limit operations temporarily  Finally, another shortfall in the current pipeline risk assessment methodology is that it does not account for natural hazards in its threat calculation, even though DHS’s definition of threat includes natural hazards, and security and resilience of critical infrastructure are often presented in the context of natural hazards  According to the NIPP, threat is a natural or manmade occurrence, individual, entity, or action that has or indicates the potential to harm life, information, operations, the environment, and/or property  As such, along with terrorism, criminal activity and cybersecurity, natural disasters are a key element of DHS’s critical infrastructure security and resilience mission  According to Pipeline Security Branch officials, there is not sufficient historical data available that would indicate a significant impact from natural disasters on specific pipeline systems  However, we identified possible sources of data for the Pipeline Security Branch to consider  For example, a 2016 RAND Corporation study examined national infrastructure systems’ exposure to natural hazards, including pipelines  Additionally, the Federal Emergency Management Agency (FEMA) has collaborated with stakeholders to develop the National Risk Index to, among other things, establish a baseline of natural hazards risk for the United States While there may not be historical data of natural hazard impact for every pipeline system, consulting other sources or experts could provide regional data or analysis to build a more comprehensive threat picture to help distinguish threats by pipeline system  According to the NIPP, hazard assessments should rely not only on historical information, but also future predictions about natural hazards to assess the likelihood or frequency of various hazards  We also identified multiple shortfalls in the vulnerability factors used in the risk assessment methodology, such as the potential uncertainty of the number of critical facilities and incorporating a feedback mechanism to calculate overall risk reduction  Other considerations for vulnerability calculations include physical condition of the pipeline system, cybersecurity activities, and interdependencies among sectors  The number of critical facilities a pipeline system has identified is used as an input for its vulnerability calculation in the Pipeline Security Branch’s risk assessment methodology  As discussed earlier, we identified deficiencies in TSA’s criteria for identifying critical facilities, and found that well-defined criteria and consistent application of the criteria for identifying critical facilities could improve the results of the Pipeline Security Branch’s risk assessment  Nevertheless, communicating in the risk assessment the uncertainty that may be inherent in this self-reported information would better align the risk assessment with the NIPP  Measuring Effectiveness in a Voluntary Environment According to the National Infrastructure Protection Plan, the use of performance metrics is an important step in the critical infrastructure risk management process to enable assessment of improvements in critical infrastructure security and resilience  The metrics provide a basis for the critical infrastructure community to establish accountability, document actual performance, promote effective management, and provide a feedback mechanism to inform decision making  By using metrics to evaluate the effectiveness of voluntary partnership efforts to achieve national and sector priorities, critical infrastructure partners can adjust and adapt their security and resilience approaches to account for progress achieved, as well as changes in the threat and other relevant environments  Metrics are used to focus attention on areas of security and resilience that warrant additional resources or other changes through an analysis of challenges and priorities at the national, sector, and owner/operator levels  Metrics also serve as a feedback mechanism for other aspects of the critical infrastructure risk management approach  Another shortfall in the risk assessment is its inability to reliably measure the progress a pipeline system made in addressing vulnerability gaps between security reviews  The current risk assessment includes a CSR score as part of its vulnerability calculation, which was developed in part in response to our 2010 recommendation to use more reliable data to measure a pipeline system’s vulnerability gap  However, during our review, Pipeline Security Branch officials said they plan to remove pipeline companies’ CSR scores from the risk assessment calculations, because they and industry partners do not have confidence that the score appropriately measures a pipeline system’s vulnerability  For example, Pipeline Security Branch officials explained that pipeline companies consider security factors differently, which can lead to variation in implementing risk reduction activities and by extension lead to different CSR scores  However, removing the CSR score eliminates the only feedback mechanism in the risk assessment from a pipeline company’s actual security review conducted by the Pipeline Security Branch  The NIPP and DHS’s Risk Management fundamentals emphasize the important role that such feedback mechanisms play in risk management  Officials from the Pipeline Security Branch agree on the importance of a feedback mechanism tying results of reviews to a revised vulnerability metric, but said they need a better measure than the current CSR score which is unreliable for comparative and analytic purposes  Developing a feedback mechanism based on implementation of TSA’s Pipeline Security Guidelines could be an important input to the risk assessment’s vulnerability calculation  This information would also inform the amount of risk pipeline companies are reducing by implementing the guidelines and could be used to inform overall risk reduction  The physical and cyber environments in which the pipeline sector operates also present vulnerabilities not accounted for in the pipeline risk assessment  In recent years, DHS has listed the potential for catastrophic losses to dramatically increase the overall risk associated with failing infrastructure and highlighted risks due to climate change and natural hazards to pipelines  For example, DHS reported extreme temperatures—such as higher and lower temperatures over prolonged periods of time—increase vulnerability to the critical infrastructure by causing elements to break and cease to function  Pipelines that freeze and then rupture can affect the energy and transportation systems sectors  As noted above, according to the NIPP, a natural or man-made occurrence or action with the potential to harm life is considered a threat, whereas vulnerability is defined as a physical feature or operational attribute that renders an entity open to exploitation or susceptible to a given threat or hazard  While pipeline physical condition is typically thought of in context of safety, pipeline condition or location (such as above or below ground) could touch upon pipeline security as it relates to system vulnerability  For example, a pipeline system or segment of a system with a compromised physical condition due to corrosion or age could affect the system’s vulnerability to threats and affect its ability to recover from such threats by potentially increasing the time a system is offline  According to the Transportation Systems Sector-Specific Plan, vulnerabilities to damage in aging transportation infrastructure—of which pipelines are a part—are projected to increase with the continued effects of climate change  Further, according to TSA’s Pipeline Security and Incident Recovery Protocol Plan, pipeline integrity efforts—including the design, construction, operation, and maintenance of pipelines—are important to pipeline security because well-maintained, safe pipelines are more likely to tolerate a physical attack  The Pipeline Security Branch already collects information from the Pipeline and Hazardous Materials Safety Administration (PHMSA) for its risk assessment, specifically information on High Consequence Area and High Threat Urban Area mileage  By considering additional information PHMSA collects on pipeline integrity, the Pipeline Security Branch could also use the information to help pipeline operators identify security measures to help reduce the consequences related to the comparatively higher vulnerability of an aging or compromised system  This would align with the Pipeline Security Branch’s efforts to improve security preparedness of pipeline systems and could better inform its vulnerability calculations for relative risk ranking of pipeline systems  Capturing cybersecurity in the risk assessment is also an area for improvement  Pipeline Security Branch officials told us they consulted with the National Cybersecurity and Communications Integration Center to revise TSA’s Pipeline Security Guidelines to align with the National Institute of Standards and Technology (NIST) Cybersecurity Framework and that absent data specific to pipelines on their cybersecurity vulnerabilities, they are unable to include a pipelines’ vulnerability to cyber attack in the risk assessment  However, the Pipeline Security Branch recently updated the security review questions asked of pipeline operators during corporate and critical facility reviews based on the recently updated Pipeline Security Guidelines  Using these updated questions related to companies’ cybersecurity posture, the Pipeline Security Branch could collect additional information on cybersecurity vulnerabilities which could inform the risk assessment  This could be an element of the feedback mechanism described above and emphasized in the NIPP  Additionally, NIST identified several supply chain vulnerabilities associated with cybersecurity, which are not currently accounted for in TSA’s Pipeline Security Guidelines  As pipeline operators implement increasing levels of network technologies to control their systems, the Pipeline Security Branch may not be fully accounting for pipeline systems’ cybersecurity posture by not including the cybersecurity-related vulnerabilities in its risk assessment inputs  Finally, we identified shortfalls in cross-sector interdependencies, which could affect vulnerability calculations  According to the NIPP, understanding and addressing risks from cross-sector dependencies and interdependencies is essential to enhancing critical infrastructure security and resilience  The Pipeline Security Branch’s pipeline risk assessment currently considers the effects of a pipeline system’s ability to service assets such as major airports, the electric grid, and military bases  However, consequence is calculated on the loss or disruption of the pipeline system to these other assets and does not capture the dependency of the pipeline system on other energy sources, such as electricity  Weather events such as Gulf of Mexico hurricanes and Superstorm Sandy highlighted the interdependencies between the pipeline and electrical sectors  Specifically, according to a 2015 DHS annual report on critical infrastructure, power failures during Superstorm Sandy in 2012 closed major pipelines for 4 days, reducing regional oil supplies by 35 to 40 percent  The report goes on to say that the interconnected nature of infrastructure systems can lead to cascading impacts and are increasing in frequency  Pipeline Security Branch officials are considering cross-sector interdependencies and said they discuss these factors with operators as they relate to system resiliency  Considering interdependencies of sectors in both directions—such as calculating the likelihood that an input like electricity could fail and cause disruptions to critical pipelines—could improve the calculations in the pipeline risk assessment  As previously discussed, the Pipeline Security Branch last calculated relative risk among the top 100 pipeline systems in 2014  When doing so, it used pipeline systems’ throughput data from 2010 to assess relative risk  According to Pipeline Security Branch officials, the amount of throughput in pipeline systems does not change substantially year to year  However, Standards for Internal Control in the Federal Government calls for management to use quality information to achieve the entity’s objectives, including using relevant data from reliable sources obtained in a timely manner  The Pipeline Security Branch uses throughput data as a consequence factor in the risk assessment to determine a pipeline system’s relative risk score  Throughput changes could affect relative risk ranking and the Pipeline Security Branch’s ability to accurately prioritize reviews based on relative risk  Chris P  Currie at (404) 679-1875 or curriec@gao gov Nick Marinos at (202) 512-9342 or marinosn@gao gov  In addition to the contacts named above, Ben Atwater, Assistant Director; Michael W  Gilmore, Assistant Director; and Michael C  Lenington, Analyst-in-Charge, managed this assignment  Chuck Bausell, David Blanding, Dominick Dale, Eric Hauswirth, Kenneth A  Johnson, Steve Komadina, Susanna Kuebler, Thomas Lombardi, David Plocher, and Janay Sam made significant contributions to this report 

ground_truth: Congress created several authorities that provide DOD research labs with ways to increase efficiency and foster innovation. Senate report 114-255 contained a provision for GAO to study governance models used by federal labs. This report evaluates DOD labs' use of authorities to foster innovation and efficiency. GAO selected four authorities that recent work on best practices for science and technology management and expedited defense lab hiring have shown to be the most crucial for supporting innovation; administered a survey to 44 lab directors to gain insight into their use of the authorities; interviewed key lab officials and contractors; and reviewed relevant policies and guidance. Congress has provided the Department of Defense's (DOD) research labs with several authorities to enhance management and operations. Four authorities that GAO examined provide lab directors with greater ability to make their own decisions regarding the funding of projects, hiring, lab management, and purchasing of equipment or services. 1. Laboratory initiated research authority. This authority, as implemented, provides labs with a means to fund new science and technology projects that they consider a priority. Labs may use a percentage of all funds available to the lab and are permitted to charge customers of the lab a percentage fee of the costs for activities performed by the lab for the customer. 2. Direct hire authority. This authority enables labs to compete with private industry for high-quality talent. For example, it provides for streamlined hiring of applicants with relevant advanced degrees, or students enrolled in science, technology, engineering, and mathematics programs. 3. Laboratory enhancement pilot program authority. This authority generally allows lab directors to propose alternative methods that might lead to more effective lab management, and waive certain policies or procedures that might affect implementation of these methods. 4. Micro-purchase authority. This authority raises the threshold for small purchases for DOD research lab activities from $3,500 to $10,000 to facilitate acquisitions. While labs have used these authorities, their use has sometimes been limited, particularly with the laboratory initiated research authority. DOD lab directors at Air Force, Navy, and Army cited several obstacles that impede wider use of that authority, specifically: Air Force: Financial management officials at the Air Force stated that the service's accounting system does not currently have an automated capability to transfer the allowable percentage fee of costs to a central account at the Air Force Research Laboratory. This lack of capability, officials noted, creates a significant administrative burden related to charging these fees. Navy: In fiscal year 2017, Navy labs invested $7.3 million in lab infrastructure projects, compared to $32.9 million and $53.7 million at the Air Force and Army, respectively. Navy lab officials told us that they were restricted in their use of infrastructure funds available under the laboratory initiated research authority due to a lack of clear guidance as to whether and how to use this authority within the Capital Investment Program of the Navy Working Capital Fund. Army: The Army requires its labs to use a similar percentage of funds from two sources: (1) what it refers to as directly appropriated funds and (2) funds labs charge for customer activities. Some Army lab directors reported assessing a lower rate on customer funds than allowed so as not to drive customers away. The labs then generally charge a lower than desired rate on their directly appropriated funds, which further constrains the total funding available to them. GAO is making three recommendations to enhance DOD's use of laboratory initiated research authority, including that the Air Force assess potential accounting system improvements, the Navy clarify how labs can use the authority for infrastructure improvements, and the Army assess its policy to determine whether changes are needed to remove disincentives for labs to use the authority. DOD concurred with the recommendations.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: VA comprises a Veterans Affairs Central Office (VACO) and over 1,000 facilities and offices throughout the nation, as well as the U S  territories and the Philippines  As shown in figure 1, VA’s three major administrations are the Veterans Health Administration (VHA), Veterans Benefits Administration (VBA), and National Cemetery Administration (NCA)  The largest of the administrations, in terms of workforce, is VHA and its associated Veterans Integrated Service Networks (VISN)  VHA is estimated to employ about 316,800 employees in 2017, followed by the VBA and NCA with about 22,700 and 1,850 employees, respectively  The remaining 15,000 employees are in various staff offices  VA’s budget request for fiscal year 2018 of $186 4 billion includes $82 1 billion in discretionary resources and $104 3 billion in mandatory funding  The following offices are involved in addressing misconduct at VA  Office of Human Resource Management (OHRM): OHRM develops policies with regard to performance management and assesses the effectiveness of department-wide human-resource programs and policies  Office of Accountability Review (OAR): OAR was established in 2014 within VA’s Office of General Counsel and was intended to ensure leadership accountability for improprieties related to patient scheduling and access to care, whistle-blower retaliation, and related disciplinary matters that affect public trust in VA  Office of Inspector General (OIG): The VA OIG provides oversight through independent audits, inspections, and investigations to prevent and detect criminal activity, waste, abuse, and mismanagement in VA programs and operations  Office of Accountability and Whistleblower Protection (OAWP): As required by the Department of Veterans Affairs Accountability and Whistleblower Protection Act of 2017, OAWP will take on the responsibility of, among other things, receiving whistle-blower complaints  Corporate Senior Executive Management Office (CSEMO): CSEMO supports the entire life-cycle management of VA’s senior executives by developing policy and providing corporate-level personnel services, such as training and coaching to VA’s senior executive workforce  Client Services Response Team (CSRT): CSRT serves to centralize and streamline internal processes to improve VHA’s overall responsiveness to concerns of veterans, employees, and other internal and external stakeholders  This office works closely with VA and VHA program offices and facilities to review, research, and respond to inquiries sent to the Office of the Under Secretary for Health, Office of the Secretary, and other concerns received via program offices within VACO, which lack a formalized response process  National Cemetery Administration (NCA): NCA honors veterans and their families with final resting places in national shrines that commemorate their service  NCA’s Office of Management oversees and administers all human-resource management, including activities associated with labor and employee relations  Office of the Medical Inspector (OMI): OMI is responsible for assessing the quality of VA health care through investigations of VA facilities nationwide, which include employee whistle-blower allegations referred to VA by the OSC; veteran complaints referred by the OIG, Congress, or other stakeholders; and site-specific internal reviews directed by the Office of the Under Secretary for Health  Office of Research Oversight (ORO): ORO promotes the responsible conduct of research, serves as the primary VHA office in advising the Office of the Under Secretary for Health on matters of research compliance, and is to provide oversight of compliance with VA and other federal requirements related to research misconduct  Office of Resolution Management (ORM): ORM provides Equal Employment Opportunity (EEO) discrimination complaint processing services to VA employees, applicants for employment, and former employees, which include counseling, investigation, and final agency procedural decisions  Office of Security and Law Enforcement (OS&LE): OS&LE develops policies, procedures, and standards that govern VA’s infrastructure law- enforcement program  The Law Enforcement Oversight and Criminal Investigations Division is responsible for conducting investigations of serious incidents of misconduct  Veterans Benefits Administration (VBA): VBA provides benefits and services to veterans, their families and survivors  VBA’s Office of Management directs and oversees nationwide human-resources activities and supports ORM in processing EEO complaints filed by employees and applicants who allege employment discrimination  The process for addressing employee misconduct involves various components within VA that are responsible for investigating and adjudicating allegations, as shown in figure 2  Receipt of Allegation: The OIG receives allegations of criminal activity and employee misconduct from VA employees, the OSC, members of Congress, the public, or other stakeholders  The allegations received by the OIG are initially routed to the OIG Hotline Division  The OIG also receives other types of allegations outside the scope of this review, such as issues pertaining to VA employee benefits and contracts  In addition to reporting allegations of employee misconduct to the OIG, VA employees may also report allegations of misconduct directly to their supervisors  Review and Referral of Allegation: Due to the substantial number of allegations received through the OIG Hotline Division, the OIG exercises a “right of first refusal” on misconduct cases, which allows it to take no further action, refer the case to program offices within VA for review and response, or open an investigation  For example, the OIG can either decide to (1) take no further action on matters not within the OIG’s jurisdiction or too vague to warrant further review; (2) refer allegations that warrant some action to the OMI, OAR, or VA facilities or program offices within each administration to conduct an independent review of the allegations; or (3) open cases for further review for serious allegations of criminal activity, fraud, waste, abuse, and mismanagement  Cases opened by the OIG typically involve misconduct by senior officials, or matters relating to the quality of care provided by licensed professionals  In contrast, the OIG typically refers allegations to VA facility or program offices for matters where the OIG does not have sufficient resources to open an internal case  The OIG generally does not review matters that are addressed in other legal or administrative forums, such as allegations of discrimination or whistle-blower retaliation  Notice to Employee Once Allegations Are Substantiated: The type of appointment an employee holds determines whether an employee is to be provided advance notice of planned disciplinary action once misconduct is substantiated at the conclusion of an investigation  Employees holding a permanent appointment are entitled to receive a notice of proposed action that states the specific charges for which the proposed disciplinary action is based and informs the employee of his or her right to review the material that is relied upon to support the reasons for the action  Employees in the competitive service serving in a permanent appointment (who have completed their probationary period) are treated differently than those who are still in their probationary period or serving under temporary appointments  An employee serving a probationary period or under a temporary appointment does not receive a notice of proposed action and may be immediately terminated because his or her work performance or conduct fails to demonstrate fitness or qualifications for continued employment  Temporary employees are terminated by notifying employees in writing as to why they are being separated and the effective date of the action  Disciplinary Action: VA Handbook 5021, Employee/Management Relations, governs policy for disciplinary and grievance procedures for all employees  Supervisory staff or appropriate higher-level officials use the results from investigations to help determine whether any disciplinary actions are warranted and, if so, the type and severity of each action  Other VA staff, such as human-resources and general-counsel staff may also provide guidance to management in determining appropriate disciplinary actions  After determining the facts in a case, VA may employ either disciplinary or adverse action  Adverse action involves a more- severe type of discipline (e g , removal, suspension more than 14 days, or reduction in grade) as described in table 1  As a federal agency, VA is required to report department-wide information on certain disciplinary personnel actions to the Office of Personnel Management (OPM)  OPM’s Enterprise Human Resources Integration (EHRI) system currently collects, integrates, and publishes data for executive-branch employees on a biweekly basis  This system provides federal workforce data to other government systems and the public  To adhere to this reporting requirement, VA provides information on certain disciplinary actions such as terminations and removals to OPM  The Department of Veterans Affairs Accountability and Whistleblower Protection Act of 2017 also requires the Secretary to provide a report on the disciplinary procedures and actions of the department to Congress  To understand the depth and breadth of misconduct and related issues in a large entity, such as VA, comprehensive and reliable information is needed  Standards for Internal Control in the Federal Government states that an information system represents the life cycle of information used for the entity’s operational processes that enables the entity to obtain, store, and process quality information  Therefore, management should design the entity’s information system to obtain and process information to meet each operational process’s information requirements and to respond to the entity’s objectives and risks, such as the ability to systematically analyze misconduct department-wide to identify trends and make management decisions regarding misconduct  A deficiency exists when (1) a control necessary to meet an objective is missing or (2) an existing control is not properly designed, so that even if the control operates as designed, the objective would not be met  We identified 12 fragmented information systems that VA has used, or continues to use, to collect employee misconduct and disciplinary actions  Although VA has made efforts to develop repositories to collect information pertaining to misconduct and disciplinary action, none of the 12 information systems contain complete information  Six of these systems collect partial misconduct and disciplinary action information and contain fields that could potentially be shared with other systems to obtain additional information, while the other six systems are intended for internal office use only, each containing their own unique fields and values tailored to the needs of that particular office, which are not shared  Therefore, the number of eligible fields for each information system was also limited to those not specifically designated for internal use  On the basis of our review, the 12 information systems are not currently able to communicate, or interoperate, with one another to provide a complete picture of misconduct and disciplinary actions across VA  Table 2 provides an overview of VA’s six information systems and associated data files that collect partial misconduct and disciplinary-action data that could potentially be shared with other systems  According to OHRM officials, VA’s information system for recording adverse disciplinary actions—the Personnel and Accounting Integrated Data (PAID) system—was not designed to track all misconduct cases  In addition, OHRM stated that the 53-year-old PAID system was developed primarily to track payroll actions for all employees and is the system of record that holds department-wide personnel information that is reported to OPM’s EHRI system  It contains information about adverse disciplinary actions that affect employee leave or salary, or result in a Notification of Personnel Action Form (Standard Form 50)  However, PAID does not track comprehensive information on instances of misconduct such as the offense, or the date of occurrence, and it does not include instances of other types of disciplinary actions, such as admonishments or reprimands that would not affect leave or salary, or result in a Standard Form 50  OHRM officials stated VA implemented a system called HR Smart in June 2016 that is intended to replace PAID, but the agency does not plan to upgrade the functionality of the new system to enable reliable collection of misconduct information  According to OHRM, HR Smart includes the same personnel-processing functions as PAID but will allow for tracking data changes and transaction history over time  However, as with the PAID system, adverse disciplinary actions involving leave and salary will be tracked, but other actions, such as reprimands and admonishments, will not  It also will not track information related to the offense that prompted the disciplinary action  While the HR Smart system has the capability to include modules to enhance performance features, such as the ability to track misconduct, according to OHRM officials VA does not currently have plans to implement these modules  As a result, the HR Smart system will not have the capability to track all employee misconduct department-wide and will not improve VA management’s visibility over the depth and breadth of misconduct so that it can systematically understand misconduct department-wide  VA has five additional information systems for tracking complaints or allegations of misconduct and disciplinary actions, but, similar to the agency’s PAID information system, each of these information systems contains a subset of the information that would be needed to understand all misconduct department-wide  According to Standards for Internal Control in the Federal Government, systems should include relevant data from reliable internal sources that are reasonably free from error and faithfully represent what they purport to represent  Additionally, management is advised to process data into quality information that is appropriate, current, complete, accurate, accessible, and provided on a timely basis  Management should also evaluate processed information, make revisions when necessary so that the information is quality information, and use the information to make informed decisions  Additionally, according to Standards for Internal Control in the Federal Government, management should design the entity’s information system and related control activities to achieve objectives and respond to risks  The standards add that the information system design should consider defined information requirements for each of the entity’s operational processes  Defined information requirements allow management to obtain relevant data from reliable internal and external sources  In order to achieve complete and accurate data, internal controls are needed, among other things, to ensure that fields are not left blank, data elements are clearly defined and standardized, and common data elements are included across data systems to allow for interoperability and aggregation  Our analysis of VA’s 14 data files identified the following three categories that reduced the reliability of the data: lack of data standardization, and lack of identifiers  We found that 7 data files we reviewed contained a majority of data within fields, but 5 data files were missing a significant amount of data within certain fields  We were unable to analyze the remaining two files due to a number of data-quality issues  Among the fields that were missing data were several that would be useful for analyzing misconduct, including complainant name, proposed action, and person of interest, as shown in table 4  For example, we found that in the OAR Legacy Referral Tracking List, 97 percent of the entries for the Proposed Action field (1,210 of 1,245) and 96 percent for the Disciplinary Action field (1,190 of 1,245) were blank  If available, comparison of the proposed and disciplinary action-taken fields would allow VA to assess whether actions are consistently implemented department-wide  However, the high percentages of blank values in multiple fields impair VA’s ability to conduct a comprehensive analysis of misconduct to identify and address trends  See appendix II for a further listing of the five data files and the corresponding fields that were missing data  In addition, we found that several data files had options such as “not applicable” or “no” for certain fields so that the field would not be left blank, but these options were not consistently used  For example, the Complainant Name field within the Legacy Referral Tracking List was blank for some entries and not applicable (N/A) for others  Accordingly, we did not know whether data were intentionally omitted or not entered by mistake  In addition, the Offense Sustained field found within the VA-Wide Adverse Employment Action and Performance Improvement Plan Database was blank for some entries and either a yes or no for others  Eight of the 14 data files we reviewed did not have key data elements that were defined within and across information systems  In other words, the data files contained entries that described similar information in different ways  For example: The Complaints Automated Tracking System (CATS) Employment data were not mutually exclusive, or independent of one another  For example, this field includes two distinct categories of information: employment status, such as full time or part time; or hiring authority, such as Title 5 or Title 38  This method of storing information resulted in undercounting each of the separate values due to the system’s inability to account for expected overlap  For instance, an employee could be both a full-time and Title 5 employee and the field only tracks one or the other  ORM officials stated that this field has since been modified to capture more options to account for the overlap  The NCA data file’s Action Proposed/Decided/Taken data were tracked in a single field and updated with the most-recent action, rather than capturing proposed actions, decided actions, and actions taken in separate fields  We also identified standardization issues with the newly updated VA- Wide Adverse Employment Action and Performance Improvement Plan Database  For example, we found 15 different variations of Registered Nurse, such as “Registered Nurse,” “Staff RN,” and “RN” position names  In addition, we found 28 alternate values that identified Diagnostic Radiologic Technologists (e g , Diagnostic Radiologic Technologist, Diagnostic Radiological Technologist, and Radiologic Technologist)  See appendix III for a description of the fields that did not have standardization within the eight data files  We determined that 5 out of the 14 data files did not have identifiers that would allow comparisons of information across systems  Identifiers are important because they reference one unique individual or case, which makes it possible to analyze historical data pertaining to all records with the same identifier and analyze trends in employee misconduct over time  For example: The OAR VA-Wide Adverse Employment Action Database and the VBA data files had a combined total of 4,487 closed cases of misconduct that received adverse corrective action during the combined period of November 2013 through December 2016  We found that these two data files did not contain unique identifiers for complainants or accused individuals for a given case  OAR officials stated that this system does not contain Personal Identifying Information since its purpose is to track proposed and taken adverse actions  If more-specific information is needed, OAR staff coordinate with the human-resource point of contact  Although OAR may obtain additional information, this information is not entered into the OAR VA-Wide Adverse Employment Action Database, which would assist with conducting analysis  VAPS tracks misconduct in two separate subsystems, one of which tracks traffic violations and other administrative offenses, and one of which also tracks more-egregious offenses such as criminal violations  These subsystems also do not have unique identifiers that would allow data matching between the two subsystems, which could impede the analysis of this information  Even with both files, there is no ready way to capture the complete number of individuals with misconduct in both files due to the lack of a shared identifier  The newly updated VA-Wide Adverse Employment Action and Performance Improvement Plan Database does not contain unique identifiers, such as employee identification number, name of the complainant or accused, or other linking variables, to allow for the analysis of historical trends or comparison of information among other information systems  The data-quality issues described above are due in part to most of VA information systems not having data dictionaries, field definitions, or other documented guidance and procedures on data entry and automated edit checks to control for erroneous entries and blank fields  Absent guidance and procedures, VA lacks assurance that employees will enter complete and accurate information in the various data systems  Further, the lack of unique identifiers such as employee identification number, case number, or other linking variables for each of the records does not allow for analysis of historical trends or comparison of information among different information systems  Consequently, this precludes VA from determining the frequency and nature of allegations by specified category, or identifying trends, thus impeding senior officials’ ability to analyze misconduct department-wide and develop corrective actions  VA Directive 5021, Employee/Management Relations, governs policy for disciplinary procedures for all employees and outlines the provisions for the adjudication of each disciplinary action and associated file documentation requirements  Specifically, files must be established before a notice of proposed adverse action is issued to the employee to document that the adjudication procedures were followed  The file must contain all available evidence upon which the notice of proposed action is based and that supports the reasons in that notice  In addition, each file should contain specific documentation related to the adjudication of employee misconduct  VA Handbook 5021 states that disciplinary actions and associated adjudication procedures for all VA employees appointed under Title 5 are governed by three basic principles: (1) an employee shall be informed in writing honestly and specifically why the action is being brought against him or her; (2) an employee shall be given a reasonable opportunity to present his or her side of the case; and (3) the employee and representative shall have assurance of freedom from restraint, interference, coercion, discrimination, or reprisal in discussing, preparing, and presenting a defense  Our review of a generalizable sample of 544 misconduct case files (from a universe of 23,622 files) associated with disciplinary actions that affect pay from October 2009 through May 2015 revealed that VA officials did not consistently adhere to VA’s policy for retaining files containing evidence of misconduct  Specifically, VA was unable to provide the files for 10 percent (55 of 544) of the files we requested  We determined that administrations and program offices within VA have various record- retention schedules  Offices that have not established a record-retention schedule refer to the general records schedule developed by the National Archives and Records Administration (NARA)  However, we found that some offices are misinterpreting OPM and NARA guidance and specify the record retention period for adverse action files as a range between 4 to 7 years rather than selecting a specific number of years in their record- retention schedules  All of the files were within VA’s record-retention range specified during the time of our review  The files that were unaccounted for were dispersed throughout most of the VISNs, but one VISN was not able to account for 19 of the missing files in our sample  On the basis of our weighted analysis of the generalizable sample, we estimate that VA would not be able to account for approximately 1,800 files in the full population that were within the record-retention period specified  In addition, VA officials did not consistently adhere to VA’s policy for documenting that procedures were followed in the adjudication of misconduct cases  We identified 22 out of 36 file requirements where VA was not able to consistently demonstrate compliance with VA policy due to the lack of documentation contained in files, based on our generalizable sample  Specific to both Title 5 and Title 38 permanent- employee misconduct case files, table 5 shows the estimated number and percentage that deviated from file documentation requirements  A list of the 22 identified requirements and the percentage of files not in compliance can be found in appendix IV  As table 5 indicates, Title 5 and Title 38 permanent employee files did not always contain documentation that employees were informed of the reason the action was brought against them  For example, on the basis of our generalizable sample, we estimate that the advance notice of proposed action, which includes a statement of the specific alleged misconduct upon which the proposed action is based, was not included in 16 percent of the files department-wide  A final decision letter, which contains a statement of the decision official’s determination regarding which charges, if any, in the advance notice were sustained, was not included for an estimated 15 percent of all files  Further, an estimated 35 percent of all files did not include a written acknowledgement from the employees that they received the final decision letter in person, and an estimated 23 percent of all files did not include the required return receipt for certified mail indicating that the decision letter was mailed to the employee  In addition, Title 5 and Title 38 permanent employee files did not always contain documentation that employees were provided a reasonable opportunity to present their side of the case  Our review found that permanent-employee disciplinary files did not adhere to basic principles outlined in VA Handbook 5021 and lacked evidence to demonstrate that employees were adequately informed regarding their rights during the adjudication procedure  Specifically, our generalizable sample found that an estimated 21 percent of all files did not include statements regarding the employee’s rights to due process, such as his or her entitlement to be represented by an attorney or other representative  In addition, an estimated 8 percent of all files did not mention that more information regarding appeal rights could be obtained by consulting Human Resources Management offices  For files where the employee provided an oral reply in response to proposed disciplinary action, an estimated 29 percent of files did not include the required written summary, which is to be signed by the official hearing the oral reply  Where a written reply was submitted, an estimated 11 percent of files did not include a copy of the employee’s written reply  Further, VA officials did not consistently adhere to VA’s best practices specific to Title 5 permanent employees only  We found that in a majority of these files (an estimated 72 percent) the proposal letters did not include a statement that assured the employee he or she had freedom from restraint, discrimination, or reprisal in discussing, preparing, and presenting a defense  VA Handbook 5021, Employee/Management Relations, also states that Title 5 employees should provide their written responses through supervisory channels to the decision official  We estimate that a total of 6,819 files (47 percent) of Title 5 permanent employee files did not provide their written reply through supervisory channels to the decision official  Although OHRM is responsible for assessing the effectiveness of department-wide human resource programs and policies of VA Handbook 5021, according to OHRM officials, each facility is responsible for oversight of implementing policies and guidelines pertaining to how disciplinary actions are processed  We found no evidence that OHRM has assessed whether documentation exists that demonstrates adherence to policy governing cases involving disciplinary actions or provided oversight of VA’s implementation of record-retention requirements, or that human- resource personnel adhere to basic principles outlined in policy to ensure employees are informed of their rights during the adjudication process  The resulting lack of oversight to HR policies increases the risk that employees will not be adequately informed of their rights during the adjudication process  Accordingly, employees may not (1) be provided with information on why an action is being brought against them, (2) be provided with a reasonable opportunity to present their case, and (3) be adequately protected from potential reprisal in preparing their defense  Regarding retention of records, according to NARA, disciplinary and adverse action case files should be destroyed no sooner than 4 years but no later than 7 years after the case is closed  According to OPM, to implement this authority, each agency must select one fixed retention period between 4 and 7 years and publish the retention in the agency’s records disposition manual  We determined that some offices are misinterpreting OPM and NARA guidance by not selecting a specific number of years in their record- retention schedules  For example, three of the six policy record-retention schedules we reviewed did not establish a specific number of years for record retention  Specifically, record-retention policies for the Office of Information and Technology, VACO staff offices, and VBA specified the record-retention period for adverse action files as a range between 4 to 7 years rather than selecting a fixed retention period  For example, we found that the Records Control Schedule pertinent to VACO was dated June 30, 1967, without references to new or revised items since 1969  Our results are consistent with an October 2016 inspection conducted by NARA  The inspection report contained 16 findings and 19 recommendations for improvement of the records-management program at VA  Among the findings and recommendations were the following  Finding: The VA records management program has not ensured that the VACO maintains a current Records Management Handbook and a current Records Control Schedule, which together establish program objectives, responsibilities, and authorities for the creation, maintenance, and disposition of agency records  Recommendation: The Department Records Office must update and maintain the VACO handbook and the Records Control Schedule for Central Office Staff Offices and the Offices of the Assistant Secretaries to include specific Records Management roles and responsibilities for all VACO staff and to include mandates for implementation of records management policies and procedures in accordance with Federal statutes and regulations  Finding: The VA Departmental Records Management program does not conduct regular records management evaluations within VACO and the Offices of the Secretary and Assistant Secretaries or monitor the oversight activities of the Administrations  Recommendation: The VA Departmental Records Management program, working with the Administrations, VACO, and Enterprise Risk Management, must establish effective Records Management evaluation programs to monitor VA compliance with Federal regulations  Recommendation: The VA Departmental Records Management program, working with the Senior Agency Official for Records Management, must establish effective Records Management evaluation programs to monitor the records management practices within the Office of the Secretary and Assistant Secretaries to ensure compliance with Federal regulations  Finding: VACO Staff Offices and the Offices of the Assistant Secretaries are not routinely conducting records inventories  Recommendation: VACO Staff Offices and the Offices of the Assistant Secretaries, with support from the Department Records Management program, must conduct inventories of existing electronic and non-electronic records to identify scheduled, unscheduled, and vital records  In response to NARA findings, VA is to submit a plan of corrective action that specifies how the agency will address each inspection report recommendation, including a timeline for completion and proposed progress reporting dates  VA does not have a method in place to evaluate the implementation of records-management practices outside of those being conducted by VHA and VBA  Accordingly, VA has not been conducting records-management oversight with any uniformity department-wide  Further, VA’s use of multiple retention periods for adverse action files, and in some cases the lack of adherence to OPM and NARA guidance in defining a specific retention period for these files, results in inconsistent retention of these files across VA  The OIG receives allegations of employee misconduct from VA employees, the OSC, members of Congress, the public, and other stakeholders  When the OIG receives allegations it can either take no further action, open an investigation, or refer the case to facility or program offices within VA for review and response  For cases referred to facility or program offices, the OIG has developed a policy for VA facilities and program offices to use when investigating allegations of misconduct  This policy includes six elements that VA facility and program officials are to incorporate in their investigations, as shown in table 6  According to OIG officials, if the reviewing employees have concerns about the adequacy of the response provided, the OIG can either ask for additional information to supplement the response or open an internal case  Departmental heads (Under Secretaries for Health, Benefits, and Memorial Affairs, Assistant Secretaries, and other key officials) are responsible for ensuring that referrals are properly reviewed, documented, and answered within specified time frames  Our review of the 23 OIG cases of alleged misconduct between calendar years 2011 and 2014 involving senior officials found that VA facility and program offices did not consistently follow policies and procedures established by the OIG for investigating such allegations  In several instances, VA facility and program offices did not include one or more of the six elements required in their investigative response to allegations of misconduct  In addition, our review of the 23 cases found instances in which VA facility and program offices did not include sufficient documentation for their findings, or provide a timely response to the OIG  The OIG was not able to produce the documentation provided by the facility or program office that was used to close 2 of the 23 cases in our review  All of the requested files were within the OIG’s 7-year record- retention period during the time of our review  As shown in table 7, we identified four cases that did not contain evidence of an independent review by an official separate from and at a higher pay grade than the accused  In three of the four cases that were not reviewed by an independent official at a higher grade, the review was performed by the medical center director, who was one of the accused named in the allegation  For example, in one case involving alleged time-and-attendance abuse by a physician, the medical center director, who was also named in the allegation as having received a similar complaint against the physician 2 years earlier, reviewed the allegations made against the physician and himself  The documentation provided showed that the medical center director conducted the investigation of allegations and found the allegations were not substantiated and no corrective actions were implemented  In all four cases, both the independence and higher-grade criteria were not followed when the accused senior officials investigated allegations against themselves  As shown in table 8, we generally found that VA facility and program offices reviewed each allegation contained in the original referrals, although in one case the reviewer did not respond directly to all allegations  As shown in table 9, VA facility and program offices clearly indicated their findings for each allegation in 14 of the 21 cases of misconduct involving senior officials for which files could be located, as well as their assessment of whether the allegations were substantiated or unsubstantiated  However, we identified seven cases in which VA discussed its findings but did not provide a clear indication of whether all allegations were substantiated or unsubstantiated  Responses lacking a clear statement of substantiation may be more difficult for subsequent reviewers, including OIG and OAR investigators, to track and perform follow-up where necessary  For example, in one case involving 11 allegations, no statement of substantiation was provided, but VA’s response included seven recommendations, three of which involved disciplinary action  We did not find evidence in the case file that follow-up was performed by the OIG personnel to clarify this discrepancy, and the case was closed  As shown in table 10, most allegations involving senior officials (16 of 21 cases for which files could be found) were not formally substantiated and did not require a recommendation for corrective action based on OIG case-referral criteria  Specifically, the criteria require a description of corrective actions taken or proposed as a result of substantiated allegations, but make no mention of allegations that were not substantiated as part of VA’s response  For one substantiated allegation, however, we found no evidence of a recommendation for corrective action  Table 11 shows that 17 cases from VA facility and program offices did not provide the supporting documentation they used to reach their conclusions about the OIG case referrals  In 17 cases, including one case reviewed by an AIB panel, VA referenced documents reviewed but did not attach any of the supporting evidence  OIG case-referral criteria state that VA facility or program offices must provide supporting documentation used in their review, such as copies of pertinent documents  However, the criteria do not specify whether copies of all documentation reviewed must be included in the file  Supporting documentation, which must be provided according to OIG policy, will vary depending on the circumstances of the case, but those used to support the findings and recommendations should be included  For example, we reviewed one case where pertinent documents were referenced to support the allegation, but documents supporting the findings and recommendation were not included  The case contained allegations involving false patient wait-time documentation and abuse of authority  Specifically, a medical center director instructed staff to review patient wait times between follow-up appointments in order to meet VA’s 14-day timeliness metric  The investigation revealed that VA staff had changed several hundred veteran appointment wait times  The investigation concluded that the false documentation allegation was substantiated, but attributed the cause to the staff not understanding how to enter a follow-up appointment date into the system  However, there was no documentation in the files to support (1) that the medical center director had not abused his authority by instructing staff to review wait times greater than 14 days to determine how they could be reduced, and (2) findings for the conclusion that the original wait times were entered in error  Absent supporting documentation, it is difficult for the OIG to determine whether enough evidence was gathered before closing alleged cases of misconduct that were found to be unsubstantiated or closing substantiated cases of misconduct that required further action  VA Directive 0701 states that copies of voluminous transcripts of interviews, the entire claims folder, and medical charts are not necessary  However, VA Directive 0701 further states that such materials should be available if the OIG subsequently requests them within the record-retention period  Case examples of allegations reviewed, and subsequently closed, by the VA OIG based on its evaluation of evidence provided by facility and program offices in response to allegations of misconduct can be found in appendix V  As shown in table 12, VA facilities’ or program offices’ response letters, which were sent to the OIG, included a point of contact for further questions in 15 of the 21 OIG case referrals involving senior officials, including the individual’s name and a means of contact (phone or e-mail)  In 2 of the 15 cases where a point of contact was provided, the contact was also one of the accused in the allegation  Although that is not technically a violation of OIG criteria, it likely presents a conflict of interest in regard to independent reviewers obtaining objective case information  In six other cases, no contact was listed, although the letter was signed by the reviewer  If a specific point of contact is not identified, including position title, it may be assumed erroneously by employees involved in the case, or following up on the case, that the default contact is the reviewer, who may not be the appropriate point of contact, and may or may not be able to provide objective case information  OIG guidelines state that VA facilities and program offices assigned Hotline case referrals are responsible for reporting written findings to the Hotline Division within 60 days, unless an extension is requested  Our review of the 21 cases found instances in which VA facility or program offices did not always provide a timely response to the OIG  Table 13 shows five instances in which VA facility or program offices submitted a response after the deadline requested by the OIG  One response was not reported timely after an extension was provided by the OIG  In one of the five cases involving allegations of abuse of authority by a VA medical center director, the reviewer requested an extension, which is permitted by OIG policy, but still missed the revised deadline  The five case files did not contain any information regarding any follow-up actions taken in response to delays  According to OIG officials, when a case has been referred to a program office for investigation, the OIG reviews the program office’s response for completeness and sufficiency before closing the case  However, there is no requirement for the OIG to ensure that the responses contain the six elements listed in VA Directive 0701 and confirm that case referral allegations have been addressed  Consequently, the lack of verification could have contributed to insufficient evidence that does not meet the requirements outlined by the OIG  Additionally, VA facility and program offices have not consistently adhered to VA Directive 0701 policy and do not always provide supporting documentation for their findings and recommendations, or always provide a timely response when reporting findings to the OIG’s Hotline division  Inconsistent adherence to the reporting standards provided by the OIG to VA facilities and program offices for investigating and resolving misconduct case referrals from the OIG Hotline impedes VA’s ability to ensure that misconduct cases are being handled appropriately  According to OIG officials, the OIG has taken steps to enhance the review of case responses  Specifically, OIG officials stated that in April 2018 the OIG implemented a new Enterprise Management System to reduce reliance on certain manual processes  According to OIG officials, Hotline analysts will now have more time to review their work and perform other quality-assurance activities  In implementing this new system, it will be important for the OIG to consider how the system can assist in ensuring requirements are met and responses are received timely  Our review of VA’s information systems that track misconduct involving senior officials department-wide indicates that they may not always be held accountable for misconduct  Specifically, (1) misconduct was sometimes substantiated, but the proposed disciplinary action was not taken; (2) misconduct was sometimes substantiated, but no disciplinary action was recommended; (3) previous penalties did not have the corrective effect for officials found to have engaged in repeated acts of misconduct and who have remained in VA management positions; and (4) senior officials violated separation-of-duty policy when taking disciplinary action  VA Handbook 5021 allows the deciding official to determine the appropriate disciplinary action if one or more allegations are substantiated  However, the disciplinary action may not be more severe than what had been proposed  In several cases, misconduct was substantiated, but the proposed action was not always taken  Our review of the OAR Legacy Referral Tracking List identified 17 officials between calendar year 2011 through May 2015 with substantiated misconduct where action was proposed  However, in some of these cases, the officials were given a lesser penalty than the one proposed, while in other cases there is no evidence that action was taken  As shown in figure 3, we found that for 12 of the 17 officials with substantiated misconduct, an adverse disciplinary action (removal) was proposed  Of those 12 officials, 3 were removed, 2 received a suspension, 4 received a reprimand or admonishment, 2 were allowed to resign or retire before receiving disciplinary action, and we found no evidence of disciplinary action for the remaining individual  For the other 5 officials, actions such as counseling, admonishment, suspension, or reprimand were proposed  Of the 5 officials, 2 received the actions that were proposed, 1 received a lesser penalty than what had been proposed, 1 was allowed to retire before receiving action, and we found no evidence of the proposed action for the remaining individual  For the two officials for whom there was no evidence that disciplinary action was taken, we found no evidence within the PAID information system or personnel files that these officials received the action proposed in the OAR Legacy Referral Tracking List  Counseling was proposed for one official, and removal from the position for the other official  OAR did not provide us with evidence that the officials had received the action proposed  We also reviewed an additional 15 cases that involved a fact-finding or an AIB  Our review of these cases found that 11 out of 23 officials were associated with instances of substantiated misconduct and proposed action was recommended  For 4 of the 23 officials where the proposed action field was populated, the information within the OAR Legacy Referral Tracking List reflected the action recommended  The applicable data fields for the remaining 19 officials within the OAR Legacy Referral Tracking List were not in agreement with the action recommended, or blank  This review also identified two officials with substantiated misconduct where OAR did not provide evidence that the disciplinary action proposed was taken: Two officials were involved in a case concerning alleged whistle- blower retaliation at the Phoenix VA Health Care System  The investigative report documented that allegations were sustained  The retaliation included allegations of involuntary reassigning the whistle- blower to another position, placement of the whistle-blower on administrative leave, and lowered performance pay ratings following disclosures regarding poor patient care and nursing triage errors  Appropriate administrative action for persons identified as having engaged in retaliation was recommended  We did not find any evidence in the PAID system that these two officials involved in retaliation received disciplinary action  OAR provided documentation to show that no action was taken against one official, and was unable to provide documentation to show that the disciplinary action had been taken for the other  The official who received no action received approximately $11,500 in performance pay during a 2-year period following the allegations  OAR’s quality-review process for investigative reports does not ensure that reports with findings of substantiated misconduct include recommendations for action  Our review of OAR’s Legacy Referral Tracking List identified 70 out of 1,245 closed cases involving officials where misconduct was either substantiated, or partially substantiated, but no disciplinary action was recommended  One case involved three allegations of poor dental care provided to patients by three different senior officials  One physician cut underneath a patient’s tongue with the bur of a hand-piece drill (substantiated), another administered medication the patient was allergic to (partially substantiated), and the final senior official extracted the wrong tooth (substantiated)  We did not find any evidence in the PAID system that these senior officials received disciplinary action  Further, OAR did not provide documentation to show that any disciplinary action had been proposed or taken  The physician that cut underneath a patient’s tongue received performance pay totaling $15,000 approximately 6 days after the investigation had concluded that misconduct was substantiated  As of March 2018, two of these senior officials received performance pay, and appear to still be employed at VA  While an investigation was conducted that substantiated (or partially substantiated) the allegations, there is an increased risk that some substantiated misconduct will go unaddressed if there is no recommendation for corrective action  Our review of OAR’s Legacy Referral Tracking List indicated that some officials who had been disciplined for misconduct remained in positions where they were responsible for proposing or deciding disciplinary action for other employees  We identified 15 officials in the OAR VA-Wide Adverse Employment Action Database who received disciplinary action between 14 days to 1 year prior to proposing disciplinary action for another employee  Most of the 15 officials (12 officials) had received a suspension  We also found that five officials in the OAR Legacy Referral Tracking List had received prior disciplinary actions for offenses unrelated to the new OAR allegations  A prior history of disciplinary actions indicates that some officials may be repeat offenders for whom the previous penalties did not have the desired corrective effect  For example, 4 out of 5 officials were suspended for a different offense prior to being the subject of a new allegation  One of the four officials was suspended less than 2 months prior to being the subject of a new allegation, while another received a suspension before, and again approximately 7 months following, the OAR allegation  According to VA Handbook 5021, the deciding official must use the “Douglas” factors, which include the employee’s past disciplinary record, to determine a reasonable penalty  One of the five VA officials was eventually removed approximately 6 months after the new OAR allegation  In analyzing cases involving senior management, we noted that the OAR Legacy Referral Tracking List often did not accurately reflect the disciplinary action that was decided based on the results of the investigation  In numerous instances for the OAR Legacy Referral Tracking List, the applicable data fields indicating the proposed and final disciplinary action were blank  In these cases where the disciplinary fields were populated, the data usually did not agree  Specifically, for 32 out of the 40 records we reviewed where misconduct was substantiated, the final disciplinary action taken did not reflect the information within the OAR Legacy Referral Tracking List  When disciplinary actions are taken in response to findings of misconduct but are not entered within an appropriate information system, or are inaccurately recorded, it is more difficult to (1) monitor whether disciplinary actions have been implemented, and (2) ensure information relevant to management for making decisions is available  Further, without a prior record of misconduct or disciplinary action, senior officials who are repeat offenders may not receive the appropriate penalty required  Pursuant to the Department of Veterans Affairs Accountability and Whistleblower Protection Act of 2017, OAWP will now be responsible for receiving, reviewing, and investigating allegations of misconduct, retaliation, or poor performance involving senior officials  According to OAWP officials, their office investigates allegations of misconduct at the senior level only  OAWP officials also stated that misconduct issues that occur below the senior level will be referred to each of the three major VA administrations for investigation and resolution  In addition to the VA- Wide Adverse Employment Action and Performance Improvement Plan Database, OAWP officials stated that it has implemented two additional information systems that are used concurrently to capture case information  OAWP officials stated that they are currently working with VA Information Technology to assess options for other case-management systems that could consolidate these three information systems into one comprehensive system  VA Handbook 5021, Employee/Management Relations, states that the decision on a proposed major adverse action will be made by an official who is in a higher position than the official who proposed the action, unless the action is proposed by the Secretary  Standards for Internal Control in the Federal Government states that management should divide or segregate duties among different people  Our review of the OAR VA-Wide Adverse Employment Action Database, OAWP VA-Wide Adverse Employment Action and Performance Improvement Plan Database, and VBA data file identified examples where VA officials did not follow separation-of-duty requirements  As shown in table 14, 73 (out of an estimated 7,886) VA officials acted as both the proposing and deciding official in cases involving removals for employees found to have engaged in misconduct  Fourteen VA officials acted as both the proposing and deciding official in two or more cases  One of these 14 officials acted as both the proposing and deciding official for seven different removal cases  Further, our review of 29 VA officials found that none received disciplinary action for violating separation-of-duty policy  The systemic lack of adherence to VA’s separation-of-duty policy is reflective of a lack of controls that would allow such activity to occur  Focusing on ensuring such controls are implemented would help ensure that VA decreases the risk of abuse when officials act as both proposing and deciding officials  VA has procedures in place to ensure that allegations of misconduct are investigated, but these procedures allow VA program offices or facilities where a whistle-blower has reported misconduct to conduct the investigation  According to VA officials, investigations that are deemed necessary are occasionally ordered directly from the head of the facility or VA leadership, which takes the lead on an investigation into the allegation  Alternatively, an OIG official stated if allegations of misconduct are received by the OIG, the OIG has the option of investigating the allegation or exercising a “right of first refusal” whereby it refers allegations of misconduct to VA facilities or program offices where the allegation originated to complete an independent review and provide a response to the OIG  As shown in figure 4, the majority of contacts the OIG received (127,265 out of 133,435) from calendar years 2010 through 2014 were not investigated due to several reasons, such as insufficient evidence or lack of jurisdiction  Of those contacts that were investigated, the majority (4,208 of 6,170 investigated contacts) were not investigated by the OIG but rather were referred to facility or program offices for investigation  Whistle-blowers also have the option of reporting alleged misconduct outside VA by filing a disclosure with the OSC, and may do so if they believe there has not been a resolution to their complaint internally  If the OSC determines that there is substantial likelihood of wrongdoing, it may refer the disclosure back to the Secretary of Veterans Affairs for further investigation  According to OSC officials, as a general policy, the OSC will not refer a disclosure to the Secretary if the OIG is already conducting an investigation of that particular complaint and defers to the OIG to finalize the investigation  According to OIG officials, the OIG may, in turn, exercise its “right of first refusal” when cases are referred from the OSC  Consequently, this process can result in a disclosure that was originally made to the OSC being referred back to the facility or program office where the allegation originated  As shown in figure 4, the OSC referred 172 of 942 disclosures (18 percent) filed by VA employees back to the Secretary of Veterans Affairs for further investigation from calendar years 2010 through 2014  Of the 172 disclosures referred, VA referred 53 back to the facility or program offices where the complaint originated and 119 to the OIG  The independence of officials conducting or reviewing the results is paramount to the integrity of the process both in deed and appearance  According to VA Directive 0700, the decision whether to conduct an investigation should not be made by an official who may be a subject of the investigation, or who appears to have a personal stake or bias in the matter to be investigated  Moreover, according to OIG policy, investigations referred to VA offices must be reviewed by an official independent of and at least one level above the individual involved in the allegation  VA does not have oversight measures to ensure that all referred allegations of misconduct are investigated by an entity outside the control of the facility or program office involved in the misconduct, to ensure independence  VA OIG officials acknowledged that there have been concerns about referring cases back to the chain of command because the OIG is unsure where cases go once they are referred  The investigation of allegations of misconduct by the program office or facility where the complaint originated may present the appearance of a conflict of interest in which managers and staff at facilities may investigate themselves or other allegations where they may have a personal stake or bias in the matter to be investigated  Consequently, there may be an increased risk that the results of the investigation are minimized, not handled adequately, or questioned by the OSC or the individual who made the original allegation  According to VA Directive 0700, significant incidents occurring, and issues arising, within VA facilities or offices shall be reported and investigated as necessary to meet the informational and decision-making needs of VA  Primary responsibility in this regard rests with the chief executives of the facility or staff office involved, and with their supervisors in VA and its administrations  According to an OIG official, VA (the Secretary or a delegate) sends disclosures received from the OSC to the OIG, which may then refer to VA facility or program offices for further review and investigation  According to OSC officials, for cases that are referred to a program office, the OSC requires that the Secretary or delegate provide a report that outlines its conclusions and findings  This reporting is not required for disclosures where an ongoing OIG investigation is already under way  According to OSC officials, for each disclosure, the OSC is to review the report for statutory sufficiency and determine whether the findings of the agency head appear reasonable  The OSC is to send its final determination, report, and any comments made by the whistle-blower to the President and responsible congressional oversight committees  The OSC has raised concerns in its reports to the President about investigations conducted by VA program offices and facilities  Of the 172 whistle-blower disclosures referred by the OSC between calendar years 2010 through 2014, the Secretary of Veterans Affairs referred 53 to facility and program offices  Our review of these 53 OSC reports found that the OSC had concerns about the conclusions VA reached in 21 (40 percent) of the 53 disclosure cases  For example, the OSC found that the conclusions in some VA reports were unreasonable because VA reached its conclusion without interviewing the witness, provided shifting explanations that strained credibility and did not provide evidence of an unbiased investigation, ignored whistle-blower concerns by refusing to investigate allegations, and refused to acknowledge the impact on the health and safety of veterans seeking care after confirming problems in these areas  For disclosure cases that were referred from the OIG to facility and program offices during the 2010–2014 time frame of our review, the OIG acknowledged that these concerns arose because of a lack of communication between the department and the OIG regarding the scope of the review  At the time of our review, VA did not have a procedure in place to ensure the conclusions reached for investigations involving OSC disclosure cases are reasonable and meet the informational and decision- making needs of VA whereby all allegations are addressed  More recently, the OIG has started to communicate the scope of its reviews that involve matters referred by the OSC to the Office of the Secretary  In implementing this new process, it will be important for the Office of the Secretary to ensure that any allegations outside the purview of the OIG’s investigation are fully addressed by a departmental entity in accordance with OSC requirements  As shown in figure 4, of the 172 disclosure cases referred to VA by the OSC, a total of 119 cases were referred to the OIG  The OIG had conducted, or was already conducting, an investigation of the particular allegations for all 119 disclosures  Since these 119 disclosure cases were already under investigation by the OIG, the OSC deferred to the OIG’s investigation for these cases  A total of 37 of these 119 disclosure cases that were referred to the VA OIG were submitted to the OSC anonymously  Therefore, we were unable to conduct a review of these investigations because there was no information available to identify the individuals involved  According to Standards for Internal Control in the Federal Government, management’s ability to make informed decisions is affected by the quality of information  Accordingly, the information should be appropriate, timely, current, accurate, and accessible  The oversight body oversees management’s design, implementation, and operation of the entity’s organizational structure so that the processes necessary to enable the oversight body to fulfill its responsibilities exist and are operating effectively  Our review of the remaining 82 disclosure cases determined that the OIG does not have procedures in place to track cases that were referred from the OSC for further investigation  According to OIG officials, the OIG’s information system did not have a method in place to ensure that OSC case numbers are linked to the OIG investigative case number and final report  Consequently, the OIG was unable to produce the investigative documentation for these 82 disclosures  According to OIG officials, OSC case numbers and associated Hotline numbers are currently tracked in a spreadsheet until the implementation of a new system  The inability to locate investigative documentation prevents a third party from verifying whether the OIG examined the disclosures, whether any recommendations were addressed, or whether appropriate disciplinary action was taken for these cases  In addition, because the OSC defers to the OIG’s investigation for allegations that were already conducted, or being conducted, the OSC and individuals that made the allegations do not have documentation to demonstrate that the allegations were addressed  This information, or lack of it, has direct influence on management’s ability to make sound decisions relating to investigative matters  Pursuant to the Department of Veterans Affairs Accountability and Whistleblower Protection Act of 2017, OAWP will be responsible for recording, tracking, reviewing, and confirming implementation of recommendations from audits and investigations involving whistle-blower disclosures, including the imposition of disciplinary actions and other corrective actions contained in such recommendations  According to OAWP officials, the whistle-blower disclosure process will be similar to the current process when cases are referred to facility and program offices for investigation  OAWP will follow up on any open points with the level of leadership that is most appropriate in each case, such as the medical center or VISN director  Case details will be tracked through the three active databases that are being used concurrently  OAWP is currently working to develop an internal process that will bring the investigative communities together  For instance, OAWP would like to monitor cases that are referred to VA facility and program offices, but it does not currently have documented criteria to guide the process  According to OAWP officials, OAWP is finalizing new policies in the form of a policy manual and handbook  However, these officials were unable to provide a time frame for completion of the published guidance  Our analysis of VA data shows that individuals who filed a disclosure of misconduct with the OSC received disciplinary action, and left the agency, at a higher rate than the peer average for the rest of VA  We identified 135 disclosure cases that were received by the OSC between calendar years 2010 and 2014 and were alleging misconduct  Of the 135 disclosures, a total of 129 employees made a total of 130 disclosures nonanonymously  We compared the 129 employees who made nonanonymous disclosures to the PAID information system using the complainants’ information  As shown in table 15, on average approximately 1 percent of all employees in the VA roster received an adverse action in any given fiscal year  For the 129 nonanonymous whistle-blowers, we found that approximately 2 percent received an adverse action in the fiscal year prior to their disclosure, while 10 percent had received an adverse action in the fiscal year of their disclosure, and 8 percent received an adverse action in the year subsequent to this disclosure  While the fact that nonanonymous whistle-blowers faced higher rates of adverse action subsequent to their disclosure than the VA population as a whole is consistent with a pattern of retaliation for nonanonymous whistle-blowers, it is only an indication that retaliation could be occurring  Our analysis also showed that among employees who could be matched to the PAID end-of-year roster, attrition rates were higher for those individuals who filed a nonanonymous disclosure with the OSC  On average, approximately 9 percent of all VA employees on the end-of-year roster in one fiscal year were not on the subsequent year’s roster  In contrast, 66 percent of the 129 nonanonymous whistle-blowers did not appear in the subsequent year’s roster  Attrition rates were higher among employees who had filed a disclosure than among their peers who had not filed disclosures, for all fiscal years in our review (see table 16)  Our analysis did not confirm the reasons for disciplinary action or attrition involving any of the 129 employees who made nonanonymous disclosures to the OSC  According to VA officials, employees who have a history of poor performance or conduct may be more likely to file a disclosure with the OSC or allege misconduct, which could explain some of the disparities between whistle-blowers and other employees  However, we also could not rule out instances where retaliation by senior officials may have occurred after misconduct was disclosed  The Civil Service Reform Act of 1978, as amended, states, among other things, that federal personnel management should be free from prohibited personnel practices (PPP)  The law also authorizes the OSC to investigate allegations involving PPP that include reprisals against employees for the lawful disclosure of certain information pertaining to individuals who engage in such conduct or other wrongdoing  According to Standards for Internal Control in the Federal Government, laws and regulations may require entities to establish separate lines of communication, such as whistle-blower and ethics hotlines, for communicating confidential information  Management informs employees of these separate reporting lines, how they operate, and how they are used, and how the information will remain confidential  Reporting lines are defined at all levels of the organization and provide methods of communication that can flow down, across, up, and around the structure  Our interviews with six VA whistle-blowers who claim to have been retaliated against provided anecdotal evidence that retaliation may be occurring  Whistle-blowers we spoke to alleged that managers in their chain of command took a number of actions that were not traceable to retaliate against the whistle-blowers after they reported misconduct  These alleged actions included being reassigned to other duty locations or denied access to computer equipment necessary to complete assignments, and socially isolating these individuals from their peers, among other things  Whistle-blowers we spoke to also expressed concerns regarding the lack of guidance available to employees about how to file a disclosure through VA and the OSC  Whistle-blowers stated that employees are not provided adequate information on how to document or file a claim of misconduct or retaliation  Employees can file disclosures regarding misconduct and complaints of retaliation through multiple reporting lines  As mentioned previously, however, the OSC will generally not refer to the Secretary under its statutory process a disclosure if the OIG has conducted or is already conducting an investigation of that particular complaint  Thus, whistle-blowers may limit their chance of having an independent, non-VA entity oversee their complaint if they file a complaint with the OIG first  The Department of Veterans Affairs Accountability and Whistleblower Protection Act of 2017 requires the Secretary, in coordination with the Whistleblower Protection Ombudsman, to provide training regarding whistle-blower disclosures to each employee of VA  This information shall include, among other items, an explanation of each method established by law in which an employee may file a whistle-blower disclosure, an explanation that the employee may not be prosecuted or reprisal taken against him or her for disclosing information, and language that is required to be included in all nondisclosure policies, forms, and agreements  The Secretary shall also publish a website and display the rights of an employee making a whistle-blower disclosure  In August 2017, VA began providing additional information on its website for potential whistle-blowers who wish to report criminal or other activity to the OIG  The information provided focuses on reporting misconduct to the OIG and provides other lines of reporting established by law in which an employee may file a whistle-blower disclosure, such as directly to an immediate supervisor or the OSC  In addition, the information provided explains the process after misconduct is reported through the OIG Hotline, but does not clarify the process for referred disclosure cases received from the OSC  As mentioned previously, OIG officials stated that a disclosure made to the OSC or the OIG can be referred back to the facility or program office where the allegation originated, which may compromise confidentiality  Consequently, employees may not be aware that their information may be shared among the OSC, the OIG, OAWP, or VA facility and program offices when a disclosure is made to the OSC  Adequately communicating the investigative process to employees may alter their decision to report wrongdoing  Without a clear understanding of the lines for reporting misconduct and how they operate, whistle-blowers may be uncertain as to their options for reporting misconduct, which increases the risk that they may not report workplace misconduct  According to OSC, it has learned through its cases that OAWP has a practice of allowing VA employees, who are the subject of the allegations brought forward by whistle-blowers to review or participate in investigations, or both, which could make the whistle-blower feel uncomfortable or intimidated  This practice has led to confusion regarding the role and responsibilities of OAWP personnel  OAWP’s use of VA employees that are employed at the facility under investigation in the review of allegations creates the possibility of a conflict of interest or an appearance of a conflict of interest  For example, in a case OSC described in its comments on a draft of our report, an OAWP representative who was also associated with the human-resource office at the VISN that oversees the whistle-blower’s facility, placed the whistle- blower under oath and questioned her about issues unrelated to the referred allegations  OSC has since sought clarification of OAWP’s role and the OAWP employee’s possible connection to the VISN  While VA collects data on some types of disciplinary actions, it is limited in its ability to use those data because it does not collect all misconduct and associated disciplinary-action data through a single information system, or multiple interoperable systems  Absent a process to collect such data department-wide, VA does not have the ability to analyze and report data systematically  In addition, the data currently collected are not always reliable or useful  The inclusion of appropriate documented guidance and standardized field definitions would help to ensure VA collects reliable misconduct and associated disciplinary-action data  With high-quality information that is accurate and comprehensive, VA management would be better positioned to make knowledgeable decisions regarding the extent of misconduct occurring and how it was addressed, department-wide  VA has not ensured that program and facility human-resources personnel adhere to policy governing documentation contained within evidence files to support conclusions reached  In addition, VA often had no record of the evidence involved with the adjudication of these actions and could not verify whether these individuals received reasonable and fair due process  The absence of documentation in some files also raises the possibility that VA may not always be in compliance with its procedures for governing the adjudication of alleged employee misconduct  Additionally, ensuring that human-resources personnel adequately inform employees of their rights during the adjudication process would provide them with a reasonable opportunity to present their case when preparing their defense  VA also does not consistently adhere to OPM and NARA guidance in defining a specific retention period for adverse action files  This results in an inconsistent retention of these files across VA which complicates department-wide analysis  VA’s inconsistent adherence to the standards provided by the OIG to facilities and program offices for investigating and resolving misconduct cases increases the risk that misconduct case are not being handled appropriately  Additionally, the lack of verification of responses received to ensure documentation supports findings and recommendations has contributed to evidence that does not always meet the requirements outlined by the OIG  Finally, timely responses are not consistently provided when facility and program offices report findings to the OIG’s Hotline Division  OAR did not monitor whether substantiated instances of misconduct involving senior officials received disciplinary action  OAR’s Legacy Referral Tracking List also did not accurately reflect the disciplinary action that was decided based on the results of the investigation  When disciplinary actions are taken, in response to findings of misconduct, but are not entered within an appropriate information system, or are inaccurately recorded, it is more difficult to monitor whether disciplinary actions have been implemented in substantiated instances of misconduct involving senior officials  As demonstrated, this may result in no action being taken for substantiated misconduct or the previous penalties not having the corrective effect for repeat offenders  There is also an increased risk that substantiated misconduct will go unaddressed if there is no recommendation for corrective action  Further, VA also does not have internal controls to ensure adherence to proper separation-of-duty standards involving the removal of an employee


Report 2:  Such controls would minimize the risk of abuse when officials act as both proposing and deciding officials  In addition, VA does not have oversight measures to ensure that all allegations of misconduct referred by the OIG to facility and program offices are investigated by an entity outside the control of the facility or program office involved in the misconduct  The investigation of allegations of misconduct by the program office or facility where the complaint originated may present the appearance of a conflict of interest in which managers and staff at facilities may investigate themselves or other allegations where they may have a personal stake or bias in the matter to be investigated  Therefore, the risk that the results of the investigation are minimized, or not handled adequately, is increased  VA’s newly developed process to communicate the scope of its reviews that involve matters referred by the OSC to the Office of the Secretary will be important to ensure any allegations outside the purview of the OIG’s investigation are fully addressed by a departmental entity in accordance with OSC requirements  Further, the OIG’s inability to locate investigative documentation prevents a third party from verifying whether the OIG examined the disclosures, whether any recommendations were addressed, or whether appropriate disciplinary action was taken for these cases  This lack of information has direct influence on management’s ability to make sound decisions relating to investigative matters  According to OIG officials, a spreadsheet is being used for tracking case numbers associated with disclosures, but plans to implement a process within the new system  Nonanonymous whistle-blowers faced higher rates of adverse action subsequent to their disclosure than the VA population as a whole  In addition, these individuals also had attrition rates higher than their peers who had not filed a disclosure  The disparities between whistle-blowers and other employees may be an indication that retaliation by senior officials may have occurred after misconduct was disclosed  Although VA has started to provide additional information for potential whistle-blowers who wish to report criminal or other activity to the OIG, VA does not have a process to inform employees of how their information may be shared between organizations when misconduct is reported  Without a clear understanding of how the lines for reporting misconduct operate, whistle- blowers may be uncertain as to their options for reporting misconduct, increasing the risk that they may not report workplace misconduct  We are making the following 16 recommendations to VA  The Secretary of Veterans Affairs should develop and implement guidance to collect complete and reliable misconduct and associated disciplinary-action data department-wide, whether through a single information system, or multiple interoperable systems  Such guidance should include direction and procedures on addressing blank data fields, lack of personnel identifiers, and standardization among fields, and on accessibility  (Recommendation 1) The Secretary of Veterans Affairs should direct applicable facility and program offices to adhere to VA’s policies regarding employee misconduct adjudication documentation  (Recommendation 2) The Secretary of Veterans Affairs should direct the Office of Human Resource Management (OHRM) to routinely assess the extent to which misconduct-related files and documents are retained consistently with VA’s applicable documentation requirements  (Recommendation 3) The Secretary of Veterans Affairs should direct OHRM to assess whether human-resources personnel adhere to basic principles outlined in VA Handbook 5021 when informing employees of their rights during the adjudication process for alleged misconduct  (Recommendation 4) The Secretary of Veterans Affairs should adhere to OPM and NARA guidance and establish a specific record-retention period for adverse action files  In doing so, the Secretary should direct applicable administration, facility, and program offices that have developed their own record-retention schedules to then adhere to the newly established record-retention period  (Recommendation 5) The Department of Veterans Affairs (VA) Inspector General should revise its policy to include a requirement to verify whether evidence produced in senior-official case referrals demonstrates that the six elements required in VA Directive 0701 have been addressed  (Recommendation 6) The Secretary of Veterans Affairs should direct the Office of Accountability and Whistleblower Protection (OAWP) to review responses submitted by facility or program offices to ensure evidence produced in senior-official case referrals demonstrates that the six elements required in VA Directive 0701 have been addressed  (Recommendation 7) The Secretary of Veterans Affairs should direct OAWP to issue written guidance on how OAWP will verify whether appropriate disciplinary action has been implemented for all substantiated misconduct by senior officials  (Recommendation 8) The Secretary of Veterans Affairs should direct OAWP to develop a process to ensure disciplinary actions proposed in response to findings of misconduct are recorded within appropriate information systems to maintain their relevance and value to management for making decisions and take steps to monitor whether the disciplinary actions are implemented  (Recommendation 9) The Secretary of Veterans Affairs should direct OAWP to issue written guidance on how OAWP will review the disposition of accountability actions for all substantiated misconduct cases involving senior officials resulting from investigations  (Recommendation 10) The Secretary of Veterans Affairs should implement internal controls to ensure that proper adherence to separation-of-duty standards involving the removal of an employee are consistent with policy  (Recommendation 11) The Secretary of Veterans Affairs should develop oversight measures to ensure all investigations referred to facility and program offices are consistent with policy and reviewed by an official independent of and at least one level above the individual involved in the allegation  To ensure independence, referred allegations of misconduct should be investigated by an entity outside the control of the facility or program office involved in the misconduct  (Recommendation 12) The VA Inspector General, in consultation with the Assistant Secretary of OAWP, should develop a process to ensure that OSC case numbers are linked to the investigative case number and final report  (Recommendation 13) The Secretary of Veterans Affairs should direct OAWP to develop a time frame for the completion of published guidance that would develop an internal process to monitor cases referred to facility and program offices  (Recommendation 14) The Secretary of Veterans Affairs should ensure that employees who report wrongdoing are treated fairly and protected against retaliation  (Recommendation 15) The Secretary of Veterans Affairs should direct OAWP to develop a process to inform employees of how reporting lines operate, how they are used, and how the information may be shared between the OSC, the OIG, OAWP, or VA facility and program offices when misconduct is reported  (Recommendation 16) We provided a draft of this report to the Department of Veterans Affairs (VA), VA Office of Inspector General (OIG), and the Office of Special Counsel (OSC) for review and comment  In its comments, VA concurred with nine of our recommendations and partially concurred with five (see app  VI for a copy of VA’s letter)  Regarding our recommendations to the Inspector General, the OIG concurred with one recommendation and partially concurred with the other  The OIG also provided comments on our findings (see app  VII for a copy of the OIG’s letter)  We received technical comments by e-mail from OSC’s Principal Deputy Special Counsel, which we incorporated in the report as appropriate  Regarding VA’s comments, in its response to our first recommendation that the Secretary develop and implement guidance to collect complete and reliable misconduct and associated disciplinary-action data department-wide, VA concurred and outlined steps it plans to take to address our recommendations  These steps include the creation of new policies to address blank data fields, lack of personnel identifiers, lack of standardization among fields, and accessibility issues related to misconduct and associated disciplinary-action data department-wide  The target date for system implementation, dependent on approved funding and acquisition-related requirements, is January 1, 2020  On our second recommendation, that the Secretary direct applicable facility and program offices to adhere to VA’s policies regarding employee-misconduct adjudication documentation, VA concurred  It stated that a memorandum will be distributed to reiterate facility and program-office requirements to adhere to VA Handbook 5021, Employee/Management Relations, no later than October 1, 2018  VA also concurred with our third recommendation, that the Secretary direct the Office of Human Resource Management (OHRM) to routinely assess the extent to which misconduct-related files and documents are retained  According to VA, OHRM will assess, during periodic Oversight and Effectiveness Service reviews, the extent to which misconduct- related files and documents are retained  The first assessment is to be incorporated into the fiscal year 2019 Oversight and Effectiveness Service schedule no later than November 1, 2018  VA concurred with our fourth recommendation, that the Secretary direct OHRM to assess whether human-resources personnel adhere to basic principles outlined in VA Handbook 5021  VA stated that OHRM will assess, during periodic Oversight and Effectiveness Service reviews, whether human-resources and administration personnel adhere to basic principles outlined in VA Handbook 5021  The first assessment is to be incorporated into the fiscal year 2019 Oversight and Effectiveness schedule no later than November 1, 2018  In its response to our fifth recommendation, that the Secretary adhere to Office of Personnel Management (OPM) and National Archives and Records Administration (NARA) guidance and establish a specific record- retention period for adverse-action files, VA concurred and indicated that the Human Resources and Administration Assistant Secretary will establish VA guidance regarding the retention period for adverse-action files  In addition, the Human Resources and Administration Assistant Secretary is to advise applicable administration, facility, and program offices that have developed their own record-retention schedules to adhere to the newly established directive  According to VA, the directive will be established no later than November 1, 2018  VA partially concurred with our seventh recommendation, that the Secretary direct departmental heads to review responses submitted by facility or program offices to ensure evidence produced in senior-official case referrals demonstrates that the six elements required in VA Directive 0701 have been addressed  VA stated that the process described in our report pertaining to OIG findings or results will be changed to require all such reports to be submitted to OAWP  VA also indicated that it expects to publish new guidance by October 1, 2018, that will require the Office of Accountability and Whistleblower Protection (OAWP) to review responses and recommendations from facilities or program offices  Given VA’s comments, we have revised our draft recommendation to have the Secretary direct OAWP, not the department heads, to ensure evidence demonstrates that the six elements have been addressed  VA also partially concurred with our eighth recommendation, that the Assistant Secretary of OAWP review all substantiated misconduct by senior officials to verify whether disciplinary action has been implemented  VA stated that all substantiated misconduct by senior leaders in VA is handled by OAWP from intake, through investigation, working with the proposing and deciding officials  VA also stated that it expects to publish written guidance by October 1, 2018, that will clarify how OAWP will work with the appropriate servicing personnel office to ensure that the recommended disciplinary actions decided are implemented for substantiated misconduct involving senior officials  Given VA’s comments, we have revised our draft recommendation to have the Secretary of Veterans Affairs direct OAWP to issue written guidance on how OAWP will verify that appropriate disciplinary action has been implemented for all substantiated misconduct by senior officials  VA partially concurred with our ninth recommendation, that the Assistant Secretary of OAWP develop a process to ensure disciplinary actions proposed are recorded within appropriate information systems  VA stated that the VA-wide discipline tracking system currently used by OAWP will eventually be phased out  It added that once the Human Resources Information System (HRSmart) is capable of capturing and recording similar data, it will be used for this purpose  Accordingly, we have not revised our draft recommendation  VA partially concurred with our 10th recommendation, that the Assistant Secretary of OAWP assess all misconduct cases involving senior officials to ensure investigative reports with findings of substantiated misconduct include recommendations for action  According to VA, OAWP has instituted several processes since our review  VA plans to issue written guidance that outlines the process for the review and disposition of appropriate accountability actions for allegations of misconduct by senior officials by October 1, 2018  Given VA’s comments, we have revised our draft recommendation to have the Secretary of Veterans Affairs direct OAWP to issue written guidance on how OAWP will review the disposition of accountability actions for all substantiated misconduct cases involving senior officials resulting from investigations  In its response to our 11th recommendation, that the Secretary implement internal controls to ensure that separation-of-duty standards involving the removal of an employee are consistent with policy, VA concurred  It stated that it will also establish and distribute internal controls to ensure that separation-of-duty standards involving the removal of an employee are consistent with policy no later than November 1, 2018  VA partially concurred with our 12th recommendation, that the Secretary take steps to ensure independence of referred allegations of misconduct by requiring that investigations be conducted outside the control of the facility or program office involved in the misconduct  VA stated that OAWP is responsible for recording, tracking, reviewing, and confirming the implementation of recommendations from audits and investigations  However, VA did not address how it will ensure the independence of the entity responsible for conducting an investigation  As we discuss in our report, during the review OAWP officials stated that the process of referring cases of misconduct back to facilities and program offices where the misconduct occurred will continue  Accordingly, we have not revised our draft recommendation and believe implementation of it will help ensure independence  VA concurred with our 14th recommendation, that the Assistant Secretary of OAWP develop a time frame for the completion of published guidance for the development of an internal process to monitor cases referred to facility and program offices  VA provided an expected date of October 1, 2018, for publishing the internal VA guidance, with the subsequent Directive and Handbook to be published as rapidly as staff coordination permits  In its response to our 15th recommendation, that the Secretary ensure that employees who report wrongdoing are treated fairly and protected against retaliation, VA concurred  It stated that OAWP and OSC have developed a functional process to ensure whistle-blower protections are implemented, but did not indicate what the process entails  The VA Secretary has also delegated authority to the Executive Director, OAWP, to put individual personnel actions on hold if the actions appear motivated by whistle-blower retaliation  VA added that OAWP has also hired two whistle-blower program specialists specifically to increase awareness of whistle-blower protections and work with individuals that disclose employee wrongdoing to ensure individuals are treated fairly and protected from retaliation for their disclosures  VA concurred with our 16th recommendation, that the Assistant Secretary of OAWP develop a process to inform employees of how reporting lines operate  VA stated that it will provide whistle-blower training to all employees on a biennial basis, which will include the reporting lines for disclosures of wrongdoing, the manner in which disclosures flow once they are made, how information is shared among the whistle-blower entities, and what protections exists for those who disclose wrongdoing  Regarding our recommendations to the Inspector General, the OIG partially concurred with our sixth recommendation, to revise its policy to include a requirement to verify whether evidence produced in senior- official case referrals demonstrates that the six elements required in VA Directive 0701 have been addressed  The OIG indicated that VA Directive 0701 is currently being updated to require a written or electronic signature from the person preparing the responses as an attestation that the specific requirements of the directive were met  The OIG also indicated in its letter that the OIG’s Hotline staff carefully review the case response but Hotline staff are not required to request an updated response from VA to address matters not necessary to the resolution to the referral  The OIG asserted that requesting an update would detract from the resources for other important VA activities  On page 4 of the OIG’s letter, the OIG states that Hotline analysts are allowed to exercise some discretion in accepting responses that may include minor departures from the six elements  We continue to believe that, in order to have a complete response to a referral, all six elements required by the directive should be addressed  In addition, Directive 0701 does not allow for the use of professional judgement to decide which elements to include or not to include in a response  While we agree that requiring a written or electronic signature from the person preparing the responses as an attestation will help ensure that the specific requirements of the directive were met, we maintain that not requiring Hotline analysts to review responses to ensure that all elements of the directive are addressed is inconsistent with the intent of the directive  In its response to our 13th recommendation, that the OIG develop a process to ensure that an OSC case number is linked to the investigative case number and the final report, the OIG concurred  It stated that it will engage with the Executive Director of OAWP to develop a process to ensure that OSC case numbers are linked to OIG and OAWP investigative case numbers, as appropriate, and linked to any final report of investigation  In addition to its response to recommendations, the VA OIG also raised a number of concerns with our findings  Page 1 of its letter summarizes some of these concerns and then provides more detail on each concern raised, starting on page 2  Our responses to each of these detailed concerns are provided below  The OIG stated that our report does not focus on the most important cases, but focuses primarily on case referrals regarding senior officials that were not handled by the OIG because the allegations were lower risk or because of resource constraints  In addition, the OIG stated that GAO risks presenting a skewed picture of the OIG’s oversight work  We disagree with this characterization of our findings  We requested that the OIG provide us with data from the OIG’s Master Case Index (MCI) information system that would allow us to select a sample of cases, in accordance with the scope of our review  The OIG was unable to provide the requested information due to several reasons  Instead, the OIG provided data from the OIG Hotline and Office of Investigations case- management systems (subsystems within MCI) that contained a limited number of fields for analysis and 23 cases pertaining to SES misconduct that were referred to VA for investigation during GAO’s period of review  Therefore, as we discuss in the report, we were only able to review the 23 senior-official misconduct cases included in our report because the OIG was only able to provide related documentation for these cases  The OIG stated that we only reviewed a sample of just 23 case referrals from fiscal years 2011 through 2014  As described, we reviewed all 23 senior-official misconduct cases that were referred to VA for investigation that the OIG was able to provide us, not a sample  The OIG stated that our report inaccurately states that the extracts received from the MCI information system contained missing information  We disagree with this characterization of our findings  Our review included a comprehensive assessment of the reliability of the OIG’s data  To conduct this assessment, we requested an explanation of each data field to clarify when fields are normally populated and how they are used  Our findings are consistent with the information provided in response to this request  For example, the OIG’s response to our data-reliability assessment stated that the data field used to identify the type of allegations being investigated should never be blank  However, we found that field to be blank in some cases in the data that were provided to us, though the OIG asserted that the MCI information system is a relational database where each case may be associated with multiple allegations and codes  In response to the OIG’s comments on our report, we requested supporting documentation to demonstrate that the fields analyzed during our period of review did not contain missing data  The OIG provided the MCI information system user’s manual that contains detailed procedures for accessing and entering data into the MCI information system, and a compilation of various internal documents  However, the documentation did not provide evidence of the completeness of data entered into the MCI information system as part of quality-assurance reviews performed by the OIG or other designated entity  Absent evidence of data-quality reviews aimed at assessing the accuracy and completeness of data contained in the MCI information system, we did not change the conclusions based on our previous analysis  The OIG stated that our report provides incomplete information regarding sampled cases and mischaracterized one of the OIG’s case referrals in the body of the report  We disagree with this characterization of our findings  Specifically, the OIG said that we inaccurately stated that a medical center director conducted the investigation into his own alleged misconduct and found no allegations were substantiated  The synopsis included in our report clearly articulates that the medical center director was named in the allegation for having received a similar complaint involving time and attendance abuse by a physician  The medical center director, who provided the response to the OIG, was implicated in the allegation as having not addressed a similar time and attendance complaint regarding the same physician 2 years earlier  The OIG did not provide any supporting documentation to demonstrate that the alleged time and attendance abuse allegations against the physician were not substantiated  The OIG stated that our report inaccurately stated that the medical center director conducted his own investigation of himself and found no allegations were substantiated  We disagree  In response to the OIG’s comments on our report, we requested that the OIG provide additional support used to determine that the medical center director did not investigate the allegation in which he was named  The additional case documentation provided by the OIG further reaffirmed our assessment that the medical center director performed his own investigation and found no allegations were substantiated  Additional documentation provided by the OIG indicated that the OIG referred the case to the Veterans Integrated Service Network (VISN) for a response  However, documentation we examined during the course of our audit, and the documentation provided in response to our draft report, indicates that the medical center director performed the investigation of the allegations and then the results were routed through the VISN back to the OIG  The OIG stated that routing the response through the VISN should address our concerns of independence  This process does not address our concerns regarding independence because VA Directive 0701 states that all responses to Hotline case referrals must contain evidence of an independent review by an official separate from and at a higher grade than the subject / alleged wrongdoer  In this case, the name of the medical center director who signed the facility response provided to the OIG was the same individual named in the allegations  The OIG stated that the report does not provide a balanced presentation of the rigor with which the OIG reviews all incoming Hotline contacts and case responses  We disagree with this characterization of our findings  As described above, the OIG was unable to provide comprehensive data to select a sample of OIG audits, evaluations, and inspections for review due to the limitations cited  We focused on misconduct involving senior officials consistently with the scope of our review and thoroughly reviewed all 23 senior-official misconduct cases that were referred to VA for investigation, which were the only cases that the OIG was able to provide  The OIG stated that the description for one of the cases included in appendix V of the draft report was incomplete because we misunderstood the OIG’s process  We disagree with this characterization  In response to the OIG’s comments on our report, we requested additional supporting documentation  The documentation provided reaffirmed our assessment that another medical center director performed his own investigation and found no allegations were substantiated  Similar to the case described above, the medical center director completed his own investigation and then routed the response through the VISN back to the OIG  In contrast to the previous case, however, the Hotline Workgroup reviewed the response to the OIG and found it to be insufficient  Specifically, the OIG noted that the medical center director who provided the response was the subject of the complaint, despite the response being directed to the VISN, and requested clarification  The VISN informed the OIG that it is not its policy for the complainant to have any involvement in the review and data submission on a case in which the complainant is involved  The VISN stated that while this did obviously occur in this instance, it has taken steps to ensure it does not occur in the future  The supporting evidence provided for the case included in appendix V also contradicts the OIG’s previous assertion that the routing of a response through the entity with oversight (VISN) over the medical center director should have addressed GAO’s concerns of independence  The OIG stated our report provides a misrepresentation of the OIG’s failure to follow internal policies for department responses  We disagree  According to OIG statements on page 4 of the OIG’s letter, Hotline analysts are allowed to exercise some discretion in accepting responses that may include minor departures from the six elements required in VA Directive 0701  We continue to believe that in order to have a complete response to a referral, all six elements required by the directive should be addressed  On the basis of our review, the OIG does not have an effective method to ensure that cases referred to VA are reviewed in accordance with VA Directive 0701  Out of the 23 cases we reviewed, only 4 included sufficient documentation needed to support VA’s findings, and we could not identify a case that contained all six elements required in VA Directive 0701  This suggests that the current OIG review process is not adequately resolving case referrals, as asserted by the OIG’s response  In addition, VA Directive 0701 does not currently include a provision that would allow Hotline analysts to deviate from the six required elements  As stated in our report, the OSC also raised concerns regarding 40 percent of disclosure cases that were referred to VA facility and program offices  The OIG stated that much of the information in the draft report is dated and ignores system updates, specifically several key Hotline-related process improvements since 2014  Although our review began in 2015, we disagree with this characterization of our findings  In our report, we included relevant improvements to demonstrate where the OIG was able to provide support for those improvements  For example, our report discusses: (1) a new process for communicating the scope of reviews that involve matters referred by the OSC to the Office of the Secretary, (2) a description on the VA website of the process for employees who wish to report criminal or other activity to the OIG, (3) a new Enterprise Management System, and (4) a new process for receiving whistle-blower disclosures by the Secretary  In response to the OIG’s comments on our report, we requested additional documentation for any systems, practices, or personnel changes that have been implemented since 2011, including improvements to Hotline-related processes since 2014 that were not included in our report  In response, the OIG provided a copy of the OIG’s organizational chart (current as of Apr  23, 2018), described the oversight responsibilities of each OIG component, and summarized the pertinent staff positions within each component  On the basis of this documentation, we identified a new office (the Office of Special Reviews), a promotion, staff reassignments, and numerous vacancies during our review period  However, the OIG did not provide evidence of any measures to improve the MCI information system, case-referral processes, or relevant staff roles that were not already included in our report  As described above, the OIG was unable to provide comprehensive data to select a sample of OIG audits, evaluations, and inspections for our initial review due to the limitations cited  In response to the OIG’s comments on our report, we requested documentation related to any significant changes that have been made to the MCI information system that allows the OIG to identify all allegations of misconduct for export and analysis  The OIG provided additional information regarding overall departmental achievements that are highlighted in the OIG’s Semiannual Report to Congress, and other products from its website published between fiscal years 2011 through 2018  We recognize the OIG’s broader administrative and oversight work described in the published reports  However, this information does not address changes specifically made to the MCI information system that would enable the OIG to analyze cases pertaining to alleged misconduct by senior officials that we requested  We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-5045 or larink@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  Our objectives were to determine the extent to which the Department of Veterans Affairs (VA) (1) collects reliable information associated with employee misconduct and disciplinary actions that is accessible and could be used to analyze misconduct department-wide; (2) retains documentation that demonstrates VA adheres to its policies when adjudicating cases of employee misconduct; (3) ensures allegations of misconduct involving senior officials are reviewed in accordance with VA investigative standards, and these officials are held accountable; (4) has procedures to investigate whistle-blower allegations of misconduct; and the extent to which (5) data and whistle-blower testimony indicate whether retaliation for disclosing misconduct occurs at VA  For the first objective, we obtained VA employee misconduct data from 12 information systems operated by various VA components covering October 2009 through July 2017, where available  To determine the reliability of VA’s misconduct data, we analyzed the contents of the 12 information systems operated by various offices across VA  These data encompass each of the three major administrations that constitute VA— the National Cemetery Administration (NCA), Veterans Benefits Administration (VBA), and Veterans Health Administration (VHA)  We selected the information systems based on our discussions with VA officials and staff that oversee the data and, hence, identified databases capable of collecting information pertaining to either employee misconduct or disciplinary actions  Data fields were selected based on whether they would provide beneficial information to better understand the disciplinary process  VA’s Personnel and Accounting Integrated Data (PAID) system, which was developed to track payroll actions, contains information about adverse disciplinary actions that affect employee salary department-wide  We obtained an extract of all adverse disciplinary actions from the PAID system  We assessed the reliability of each system for the purposes of identifying and tracking misconduct cases  To do this, we performed electronic tests on 12 information systems to determine the completeness and accuracy of the fields contained in the data files  We also submitted to the overseeing offices for all 12 information systems general data-quality questions regarding the purpose of the data, their structure, definitions and values for certain fields, automated and manual data-quality checks to ensure the accuracy of the data, and limitations  As discussed further, the data were generally not reliable for a department-wide assessment of all misconduct and disciplinary actions due to the lack of completeness and compatibility of the data across all information systems  VA staff could not confirm whether some of the missing data we identified were artifacts of the database extraction process VA used to assemble the data files we used in our review  Despite challenges with aspects of the data, we found the data sufficiently reliable for conducting analysis where fields were populated and field definition concurrence was obtained by program offices  For the second objective, we selected a generalizable stratified random sample of 544 misconduct cases from October 2009 through May 2015  Where available, we reviewed the employees’ disciplinary-action files and Electronic Official Personnel Folders to determine the extent to which VA’s actions were consistent with disciplinary policy outlined in VA Handbook 5021, Employee/Management Relations  These data encompass each of the three major administrations that constitute VA— NCA, VBA, and VHA  We determined the data to be sufficiently reliable for analysis of disciplinary actions affecting salary that resulted from misconduct that was not reported to supervisors directly from employees  Accordingly, our sample only includes misconduct cases that resulted in a change in salary or were reported to departmental organizations within the 12 information systems selected  We developed a data-collection instrument to document the results of our case reviews  We revised our data-collection instrument to address issues found during the course of our analysis, and developed a companion document that outlined the decision rules for reviewing cases  We also designated two primary reviewers to ensure the decision rules were consistently applied across all cases  Our review of laws and regulations revealed that disciplinary rules sometimes vary depending on whether employees fall under Title 5, Title 38, or hybrid Title 5 and Title 38 hiring authority  To minimize confusion associated with these differences, we incorporated criteria into our data- collection instrument  In addition, we were unable to obtain complete case information for 25 percent of the cases  For these cases, we obtained direct access to the Office of Personnel Management’s (OPM) Electronic Official Personnel Folders system to attempt to recover some of the missing information  Ultimately, we were unable to complete our review for 10 percent of cases in our sample because of missing files  In addition to reporting missing case information, we used our generalizable analysis results to project VA-wide figures for several data elements that were not in compliance with VA policy  Unless otherwise noted, estimates in this report have a margin of error of +/-7 4 percentage points or less for a 95 percent confidence interval  For the third objective, we analyzed data from the Office of Accountability Review (OAR) Legacy Referral Tracking List and VA Office of Inspector General (OIG) case-referral and investigative case-management systems, and we selected cases for in-depth review  We selected these two systems based on discussions with VA officials who were knowledgeable with databases that have the capacity to track misconduct information pertaining to senior officials  The OAR Legacy Referral Tracking List comprises referrals from January 2011 through May 2015  The OIG provided 23 case-referral files involving senior officials from calendar years 2011 through 2014  As part of our review of the OIG case files, we evaluated specific data elements contained in VA’s response documents using VA policy for referring and reviewing misconduct cases  We assessed the reliability of the OAR Legacy Referral Tracking List and OIG case-management systems for the purposes of identifying and tracking misconduct cases  To do this, we performed electronic tests on each database to determine the completeness and accuracy of the fields contained in the data files, including senior-official indicators  Where feasible, we opted to match individual datasets to PAID to determine whether disciplinary actions were administered as prescribed  We also submitted to OAR and the OIG general data-quality questions regarding the purpose of the data, their structure, definitions and values for certain fields, automated and manual data-quality checks to ensure the accuracy of the data, and limitations  On the basis of this information, we found the OAR data to be sufficiently reliable for conducting analysis where fields were sufficiently populated  For the OAR data, we matched the persons of interest to adverse-action files from PAID to determine whether adverse disciplinary actions were administered as prescribed during the available time frame (January 2011 through May 2015)  We also obtained VA’s response documents for the 23 case-referral files provided by the OIG to evaluate whether VA was adhering to its own policy for referring and reviewing misconduct cases  Through our OAR Legacy Referral Tracking List analysis, we identified illustrative case examples of misconduct involving senior officials  Further, based on our evaluation of the 23 OIG case referrals using VA’s referral policy, we developed several illustrative case examples  For the fourth objective, we interviewed senior officials from VA and the OSC responsible for investigating whistle-blower complaints  We obtained the OSC’s procedures for referring disclosure complaints and VA’s policy for investigating these complaints once received at the agency  In addition, we obtained whistle-blower disclosure data from the Office of Special Counsel (OSC) covering calendar years 2010 through 2014  To determine the reliability of the data, we conducted electronic testing and traced data elements to source documentation  We determined the data to be sufficiently reliable to identify the total number of cases that were investigated by the OIG, or referred to facility and program offices  We also observed a course to assess VA’s training provided to VA employees conducting investigations  We identified 135 OSC disclosure cases for analysis based on two criteria: (1) they contained at least partial complainant information (i e , the allegations were not anonymously reported or could be identified with supplemental information) and (2) they contained an indicator that the case had been closed by the OSC pending an ongoing investigation by VA or the OIG  These cases represent the universe of VA disclosures accepted by the OSC  Of the 135 disclosure cases referred to VA, 53 cases were referred to VA facility and program offices for further investigation  The remaining 82 disclosure cases indicated that they were investigated by the OIG  We reviewed the results of OSC’s assessment of investigative documentation developed by VA for these whistle-blower disclosure cases  For the fifth objective, we analyzed the 135 whistle-blower disclosure cases obtained from the OSC  These cases represent the universe of VA disclosures accepted by the OSC from calendar years 2010 through 2014, which were investigated by VA  We obtained an extract of year-end rosters from the PAID system as of September for fiscal years 2010 through 2014, with a final extract through May 30, 2015  Finally, we interviewed representatives from whistle-blower advocacy groups, as well as established whistle-blowers who disclosed wrongdoing or retaliation at VA and who were referred to us by one advocacy group  Of the 135 disclosures received by the OSC, a total of 129 employees made a total of 130 disclosures nonanonymously  For these 130 disclosure cases, we reviewed OSC and OIG investigative reports, as well as PAID roster files, to gather additional information to perform analysis of potential retaliation  We also interviewed six individual whistle- blowers with formal disclosure cases accepted by the OSC, indicating that the OSC had previously reviewed the case and determined that it contained sufficient evidence and merit to warrant further investigation  Our analysis of potential retaliation comprised two parts  First, we compared the 129 employees associated with the selected OSC cases to the PAID rosters using the complainants’ information to determine whether employees associated with the selected OSC cases were more likely to leave the agency  We identified the overall count and proportion (across years) of roster-matched employees who made a disclosure between fiscal years 2010 through 2014 but were not employed at VA the following fiscal year  Second, to determine whether employees associated with the selected OSC cases were more likely to receive disciplinary action, we also calculated the yearly totals and proportion of roster-matched employees identified above for whom a record existed in the PAID disciplinary action information system  We did this by comparing the proportion of employees who received one or more disciplinary actions in the year prior to the appearance in the roster, in the same fiscal year as the roster, and in the subsequent fiscal year  We also completed this analysis utilizing the PAID roster file to determine the yearly proportion of all VA employees who left the agency  On the basis of the results of our analysis, we reported by fiscal year the percentage of whistle-blowers that received disciplinary action or left VA at a higher rate than the overall VA population following a disclosure  To address all objectives, we interviewed senior officials from VA’s major components responsible for investigating and adjudicating cases of employee misconduct  We also reviewed standard operating procedures, policy statements, and guidance for staff charged with investigating and adjudicating allegations of employee misconduct  We conducted this performance audit from January 2015 to May 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  This data file is designed to track referrals made to OAR, including allegations of misconduct related to senior officials  Through our analysis of this spreadsheet, we identified 11 fields out of 92 that were missing information that could be used to analyze misconduct  Complainant #1 (First Name) (20 percent of 1,245 blank)— According to OAR, this field is populated when there is a known complainant for a matter  Some matters referred to OAR may be anonymously disclosed and not contain complainant information  This file also contains a case-origin field that specifies whether a case was anonymous  Our review of both the complainant and case-origin field indicated that only 11 percent of the complaints were generated from an anonymous source, and the remaining records should have included a complainant name  Disciplinary Action (96 percent of 1,245 blank)—This field should be populated to indicate whether a disciplinary action was taken after the completion of an investigation  Grade (97 percent of 1,245 blank)—According to OAR, this field should be populated with the grade of the Person of Interest (POI), if known  Our review of this field indicated that 97 percent were blank, therefore we were unable to analyze the variation in grade level for officials who were the subject of complaints  OAR Action (77 percent of 1,245 blank)—According to OAR, this field should be populated as an internal reference to describe what stage in the administrative process the matter was in when received  The lack of information did not allow for analysis of the types of actions taken for each case  Person of Interest (POI) (47 percent of 1,245 blank)—This field specifies the first and last name of the Department of Veterans Affairs (VA) employee who is the subject of an allegation  POI (Person of Interest) Last Name 1–5 (51 to 99 percent of 1,245 blank)—This field specifies the last name of VA employee who is the subject of an allegation  There are a total of five person-of-interest (POI) fields for each matter  According to OAR, blank POI fields occur when the case has fewer than five POIs or the POI was not specified in the matter referred  Our review of the five POI fields indicated that more than half of the records did not contain at least one POI because no individual was specified  The lack of information did not allow for further investigation of senior-level officials involved in misconduct  Proposed Action (97 percent of 1,245 blank)—According to OAR, this field should be populated to notify OAR staff if any disciplinary action was proposed  Our review of this field indicated that 97 percent were blank, suggesting that very few high-level officials received corrective action, or the field was not consistently completed for each record  Due to the large share of blank values, the data posed limitations when analyzing how many senior officials received corrective action as a result of a complaint  This data file is designed to track misconduct and disciplinary actions taken against VA employees  Through our analysis of this spreadsheet, we identified 9 fields out of 21 that were missing information that could be used to analyze misconduct  Action Taken (3 percent of 9,851 blank)—According to OAR, this field should be populated with the action that the deciding official takes, with the exception of pending actions  If there is a pending action, this field will remain blank  Our review of the action-taken field found three records that were annotated as a “pending decision” within this field, which indicates that there is an option for entering information into this field when there is an action pending and the field should never be blank  Admin Leave (30 percent of 9,851 blank)—According to OAR, this field should be populated if an employee is placed on administrative leave while an adverse action is pending  If an employee is not placed on administrative leave, the field may be left blank  Our review of the admin-leave field found that about 66 percent of the records were annotated as “no” within this field, which indicates that there is an option for entering information into this field when an employee was not placed on administrative leave  Date Proposed (11 percent of 9,851 blank)—According to OAR, this field is used when an adverse action is proposed for an employee  There are some actions that are not proposed, such as probationary terminations or admonishments that may be taken without being proposed, and therefore result in this field being blank  Our analysis found that about 95 percent of these records containing blank proposed date fields also had an entry in the proposed adverse-action field, which contained such entries as removals, suspensions, and demotions that require a proposed date  Deciding Official (14 percent of 9,851 blank)—According to OAR, this field should be populated with the name of the official who issued the action taken  Effective Date (5 percent of 9,851 blank)—According to OAR, this field should be populated with the date of the action taken  Some entries will not have an effective date if an entry is pending decision  Also, if no action is taken, the decision was counseling, or the proposed action was rescinded, this field may not have an effective date  Our review found that about 14 percent of the cases that included adverse actions, such as a suspension, removal, reassignment, or demotion, and that should have included a date of action, were blank  Offense 2 and 3 (79 and 95 percent of 9,851 blank)—According to OAR, this field tracks the second- and third-most-significant charge against the employee when applicable  Proposing Official (16 percent of 9,851 blank)—According to OAR, this field should be populated with the name of the official who makes the proposed adverse action  Instances where a proposing official has left the agency at the time of entry and could not be found in the lookup feature that relies on the e-mail global address list may produce blank fields  Also, disciplinary actions that were taken without proposal would not have a proposing official  In these instances, the human-resources specialists who enter the actions are instructed to include the name in the other-comments box  Our review of these records indicated that a majority of records lacking an entry in the proposing official field also lacked an annotation in the other- comments field to accurately identify the proposing official  Settlement (14 percent of 9,851 blank)—According to OAR, this field tracks whether a settlement agreement occurred  This data file is designed to collect allegations of criminal activity, waste, abuse, and mismanagement received by the OIG Hotline Division  Through our analysis of this information system, we identified one field out of seven that was missing information that could be used to analyze misconduct  Nature of Complaint (54 percent of 896 blank)—According to the OIG, this field should contain a brief description of the issue that most closely matches the allegation  Each case can have more than one nature of complaint and corresponding administrative action, if any  OIG officials stated that this field identifies the type of allegations being investigated and should never be blank  Our review of these cases found that over half of the cases involving the OIG contained entries for administrative action taken, but the nature-of-complaint fields corresponding to these actions were blank  This data file is designed to track misconduct and disciplinary action taken against VBA employees  Through our analysis of this spreadsheet, we identified 3 fields out of 20 that were missing information that could be used to analyze misconduct  Alleged Offense 2 and 3 (92 and 99 percent of 1,375 blank)— According to VBA officials, this field should be populated if an individual is charged with multiple offenses, or has additional offenses in the same reporting period  In most instances, there is typically only one offense at the time of reporting  Sustained (52 percent of 1,375 blank)—According to VBA, this field should be populated if an offense is sustained at the time of reporting  This data file is designed to track all allegations of misconduct and associated disciplinary actions taken against VA employees  Through our analysis of this spreadsheet, we identified 8 fields out of 34 that were missing information that could be used to analyze misconduct  Deciding Official (3 percent of 5,571 blank)—According to OAWP, this field should be populated with the name of the official who makes the decision for adverse action  Detail Position (89 percent of 5,571 blank)—According to OAWP, this field should be populated with the position an employee was detailed to if removed from official position  Offense 2 and 3 (69 and 91 percent of 5,571 blank)—According to OAWP, this field tracks the most-significant charges against the employee  If there are fewer than three charges, these fields are left blank  Offense 1 Sustained (14 percent of 5,571 blank)—According to OAWP, this field should be populated if an individual’s first offense has been sustained  Offense 2 and 3 Sustained (73 and 91 percent of 5,571 blank)— According to OAWP, these fields should be populated if an individual’s second and third offenses have been sustained  The majority of cases only involve one offense  Proposing Official (9 percent of 5,571 blank)—According to OAWP, this field should be populated with the name of the official who makes the proposed adverse action  This data file tracks Equal Employment Opportunity (EEO) discrimination complaints  Through our analysis of this information system, we identified 1 field out of 66 that did not have standardization that could be useful to analyze misconduct  Employment—We found that the values for this field were not mutually exclusive, or independent of one another  For example, this field includes two distinct categories of information: employment status, such as full time or part time; and hiring authority, such as Title 5 or Title 38  This method of storing information resulted in undercounting each of the separate values due to the system’s failure to account for expected overlap  For instance, an employee could be both a full-time and Title 5 employee and the field only tracks one or the other  ORM officials stated that this field has since been modified to capture more options to account for the overlap  This data file is designed to track misconduct and disciplinary actions taken against Department of Veterans Affairs (VA) employees  Through our analysis of this spreadsheet, we identified 1 field out of 21 that did not have standardization that could be useful to analyze misconduct  Position—We found the VA-Wide Adverse Employment Action Database contained variations within this field, such as multiple values for the “Cemetery Caretaker” position name  According to OAR, this field is a free-text field, and the office conducts manual searches to review and analyze position titles when needed  Our review found that the different variations in position titles made it difficult to successfully determine the frequency and nature of allegations by position  This is an information system designed to collect allegations of criminal activity, waste, abuse, and mismanagement received by the OIG Hotline Division  Through our analysis of this data file, we identified 1 field out of 7 that did not have standardization that could be useful to analyze misconduct  Nature of Complaint—Our review of the Master Case Index file found variations of similar values in this field  For example, this field contained 21 different claim types pertaining to similar types of fraud, which made it difficult to assess the frequency and nature of claims entered into the system  OIG officials stated that they do not attempt to account for these variations or assess the frequency of use because they are assigned based on a “best match” to the allegations of the case  This data file is designed to track misconduct and disciplinary action taken against VBA employees  Through our analysis of this spreadsheet, we identified 1 field out of 20 that did not have standardization that could be useful to analyze misconduct  Position—Our review found some duplication and overlapping values among this field  For example, the position title for “service representative” contained 21 similar categories with numerous variations in spelling (i e , Veteran Service Representative vs  Veterans Service Representative, and Rating Veteran Service Representative vs  Rating Veterans Service Representative)  We were unable to verify the number of distinct positions due to the lack of standardization within this field  This data file is a tracking spreadsheet for monitoring misconduct and disciplinary action workload  Through our analysis of this spreadsheet, we identified 5 fields out of 12 that did not have standardization that could be useful to analyze misconduct  Action Proposed/Decided/Taken—We were unable to analyze this data field because the action taken was tracked in one single field and updated with the most-recent action, rather than each distinct action being entered in a separate field  Consequently, we were not able to distinguish those cases where corrective action may have been taken, to verify whether the corrective action had been implemented  Current Status—We were unable to analyze this data field because it was not a standardized field  For example, we were unable to determine the total number of cases that were closed, open, or pending due to the variations in the data field (e g , Open, open, open – pending, open-pending)  Full Name of Employee, Grievant, Appellant, Complainant, Non- Employee—We were unable to distinguish whether the individual filing a complaint was an employee, grievant, appellant, complainant, or nonemployee because the information entered into this single field only provided the employee’s full name and did not provide a distinction as to which category the record was assigned, as indicated by the field name  NCA Facility—We were unable to analyze this data field because it was not a standardized field  For example, we were unable to run demographic information on the different facilities involved because this field contained erroneous information  Examples of erroneous information included the name of the Memorial Service Network in one case, and the region, rather than the facility name, in another  Supervisor Name—We were unable to analyze this data field because it was not a standardized field  For example, we were unable to determine the total number of supervisors that were associated with each case due to the variations in the names entered within this field, which included misspelled first or last names, addition/omission of middle initials, or no first name  This data file is a tracking spreadsheet for all allegations received by the VA Secretary regarding misconduct, patient care, or other wrongdoing  Through our analysis of this spreadsheet, we identified 1 field out of 9 that did not have standardization that could be useful to analyze misconduct  Subject—Our review of this tracking spreadsheet found over 380 different possible categories that could be assigned to one record  These categories contained a significant number of variations  For example, we found 38 different categories that contained possible EEO-related issues such as “EEO/Whistleblower,” “Potential EEO,” and “EEO Violations ” We were unable to distinguish the different subject categories for this field due to the lack of standardization  CSRT officials stated that ExecVA reports contain only data corresponding to specific search criteria  This is an information system for tracking allegations of misconduct at all VA facilities that include violation of law and misdemeanors  Through our analysis of this data file, we identified 3 fields out of 29 that did not have standardization that could be useful to analyze misconduct  Classification—We found that this field contained at least three different variations of assault categories (i e , assault, assault-other, and assault-aggravated)  VAPS officials stated that this field is determined and entered by the user  Crime Type—We found this field contained at least five different variations of alcohol-consumption categories, such as “entering premises under the influence” and “alcohol – under the influence ” VAPS officials stated that this field is determined and entered by the user  Final Disposition—We found this field contained at least two different variations of charge type (i e , charged, charged – Issued Ticket), six different variations of open type (for example, open/referred to Court, open/cvb), and two different variations of closed type (i e , closed, case closed)  VAPS officials stated that this field is determined and entered by the user  This data file is designed to track all allegations of misconduct and associated disciplinary actions taken against VA employees  Through our analysis of this spreadsheet, we identified 1 field out of 34 that did not have standardization that could be useful to analyze misconduct  Position Title—We found this field contained at least 15 different variations of Registered Nurse, such as “Registered Nurse,” “Staff RN,” and “RN ” Allegations surrounding inadequate staffing, patient care, and safety at a Department of Veterans Affairs (VA) emergency room were investigated by the medical center director of the facility  The medical center director found that the inadequate patient care and safety issue was unsubstantiated based on a review of patient safety incidents for the last 6 months  The medical center director did not provide a copy of her report review to support this conclusion  She also indicated that an external consultant was hired to assess staffing issues, and found generally that improvements could be made for staffing to address surge capacity  The director stated the medical center was in the process of implementing the recommendations made by the consultant, but her response did not discuss the specific improvements planned or include the external consultant’s report  A fact-finding was performed by a panel comprised of VA Connecticut Healthcare System officials in response to alleged violations of law, gross mismanagement, and waste of funds that included the improper billing of services for a Las Vegas conference and paying contracts through a VA nonprofit corporation to handle such expenditures  The allegations specifically requested a cost-benefit analysis for the conference location  The response received from the program office stated that an outside accounting firm performs an annual financial audit of the VA nonprofit corporation and found no material issues  Neither a copy of the annual financial audit nor a cost-benefit analysis was provided in the response as support  Additionally, the response did not address allegations regarding the status of several essential positions vacated over the prior 3 years  Allegations involved time-and-attendance abuse by a physician who was accused of not responding to calls from peers or coming into the clinic, in favor of his private practice  According to the complainant, physician assistants (PA) examine the physician’s patients at the clinic for him  The medical center director, who was also named in the allegation as having received a similar complaint against the physician 2 years earlier, reviewed the case against the physician and himself  The medical center director’s response claimed that the location indicated in the allegation was not a private practice, but rather a location where the physician reviews medical records and sometimes serves as an expert witness  He did not provide evidence to support his claim  The medical center director also stated that he had not received any reports against the physician for missed calls or clinics and that PAs are expected to participate in these activities  However, he did not provide the physician’s work log, or the PA position descriptions showing that they are allowed to perform these functions autonomously  Finally, the medical center director claimed he did not recall the allegation made against the physician 2 years prior and neither formally substantiated nor disproved the current allegations against the physician  No recommendations were made  The medical center director was accused of hiring an unqualified individual to a Quality Manager position due to their romantic relationship  The response received from the human-resources consultant noted that it was unusual to find a Nurse II manager with only an associate’s degree, but was not illegal, and the employee was qualified based on prior experience  Concerns were also raised concerning the medical center director’s use of over $400 in government funds to “soundproof” the Quality Manager’s office, including having panels attached to one wall and the hollow office door replaced with a solid door  The Chief of Engineering Services was interviewed regarding the request and stated it was an odd request, and the first time he was asked to soundproof an administrative employee’s office  The response provided by the program office did not address why the director used government funds to soundproof the Quality Manager’s office  The response provided also did not address whether recommendations that the Quality Manager’s retention allowance be reviewed for compliance and that she be counseled for appropriate office dress code were implemented  The medical center director was alleged to have misrepresented a plan to track and provide mental-health services to veterans in non-VA hospitals and created a hostile work environment against African-American veterans and employees  The medical center director investigated the allegations against himself and provided a response that was eventually submitted late to the OIG  His response indicated that a review was completed and all allegations were unsubstantiated  Several documents provided with his response showed that only 12 contacts were made to veterans with mental-health care needs during the requested 24-month period, the percentage of patients experiencing wait times greater than 14 days before receiving mental-health services averaged 18 percent, and two veteran suicides occurred  The medical center director did not address allegations of creating a hostile work environment for African American veterans and employees  The medical center director improperly reannounced a vacancy in order to hire an individual with whom he allegedly had a close personal relationship to an Assistant Director position  He also requested the master key to the facility be issued to her against regulations  The allegation involving the master key was substantiated, but the Deputy Under Secretary who conducted the investigation stated that while there was no record of the key being returned, the key was returned and the general engineer brought the facility into compliance with VA regulations  Nonetheless, the Deputy Under Secretary found that a master key was issued in violation of policy, but no recommendations were made to the medical center director for corrective action  Allegations involved false patient wait-time documentation and abuse of authority  Specifically, a medical center director instructed staff to falsify patient wait times between follow-up appointments in order to meet VA’s 14-day timeliness metric  The investigation concluded that the false documentation allegation was substantiated, but attributed the cause to staff not understanding how to enter a follow-up appointment date into the system  It was also concluded that the correction of several hundred dates in the system improved the performance of the department for the national wait-time metric  However, no documentation was provided to (1) prove the medical center director had not abused his authority by instructing staff to review wait times greater than 14 days to determine how they could be reduced, and (2) support the conclusion that the original wait times were entered in error  In addition to the contact above, Dave Bruno (Assistant Director), Erica Varner (Analyst in Charge), Hiwotte Amare, Chris Cronin, Carrie Davidson, Ranya Elias, Colin Fallon, Mitch Karpman, Grant Mallie, Anna Maria Ortiz, Sabrina Streagle, Reed Van Beveren, and April Van Cleef made key contributions to this report 

ground_truth: More than 2.7 million miles of pipeline transport and distribute oil, natural gas, and other hazardous products throughout the United States. Interstate pipelines run through remote areas and highly populated urban areas, and are vulnerable to accidents, operating errors, and malicious physical and cyber-based attack or intrusion. The energy sector accounted for 35 percent of the 796 critical infrastructure cyber incidents reported to DHS from 2013 to 2015. Several federal and private entities have roles in pipeline security. TSA is primarily responsible for the oversight of pipeline physical security and cybersecurity. GAO was asked to review TSA's efforts to assess and enhance pipeline security and cybersecurity. This report examines, among other objectives: (1) the guidance pipeline operators reported using to address security risks and the extent that TSA ensures its guidelines reflect the current threat environment; (2) the extent that TSA has assessed pipeline systems' security risks; and (3) the extent TSA has assessed its effectiveness in reducing pipeline security risks. GAO analyzed TSA documents, such as its Pipeline Security Guidelines ; evaluated TSA pipeline risk assessment efforts; and interviewed TSA officials, 10 U.S. pipeline operators—selected based on volume, geography, and material transported—and representatives from five industry associations. Pipeline operators reported using a range of guidelines and standards to address physical and cybersecurity risks, including the Department of Homeland Security's (DHS) Transportation Security Administration's (TSA) Pipeline Security Guidelines , initially issued in 2011. TSA issued revised guidelines in March 2018 to reflect changes in the threat environment and incorporate most of the principles and practices from the National Institute of Standards and Technology's Framework for Improving Critical Infrastructure Cybersecurity . However, TSA's revisions do not include all elements of the current framework and TSA does not have a documented process for reviewing and revising its guidelines on a regular basis. Without such a documented process, TSA cannot ensure that its guidelines reflect the latest known standards and best practices for physical security and cybersecurity, or address the dynamic security threat environment that pipelines face. Further, GAO found that the guidelines lack clear definitions to ensure that pipeline operators identify their critical facilities. GAO's analysis showed that operators of at least 34 of the nation's top 100 critical pipeline systems (determined by volume of product transported) deemed highest risk had identified no critical facilities. This may be due, in part, to the guidelines not clearly defining the criteria to determine facilities' criticality. To assess pipeline security risks, TSA conducts pipeline security reviews—Corporate Security Reviews and Critical Facility Security Reviews—to assess pipeline systems' vulnerabilities. However, GAO found that the number of TSA security reviews has varied considerably over the last several years, as shown in the table on the following page. TSA officials stated that staffing limitations have prevented TSA from conducting more reviews. Staffing levels for TSA's Pipeline Security Branch have varied significantly since fiscal year 2010 with the number of staff ranging from 14 full-time equivalents in fiscal years 2012 and 2013 to 1 in 2014. Further, TSA does not have a strategic workforce plan to help ensure it identifies the skills and competencies—such as the required level of cybersecurity expertise—necessary to carry out its pipeline security responsibilities. By establishing a strategic workforce plan, TSA can help ensure that it has identified the necessary skills, competencies, and staffing. GAO also identified factors that likely limit the usefulness of TSA's risk assessment methodology for prioritizing pipeline system reviews. Specifically, TSA has not updated its risk assessment methodology since 2014 to reflect current threats to the pipeline industry. Further, its sources of data and underlying assumptions and judgments regarding certain threat and vulnerability inputs are not fully documented. In addition, the risk assessment has not been peer reviewed since its inception in 2007. Taking steps to strengthen its risk assessment, and initiating an independent, external peer review would provide greater assurance that TSA ranks relative risk among pipeline systems using comprehensive and accurate data and methods. TSA has established performance measures to monitor pipeline security review recommendations, analyze their results, and assess effectiveness in reducing risks. However, these measures do not possess key attributes—such as clarity, and having measurable targets—that GAO has found are key to successful performance measures. By taking steps to ensure that its pipeline security program performance measures exhibit these key attributes, TSA could better assess its effectiveness at reducing pipeline systems' security risks. Pipeline Security Branch officials also reported conducting security reviews as the primary means for assessing the effectiveness of TSA's efforts to reduce pipeline security risks. However, TSA has not tracked the status of Corporate Security Review recommendations for the past 5 years. Until TSA monitors and records the status of these reviews' recommendations, it will be hindered in its efforts to determine whether its recommendations are leading to significant reduction in risk. GAO makes 10 recommendations to TSA to improve its pipeline security program management (many are listed on the next page), and DHS concurred. GAO recommends, among other things, that the TSA Administrator take the following actions: implement a documented process for reviewing, and if deemed necessary, for revising TSA's Pipeline Security Guidelines at defined intervals; clarify TSA's Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities; develop a strategic workforce plan for TSA's Security Policy and Industry Engagement‘s Surface Division; update TSA's pipeline risk assessment methodology to include current data to ensure it reflects industry conditions and threats; fully document the data sources, underlying assumptions and judgments that form the basis of TSA's pipeline risk assessment methodology; take steps to coordinate an independent, external peer review of TSA's pipeline risk assessment methodology; ensure the Security Policy and Industry Engagement‘s Surface Division has a suite of performance measures which exhibit key attributes of successful performance measures; and enter information on Corporate Security Review recommendations and monitor and record their status.