Information Security: TVA Needs to Address Weaknesses in Control Systems and Networks
Published: 2008-05-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Securing the control systems that regulate the nation's critical infrastructures is vital to ensuring our economic security and public health and safety. The Tennessee Valley Authority (TVA), a federal corporation and the nation's largest public power company, generates and distributes power in an area of about 80,000 square miles in the southeastern United States. GAO was asked to determine whether TVA has implemented appropriate information security practices to protect its control systems. To do this, GAO examined the security practices in place at several TVA facilities; analyzed the agency's information security policies, plans, and procedures against federal law and guidance; and interviewed agency officials who are responsible for overseeing TVA's control systems and their security.
section_title: What GAO Found
paragraphs: TVA has not fully implemented appropriate security practices to secure the control systems and networks used to operate its critical infrastructures. Both its corporate network infrastructure and control systems networks and devices were vulnerable to disruption. The corporate network was interconnected with control systems networks GAO reviewed, thereby increasing the risk that security weaknesses on the corporate network could affect those control systems networks. On TVA's corporate network, certain individual workstations lacked key software patches and had inadequate security settings, and numerous network infrastructure protocols and devices had limited or ineffective security configurations. In addition, the intrusion detection system had significant limitations. On control systems networks, firewalls reviewed were either inadequately configured or had been bypassed, passwords were not effectively implemented, logging of certain activity was limited, configuration management policies for control systems software were inconsistently implemented, and servers and workstations lacked key patches and effective virus protection. In addition, physical security at multiple locations did not sufficiently protect critical control systems. As a result, systems that operate TVA's critical infrastructures are at increased risk of unauthorized modification or disruption by both internal and external threats. An underlying reason for these weaknesses is that TVA had not consistently implemented significant elements of its information security program. Although TVA had developed and implemented program activities related to contingency planning and incident response, it had not consistently implemented key activities related to developing an inventory of systems, assessing risk, developing policies and procedures, developing security plans, testing and monitoring the effectiveness of controls, completing appropriate training, and identifying and tracking remedial actions. For example, the agency lacked a complete inventory of its control systems and had not categorized all of its control systems according to risk, thereby limiting assurance that these systems were adequately protected. Agency officials stated that they plan to complete these risk assessments and related activities but have not established a completion date. Key information security policies and procedures were also in draft or under revision. Additionally, the agency's patch management process lacked a way to effectively prioritize vulnerabilities. TVA had only completed one system security plan, and another plan was under development. The agency had also tested the effectiveness of its control systems' security using outdated federal guidance, and many control systems had not been tested for security. In addition, only 25 percent of relevant agency staff had completed required role-based security training in fiscal year 2007. Furthermore, while the agency had developed a process to track remedial actions for information security, this process had not been implemented for the majority of its control systems. Until TVA fully implements these security program activities, it risks a disruption of its operations as a result of a cyber incident, which could impact its customers.

section_title: Letter
paragraphs: Securing the control systems that perform vital functions in the complex  networks of digital information systems on which the nation’s critical  infrastructures rely is critical to ensuring our national and economic  security and public health and safety. Control systems are computer-based  systems used by critical infrastructure sectors and industries to monitor  and control sensitive processes and physical functions such as electric  power generation and its transmission, oil and gas refining, water  treatment and its distribution, and transportation.
We have previously reported that critical infrastructure control systems  face increasing risks due to cyber threats, system vulnerabilities, and the  serious potential impact of attacks as demonstrated by reported incidents.  If control systems are not adequately secured, their system vulnerabilities  could be exploited, and our critical infrastructures could be disrupted or  disabled, possibly resulting in loss of life, physical damage, or economic  losses.
The majority of our nation’s critical infrastructures are owned by the  private sector; however, the federal government owns and operates  critical infrastructure facilities including those in energy, water treatment  and distribution, and transportation. One such entity, the Tennessee Valley  Authority (TVA)—a federal corporation and the nation’s largest public  power company—generates electricity using its 52 fossil, hydro, and  nuclear facilities—all of which use control systems. As a wholly owned  government corporation, TVA must comply with the Federal Information  Security Management Act of 2002 (FISMA) by developing a risk-based  information security program and implementing information security  controls for its computer systems.
Our objective was to determine whether TVA has effectively implemented  appropriate information security practices for the control systems used to  operate its critical infrastructures. To accomplish this objective, we  examined the security practices in place at six TVA facilities. In addition,  we analyzed the agency’s information security policies, plans, and  procedures and interviewed agency officials who are responsible for  overseeing TVA’s control systems and their security. See appendix I for a  complete description of our objective, scope, and methodology.
We conducted this performance audit from March 2007 to May 2008 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: Information security is a critical consideration for any organization that  depends on information systems and computer networks to carry out its  mission or business. It is especially important for government agencies,  where maintaining the public’s trust is essential. The dramatic expansion  in computer interconnectivity and the rapid increase in the use of the  Internet have changed the way our government, the nation, and much of  the world communicate and conduct business. However, without proper  safeguards, systems are unprotected from individuals and groups with  malicious intent to intrude and use the access to obtain sensitive  information, commit fraud, disrupt operations, or launch attacks against  other computer systems and networks. This concern is well-founded for a  number of reasons, including the increase in reports of security incidents,  the ease of obtaining and using hacking tools, the steady advance in the  sophistication and effectiveness of attack technology, and the dire  warnings of new and more destructive attacks to come. Computer- supported federal operations are likewise at risk. Our previous reports and  those of agency inspectors general describe persistent information  security weaknesses that place a variety of federal operations at risk of  disruption, fraud, and inappropriate disclosure. Thus, we have designated  information security as a governmentwide high-risk area since 1997, a  designation that remains in effect.
subsections: section_title: Control Systems Are Used in Critical Infrastructures, Including Those Operated by the Federal Government
paragraphs: We have specifically recognized the importance of information security  related to critical infrastructures. Critical infrastructures are physical or  virtual systems and assets so vital to the nation that their incapacitation or  destruction would have a debilitating impact on national and economic  security and on public health and safety. These systems and assets—such  as the electric power grid, chemical plants, and water treatment  facilities—are essential to the operations of the economy and the  government. Recent terrorist attacks and threats have underscored the  need to protect these critical infrastructures. If their vulnerabilities are  exploited, our nation’s critical infrastructures could be disrupted or  disabled, possibly causing loss of life, physical damage, and economic  losses.
Although the majority of our nation’s critical infrastructures are owned by  the private sector, the federal government owns and operates key facilities  that use control systems, including oil, gas, water, energy, and nuclear  facilities. Control systems are used within these infrastructures to monitor  and control sensitive processes and physical functions. Typically, control  systems collect sensor measurements and operational data from the field,  process and display this information, and relay control commands to local  or remote equipment. Control systems perform functions that range from  simple to complex. They can be used to simply monitor processes—for  example, the environmental conditions in a small office building—or to  manage the complex activities of a municipal water system or a nuclear  power plant. In the electric power industry, control systems can be used to  manage and control the generation, transmission, and distribution of  electric power. For example, control systems can open and close circuit  breakers and set thresholds for preventive shutdowns.
section_title: Control Systems: Types and Components
paragraphs: There are two primary types of control systems: distributed control  systems and supervisory control and data acquisition (SCADA) systems.  Distributed control systems typically are used within a single processing  or generating plant or over a small geographic area and communicate  using local area networks, while SCADA systems typically are used for  large, geographically dispersed operations and rely on long-distance  communication networks. In general, critical infrastructure sectors and  industries depend on both types of control systems to fulfill their missions  or conduct business. For example, a utility company that serves a large  geographic area may use distributed control systems to manage power  generation at each power plant and a SCADA system to manage power  distribution to its customers.
A SCADA system is generally composed of these six components (see  fig. 1): (1) operating equipment, which includes pumps, valves, conveyors,  and substation breakers; (2) instruments, which sense conditions such as  pH, temperature, pressure, power level, and flow rate; (3) local processors,  which communicate with the site’s instruments and operating equipment,  collect instrument data, and identify alarm conditions; (4) short-range  communication, which carries analog and discrete signals between the  local processors and the instruments and operating equipment; (5) host  computers, where a human operator can supervise the process, receive  alarms, review data, and exercise control; and (6) long-range  communication, which connects local processors and host computers  using, for example, leased phone lines, satellite, and cellular packet data.  A distributed control system is similar to a SCADA system but does not  operate over a large geographic area or use long-range communications.
section_title: Control Systems for Critical Infrastructures Face Increasing Cyber Threats
paragraphs: We have previously reported that critical infrastructure control systems  face increasing risks due to cyber threats, system vulnerabilities, and the  potential impact of attacks as demonstrated by reported incidents. Cyber  threats can be intentional or unintentional, targeted or nontargeted, and  can come from a variety of sources. The Federal Bureau of Investigation  has identified multiple sources of threats to our nation’s critical  infrastructures, including foreign nation states engaged in information  warfare, domestic criminals and hackers, and disgruntled employees  working within an organization. Table 1 summarizes those groups or  individuals that are considered to be key sources of threats to our nation’s  infrastructures.
Control systems are more vulnerable to cyber threats, including  intentional attacks and unintended incidents, than in the past for several  reasons, including their increasing standardization and their increased  connectivity to other systems and the Internet. For example, in August  2006, two circulation pumps at Unit 3 of the Browns Ferry, Alabama,  nuclear power plant operated by TVA failed, forcing the unit to be shut  down manually. The failure of the pumps was traced to an unintended  incident involving excessive traffic on the control system network caused  by the failure of another control system device.
Critical infrastructure owners face both technical and organizational  challenges to securing control systems. Technical challenges—including  control systems’ limited processing capabilities, real-time operations, and  design constraints—hinder an infrastructure owner’s ability to implement  traditional information technology (IT) security processes, such as strong  user authentication and patch management. Organizational challenges  include difficulty in developing a compelling business case for investing in  control systems security and differing priorities of information security  personnel and control systems engineers.
section_title: Federal Regulations, Standards, and Guidance Establish Requirements to Secure Control Systems
paragraphs: To address the increasing threat to control systems governing critical  infrastructures, both federal and private organizations have begun efforts  to develop requirements, guidance, and best practices for securing control  systems. For example, FISMA outlines a comprehensive, risk-based  approach to securing federal information systems, which encompass  control systems. Federal organizations, including the National Institute of  Standards and Technology (NIST), the Federal Energy Regulatory  Commission (FERC), and the Nuclear Regulatory Commission (NRC),  have used a risk-based approach to develop guidance and standards to  secure control systems. NIST guidance has been developed that currently  applies to federal agencies; however, much FERC and NRC guidance and  many standards have not been finalized. Once implemented, FERC and  NRC standards will apply to both public and private organizations that  operate covered critical infrastructures.
We have previously reported on the importance of using a risk-based  approach for securing critical infrastructures, including control systems.  Risk management has received widespread support within and outside  government as a tool that can help set priorities on how to protect critical  infrastructures. While numerous and substantial gaps in security may  exist, resources for closing these gaps are limited and must compete with  other national priorities.
subsections: section_title: FISMA Established Requirements to Strengthen Information Security Practices at Federal Agencies
paragraphs: Recognizing the importance of securing federal agencies’ information and  systems, Congress enacted FISMA to strengthen the security of  information and information systems within federal agencies, which  include control systems.
FISMA requires each agency to develop, document, and implement an  agencywide information security program to provide security for the  information and information systems that support the operations and  assets of the agency, including those provided or managed by another  agency, contractor, or other source. Specifically, this program is to include    periodic assessments of the risk and magnitude of harm that could result  from the unauthorized access, use, disclosure, disruption, modification, or  destruction of information or information systems; risk-based policies and procedures that cost effectively reduce  information security risks to an acceptable level and ensure that  information security is addressed throughout the life cycle of each  information system;    subordinate plans for providing adequate information security for  networks, facilities, and systems or groups of information systems;   security awareness training for agency personnel, including contractors  and other users of information systems that support the operations and  assets of the agency;   periodic testing and evaluation of the effectiveness of information security  policies, procedures, and practices, performed with a frequency depending  on risk, but no less than annually, and that includes testing of  management, operational, and technical controls for every system  identified in the agency’s required inventory of major information systems;   a process for planning, implementing, evaluating, and documenting  remedial action to address any deficiencies in the information security  policies, procedures, and practices of the agency;   procedures for detecting, reporting, and responding to security incidents;    plans and procedures to ensure continuity of operations for information  systems that support the operations and assets of the agency.
Furthermore, FISMA established a requirement that each agency develop,  maintain, and annually update an inventory of major information systems  (including major national security systems) operated by the agency or  under its control. This inventory is to include an identification of the  interfaces between each system and all other systems or networks,  including those not operated by or under the control of the agency.
section_title: NIST Has Developed Standards and Guidance to Implement FISMA
paragraphs: FISMA also directs NIST to develop standards and guidelines for systems  other than national security systems. As required by FISMA and based on  the objectives of providing appropriate levels of information security,  NIST developed    standards for all agencies to categorize their information and information  systems according to a range of risk levels,   guidelines recommending the types of information and information  systems to be included in each category, and   minimum information security requirements for information and  information systems in each category.
NIST standards and guidelines establish a risk management framework  that instructs agencies on providing an acceptable level of information  security for all agency operations and assets and that guides the testing  and evaluation of information security control effectiveness within an  agencywide information security program. Recognizing the importance of  documenting standards and guidelines as part of an agencywide  information security program, NIST emphasizes that agencies must  develop and promulgate formal, documented policies and procedures in  order to ensure the effective implementation of security requirements.
NIST also collaborates with federal and industry stakeholders to develop  standards, guidelines, checklists, and test methods to help secure federal  information and information systems, including control systems. For  example, NIST is currently developing guidance for federal agencies that  own or operate control systems to comply with federal information system  security standards and guidelines. The guidance identifies issues and  modifications to consider in applying information security standards and  guidelines to control systems. In December 2007, NIST released an  augmentation to Special Publication (SP) 800-53, Recommended Security  Controls for Federal Information Systems, which provides a security  control framework for control systems. According to NIST officials, while  most controls in SP 800-53 are applicable to control systems as written,  several controls do require supplemental guidance and enhancements.
section_title: FERC Has Recently Approved Reliability Standards That Address Control Systems Security
paragraphs: Under the Energy Policy Act of 2005, FERC was authorized to (1) appoint  an electricity reliability organization to develop and enforce mandatory  electricity reliability standards, including cyber security, and (2) approve  or remand each proposed standard. The commission may also direct the  reliability organization to develop a new standard or modify approved  standards. Both the commission and the reliability organization have the  authority to enforce approved standards, investigate incidents, and impose  penalties (up to $1 million a day) on noncompliant electricity asset owners  or operators.
FERC has conducted several activities to begin implementing the  requirements of the act. In July 2006, FERC certified the North American  Electric Reliability Corporation (NERC) as the national electric reliability  organization. In August 2003, prior to passage of the Energy Policy Act of  2005, NERC adopted Urgent Action 1200, a temporary, voluntary cyber  security standard for the electric industry. Urgent Action 1200 directed  electricity transmission and generation owners and operators to develop a  cyber security policy, identify critical cyber assets, and establish controls  for and monitor electronic and physical access to critical cyber assets.  Urgent Action 1200 remained in effect on a voluntary basis until June 1,  2006, at which time NERC proposed eight critical infrastructure protection  reliability standards to replace the Urgent Action 1200 standard.
In July 2007, FERC issued a notice of proposed rulemaking in which it  proposed to approve eight critical infrastructure reliability standards,  which included standards for control systems security. FERC also  proposed to direct NERC to modify the areas of these standards that  required improvement. In January 2008, after considering public  comments on the notice of proposed rulemaking, FERC approved the  reliability standards and the accompanying implementation plan. It also  directed NERC to develop modifications to strengthen the standards and  to monitor the development and implementation of the NIST standards to  determine if they contain provisions that will protect the bulk-power  system better than NERC’s reliability standards. The organizations subject  to the standards, including utilities like TVA, must be auditably compliant  with the standards by 2010.
section_title: NRC Is Conducting A Rulemaking Process on Cyber Security, Including Control Systems
paragraphs: The NRC, which has regulatory authority over nuclear power plant safety  and security, has conducted several activities related to enhancing the  cyber security of control systems. In 2005, an industry task force led by the  Nuclear Energy Institute (NEI) developed and released the Cyber Security  Program for Power Reactors (NEI 04-04) to provide nuclear power reactor  licensees a means for developing and maintaining effective cyber security  programs at their sites. In December 2005, the commission staff accepted  the method outlined in NEI 04-04 for establishing and maintaining cyber  security programs at nuclear power plants. TVA officials stated that the  agency has begun a program to comply with NEI 04-04 guidelines and  plans to complete implementation of corrective actions identified as a  result of these guidelines over the next 3 years, consistent with planned  plant outages and upgrade projects.
In January 2006, the commission issued a revision to Regulatory Guide  1.152, Criteria for Use of Computers in Safety Systems of Nuclear Power  Plants, which provides cyber security-related guidance for the design of  nuclear power plant safety systems. In April 2007, the commission  finalized a rule that added “external cyber attack” to the events that power  reactor licensees are required to prepare to defend against. In addition, the  commission initiated a rulemaking process that provides cyber security  requirements for digital computer and communication networks, including  systems that are needed for plant safety, security, or emergency response.  The public comment period for this rulemaking closed in March 2007.  Commission officials stated that they estimate this rulemaking process  will be completed in early 2009. Once the rulemaking process is completed  and requirements for nuclear power plant cyber security programs are  finalized, the commission is planning to conduct a range of oversight  activities, including inspections at power plants. According to commission  officials, all nuclear plant operators have committed to complete  implementation of the NEI-04-04 program at their sites.
section_title: TVA Provides Power to the Southeastern United States
paragraphs: The TVA is a federal corporation and the nation’s largest public power  company. Its mission is to supply affordable, reliable power, support a  thriving river system, and stimulate sustainable economic development in  the public interest. In addition to generating and transmitting power, TVA  also manages the nation’s fifth-largest river system to minimize flood risk,  maintain navigation, provide recreational opportunities, and protect water  quality. TVA is governed by a nine-member Board of Directors that is led  by the Chairman. Each board member is nominated by the President of the  United States and confirmed by the Senate. The TVA Chief Executive  Officer reports to the TVA Board of Directors.
TVA’s power service area covers 80,000 square miles in the southeastern  United States, an area that includes almost all of Tennessee and parts of  Mississippi, Kentucky, Alabama, Georgia, North Carolina, and Virginia, and  has a total population of about 8.7 million people (see fig. 2).
TVA operates 11 coal-fired fossil plants, 8 combustion turbine plants,  3 nuclear plants, and a hydroelectric system that includes 29 hydroelectric  dams and one pumped storage facility (see fig. 2 and fig. 3). Fossil plants  produce about 60 percent of TVA’s power, nuclear plants about 30 percent,  and the hydroelectric system about 10 percent. TVA also owns and  operates one of the largest transmission systems in North America. TVA’s  transmission system moves electric power from the generating plants  where it is produced to distributors of TVA power and to industrial and  federal customers across the region.
TVA provides power to three main customer groups: distributors, directly  served customers, and off-system customers. There are 159 distributors— 109 municipal utility companies and 50 cooperatives—that resell TVA  power to consumers. These groups represent the base of TVA’s business,  accounting for 85 percent of their total revenue. Fifty-three large industrial  customers and six federal installations buy TVA power directly. They  represent 11 percent of TVA’s total revenue. Twelve surrounding utilities  buy power from TVA on the interchange market. Sales to these utilities  represent 4 percent of TVA’s total revenue.
Control systems are essential to TVA’s operation. TVA uses control  systems to both generate and deliver power. In generation, control  systems are used within power plants to open and close valves, control  equipment, monitor sensors, and ensure the safe and efficient operation of  a generating unit. Many control systems networks connect with TVA’s  corporate network to transmit information about system status.
To deliver power, TVA monitors the status of its own and surrounding  transmission facilities from two operations centers. Each center is staffed  24 hours a day and can serve as a backup for the other center. Control  systems at these centers are used to open and close breakers and balance  the transmission of power across the TVA network while accounting for  changes in network capacity due to outages and changes in demand that  occur continuously throughout the day. TVA’s control systems range in  capacity from simple systems with limited functionality located in one  facility to complex, geographically dispersed systems with multiple  functions. The ages of these control systems range from modern systems  to systems dating back 20 or more years to the original construction of a  facility.
As shown in table 2, TVA has designated certain senior managers to serve  the key roles in information security designated by FISMA.
Responsibility for control systems security is distributed throughout TVA  (see fig. 4). TVA’s Information Services organization provides general  guidance, assistance in FISMA compliance, and technical assistance in  control systems security. The Information Services organization also  manages the overall TVA corporate computer network that links facilities  throughout the TVA service area and is connected to the Internet. As of  February 2008, the Enterprise IT Security organization within Information  Services was given specific responsibility for cyber security throughout  the agency.
However, the control systems located within a plant are integrated with  and managed as part of the generation equipment, safety and  environmental systems, and other physical equipment located at that  plant. This means that development, day-to-day maintenance and  operation, and upgrades of control systems are handled by the business  units responsible for the facilities where the systems are located.  Specifically, nuclear systems are managed by the Nuclear Power Group;  coal and combustion turbine control systems are managed by the Fossil  Power Group; and hydroelectric facilities are managed by River  Operations. Transmission control systems are managed by TVA’s  Transmission and Reliability Organization, located within its Power  Systems Operations business unit.
The Transmission and Reliability Organization is highly dependent on  control systems. To comply with NERC Urgent Action 1200, and in an  effort to ensure its systems are secure, the Transmission and Reliability  Organization has handled additional aspects of information security  compared with other TVA organizations. For example, the organization  manages portions of its own network infrastructure. It also has arranged  for both internal and external security assessments in order to enhance  the security of its control systems.
section_title: TVA Had Not Fully Implemented Appropriate Security Practices to Protect Its Critical Infrastructures
paragraphs: TVA had not fully implemented appropriate security practices to secure  the control systems used to operate its critical infrastructures. Both the  corporate network infrastructure and control systems networks and  devices at individual facilities and plants were vulnerable to disruption. In  addition, physical security controls at multiple locations did not  sufficiently protect critical control systems. The interconnections between  TVA’s control system networks and its corporate network increase the risk  that security weaknesses on the corporate network could affect control  systems networks. For example, because of weaknesses in the separation  of lower security network segments from higher security network  segments on TVA networks, an attacker who gained access to a less  secure portion of a network such as the corporate network could  potentially compromise equipment in a more secure portion of the  network, including equipment that has access to control systems. As a  result, TVA’s control systems that operate its critical infrastructures are at  increased risk of unauthorized modification or disruption by both internal  and external threats.
subsections: section_title: TVA Corporate Network Was Vulnerable to Disruption
paragraphs: The TVA corporate network infrastructure had multiple weaknesses that  left it vulnerable to intentional or unintentional compromise of the  confidentiality, integrity, and availability of the network and devices on  the network. These weaknesses applied both at TVA headquarters and to  the portions of the corporate network located at the individual facilities  we reviewed. For example, one remote access system used for the  network that we reviewed was not securely configured. Further, individual  servers and workstations lacked key patches and were insecurely  configured. In addition, the configuration of numerous network  infrastructure protocols and devices provided limited or ineffective  security protections. Moreover, the intrusion detection system that TVA  used had significant limitations. As a result, TVA’s control systems were at  an increased risk of unauthorized access or disruption via access from the  corporate network. Furthermore, weaknesses in the intrusion detection  system could limit the ability of TVA to detect malicious or unintended  events on its network.
subsections: section_title: TVA Remote Access Was Insecurely Configured
paragraphs: Remote access is any access to an organizational information system by a  user (or an information system) that communicates through an external,  nonorganization-controlled network (e.g., the Internet). NIST guidance  states that information systems should establish a trusted communications  path between remote users and an information system and that two-factor  authentication should be part of an organization’s remote access  authentication requirements. Additionally, TVA policy requires that if  remote access technology is used to connect to the network, it must be  configured securely. One device used for remote access is a virtual private  network (VPN).
TVA did not configure a VPN system to include effective security  mechanisms. This could allow an attacker who compromised a remote  user’s computer to remotely access the user’s secure session to TVA,  thereby increasing the risk that unauthorized users could gain access to  TVA systems and sensitive information.
section_title: Individual Servers and Workstations Were Insecurely Configured
paragraphs: Federal and agency guidance call for effective patch management, firewall  configuration, and application security settings. TVA has a patch  management policy that requires it to regularly monitor, identify, and  remediate vulnerabilities to applications in its software inventory. NIST  guidance also states that firewalls should be carefully configured to  provide adequate protection. Furthermore, NIST guidance states that  organizations should effectively configure security settings in key  applications to the highest level possible.
However, almost all of the workstations and servers that we examined on  the corporate network lacked key security patches or had inadequate  security settings. Furthermore, TVA did not effectively implement host  firewall controls on its laptops. In addition, inadequate security settings  existed in key applications installed on laptops, servers, and workstations  we examined. Consequently, TVA is at an increased risk that known  vulnerabilities in these applications could allow an attacker to execute  malicious code and gain control of or compromise a system.
section_title: Network Infrastructure Protocols and Devices Provided Limited or Ineffective Protections
paragraphs: Federal and agency guidance state that organizations should have strong  passwords, identification and authentication, and network segmentation.  National Security Agency guidance states that Windows passwords should  be 12 or more characters long, include upper and lower case letters,  numbers, and special characters, and not consist of dictionary words and  has advised against the use of weak encryption. NIST guidance states that  systems should uniquely identify and authenticate users with passwords or  other authentication mechanisms or implement other compensating  controls. NIST guidance also states that organizations should take steps to  secure their e-mail systems. Finally, NIST guidance states that  organizations should partition networks containing higher risk systems  from lower risk systems and configure interfaces between those systems  to manage risk.
However, the TVA corporate network used several protocols and devices  that did not provide sufficient security controls. For example, certain  network protocols and devices were not adequately protected by  password or authentication controls or encryption. In addition, TVA had  network services that spanned different security network segments. As a  result, a malicious user could exploit these weaknesses to gain access to  sensitive systems or to otherwise modify or disrupt network traffic.
section_title: Intrusion Detection System Had Significant Limitations
paragraphs: Even strong controls may not block all intrusions and misuse, but  organizations can reduce the risks associated with such events if they take  steps to promptly detect, report, and respond to them before significant  damage is done. In addition, analyzing security events allows organizations  to gain a better understanding of the threats to their information and the  costs of their security-related problems. Such analyses can pinpoint  vulnerabilities that need to be eliminated so that they will not be exploited  again. NIST states that intrusion detection is the process of monitoring  events occurring in a computer system or network and analyzing the  events for signs of intrusion, which it defines as an attempt to compromise  the confidentiality, integrity, or availability of a computer or network.  NIST guidance prescribes network and host-based intrusion detection  systems as a means of protecting systems from the threats that come  with increasing network connectivity.
TVA had limited ability to effectively monitor its network with its intrusion  detection system. Although a network intrusion detection system was  deployed by TVA to monitor network traffic, it could not effectively  monitor key computer assets. As a result, there is an increased risk that  unauthorized access to TVA’s networks may not be detected and mitigated  in a timely manner.
section_title: TVA Control System Networks and Devices Were Vulnerable to Disruption
paragraphs: TVA’s control system networks and devices on these networks were  vulnerable to disruption due to inadequate information security controls.  Specifically, firewalls were either bypassed or inadequately configured,  passwords were either weak or not used at all, logging of certain activity  was limited, configuration management policies for control systems  software were not consistently implemented, and servers and  workstations lacked key patches and effective virus protection. The  combination of these weaknesses with the weaknesses in the TVA  corporate network identified in the previous section places TVA’s control  systems that operate its critical infrastructures at increased risk of  unauthorized modification or disruption by both internal and external  threats.
subsections: section_title: Firewalls Were Either Bypassed or Inadequately Configured
paragraphs: A firewall is a hardware or software component that protects given  computers or networks from attacks by blocking network traffic. NIST  guidance states that firewalls should be configured to provide adequate  protection for the organization’s networks and that the transmitted  information between interconnected systems should be controlled and  regulated.
TVA had implemented firewalls to segment control systems networks from  the corporate network at all facilities we reviewed with connections  between these two networks. However, firewalls at three of six facilities  reviewed were either bypassed or inadequately configured. As a result, the  hosts on higher security control system networks were at increased risk of  compromise or disruption from the other lower security networks.
section_title: Passwords or Other Compensating Controls Were Not Effectively Implemented
paragraphs: Passwords are used to establish the validity of a user’s claimed identity by  requesting some kind of information that is known only by the user—a  process known as authentication. The combination of identification, using,  for example, a unique user account, and authentication, using, for  example, a password, provides the basis for establishing individual  accountability and for controlling access to the system. In cases where  passwords cannot be implemented because of technological limitations or  other concerns, such as impact on emergency response, NIST states that  an organization should document controls that have been put in place to  compensate for this weakness. TVA policy requires authentication of users  except where security requirements or limitations in the hardware or  software preclude it. In addition, agency policy requires users to establish  complex passwords.
TVA did not have effective passwords or other documented compensating  controls governing control systems we reviewed. According to agency  officials, in certain cases, passwords were not technologically possible to  implement but in these cases, there were no documented compensating  controls. Until the agency implements either effective password practices  or documented compensating controls, it faces an increased risk of  unauthorized access to its control systems.
section_title: Audit Controls Did Not Effectively Log Certain Activity on Control Systems
paragraphs: Determining what, when, and by whom specific actions are taken on a  system is crucial to establishing individual accountability, monitoring  compliance with security policies, and investigating security violations.  Audit and monitoring involves the regular collection, review, and analysis  of auditable events for indications of inappropriate or unusual activity and  the appropriate investigation and reporting of such activity. Audit and  monitoring can help security professionals routinely assess computer  security, perform investigations during and after an attack, and even  recognize an ongoing attack. Federal guidance states that organizations  should develop formal audit policies and procedures. TVA guidance states  that sufficient audit logs should be maintained that allow monitoring of  key user activities.
While TVA had taken steps to establish audit logs for its transmission  control centers, it had not established effective audit logs or compensating  controls at other facilities we reviewed. According to agency officials,  system limitations at these facilities have historically meant that multiple  users shared a single account to access these control systems. Therefore,  audit logs would not have served a useful purpose because activities could  not be traced to a single user. Until TVA establishes detailed audit logs for  its control systems at these facilities or compensating controls in cases  where such logs are not feasible, it risks being unable to determine if  malicious incidents are occurring and, after an event occurs, being able to  determine who or what caused the incident.
section_title: Configuration Management Policies Were Not Consistently Implemented on TVA Control Systems
paragraphs: Federal guidance states that all applications and changes to those  applications should go through a formal, documented process that  identifies all changes to the baseline configuration. Also, procedures  should ensure that no unauthorized software is installed. TVA has  established configuration management policies and procedures for its  information technology systems. Specifically, its policies define the roles  and responsibilities of application owners and developers; require  business units to implement procedural controls that define  documentation and testing required for software changes; and establish  procedures to ensure that all changes relating to infrastructure and  applications be managed and controlled.
However, TVA did not consistently apply its configuration management  policies and procedures to control systems. The transmission control  system had a configuration management process, and the hardware at  individual plants was governed by a configuration management process,  including plant drawings that tracked individual pieces of equipment.  However, there was no formal configuration management process for  software that was part of the control systems at the hydroelectric and  fossil facilities that we reviewed. As a result, increased risk exists that  unapproved changes to control systems could be made.
section_title: Software Patches on Control Systems Were Not Current
paragraphs: Patch management, including up-to-date patch installation, helps to  mitigate vulnerabilities associated with flaws in software code, which  could be exploited to cause significant damage. According to NIST,  agencies should identify, report, and correct their information system  flaws. According to NIST, tracking patches allows organizations to identify  which patches are installed on a system and provides confirmation that  the appropriate patches have been applied. Moreover, TVA policy requires  the agency to remediate these vulnerabilities in a timely manner.
TVA had not installed current versions of patches for key applications on  computers on control systems networks. While TVA had an agencywide  policy and procedure for patch management, these policies did not apply  to individual plant-level control systems. According to the operators at two  of the facilities we reviewed, they applied vendor-approved patches to  control systems but did not track versions of patches on these machines.  Failure to keep software patches up-to-date could allow unauthorized  individuals to gain access to network resources or disrupt network  operations.
section_title: Virus Protection Software Was Not Consistently Implemented
paragraphs: Virus and worm protection for information systems is a serious  challenge. Computer attack tools and techniques are becoming  increasingly sophisticated; viruses are spreading faster as a result of the  increasing connectivity of today’s networks; commercial off-the-shelf  products can be easily exploited for attack by their users; and there is no  single solution such as firewalls or encryption to protect systems. To  combat viruses and worms specifically, entities should keep antivirus  programs up-to-date. According to NIST, agencies should implement  malicious code protection that includes a capability for automatic updates  so that virus definitions are kept up-to-date on servers, workstations, and  mobile computing devices. Virus-scanning software should be provided at  critical entry points, such as remote-access servers, and at each desktop  system on the network.
Although TVA implemented antivirus software on its transmission control  systems network, it did not consistently implement antivirus software on  other control systems we reviewed. In one case, according to agency  officials, the vendor that developed the control systems software would  not support an antivirus application, and the agency did not have plans to  require the vendor to address this weakness. In another case, antivirus  software was implemented, but it was not up-to-date. In the event that  using antivirus software is infeasible on a control system, the agency must  document the controls, such as training or physical security, that would  compensate for this deficiency. TVA had not done this. According to  agency officials, such documentation is under way for its hydroelectric  facilities, but not for other facilities. As a result, there is increased risk that  the integrity of these networks and devices could be compromised.
section_title: Physical Security Controls Did Not Effectively Limit Access to Sensitive Control Systems
paragraphs: Physical security controls are important for protecting computer facilities  and resources from espionage, sabotage, damage, and theft. These  controls restrict physical access to computer resources, usually by limiting  access to the buildings and rooms in which the resources are housed and  by periodically reviewing the access granted in order to ensure that access  continues to be appropriate. TVA policy requires that appropriate physical  and environmental controls be implemented to provide security  commensurate with the level of risk and magnitude of harm that would  result from loss, misuse, unauthorized access, or modification of  information or information systems. Further, NIST policy requires that  federal organizations implement a variety of physical security controls to  protect information and industrial control systems and the facilities in  which they are located.
TVA had taken steps to provide physical security for its control systems.  For example, it had issued electronic badges to agency personnel and  contractors to help control access to many of its sensitive and restricted  areas. TVA had also established law enforcement liaisons that help ensure  additional backup security and facilitate the accurate flow of timely  security information between appropriate government agencies. In  addition, the agency had implemented physical security training for its  employees to help achieve greater security awareness and accountability.
However, the agency had not effectively implemented physical security  controls at various locations, as the following examples illustrate:    Live network jacks connected to TVA’s internal network at certain  facilities we reviewed had not been adequately secured from access by the  public.
TVA did not adequately control or change its keys to industrial control  rooms containing sensitive equipment at one facility we reviewed. For  example, the agency could neither account for all keys issued at the  facility, which relies on manual locks for the security of rooms containing  sensitive computer and control equipment, nor could it determine when  keys had last been changed.
TVA did not have an effective visitor control program at one facility we  reviewed. For example, the agency had not maintained a visitor log  describing visitors’ names, organizations, purpose of visits, forms of  identification, or the names of the persons visited.
Physical security policies and plans were either in draft form or were  nonexistent.
Rooms containing sensitive IT equipment had not been adequately  environmentally protected. For example, sufficient emergency lighting  was not available outside the control room at one facility we reviewed, a  server room at the facility had no smoke detection capability, a control  room at the facility contained a kitchen (a potential fire and water hazard),  and a communications room had batteries collocated with sensitive  communications gear.
TVA had not always ensured that access to sensitive computing and  industrial control systems resources had been granted to only those who  needed it to perform their jobs at one facility we reviewed. About 75  percent of those who were issued facility badges had access to a facility  computer room, but the vast majority of these badgeholders did not need  access to the room. While TVA officials stated that all of those with access  had been through the background investigation and training process  required for all employees at the facility, an underlying principle for secure  computer systems and data recommended by NIST is that users should be  granted only those access rights and permissions needed to perform their  official duties.
As a consequence of weaknesses such as these, increased risk exists that  sensitive computing resources and data could be inadvertently or  deliberately misused or destroyed.
section_title: Cumulative Effect of Inconsistencies and Weaknesses in Layered Network Defense Placed Critical Infrastructure Control Systems at Risk
paragraphs: Federal guidance and best practices in information security call for the use  of multiple layers of defense to secure information resources. These  multiple layers include the use of protection mechanisms and key network  control points such as firewalls, routers, and intrusion detection systems  to segment and control access to networks. Higher risk networks and  devices, such as critical infrastructure control systems, may require  additional security controls and should be on networks that are separate  from lower risk devices.
TVA had deployed a layered defense model to control access between and  among the corporate and control systems networks. For example, in all  cases we examined, control systems were located on networks that had  been segmented from business computing resources. The agency had also  deployed protection mechanisms such as firewalls, router access control  lists, virtual local area networking, and physical security controls at  multiple locations throughout its network. For example, TVA’s  transmission control organization used layered networks with increasing  levels of security to separate critical control devices from the corporate  network.
However, these mechanisms and information security controls had been  inconsistently applied. As a result, the effectiveness of the multiple layers  of defense was limited. For example, while the transmission control  organization network restricted access to control systems using multiple  firewalls at outer and inner network boundaries, some plant systems had  significantly fewer levels of security to reach control systems that  impacted the same facilities. In addition, specific weaknesses in security  configurations on key systems further reduced the overall effectiveness of  security controls. The cumulative effect of these individual weaknesses  and the interconnectedness of TVA critical infrastructure control systems  places these systems at risk of compromise or disruption from internal  and external threats.
section_title: Information Security Management Program Was Not Consistently Implemented across TVA’s Critical Infrastructure
paragraphs: An underlying reason for TVA’s information security control weaknesses is  that it had not consistently implemented significant elements of its  information security program. The effective implementation of an  information security program includes implementing the key elements  required under FISMA and the establishment of a continuing cycle of  activity—which includes developing an inventory of systems, assessing  risk, developing policies and procedures, developing security plans,  testing and monitoring the effectiveness of controls, identifying and  tracking remedial actions, and establishing appropriate training. TVA had  not consistently implemented key elements of these activities. As a result  of not fully developing and implementing its information security program,  an increased potential for disruption or compromise of its control systems  exists.
subsections: section_title: Inventory of Systems Was Not Complete or Accurate
paragraphs: FISMA requires that each agency develop, maintain, and annually update  an inventory of major information systems operated by the agency or that  are under its control. A complete and accurate inventory of major  information systems is a key element of managing the agency’s  information technology resources, including the security of those  resources. The inventory can be used to track agency systems for  purposes such as periodic security testing and evaluation, patch  management, contingency planning, and identifying system  interconnections. TVA requires that the senior agency information security  officer maintain an authoritative inventory of general support systems,  major applications, major information systems, and minor applications.
TVA did not have a complete and accurate inventory of its control  systems. In its fiscal year 2007 FISMA submission, TVA included in its  inventory of major applications the transmission and the hydro  automation control systems. Although TVA stated that the plant control  systems at its nuclear and fossil facilities were minor applications, these  applications had not been included in TVA’s inventory of minor  applications or accounted for as part of a consolidated general support  system. These systems are essential to automated operation of generation  facilities. At the conclusion of our review, agency officials stated they had  developed a plan to develop a more complete and accurate system  inventory by September 2008. Until TVA has a complete and accurate  inventory of its control systems, it cannot ensure that the appropriate  security controls have been implemented to protect these systems.
section_title: TVA Had Not Assessed Risk for Almost All of Its Control Systems
paragraphs: FISMA mandates that agencies assess the risk and magnitude of harm that  could result from the unauthorized access, use, disclosure disruption,  modification, or destruction of their information and information systems.  The Federal Information Processing Standard (FIPS) 199, Standards for  Security Categorization of Federal Information and Information  Systems, and related NIST guidance provide a common framework for  categorizing systems according to risk. The framework establishes three  levels of potential impact on organizational operation, assets, or  individuals should a breach of security occur—high (severe or  catastrophic), moderate (serious), and low (limited)—and it is used to  determine the impact for each of the FISMA-specified security objectives  of confidentiality, integrity, and availability. Once determined, security  categories are to be used in conjunction with vulnerability and threat  information in determining minimum security requirements for the system  and in assessing the risk to an organization. Risk assessments help ensure  that the greatest risks have been identified and addressed, increase the  understanding of risk, and provide support for needed controls. Office of  Management and Budget (OMB) Circular A-130, appendix III, prescribes  that risk be assessed when significant changes are made to major systems  and applications in an agency’s inventory or at least every 3 years.  Consistent with NIST guidance, TVA policy states that risk assessments  should be updated to reflect the results of security tests and evaluations.
TVA had not completed assigning risk levels or assessing the risk of its  control systems. While TVA categorized the transmission and hydro  automation control systems as high-impact systems using FIPS 199, its  nuclear division and fossil business unit, which include its coal and  combustion turbine facilities, had not assigned risk levels to their control  systems. Further, although TVA had performed a risk assessment for the  transmission control system, the risk assessment did not include the risks  associated with the newly identified vulnerabilities identified during the  latest security test and evaluation. TVA had not completed risk  assessments for the control systems at their nuclear, hydroelectric, coal,  and combustion turbine facilities. According to TVA officials, the agency  plans to complete risk assessments by May 2008 at the nuclear facility and  June 2008 at the hydroelectric facility. For the fossil facility and all  remaining control systems throughout TVA, agency officials stated that  they would complete the security categorization of these systems by the  end of September 2008. However, no date has been set for completion of  risk assessments. Without assigned risk levels, TVA cannot make risk- based decisions on the security needs of their information and information  systems. Moreover, until TVA assesses the risks of all its control systems,  the agency cannot be assured that its control systems apply the  appropriate level of controls to help prevent their unauthorized access,  use, disclosure, disruption, modification, or destruction.
section_title: Security Policies Existed but Were Not Always Consistent and Did Not Clearly Define All Roles and Responsibilities
paragraphs: A key task in developing, documenting, and implementing an effective  information security program is to establish and implement risk-based  policies, procedures, and technical standards that cover security over an  agency’s computing environment. If properly implemented, policies and  procedures can help to reduce the risk that could come from unauthorized  access or disruption of services. Because security policies are the primary  mechanism by which management communicates its views and  requirements, it is important to document and implement them.
Several shortcomings existed in TVA’s information security policies. First,  the agency had not consistently applied information security policies to its  control systems. Second, business unit security policies were not always  consistent with overall agency information security policies. Third, cyber  security responsibilities for interfaces between TVA’s transmission control  system and its fossil and hydroelectric generation units had not been  documented. Fourth, TVA’s patch management process was not in  compliance with federal guidance. Finally, physical security standards for  control system sites were in draft.
subsections: section_title: TVA Had Not Consistently Applied Information Security Policies to Control Systems
paragraphs: TVA had developed and documented policies, standards, and guidelines  for information security; however, it had not consistently applied these  policies to its control systems. Although neither FISMA nor TVA’s  agencywide IT security policy explicitly mentions control systems, our  analysis of NIST guidance and the stated position of NIST officials is that  the guidance does apply to industrial control systems, such as the systems  that TVA uses to operate critical infrastructures. Furthermore, NIST has  recently developed and released guidance to assist agencies in applying  federal IT security requirements to control systems. As a result of not  applying this guidance with the same level of rigor to its control systems,  numerous shortfalls existed in TVA’s information security management  program for its control systems, including outdated risk assessments;  incomplete system security categorizations, system security plans, and  testing and evaluation activities; and an ineffective remediation process.  TVA officials stated that they are in the process of applying current NIST  criteria to their control systems and plan to complete this process by the  end of fiscal year 2008. Until TVA consistently applies federal IT security  policies to its control systems and addresses identified weaknesses, its  control systems will remain at risk of compromise and disruption.
section_title: Business Unit Policies Were Not Consistent with Overall Agency Policy
paragraphs: While two TVA business units had developed IT security policies to  address anticipated cyber security guidance from their respective  industries, these policies were not always consistent with agencywide IT  security policy. According to TVA policy, business units may establish  their own IT security policies but must still comply with agencywide IT  security policy. For example, TVA’s Nuclear Power Group had developed a  cyber security policy and the Power Systems Operations business unit had  developed two cyber security policies—one business unit policy that was  in draft, and one approved policy developed by and applicable to the unit’s  Transmission and Reliability Organization. These policies addressed many  of the same issues as TVA’s agencywide IT security policy, including  establishing roles and responsibilities, access controls, configuration  management, training, and emergency planning and response. However,  the policies were not always consistent with the agencywide IT security  policy. For example, although both the Nuclear Power Group and the  Transmission and Reliability Organization policies had been developed to  establish requirements for cyber security of plant systems, neither policy  directed system security officers to implement minimum baseline security  controls to protect the confidentiality, integrity, and availability of these  systems, as is required by agency policy, nor did they establish a link or  reference to agencywide IT security policy or federal IT security  requirements. Although the Power System Operations cyber security  policy reiterated requirements outlined by FISMA and the TVA IT security  policy, this policy remained in draft. The existence of inconsistent policies  at different levels of TVA could hinder its ability to apply IT security  requirements consistently across the agency. Without developing and  implementing consistent policies, procedures, and standards across all  agency divisions and groups, TVA has less assurance that its systems  controlling critical infrastructure are protected from unauthorized access  and cyber threats.
section_title: Cyber Security Responsibilities for Interfaces with Transmission Organization Were Not Defined
paragraphs: NIST guidance states that organizations should authorize all connections  from an information system to another information system through the use  of system connection agreements. Documentation should include  security roles and responsibilities and any service level agreements, which  should define the expectations of performance for each required security  control, and remedy and response requirements for any identified instance  of noncompliance.
The agreements established by TVA’s Transmission and Reliability  Organization with other TVA business units did not fully address  information that should be included based on NIST guidance. For  example, the control systems operated by the Transmission and Reliability  Organization interface with power plant control systems operated by  TVA’s fossil and hydroelectric business units. Although the transmission  organization had established agreements with the fossil and hydroelectric  business units, these agreements made no mention of cyber security roles  and responsibilities, performance expectations for security controls, and  remedy and response requirements for noncompliance. TVA officials  stated that the type of interface between the transmission control system  and individual plant systems means that, in most cases, a cyber security  incident on a plant control network would not impact the overall  transmission control network. While the likelihood of direct transmission  of malware such as a virus might be small, without clear documentation of  information required in an intergroup agreement, TVA faces the risk that  security controls may not be in place or work as intended at an individual  plant, resulting in a situation where critical generation equipment may not  be able to start, safely shut down, or otherwise be controlled by the  transmission control system when necessary. This is particularly of  concern because of the variation in cyber security controls that we  observed between the overall transmission control system and the  individual plants. Without clear documentation of cyber security-related  roles and responsibilities, TVA faces the risk that security controls may  not be in place or work as intended.
section_title: Patch Management Policies Were Not in Compliance with NIST Guidance
paragraphs: NIST guidance states that federal agencies should create a comprehensive  patch management process. The process should include    monitoring of security sources for vulnerability announcements;    an accurate inventory of the organization’s IT resources, using  commercially available automated inventory management tools whenever  possible;   prioritization of the order in which the vulnerabilities are addressed with a  focus on high-priority systems such as those essential for mission-critical  operations; and    automated deployment of patches to IT devices using enterprise patch  management tools.
TVA had not fully implemented such a comprehensive process. It had a  patch management process, including staff whose primary responsibility is  to monitor security sources for vulnerability announcements. However,  the agency lacked an accurate inventory of its IT resources produced  using an automated management tool. For example, agency staff did not  have timely access to version numbers and build numbers of software  applications in the agency, although officials stated this information could  be obtained manually. In addition, the agency’s patch management policy  did not apply to individual plant-level control systems or network  infrastructure devices such as routers and switches.
Furthermore, TVA’s written guidance on patch management provided only  limited guidance on how to prioritize vulnerabilities. For example, the  guidance did not refer to the criticality of IT resources. In addition, as  previously noted, the agency had not categorized the impact of many of its  control systems. The guidance also did not specify situations for which it  was acceptable to upgrade or downgrade a vulnerability’s priority from  that given by industry standard sources such as the vendor or third-party  patch tracking services. As a result, patches that were identified as critical,  meaning they should be applied immediately to vulnerable systems, were  not applied in a timely manner. For example, agency staff had reduced the  priority of three vulnerabilities identified as critical or important by the  vendor or a patch tracking service and did not provide sufficient  documentation of the basis for this decision. TVA also did not document  many vulnerabilities on its systems. For a 15-month period, TVA  documented its analysis of 351 reported vulnerabilities, while NIST’s  National Vulnerability Database reported about 2,000 vulnerabilities rated  as high or medium-risk for the types of systems in operation at TVA for the  same time period. Finally, the agency lacked an automated tool to assess  the deployment of many types of application patches. As a result, certain  systems were missing patches more than 6 months past TVA deadlines for  patching. Without a fully effective patch management process, TVA faces  an increased risk that critical systems may remain vulnerable to known  vulnerabilities and be open to compromise or disruption.
section_title: Physical Security Policies Remained in Draft
paragraphs: NIST guidance states that organizations should develop formal  documented physical security policies and procedures to facilitate the  implementation of physical and environmental protection controls.  However, TVA’s physical security standards for protection of its assets,  including sensitive computer and industrial control equipment, as well as  employees, contractors, visitors, and the general public, had been drafted  but not approved by management. These standards are intended to  provide clear and consistent physical security policy for all nonnuclear  facilities. According to TVA Police officials, most sites budget for and  implement their own physical security guidance and measures. Finalized  physical security standards agencywide would provide consistent  guidelines for facilities to make risk-based decisions on implementing  these recommendations. Consequently, TVA has less assurance that  control systems will be consistently and effectively protected from  inadvertent or deliberate misuse including damage or destruction.
section_title: Security Plans for Most Control Systems Had Not Been Completed
paragraphs: The objective of system security planning is to improve the protection of  IT resources. A system security plan provides a complete and up-to-date  overview of the system’s security requirements and describes the controls  that are in place—or planned—to meet those requirements. FISMA  requires that agency information security programs include subordinate  plans for providing adequate information security for networks, facilities,  and systems or groups of information systems, as appropriate. OMB  Circular A-130 specifies that agencies develop and implement system  security plans for major applications and for general support systems and  that these plans address policies and procedures for providing  management, operational, and technical controls. NIST guidance states  that minor applications that are not connected to a general support system  or major application should be described in a general support system plan  that has either a common physical location or is supported by the same  organization. Further, TVA policy states that minor applications should be  briefly described in a general support system security plan. NIST guidance  states that security plans should contain key information needed to select  the appropriate security controls, such as the FIPS 199 category and the  certification and accreditation status of the connected systems. Plans  should also be updated to include the latest security test and evaluation  and risk assessment results.
TVA had only developed a system security plan that covered two of the six  facilities we reviewed, and this plan was incomplete and not up-to-date.  The transmission control system security plan, which addressed systems  at two transmission control centers, included many elements required by  NIST, such as the description of the individuals responsible for security,  and addressed management, operational, and technical controls. Although  the plan listed interconnected systems, it did not completely address  interconnectivity with other systems operated by other organizations.  Specifically, it did not include essential information needed to select the  appropriate security controls, such as the FIPS 199 category or the  certification and accreditation status of the connected systems. Further,  the plan was not updated to include the latest security test and evaluation  or risk assessment results. According to agency officials, TVA is  developing a system security plan for its hydroelectric automation control  system as part of its certification and accreditation process. Agency  officials stated that this plan will be completed by June 2008.
TVA nuclear and fossil facilities had not developed security plans for their  control systems. Agency officials stated that they were planning to develop  security plans and complete the certification and accreditation process for  these control systems. The plan for the nuclear facility is scheduled to be  completed by June 2008. For the fossil facility, TVA officials stated that  they intend to complete a security plan and certification and accreditation  activities based on the results of security categorizations that will be  completed by September 2008. However, no time frame has been set for  completion of the plan or accreditation. Until these activities are  completed, TVA cannot ensure that the security requirements have been  identified and that the appropriate controls will be in place to protect  these critical control systems.
section_title: General Security Awareness Training Was Completed, but Training for Specific Roles Was Not Completed
paragraphs: FISMA mandates that federal employees and contractors who use agency  information systems be provided with periodic training in information  security awareness. FISMA also requires agencies to provide appropriate  training on information security to personnel who have significant security  responsibilities. This training, described in NIST guidance, should inform  personnel, including contractors and other users of information systems  supporting the operations and assets of an agency, of information security  risks associated with their activities and their roles and responsibilities to  properly and effectively implement the practices that are designed to  reduce these risks. Depending on an employee’s specific security role,  training could include specialized topics such as incident detection and  response, physical security, or firewall configuration. TVA also has a  policy that requires that all employees and others who have access to its  corporate network to complete annual security awareness training. The  policy requires that employees and contractors who do not complete the  training within a set time frame have their network access suspended.
Although for fiscal year 2007 TVA reported that 98 percent of its  employees and contractors completed its annual security awareness  training, other shortfalls existed in TVA’s training program. For example,  the agency policy of suspending network access for employees who did  not complete security awareness training did not apply to control system- specific networks, such as those at the nuclear, hydroelectric, and fossil  facilities we reviewed. At these sites, there were no controls in place to  enforce completion of the required training by employees using these  control systems.
In addition, a substantial number of TVA employees who have significant  security responsibilities did not complete role-based training in the last  fiscal year, and the required training did not include specialized technical  topics. In fiscal year 2007, TVA reported that only 25 percent of 197  applicable employees who had significant IT security responsibilities had  completed role-based training, compared with 86 percent and 72 percent  who reportedly received such training in fiscal years 2005 and 2006,  respectively. According to agency officials, training had not been  completed primarily due to a lack of staff to provide the training.  Furthermore, the role-based training that was required was focused on  management and procedural issues. TVA had technical security training  available to its information security staff, which comprised approximately  14 of the 197 employees who needed role-based training, but this training  was not required. For these 14 staff, TVA reported a 100 percent  completion rate for the technical training. At the end of our review, agency  officials provided a plan to improve the number of employees completing  role-based training and to examine adding technical training to training  requirements. The plan is to be completed by July 2008. Until this plan is  fully implemented, security lapses are more likely to occur and could  contribute to information security weaknesses at TVA.
section_title: TVA Did Not Adequately Test and Evaluate the Effectiveness of Security Practices
paragraphs: A key element of an information security program is ongoing testing and  evaluation to ensure that systems are in compliance with policies and that  the policies and controls are both appropriate and effective. Testing and  evaluation demonstrates management’s commitment to the security  program, reminds employees of their roles and responsibilities, and  identifies areas of noncompliance and ineffectiveness requiring  remediation. Starting in fiscal year 2007, OMB required agencies to  discontinue using SP 800-26 and to use NIST SP 800-53A for the  assessment of security controls effectiveness when performing periodic  security testing and evaluation of their information systems. In addition,  TVA policy requires all minor applications to be assigned to a general  support system or major application that is tested and evaluated as part of  the certification and accreditation process performed every 3 years.
TVA did not properly test and evaluate all of its control systems. Although  TVA had performed annual self-assessments of the two control systems  designated as major applications (transmission and hydro automation  control systems) in fiscal year 2007, it did so using outdated NIST  guidance contained in SP 800-26, rather than the current guidance in SP  800-53A. Of these two control systems, TVA performed a complete test and  evaluation of the security controls on one of the systems—the  transmission control system—within the last 3 years. Although TVA  officials at the nuclear and fossil facilities considered their plant-level  control systems to be minor applications, they were not part of any  general support system. As a result, TVA did not appropriately identify,  test, or evaluate the effectiveness of the security controls in place for the  control systems at these facilities. Without appropriate tests and  evaluations of all its control systems, the agency has limited assurance  that policies and controls are appropriate and working as intended.  Additionally, increased risk exists that undetected vulnerabilities could be  exploited to allow unauthorized access to these critical systems.
section_title: Most Remedial Action Plans Had Not Been Developed
paragraphs: A remedial action plan is a key component described in FISMA. Such a  plan assists agencies in identifying, assessing, prioritizing, and monitoring  progress in correcting security weaknesses that are found in information  systems. In its annual FISMA guidance to agencies, OMB requires  agencies’ remedial action plans, also known as plans of action and  milestones, to include, at a minimum, the resources necessary to correct  an identified weakness, the original scheduled completion date, the status  of the weakness as completed or ongoing, and key milestones with  completion dates.According to TVA policy, the agency should document  weaknesses found during security assessments and document any planned  remedial actions to correct any deficiencies.
TVA did not always address known significant deficiencies in its remedial  action plans. The agency had developed a plan of action and milestones  for its transmission control system; however, it did not do so for the  control systems at the fossil, hydroelectric, or nuclear facilities. In  addition, while the agency tracks weaknesses identified by the TVA  Inspector General for its transmission control system, it did not include  these weaknesses in its plan of action and milestones. Until the agency  implements an effective remediation process for all control systems, it will  not have assurance that the proper resources will be applied to known  vulnerabilities or that those vulnerabilities will be properly mitigated.
section_title: Incident Response Procedures Had Not Been Finalized
paragraphs: Even strong controls may not block all intrusions and misuse, but  organizations can reduce the risks associated with such events if they take  steps to promptly detect, report, and respond to them before significant  damage is done. In addition, analyzing security incidents allows  organizations to gain a better understanding of the threats to their  information and the costs of their security-related problems. Such analyses  can pinpoint vulnerabilities that need to be eliminated so that they will not  be exploited again. Incident reports can be used to provide valuable input  for risk assessments, can help in prioritizing security improvement efforts,  and can illustrate risks and related trends for senior management. FISMA  and NIST guidance require that agency information security programs  include procedures for detecting, reporting, and responding to security  incidents, including reporting them to the U.S. Computer Emergency  Readiness Team (US-CERT). Furthermore, NIST guidance prescribes  network and host-based intrusion detection systems as a means of  protecting systems from the threats that come with increasing network  connectivity.
TVA had developed incident detection, response, and reporting  procedures. However, while the TVA organization responsible for  operating its transmission control center had approved incident response  and reporting procedures, the agencywide incident response and reporting  procedure remained in draft form, although it is currently being used by  TVA information security personnel. According to agency officials, the  procedure is being revised and finalized to align with incident reporting  guidelines developed by US-CERT. Until TVA finalizes these procedures, it  cannot be assured that facilities are prepared to respond to and report  incidents in an effective manner.
section_title: Contingency Planning Activities Were Completed but Were Not Fully Documented
paragraphs: Contingency planning includes developing and testing plans and activities  so that when unexpected events occur, critical operations can continue  without disruption or can be promptly resumed and that critical and  sensitive data are protected. If contingency planning controls are  inadequate, even relatively minor interruptions can result in a loss of  system function and expensive recovery efforts. For some TVA control  systems, system interruptions or malfunctions could result in loss of  power, injuries, or loss of life. Given these severe implications, it is critical  that an entity have in place (1) procedures for protecting information  systems and minimizing the risk of unplanned interruptions and (2) a plan  to recover critical operations should interruptions occur. To determine  whether recovery plans will work as intended, they should be tested  periodically in disaster-simulation exercises. FISMA requires that each  federal agency implement an information security program that includes  plans and procedures to ensure continuity of operations for information  systems that support the operation and assets of the agency.
TVA had taken steps to address contingency planning for physical  incidents such as fire, explosion, and natural disasters, and for other  events such as cyber incidents. At the facilities we reviewed, staff  performed regular drills and tests to address physical contingencies.  According to agency officials, in many cases, these same drills are  applicable to cyber incidents that could have physical consequences. In  addition, the agency had developed backup procedures for key  information resources, including those that support its control systems. In  TVA’s transmission control centers, written backup procedures existed;  however, in the hydroelectric, coal, and gas turbine facilities we reviewed,  the backup procedures were not documented. Until TVA consistently  documents backup procedures across all of its facilities, it has limited  assurance that all TVA facilities will be able to respond appropriately in  the event of a physical or cyber incident.
section_title: Conclusions
paragraphs: TVA’s power generation and transmission critical infrastructures are  important to the economy of the southeastern United States and the  safety, security, and welfare of millions of people. Control systems are  essential to the operation of these infrastructures; however, multiple  information security weaknesses existed in both the agency’s corporate  network and individual control systems networks and devices. As a result,  although TVA had implemented multiple layers of information security  controls to protect its critical infrastructures, such as segmenting control  systems networks from the corporate network, in many cases, these layers  were not as effective as intended. An underlying cause for these  weaknesses is that the agency had not consistently implemented its  information security program throughout the agency. If TVA does not take  sufficient steps to secure its control systems and implement an  information security program, it risks not being able to respond properly  to a major disruption that is the result of an intended or unintended cyber  incident, which could affect the agency’s operations and its customers.
section_title: Recommendations for Executive Action
paragraphs: To improve the implementation of information security program activites  for the control systems governing TVA’s critical infrastructures, we are  recommending that the Chief Executive Officer of TVA take the following  19 actions:    Establish a formal, documented configuration management process for  changes to software governing control systems at TVA hydroelectric and  fossil facilities.
Establish a patch management policy for all control systems.
Establish a complete and accurate inventory of agency information  systems that includes each TVA control system either as a major  application, or as a minor application to a general support system.
Categorize and assess the risk of all control systems.
Update the transmission control system risk assessment to include the risk  associated with vulnerabilities identified during security testing and  evaluations and self-assessments.
Revise TVA information security policies and procedures to specifically  mention their applicability to control systems.
Ensure that any division-level information security policies and  procedures established to address industry regulations or guidance are  consistent with, refer to, and are fully integrated with TVA corporate  security policy and federal guidance.
Revise the intergroup agreements between TVA’s Transmission and  Reliability Organization and its fossil and hydroelectric business units to  explicitly define cyber security roles and responsibilities.
Revise TVA patch management policy to clarify its applicability to control  systems and network infrastructure devices, provide guidance to prioritize  vulnerabilities based on criticality of IT resources, and define situations  where it would be appropriate to upgrade or downgrade a vulnerability’s  priority from that given by industry standard sources.
Finalize draft TVA physical security standards.
Complete system security plans that cover all control systems in  accordance with NIST guidance and include all information required by  NIST in security plans, such as the FIPS 199 category and the certification  and accreditation status of connected systems.
Enforce a process to ensure that employees who do not complete required  security awareness training cannot access control system-specific  networks.
Ensure that all designated employees complete role-based security  training and that this training includes relevant technical topics.
Develop and implement a TVA policy to ensure that periodic (at least  annual) assessments of control effectiveness use NIST SP 800-53A for  major applications and general support systems.
Perform assessments of control effectiveness following the methodology  in NIST SP 800-53A.
Develop and implement remedial action plans for all control systems.
Include the results of inspector general assessments in the remedial action  plan for the transmission control system.
Finalize the draft agencywide cyber incident response procedure.
Document backup procedures at all control system facilities.
In a separate report designated “Limited Official Use Only,” we are also  making 73 recommendations to the Chief Executive Officer of TVA to  address weaknesses in information security controls.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, the Executive Vice President  of Administrative Services for TVA agreed on the importance of protecting  critical infrastructures and described several actions TVA has taken to  strengthen information security for control systems, such as centralizing  responsibility for cyber security within the agency. The Executive Vice  President concurred with all 19 recommendations in this report and  provided information on steps the agency was taking to implement the  recommendations. A copy of the agency’s response is included in  appendix II.
Additionally, in a meeting with GAO officials, TVA officials expressed  concerns about the level of detail in this report. Based on that meeting and  subsequent discussions with agency officials, we have modified the  wording in this report to address the agency’s concerns. The agency also  provided technical comments that we have incorporated where  appropriate.
We are sending copies of this report to OMB, the TVA Inspector General  and other interested parties. We will also make copies available to others  upon request. In addition, the report will be available at no charge on the  GAO Web site at http:www.gao.gov.
If you have any questions on matters discussed in this report, please  contact Gregory Wilshusen at (202) 512-6244 or Nabajyoti Barkakati (202)  512-4499, or by e-mail at wilshuseng@gao.gov and barkakatin@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: The objective of our review was to determine if the Tennessee Valley  Authority (TVA) has effectively implemented appropriate information  security practices for the control systems used to operate its critical  infrastructure. We conducted our review using our Federal Information  System Controls Audit Manual, a methodology for reviewing information  system controls that affect the confidentiality, integrity, and availability of  computerized data. We focused our work on the control systems located at  six TVA facilities. These facilities were selected to provide a cross-section  of the variety of control systems by type of generation facility (coal,  combustion turbine, hydroelectric, and nuclear) and function (generation  and transmission).
To evaluate the effectiveness of TVA’s information security practices, we  conducted tests and observations using federal guidance, checklists, and  vendor best practices for information security. Where federal  requirements or guidelines, including National Institute of Standards and  Technology (NIST) guidance, were applicable, we used them to assess the  extent to which TVA had complied with specific requirements.  Specifically, we used NIST guidance for the security of federal information  systems. For example, we    analyzed the password hashing implementation used for identification and    evaluated and reviewed the complexity and expiration of passwords on  servers to determine if strong password management was enforced;   examined user and application system authorizations to determine  whether they had more permissions than necessary to perform their  assigned functions;   analyzed system configurations to determine whether sensitive data were    observed whether system security software was configured to log  successful system changes;  inspected key servers, workstations, and network infrastructure devices to  determine whether critical patches had been installed or were up-to-date; tested and observed physical access controls to determine if computer  facilities and resources were being protected from espionage, sabotage,  damage, and theft; and   synthesized the information obtained about networks and applications to  develop an accurate understanding of overall network and system  architecture.
The Federal Information Security Management Act of 2002 (FISMA)  establishes key elements of an effective agencywide information security  program. We evaluated TVA’s implementation of these key elements by  reviewing TVA’s system inventory to determine whether it contained an  accurate and comprehensive list of control systems;   analyzing risk assessments for key TVA systems to determine whether  risks and threats were documented;   examining security plans to determine if management, operational, and  technical controls were in place or planned and whether these security  plans were updated;   analyzing TVA policies, procedures, practices, and standards to determine  their effectiveness in providing guidance to personnel responsible for  securing information and information systems; inspecting training records for personnel with significant responsibilities  to determine if they received training commensurate with those  responsibilities;   analyzing test plans and test results for key TVA systems to determine  whether management, operational, and technical controls were adequately  tested at least annually and were based on risk;    evaluating TVA’s process to correct weaknesses and determining whether  remedial action plans complied with federal guidance; and    examining contingency plans for key TVA systems to determine whether  those plans had been tested or updated.
To conduct our work, we reviewed and analyzed relevant documentation  and held discussions with key security representatives, system  administrators, and management officials to determine whether  information system controls were in place, adequately designed, and  operating effectively. We also reviewed previous reports issued by the TVA  Inspector General’s Office. We conducted this performance audit from  March 2007 to April 2008 in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Tennessee Valley Authority
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals named above, Nancy DeFrancesco and Lon  Chin, Assistant Directors; Angela Bell; Bruce Cain; Mark Canter; Heather  Collins; West Coile; Kirk Daubenspeck; Neil Doherty; Vijay D’Souza; Nancy  Glover; Sairah Ijaz; Myong Kim; Stephanie Lee; Lee McCracken; Duc Ngo;  Sylvia Shanks; John Spence; and Chris Warweg made key contributions to  this report.