Information Security: Agencies Face Challenges in Implementing Effective Software Patch Management Processes
Published: 2004-06-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Flaws in software code can introduce vulnerabilities that may be exploited to cause significant damage to federal information systems. Such risks continue to grow with the increasing speed, sophistication, and volume of reported attacks, as well as the decreasing period of the time from vulnerability announcement to attempted exploits. The process of applying software patches to fix flaws--patch management--is critical to helping secure systems from attacks. At the request of the House Committee on Government Reform and the Subcommittee on Technology, Information Policy, Intergovernmental Relations, and the Census, GAO reviewed the (1) reported status of 24 selected agencies in performing effective patch management practices, (2) tools and services available to federal agencies, (3) challenges to this endeavor, and (4) additional steps that can be taken to mitigate risks created by software vulnerabilities. This testimony highlights the findings of GAO's report, which is being released at this hearing.
section_title: What GAO Found
paragraphs: Agencies are generally implementing certain common patch management-related practices, such as inventorying their systems and providing information security training. However, they are not consistently implementing other common practices. Specifically, not all agencies have established patch management policies and procedures. Moreover, not all agencies are testing all patches before deployment, performing documented risk assessments of major systems to determine whether to apply patches, or monitoring the status of patches once they are deployed to ensure that they are properly installed. Commercial tools and services are available to assist agencies in performing patch management activities. These tools and services can make patch management processes more efficient by automating time-consuming tasks, such as scanning networks and keeping up-to-date on the continuous releases of new patches. Nevertheless, agencies face significant challenges to implementing effective patch management. These include, among others, (1) the high volume and increasing frequency of needed patches, (2) patching heterogeneous systems, (3) ensuring that mobile systems such as laptops receive the latest patches, and (4) dedicating sufficient resources to assessing vulnerabilities and deploying patches. Agency officials and computer security experts have identified several additional measures that vendors, the security community, and the federal government can take to address the risks associated with software vulnerabilities. These include, among others, adopting more rigorous software engineering practices to reduce the number of coding errors that create the need for patches, implementing successive layers of defense mechanisms at strategic points in agency information systems, and researching and developing new technologies to help uncover flaws during software development.

section_title: Letter
paragraphs: This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in  the  United  States.  It  may  be  reproduced  and  distributed  in  its  entirety  without  further  permission  from  GAO.  However,  because  this  work  may  contain  copyrighted  images  or  other  material,  permission  from  the  copyright  holder  may  be  necessary  if  you  wish  to  reproduce this material separately.
I am pleased to be here today to discuss patch management and  steps that agencies can take to mitigate information security risks  resulting from software vulnerabilities. As you know, attackers may  attempt to exploit such vulnerabilities, potentially causing  significant damage to agencies’ computer systems.
My testimony today will highlight the findings of a report requested  by the Subcommittee and full Committee, which we are releasing  today.  This report discusses: (1) the status of 23 of the agencies  under the Chief Financial Officers (CFO) Act of 1990 and the  Department of Homeland Security (DHS) in performing effective  patch management, (2) tools and services available to assist federal  agencies in this endeavor, (3) obstacles to performing effective  patch management, and (4) additional steps that can be taken to  mitigate the risks created by software vulnerabilities.
Our report is based on an extensive search of professional  information technology (IT) security literature, research studies and  reports about cybersecurity-related vulnerabilities (including our  own), and the results of a Web-based survey of the 24 agencies that  we conducted to determine their patch management practices. Our  work was conducted from September 2003 through last month, in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Patch management is a critical process used to help alleviate many  of the challenges involved with securing computing systems from  attack. A component of configuration management, it includes  acquiring, testing, applying, and monitoring patches to a computer  system. Flaws in software code that could cause a program to  malfunction generally result from programming errors that occur  during software development. The increasing complexity and size of  software programs contribute to the growth in software flaws. For  example, Microsoft Windows 2000 reportedly contains about 35  million lines of code, compared with about 15 million lines for  Windows 95. As reported by the National Institute of Standards and  Technology (NIST), based on various studies of code inspections,  most estimates suggest that there are as many as 20 flaws per  thousand lines of software code. While most flaws do not create  security vulnerabilities, the potential for these errors reflects the  difficulty and complexity involved in delivering trustworthy code.
subsections: section_title: Security Vulnerabilities and Incidents Are Increasing
paragraphs: From 1995 through 2003, the CERT Coordination Center  (CERT/CC) reported just under 13,000 security vulnerabilities that  resulted from software flaws. Figure 1 illustrates the dramatic  growth in security vulnerabilities during this period.
As vulnerabilities are discovered, attackers can cause major damage  in attempting to exploit them. This damage can range from defacing  Web sites to taking control of entire systems and thereby being able  to read, modify, or delete sensitive information; destroy systems;  disrupt operations; or launch attacks against other organizations’  systems. Attacks can be launched against specific targets or widely  distributed through viruses and worms.
The sophistication and effectiveness of cyber attacks have steadily  advanced. According to security researchers, reverse-engineering  patches has become a leading method for exploiting vulnerabilities.
By using the same tools used by programmers to analyze malicious  code and perform vulnerability research, hackers can locate the  vulnerable code in unpatched software and build to exploit it.  Reverse engineering starts by locating the files or code that changed  when a patch was installed. Then, by comparing the patched and  unpatched versions of those files, a hacker can examine the specific  functions that changed, uncover the vulnerability, and exploit it.
A spate of new worms has been released since February—most  recently last month—and more than half a dozen new viruses were  unleashed. The worms were variants of the Bagle and Netsky  viruses. The Bagle viruses typically included an infected e-mail  attachment containing the actual virus; the most recent versions  have protected the infected attachment with a password, preventing  anti-virus scanners from examining it. The recent Netsky variants  attempted to deactivate two earlier worms and, when executed,  reportedly make a loud beeping sound. Another worm known as  Sasser, like the Blaster worm discussed later, exploits a  vulnerability in the Microsoft Windows operating system, while the  Witty worm exploits a flaw in certain Internet security software  products.
The number of computer security incidents within the past decade  has risen in tandem with the dramatic growth in vulnerabilities, as  the increased number of vulnerabilities provides more opportunities  for exploitation. CERT/CC has reported a significant growth in  computer security incidents—from about 9,800 in 1999 to over  82,000 in 2002 and over 137,500 in 2003. And these are only the  reported attacks. The director of the CERT Centers has estimated  that as much as 80 percent of actual security incidents go  unreported, in most cases because  there were no indications of penetration or attack,  the organization was unable to recognize that its systems had  been penetrated, or  the organization was reluctant to report the attack.
Figure 2 shows the number of incidents reported to CERT/CC from  1995 through 2003.
According to CERT/CC, about 95 percent of all network intrusions  could be avoided by keeping systems up to date with appropriate  patches; however, such patches are often not quickly or correctly  applied. Maintaining current patches is becoming more difficult, as  the length of time between the awareness of a vulnerability and the  introduction of an exploit is shrinking. For example, the recent  Witty worm was released only a day after the announcement of the  vulnerability it attacked. As figure 3 illustrates, in the last 3 years,  the time interval between the announcement of a particular  vulnerability and the release of its associated worm has diminished  dramatically.
section_title: Exploited Software Vulnerabilities Can Result in Economic Damage and Disruption of Operations
paragraphs: Although the economic impact of a cyber attack is difficult to  measure, a recent Congressional Research Service study cites  members of the computer security industry as estimating that  worldwide, major virus attacks in 2003 cost $12.5 billion. They  further project that economic damage from all forms of digital  attacks in 2004 will exceed $250 billion.
Following are examples of significant damage caused by worms that  could have been prevented had the available patches been  effectively installed:  ●  On January 25, 2003, Slammer reportedly triggered a global  Internet slowdown and caused considerable harm through  network outages and other unforeseen consequences. As  discussed in our April 2003 testimony on the security of federal  systems and critical infrastructures, the worm reportedly shut  down a 911 emergency call center, canceled airline flights, and  caused automated teller machine failures. According to media  reports, First USA Inc., an Internet service provider, experienced  network performance problems after an attack by the Slammer  worm, due to a failure to patch three of its systems. Additionally,  the Nuclear Regulatory Commission reported that Slammer also  infected a nuclear power plant’s network, resulting in the  inability of its computers to communicate with each other,  disrupting two important systems at the facility. In July 2002,  Microsoft had released a patch for its software vulnerability that  was exploited by Slammer. Nevertheless, according to media  reports, Slammer infected some of Microsoft’s own systems.  Reported cost estimates of Slammer damage range between $1.05  billion and $1.25 billion.  ●  On August 11, 2003, the Blaster worm was launched to exploit a  vulnerability in a number of Microsoft Windows operating  systems. When successfully executed, it caused the operating  system to fail. Although the security community had received  advisories from CERT/CC and other organizations to patch this  critical vulnerability, Blaster reportedly infected more than  120,000 unpatched computers in its first 36 hours. By the  following day, reports began to state that many users were  experiencing slowness and disruptions to their Internet service,  such as the need to reboot frequently. The Maryland Motor  Vehicle Administration was forced to shut down, and systems in  both national and international arenas were also affected.  Experts consider Blaster, which affected a range of systems, to  be one of the worst exploits of 2003. Microsoft reported that the  Blaster worm has infected at least 8 million Windows computers  since last August.  ●  On May 1 of this year, the Sasser worm was reported, which  exploits a vulnerability in the Windows Local Security Authority  Subsystem Service component. This worm can compromise  systems by allowing a remote attacker to execute arbitrary code  with system privileges. According to US-CERT (the United States  Computer Emergency Readiness Team), systems infected by  this worm may suffer significant performance degradation.  Sasser, like last year’s Blaster, exploits a vulnerability in a  component of Windows by scanning for vulnerable systems.  Estimates by Internet Security Systems, Inc., place the Sasser  infections at 500,000 to 1 million machines. Microsoft has  reported that 9.5 million patches for the vulnerability were  downloaded from its Web site in just 5 days.
section_title: Federal Efforts to Address Software Vulnerabilities
paragraphs: The federal government has taken several steps to address security  vulnerabilities that affect agency systems, including efforts to  improve patch management. Specific actions include (1) requiring  agencies to annually report on their patch management practices as  part of their implementation of FISMA, (2) identifying vulnerability  remediation as a critical area of focus in the President’s National  Strategy to Secure Cyberspace, and (3) creating US–CERT.
FISMA permanently authorized and strengthened the information  security program, evaluation, and reporting requirements  established for federal agencies in prior legislation. In accordance  with OMB’s reporting instructions for FISMA implementation,  maintaining up-to-date patches is part of system configuration  management requirements. The 2003 FISMA reporting instructions  that specifically address patch management practices include  agencies’ status on (1) developing an inventory of major IT systems,  (2) confirming that patches have been tested and installed in a  timely manner, (3) subscribing to a now-discontinued  governmentwide patch notification service, and (4) addressing  patching of security vulnerabilities in configuration requirements.
The President’s National Strategy to Secure Cyberspace was issued  on February 14, 2003, to identify priorities, actions, and  responsibilities for the federal government—as well as for state and  local governments and the private sector—with specific  recommendations for action to DHS. This strategy identifies the  reduction and remediation of software vulnerabilities as a critical  area of focus. Specifically, it identifies the need for (1) a better–  defined approach on disclosing vulnerabilities, to reduce their  usefulness to hackers in launching an attack; (2) creating common  test beds for applications widely used among federal agencies; and  (3) establishing best practices for vulnerability remediation in areas  such as training, use of automated tools, and patch management  implementation processes.
US-CERT was created last September by DHS’s National Cyber  Security Division (NCSD) in conjunction with CERT/CC and the  private sector. Specifically, US-CERT is intended to aggregate and  disseminate cyber security information to improve warning and  response to incidents, increase coordination of response  information, reduce vulnerabilities, and enhance prevention and  protection. This free service—which includes notification of  software vulnerabilities and sources for applicable patches—is  available to the public, including home users and both government  and nongovernment entities.
section_title: Agencies Are Not Consistently Implementing Common Practices for Effective Patch Management
paragraphs: Common patch management practices—such as establishing and  enforcing standardized policies and procedures and developing and  maintaining a current technology inventory—can help agencies  establish an effective patch management program and, more  generally, assist in improving an agency’s overall security posture.
Our survey results showed that the 24 agencies are implementing  some practices for effective patch management, but not others.  Specifically, all report that they have some level of senior executive  involvement in the patch management process and cited the chief  information security officer (CISO) as being the individual most  involved in the patch management process. The CISO is involved in  managing risk, ensuring that appropriate resources are dedicated,  training computer security staff, complying with policies and  procedures, and monitoring the status of patching activities.
Other areas in which agencies report implementing common patch  management practices are in performing a systems inventory and  providing information security training. All 24 agencies reported that  they develop and maintain an inventory of major information  systems as required by FISMA and do so using a manual process, an  automated tool, or an automated service. Additionally, most of the  24 agencies reported that they provide both on-the-job and  classroom training in computer security, including patch  management, to system owners, administrators, and IT security  staff.
However, agencies are inconsistent in developing patch  management policies and procedures, testing of patches, monitoring  systems, and performing risk assessments. Specifically, not all  agencies have established patch management policies and  procedures. Eight of the 24 surveyed agencies report having no  policies and 10 do not have procedures in place. Additionally, most  agencies are not testing all patches before deployment. Although all  24 surveyed agencies reported that they test some patches against  their various systems configurations before deployment, only 10  agencies reported testing all patches, and 15 agencies reported that  they do not have any testing policies in place. Moreover, although all  24 agencies indicated that they perform some monitoring activities  to assess their network environments and determine whether  patches have been effectively applied, only 4 agencies reported that  they monitor all of their systems on a regular basis. Further, just  under half of the 24 agencies said they perform a documented risk  assessment of all major systems to determine whether to apply a  patch or an alternative workaround. Without consistent  implementation of patch management practices, agencies are at  increased risk of attacks that exploit software vulnerabilities in their  systems.
More refined information on key aspects of agencies’ patch  management practices—such as their documentation of patch  management policies and procedures and the frequency with which  systems are monitored to ensure that patches are installed—could  provide OMB, Congress, and agencies themselves with data that  could better enable an assessment of the effectiveness of an  agency’s patch management processes.
section_title: Automated Tools and Services Can Assist Agencies in Performing Patch Management Activities
paragraphs: Several automated tools and services are available to assist agencies  with patch management. A patch management tool is an application  that automates a patch management function, such as scanning a  network and deploying patches. Patch management services are  third-party resources that provide services such as notification,  consulting, and vulnerability scanning. Tools and services can make  the patch management process more efficient by automating  otherwise time-consuming tasks, such as keeping current on the  continuous flow of new patches.
Commercially available tools and services include, among others,  methods to  inventory computers and the software applications and patches  installed;  identify relevant patches and workarounds and gather them in  one location; group systems by departments, machine types, or other logical manage patch deployment;  scan a network to determine the status of patches and other  corrections made to network machines (hosts and/or clients); assess machines against set criteria, including required system  configurations; access a database of patches;  report information to various levels of management about the  status of the network.
In addition to automated tools and services, agencies can use other  methods to assist in their patch management activities. For  example, although labor-intensive, they can maintain a database of  the versions and latest patches for each server and each client in  their network, and track the security alerts and patches manually.  Agencies can also employ systems management tools with patch- updating capabilities to deploy the patches. This method requires  that agencies monitor for the latest security alerts and patches.  Further, software vendors may provide automated tools with  customized features to alert system administrators and users of the  need to patch and, if desired, to automatically apply patches.
We have previously reported on FedCIRC’s Patch Authentication  and Dissemination Capability (PADC), a service initiated in  February 2003 to provide users with a method of obtaining  information on security patches relevant to their enterprise and  access to patches that had been tested in a laboratory environment.   According to FedCIRC officials, this service was terminated on  February 21, 2004, for a variety of reasons, including low levels of  usage. In the absence of this service, agencies are left to  independently perform all components of effective patch  management. A centralized resource that incorporates lessons  learned from PADC’s limitations could provide standardized  services, such as testing of patches and a patch management  training curriculum.
section_title: Significant Obstacles to Effective Patch Management Remain
paragraphs: Security experts and agency officials have identified several  obstacles to implementing effective patch management; these  include the following:  ●  High volume and increasing frequency of patches. Several of the  agencies we surveyed indicated that the sheer quantity and  frequency of needed patches posed a challenge to the  implementation of the recommended patch management  practices. As increasingly virulent computer worms have  demonstrated, agencies need to keep systems updated with the  latest security patches.  ●  Patching heterogeneous systems. Variations in platforms,  configurations, and deployed applications complicate agencies’  patching processes. Further, their unique IT infrastructures can  make it challenging for agencies to determine which systems are  affected by a software vulnerability.  ●  Ensuring that mobile systems receive the latest patches. Mobile  computers—such as laptops, digital tablets, and personal digital  assistants—may not be on the network at the right time to  receive appropriate patches that an agency deploys and are at  significant risk of not being patched.  ●  Avoiding unacceptable downtime when patching systems that  require high availability. Reacting to new security patches as they  are introduced can interrupt normal and planned IT activities,  and any downtime incurred during the patching cycle interferes  with business continuity, particularly for critical systems that  must be continuously available.  ●  Dedicating sufficient resources to patch management. Despite  the growing market of patch management tools and services that  can track machines that need patches and automate patch  downloads from vendor sites, agencies noted that effective patch  management is a time-consuming process that requires dedicated  staff to assess vulnerabilities and test and deploy patches.
section_title: Additional Steps Can Be Taken to Mitigate Risks
paragraphs: As with the challenges to patch management identified by agencies,  our report also identified a number of steps that can be taken to  address the risks associated with software vulnerabilities. These  include:  ●  Better software engineering. More rigorous engineering  practices, including a formal development process, developer  training on secure coding practice, and code reviews, can be  employed when designing, implementing, and testing software  products to reduce the number of potential vulnerabilities and  thus minimize the need for patching.  ●  Implementing “defense-in-depth.” According to security experts,  a best practice for protecting systems against cyber attacks is for  agencies to build successive layers of defense mechanisms at  strategic points in their IT infrastructures. This approach,  commonly referred to as defense-in-depth, entails implementing a  series of protective mechanisms such that if one fails to thwart  an attack, another will provide a backup defense.  ●  Using configuration management and contingency planning.
Industry best practices and federal guidance recognize the  importance of configuration management when developing and  maintaining a system or network to ensure that additions,  deletions, or other changes to a system do not compromise the  system’s ability to perform as intended. Contingency plans  provide specific instructions for restoring critical systems,  including such elements as arrangements for alternative  processing facilities, in case usual facilities are significantly  damaged or cannot be accessed due to unexpected events such  as temporary power failure, accidental loss of files, or major  disaster.  ●  Ongoing improvements in patch management tools. Security  experts have noted the need for improving currently available  patch management tools. Several patch management vendors  have been working to do just that.  ●  Research and development of new technologies. Software  security vulnerabilities can also be addressed through the  research and development of automated tools to uncover hard-to- see security flaws in software code during the development  phase.  ●  Federal buying power. The federal government can use its  substantial purchasing power to demand higher quality software  that would hold vendors more accountable for security defects in  released products and provide incentives for vendors that supply  low-defect products and products that are highly resistant to  viruses.