Homepage Usability: 50 Websites Deconstructed
Authors: Jakob Nielsen, Marie Tahir
Venue:  (2001)

From the Book:Forming and Managing an Incident Response TeamFrom time to time, weve mentioned the word team in the process of covering various topics related to incident response. This chapter delves into forming and managing an incident response teamwhat a response team is, the rationale for forming an incident response team, major issues that must be addressed, and special management considerations. These topics are particularly important. Many incident response efforts fail or flounder because of mistakes made in forming and/or managing a response team. This chapter again presents the authors perspectives and real-life experiences in dealing with the many issues related to this area. We will begin by considering the most fundamental part of an incident response teamthe meaning of the term itself. What Is an Incident Response Team? In many contexts, you will see incident response equated with incident response team. Equating these two constructs might superficially appear logical, but doing so often constitutes a departure from reality. Why? People who know little or nothing about the process of incident response often become involved in dealing with security-related incidents. Users are a classic example. Suppose a worm infects numerous systems. Users might collaborate to analyze what has happened and to combat the worm, yet they can hardly be called an incident response team. The reason is that an incident response team is a capability responsible for dealing with potential or real information security incidents. A team is assigned a set of duties related to bringing each security-related incident to a conclusion, ideally in accordance with the goals of the organizationit serves. The difference, therefore, between individuals who are dealing with an incident and an incident response team is the missionin terms of job-related responsibilitiesassigned to each. Individuals might sometimes become involved in dealing with incidents, but an incident response team is assigned the responsibility of dealing with incidents as part or all of the job descriptions of the individuals involved. How many individuals must be involved in an incident response effort for them to collectively be considered a team? A team consists of one or more individuals. You might ask how a team can consist of one individual when one person is not, in most situations, sufficient to deal adequately with most incidents. The answer is that one individual can effectively serve as the coordinator of efforts by a number of people. When incident handling efforts are finished, the others involved in the incident are released from any responsibilities they might have had in dealing with incident. But the team member has the ongoing, day-to-day responsibility of handling incidents and will have to deal with the next incident that occurs. Many incident response teams have many team members, each with a specialized role. Consider, for example, the Computer Emergency Response Team Coordination Center (CERT/CC). Some of the many members of this team are engaged in daily operations, receiving reports of incidents and attempting to identify the type, source, impact, and other facets of security-related incidents that are reported. Others attempt to deal with vendors to close known vulnerabilities in operating systems, applications, and so forth. Still others examine data to identify and project incident trends, something that is more related to research. Outsourcing Incident Response Efforts Should an organization have its own incident response effort, or should it contract with a consultancy or contractor to provide incident response support? The answer in most cases is that it depends on a number of basic factors. Lets consider the alternatives. Hiring a Contractor or Consultancy. One of the many advantages of contracting with a commercial incident response team is that the overall cost of dealing with security-related incidents is likely to be lower. Why? Incident response personnelcontractors or consultantsneed to deal only with incidents that occur. Unless there is a plethora of incidents, there is no need to keep regular personnel around to wait for incidents to occur. Additionally, contractors or consultancies usually offer special kinds of expertise that are often not available within any particular organization. Be careful, however. Many consultancies and service providers offer incident response services, some of which are far superior to others. Be sure to ask for references, preferably from current and ex-customers, before signing any contract for incident response services with any consultancy or service provider. Using In-House Capability. The major rationale for developing an in-house incident response capability is to handle incidents in accordance with the policy and cultural/political needs of an organization. Security-related incidents are potentially very sensitive and political; an in-house capability is likely to deal with them in a manner that is most advantageous to the organization (provided, of course, that the individuals within this capability understand the culture and politics of the organization). Why Form an Incident Response Team? Why might some organizations want to form an incident response team? This section focuses on some possible reasons. Ability to Coordinate In general, it is easier to coordinate the efforts of individuals who are on an incident response team because they generally report to the team leader, who can direct them to become involved in one particular activity or another. Expertise Information security incidents are becoming increasingly complex; incident handling experts are thus becoming increasingly necessary. Technical gurus always come in handy when incidents occur, but pure technical expertise is not enough when it comes to many incidents. Having helped with many previous incidents, knowing what policies to consider and procedures to follow, and so forth are just as critical, if not more critical, than pure technical skills. One of the best ways to build expertise is to serve on a dedicated incident response function. Efficiency A team builds a collective knowledge that often leads to increased efficiency. An isolated individual can easily go astray in dealing with an incident, but collective wisdom accrued within a team can help incident response efforts get back on track. Additionally, a team (as opposed to any individual or a few independent individuals) is more likely to develop and follow procedures for incident response, something that boosts efficiency. Ability to Work Proactively Being proactive (that is, adopting measures that address incident response needs before incidents actually occur) is one of the keys to a successful incident response effort. Training users and system administrators to recognize the symptoms of incidents and what to do (as well as what not to do) is a good example of a proactive effort. Although it is possible for any number of individuals to engage in proactive efforts, having a team increases the likelihood that proactive efforts will occur. Having a team allows the luxury of having different persons specialize in different functions, especially in proactive activity. Additionally, successful proactive efforts are often the byproduct of successful collaboration by teams; individuals are not as likely to think of and carry out successful proactive activity. Ability to Meet Agency or Corporate Requirements Another advantage of having an incident response team is that a team is generally better suited to meeting agency or corporate requirements. The main reason is that a team has individuals who are geared toward the same mission. Note that some government agencies and corporations go one step further in that they require (through a management directive or a policy statement) that an incident response team be formed. Serving a Liaison Function Response teams are better suited to serving a liaison function than are individuals because outside entities are not likely to learn of and/or be motivated to deal with individuals. Having a team identity provides extra external visibility as well as credibility, both of which are more suited to the liaison function. Furthermore, a team, in many respects, commands a certain degree of legitimacy within internal and external organizations. Ability to Deal with Institutional Barriers Institutional politics invariably affect virtually any effort that occurs within an institution. Incident response teams (or at least incident response teams sanctioned by senior management), however, provide at least some degree of immunity from politics that provide barriers to incident response efforts. The main reason is that these teams are likely to have more authority to take actionsuch as shutting down systems that have been compromised at the superuser levelthan individuals. Additionally, teams often involve individuals from a cross-section of organizations and groups, making them more politically palatable within a range of an organizations divisions and groups. Issues in Forming a Response Team Forming an incident response team generally is not as easy as it superficially might appear. The individual(s) charged with this responsibility must deal with many key issues, including policy, whether or not a team is really necessary, defining and communicating with a constituency, defining functional requirements, defining the role of the incident response team, staffing the team appropriately, and creating and updating operational procedures. This section discusses these issues. Policy The most important issue in forming and managing an incident response team, all things considered, is policy. Any incident response team must always operate within the constraints of the policy of the organization to which it belongs or that it serves. Suppose, for example, an organization requires that no employee make contact with or answer questions from the press unless that person obtains written approval from the head of the public relations department. Another organizational policy provision might be that no system being attacked can stay connected to the network if it holds extremely valuable resources (such as proprietary data, proprietary source code, and so forth). Additionally, an incident response team might impose its own policy provisions on its own operations. A policy provision of this nature might be that no team member can spread information about any incident outside of the immediate team without the direct permission of the team leader. Failure to conform to existing policy spells catastrophe for an incident response team; consequences can range from embarrassment to termination of employment or even to dissolution of the team itself. Is a Team Really Necessary? Another extremely important issue is whether an incident response team is really necessary. Some of the advantages of forming a response team have been presented earlier in this chapter, but it is not always advantageous to create such a team. An alternative is to have individuals who are not part of an incident response team but who are available (usually on the basis of a matrix agreement1 between organizations) when incidents occur. Here are some possible advantages of adopting this alternative approach: · Smaller organizations generally do not need a team. A smaller organization, such as a small startup company, does not usually need an incident response team per se. This kind of company is not likely to have very much internal structure; creating policy and procedures, in many cases, is something that must be placed on the proverbial backburner while more immediately pressing issues (survival of the business) are addressed. Forming an incident response team would constitute overkill. · Few resources might be available. One of the major reasons for not forming an incident response team is lack of resources, particularly personnel resources. Although not a particularly good reason from a security viewpoint, lack of resources is too often a problem for information security efforts in general. · Incident response might work better as a distributed effort. In some organizations, incident response works better as a distributed effort. Different individuals from different divisions or groups can be called in whenever an incident of sufficient magnitude or impact occurs. Having this kind of arrangement can make these divisions or groups feel that they have some kind of direct control over the incident response process; some of their own staff members will be involved in handling their own incidents. Additionally, a distributed effort can help ensure that people who know and understand how individual units work, how the systems and networks are configured and maintained, how the applications work, and so forth will be involved in handling incidents. This can lead to better insight into what should and should not be done to resolve each incident satisfactorily. What if You Dont Have a Response Team Per Se? The authors of this book feel that, all things considered, it is better to have a response team to deal with security-related incidents than to call on individuals when incidents occur. Many readers of this book will never be part of an incident response team, however. If it is not possible to have such a team, you can adopt measures that will increase the likelihood of success in your efforts to handle incidents. Consider the following suggestions: · Identify key personnel (especially technical personnel), people you feel are qualified to deal with incidents and obtain contact and other information. · Establish some kind of ground rules or agreements concerning the availability of people who are likely to be needed in dealing with incidents. Try to get a commitment from management that guarantees a minimum number of hours of participation (per week, month, or year) from each individual who might be involved. Try to obtain assurance that even more hours of support will be available in the case of a severe incident. · Be wise in your dealings with organizations that provide individuals who are available for incident response support. In many cases, having these individuals participate in incident handling detracts from their own mainstream missions. Avoid being overly demanding and be prepared for a no answer. Sometimes an organization might refuse to allow someone from that organization to deal with an incident due to a pressing need such as meeting a major project milestone. Having a long list of potential incident support personnelso that if one person is not available, you can turn to anotheris thus essential. · Provide some kind of training and orientation to everyone who is likely to help in dealing with incidents. Ensure that everyone has at least a minimum level of knowledge about responding to incidents and that everyone understands the importance of cooperation and teamwork. · To the maximum possible extent, solve leadership and authority issues in advance. In many (if not most) incidents, having someone in charge is essential to success. Conversely, having several people think they are in charge is extremely counterproductive. · Do not call on the individuals who are available for incident response support unless they are genuinely needed. You are disrupting some other business unit or groups work each time you call on such individuals. You will wear out your welcome if you call on these individuals too much or if you call them into too many false alarms. · Organize a committee or board that oversees incident response activities. Have this entity analyze critical aspects such as difficulty in obtaining support personnel, efficiency of incident response activity, and others. This entity might be instrumental in pointing out to management things that need to be improved (such as resource levels) and might prove to be instrumental in helping you form a team in time. What Are the Functional Requirements and Roles? If you have ever taken a course in software engineering, you have learned that defining requirements right up front is crucial to the success of the project. Incident response teams are no exception to this principle; functional requirements and the role for this team need to be defined as early in the life of the team as possible. Basic Requirements The most fundamental requirement for an incident response team is providing incident response support to a constituency. In providing incident response support, a response team can serve several potential roles: · A team can assume full control over an incident and any computing and data resources involved. The extreme version of this role is to go to a site or area within a facility and take over all incident response efforts.2 In most settings, however, this approach does not work too well in that it alienates others, particularly the owners of the computing resources and data, causing territory wars. If mandated by senior management, however, this approach can be viable in that it establishes a clear line of authority during incidents. · Another, less extreme approach is control sharingboth the incident response team and operations or business unit staff. This generally causes less friction, but questions concerning who is in charge at any time are likely to arise. · Still another possibility is providing direct (hands-on) incident response support but limiting this support to a purely advisory role. This means that an incident response team will do something only when its constituency requests that it do so. This role ruffles fewer feathers but typically also greatly limits the role and effectiveness of the people who serve on the incident response team. · A final potential role for a response team is providing indirect rather than direct support in the form of advice but nothing more. This role is the most limiting for team members. However, it also tends to alienate others the least of any role. Additional Requirements In many circumstances, simply providing incident response support is not sufficient to keep management happy. Management too often views an incident response team as individuals who sometimes are busy but at other times have absolutely nothing to do. Management might, therefore, demand more of the team. In other cases, the individuals who attempt to create a response team can see the need for the team to perform other activities related to incident response support. Here are some additional potential types of requirements for a response team: · Interagency/corporation coordination/liaison. The response team might, for example, provide a liaison function with other response teams, an organizations business continuity organization, law enforcement agencies, or some other entity. · Serving as a clearinghouse. A clearinghouse serves as a central repository for information, patches, tools, and so forth. Although almost every response team in some way serves as a clearinghouse for information about incidents and vulnerabilities, serving as a clearinghouse for patches and tools has quite a few additional risks. What if the team provides the wrong patch, resulting in an unexpected incident or system failure? The same applies to tools. The point here is that serving as a clearinghouse for patches and tools often (but not always) poses more risk than potential benefit. · Contingency planning and business continuity services. In some organizations, an incident response team also engages in contingency planning and business continuity functions. This is potentially a good idea in that incident response personnel generally become very proficient in recognizing and dealing with emergencies. All things considered, however, the best way to meet this kind of requirement is to have one or more individuals from a business continuity team closely work with or even join an incident response team. Business continuity staff members generally know things that incident response people need to know, such as what to do to protect business interests in the case of a prolonged outage. This kind of knowledge can be well applied to security-related incidents such as massive distributed denial-of-service attacks. · Information security tool development. Another possible requirement is for the incident response staff to develop information security tools in their spare time. This kind of requirement can result in the availability of useful tools for a teams constituency. The downside is that tool development sidetracks team members from the teams main focus, namely handling incidents. A division within the teamincident handlers versus developersmight even develop. · Incident response planning and analysis. A few teams have a requirement to analyze trends and plan for incident response and security needs of the future. Although most teams are not funded sufficiently to engage in efforts of this nature, incident response planning and analysis is one of the most proactive and potentially valuable activities in which a response team can engage. · Training and awareness. We will discuss training and awareness in more detail later in this chapter. Suffice it to say, at this point, that training and awareness is one of the most proactive activities in which a response team can engage. Response team members will learn about many developments and trendssuch as new types of malicious programs, new types of attacks, new countermeasures, and so forththat are potentially of great value to the teams constituency. Training and awareness activities are a good outlet for disseminating this kind of information. Who is the Constituency? An essential issue in incident response is determining exactly whom you are supporting. In other words, you need to find out who your constituency is. The reason this is so important is that an incident response effort that does not meet the needs of those it serves is doomed to failure. If you can determine whom your constituency is, you can communicate with that constituency to learn the needs that exist. You also will know how to better focus your efforts. If, for example, you discover that your constituency consists largely of system administrators, your approach to providing incident response support will be substantially different than if you have mostly users as customers. In the first case, you will probably need to be more technical in your approach. Your communications with system administrators will, in all likelihood, be of a technical nature. In the second case, you will almost certainly take a much less technical approach, emphasizing instead things that users need and can understand. Motivating users to engage in sound computing practicessuch as updating antivirus software on desktop systems and helping users whose systems have virus or worm infections by advising them to avoid dealing with these incidents directly3would in this case be more appropriate. A response teams relationship with its constituency will make or break an incident response effort. Providing quality help to the right people will eventually result in positive feedback to both the team and its management or sponsor. Many teams (some of which are still in existence, others of which are not), however, have failed primarily because they have neither understood who their constituency is nor served their constituencys needs very well. The following sidebar describes some of the many mistakes that some incident response teams have made. Case Studies: Failing to Adequately Serve a Constituency Several incident response teams have lost most or all credibility within their constituent communities for a variety of reasons. Consider the following mistakes that these teams have made: · Failing to get back to someone who contacts a response team to report an incident or new vulnerability. Some incident response teams send an automated reply containing an incident number but do nothing more. In the perception of a constituency, this is as bad as not replying at all. Several teams have thus deservedly earned the reputation of being a black hole. People who can be an excellent source of information about new incidents and vulnerabilities often quit contacting their incident response team after just one case of failing to follow up a report of an incident. · Spreading misinformation. Recently an incident response team informed someone at the site at which the senior author of this book works that multiple systems at the site were infected by a worm. After hours of investigation, no evidence of any worm could be found in any of the four allegedly infected machines. The individuals who performed the investigation developed negative feelings toward the response team for not getting its facts straight and for wasting their time. · Becoming too intrusive. One incident response team for a government agency became intensely disliked within its constituency because it initiated a project to monitor network traffic at the external gateways at each site without the consent of management at each site. People at the sites felt that the incident response team was eavesdropping on them. · Causing embarrassment or leaking information without authorization. Another incident response team was hired to perform a security evaluation at one of its constituent sites. After finishing the evaluation (in which a considerable number of vulnerabilities were found), the response team reported the results to the head of security within the government agency that oversaw both the site and the response team. Management at that site had expected that the results would be confidential. · Betrayal. Under the edict of Congress, a certain U.S. government contractor launched a set of network attacks against several U.S. government sites. The attacks were very vigorous; the attackers not surprisingly achieved more than a minimal level of success. Not knowing the source of the attacks, those who noticed the resulting security breaches in victim systems frequently turned to their agencys response team. As the attacks progressed, people at some of the sites within one government agency noticed a strange phenomenon: After the identity of a victim system had been reported to the agencys response team, that system was never attacked again. After several weeks, the attacks ceased entirely. Soon afterward, the nature of the white hat penetration tests started to become common knowledge. Along with the news of the nature and purpose of the tests came the news that one response team was working in full cooperation with those who were launching the attacks. When a site detected an intrusion into a system and reported it to the response team, that response team forwarded the information to the attackers, who quit accessing the system in favor of launching new attacks against others. Since all this happened, virtually no one at any site has wanted to deal with this response team any more. Communicating with a Constituency After a response teams constituency is defined, establishing communication channels is essential. One-way communication, in which the response team keeps sending information to its constituency without communication being initiated by the constituency, generally does not work. An effective response team needs to obtain information about what is actually occurring within its constituency. It is possible, for example, for a response team to be unaware that a worm is circulating within part of its constituencys networks. Learning that this is happening would enable the response team to be able to better serve its constituency. The bottom line here is that an effective incident response team establishes two-way communication. It shares information about vulnerabilities and types of incidents that are occurring within its constituency. As the saying goes, You have to give information to get information. If the response teams constituency does not share information with the response team, the response team is not likely to have much worthwhile information to share with its constituency. A response team can use any or all of the following avenues of communication: · Telephone. One of the most simple and direct avenues of communication is the telephone. Calling someone can inject a personal touch into communications with a constituency. The fact that telephone conversations are in real time is also an important advantage of this means of communication. The downside is that people do not always speak and/or listen as well as they should; misunderstandings and miscommunication can occur. Another downside is that telephone communications are subject to eavesdropping, especially with cordless telephones based on radio frequency transmission and wireless telephones.4 Secure telephones solve the eavesdropping threat in that they encrypt voice transmissions from one secure telephone to another. An example of an encrypting telephone is an STU-4something that the U.S. government uses for transmitting classified information via telephone. A limitation is that not everyone can have access to a secure telephone when it is needed. Additionally, secure telephones can prove financially costly. · Email. Email is another potentially advantageous means of communicating with a constituency because of its efficiency. You can send a message to someone else in another part of the world in only a few seconds. Furthermore, the person to whom you send the message does not have to be monitoring email at that particular moment in time. Additionally, you can create mail exploders to which you send a message that is subsequently sent to an entire distribution list of email addresses. As pointed out in Chapter 3, A Methodology for Incident Response, however, email is extremely prone to eavesdropping. Email can also easily be spoofed, and incident response team members are also sometimes spammed by attackers. A better solution is secure email, which encrypts email messages sent from one system to another. Various freeware and commercial packages that deliver email encryption are available. They provide a good solution for the eavesdropping problem but tend to be plagued with problems related to using encryption particularly key distribution and key recovery. · Fax. Sending faxes is an often overlooked but potentially effective means of communicating with a constituency. A nice feature of faxes is that they generally result in an easy-to-read hard copy. Additionally, faxes can be sent when ones network or mail server is down. Some types of fax machines can even explode a single fax message to hundreds of fax numbers in only a few minutes. Faxes, like anything else, are hardly a panacea, however. One of the greatest limitations is that they do not work when the destination fax number is busy or out of order. Faxing messages can also be unduly labor intensive because it takes a while to set up a fax transmission, undo any paper clogs at both ends of the transmission, replace empty paper bins, and so forth. Additionally, fax transmissions are potentially subject to eavesdropping. Secure faxes solve this eavesdropping problem, but they tend to be more expensive. Because of all the potential complications associated with fax communications, our recommendation is to use this method of communication as a backup rather than as a primary method. · Bulletins/notices. Bulletins and notices provide an excellent way not only to...