Restricting access to a shared resource

A method for restricting access of a client to a web site hosted at first and second servers is described. A first tally that includes identification information of the client and a first number of access requests sent from the client to the first server is received. A second tally that includes the identification information of the client and a second number of access requests sent from the client to the second server is received. The first and second tallies are collated to determine a total number of access requests made by the client.

TECHNICAL FIELD

This invention relates to detecting and blocking requests from programmatic robots.

BACKGROUND

A web site is a directory of files stored on a web server or several web servers that may be accessed by a client over a network (e.g., the Internet). Both individual users and non-human programmatic sources (referred to as “robots”) may request access to a web server. Individual users who access a web server according to the intended presentation of the web site are referred to as “direct users”. Direct users often purchase items or services from the web site and view advertisements and sponsorships displayed in the web site. For these reasons, and others, access to a web server by direct users is highly desirable. Direct users represent the primary source of revenue for companies that operate web sites.

Robots, on the other hand, retrieve and index documents contained within web sites and often deliver these documents elsewhere. Robots, which are also referred to as “spiders” or “web crawlers”, may be server-based or client-based and are employed for a variety of reasons, some legitimate and many fraudulent. Robots can also be part of computer viruses, making the source of the activity difficult to track or control. Robots impose a cost on companies (both in terms of infrastructure to support the web site and whatever licensing costs are involved in presenting the content of a web page) while defeating most of the mechanisms by which a company attempts to make a profit.

Robots are often used by search engines to maintain an index of web sites. Legitimate robots follow conventions that allow web sites to mark pages, directories, or whole sites as “off limits”; pernicious robots ignore these conventions. There is a keen financial interest in minimizing access to a web server by pernicious robots.

SUMMARY

The present invention provides methods and systems, including computer program products, for restricting access of a client to a web site hosted at first and second servers.

In general, in one aspect, the invention features a method performed at a third server that includes receiving a first and second tallies associated with the client. The first tally includes identification information of the client and a first number of access requests sent from the client to the first server, and the second tally includes the identification information of the client and a second number of access requests sent from the client to the second server. The first and second tallies are collated to determine a total number of access requests made by the client.

Embodiments may include one or more of the following. A dynamic blocking instruction, an allow instruction, or a static blocking instruction may be assigned to the client's identification information. A dynamic blocking instruction causes the first and second servers to restrict access of the client to the web site. An allow instruction causes the first and second servers to always grant access to the client even if the total number of access requests exceeds the predefined threshold. A static blocking instruction causes the first and second servers to always deny access to the client even if the total number of access requests is below the predefined threshold. The dynamic blocking instruction, allow instruction, or static blocking instruction may be recorded in a configuration file which may then be sent to the first and second servers.

The first server performs functions that include: receiving the configuration file from the third server; receiving an access request from the client; recording, in a log entry, the client's identification information (e.g., internet protocol (IP) address) and information associated with the access request; determining whether the identification information of the client is associated with an instruction recorded in the configuration file; and if the identification information of the client is associated with an instruction recorded in the configuration file, controlling access to the first server from the client according to the instruction. Controlling client access, for example, may include denying the client access to the first server or granting the client access to the first server.

The first server may record, in a least-frequently-recently used (LFRU) queue, a tally associated with the client and send the tally to the third server. Collating the first and second tallies may include adding the first number of requests to the second number of requests. The first number of requests may be subtracted from the total number of requests if no further tallies associated with the client are received from the first server within an expiration period and the first server may send a tally associated with the client if the client requests access to the first server. The dynamic blocking instruction may be deleted from the configuration file if the total number of access requests minus the first number of requests is below the predefined threshold.

In general, in another aspect, the invention features a system for restricting access to a web site hosted at first and second servers. The system includes a third server in communication with the first and second servers via a communications network. The third server includes: a collated database configured to collate tallies received from the first and second servers to obtain a total number of access requests made by a client to the first and second servers; an analysis engine configured to generate a dynamic blocking instruction that causes the first and second servers to deny access to the client if the total number of access requests exceeds a predefined threshold; and a configuration file including the dynamic blocking instruction. The tallies include identification information of the client and a number of access requests sent from the client to the first and second servers.

Embodiments may include one or more of the following. The first server may include: a local log file comprising identification information associated with the client and information associated with an access requests made by the client to the first server; a least-frequently-recently-used (LFRU) queue configured to store a tally associated with the client; and a blocking engine configured to block the client from accessing the first server according to the configurable blocking plan. The third server may further include a communication device for sending the configuration file to the first and second servers over the communications network.

Advantages that can be seen in particular implementations of the invention include one or more of the following. The total frequency of requests sent from a single IP address to a server farm can be determined even if the requests are spread over many servers in the server farm. A client is blocked from accessing the server farm for a configurable period of time if the number of requests sent from the client within a given time period exceeds a predefined threshold. The period over which a client is blocked extends as long as abusive traffic from that client continues plus a configurable margin. Furthermore, the configurable margin and the threshold of traffic considered abusive may be adjusted to reduce the likelihood of blocking legitimate client IP addresses that are shared among multiple users. Instructions for denying or allowing a client access to the server farm can be changed or updated periodically.

DETAILED DESCRIPTION

Detecting and blocking requests from robots is difficult when the robots skillfully mimic real browser requests. Often the only indication that a robot, rather than a legitimate user, is requesting access to a web server is the frequency at which requests from the robot are made. If the frequency of access requests sent from an Internet protocol (IP) address exceeds an allowable frequency threshold, the server may mark the IP address as belonging to a robot and block further requests from the address. When a web site employs multiple servers (also referred to as a “server farm”), measuring the frequency of incoming requests for a particular IP address is difficult because the requests may be distributed among many different servers in the server farm. If the requests are spread out over multiple servers, the distribution of requests from a single IP address may or may not be even, making it difficult to set a threshold on a per-web server basis. In some situations, the total frequency of requests from a single IP address exceeds an allowable threshold; however, the frequency of requests to any given server in the server farm is lower than the threshold. By aggregating the requests made from a source to each server over the entire farm of servers, a complete set of request statistics for the source can be determined. The statistics may then be analyzed for indications of robot activity so that appropriate action can be taken.

FIGS. 1a-1cillustrate an example of a system10for detecting and blocking requests from a robot. Client computer12requests and receives information from one or more of the servers14a-chosting a web site. Collectively, servers14a-care referred to as “server farm14”. In some embodiments, server farm14includes hundreds or thousands of servers. Client computer12and server farm14are connected to a network20, which is the Internet. Client computer12may also be multiple client computers. In some embodiments, network20is a private network, a corporate intranet, or other similar wired or wireless network. Server farm14is also connected to network22through which communications are sent to and from mid-tier server16. Mid-tier server16and servers14a-cinclude communication devices for receiving and transmitting data over network22. Network22is a private local area network that is separate from network20. In some embodiments client12accesses server farm through an Internet service provider (ISP) that recycles temporary IP addresses among multiple clients including client12. In other embodiments, client12has a permanent IP address.

In general, client12uses a Web browser program to interact with server farm14according to hypertext transfer protocol (HTTP). Examples of browsers include Internet Explorer® and Firefox®. In the browser, a user at client12enters a Universal Resource Locators (URL) for a desired web site. Users can also request pages by clicking on hyperlinks within a hypertext markup language (HTML) document. These requests are sent to one or more servers in the server farm14. When a request is received at a server (e.g., server14a), the server identifies the IP address from which the request originates. Each server in server farm14stores, in a request queue32(FIG. 1c), dynamic tallies of requests of clients that are most actively requesting access to server farm14. A dynamic tally includes the number of requests sent by a client over a given period of time and the IP address of the client. Each of the servers14a-csend their dynamic tallies to mid-tier server16. In this way, only the most frequently requesting clients are reported to mid-tier server16. Mid-tier server16collates the dynamic tallies sent from servers14a-cand stores them in a collated database36(FIG. 1b).

As shown inFIG. 1b, mid-tier server16includes a collated database36of dynamic tallies sent from servers14a-c, a configuration file34, and an analysis engine38. After receiving dynamic tallies from server farm14, mid-tier server16collates the dynamic tallies and stores them in collated database36. For example, if client12sends one request to each of servers14a-c, each server sends a dynamic tally having a value of one to mid-tier server16. Mid-tier server16collates the tallies in collated database36, which records that client12made a total of three requests to the server farm14. In some embodiments, collated database36includes hundreds or thousands of entries. Collated database36displays all of the requests distributed over server farm14. The size of collated database36is not bounded, though it has some practical limits.

From the collated dynamic tallies associated with client12, an analysis engine38calculates the total number of requests made from client12to the entire server farm14over a given period of time. Based on this total, the analysis engine38determines whether to block further access requests from client12or to flag the client's IP address to an operator's attention. If a decision is made to block client12, the client's fingerprint information (e.g., IP address) is associated with a blocking instruction. The client's fingerprint information and associated blocking instruction are recorded in the configuration file34.

Configuration file34includes a list of client fingerprints to be blocked. In some embodiments, configuration file34includes a list of client IP addresses that are permanently blocked. Such a list is referred to as a “black list”. In other embodiments, configuration file34includes a list of client IP addresses from which requests are always allowed. Such a list is referred to as a “white list”. The IP addresses and other client information contained in a white list could, for example, belong to client machines that frequently access server farm14for legitimate purposes (e.g., server maintenance and configuration). Configuration file34also includes a list of client IP addresses to be blocked temporarily. Such a list is referred to as a “dynamic block list”. After analysis engine38updates the configuration file34, mid-tier server16sends the configuration file34to each of the servers14a-cin server farm14over network22.

The period of time over which an IP address listed in the dynamic block list is denied access to server farm14depend on the last time each server of server farm14received requests from that IP address. After receiving a request from the IP address, if a server (e.g., server14a) does not receive anymore requests from the IP address within a certain period of time (referred to as an expiration period), the number of requests from IP address that were previously reported by the server is subtracted from the total number of requests recorded for IP address in collated database36. For example, if servers14areports100requests sent from client12, server14breports40requests sent from client12, and server14creports25requests sent from client12in a given time period, the total number of requests sent from client12to server farm14is 165. If in the next time period, for example, server14areports another 27 requests sent from client12, server14breports13more requests sent from client12, and server14creports10more requests sent from client12, the overall total becomes 215. The collated database36arrives at this total by keeping track of the subtotals for each server14a-cand adding these subtotals. In the previous example, the subtotals for servers14a,14b, and14cafter the second time period are 127, 53, and 35 requests, respectively. If, for example, client12makes no more requests to server14cwithin the expiration period—even if it still sends requests to servers14aand14b—the subtotal of previous requests sent to server14cfrom the client (i.e., 35) is subtracted from the total while new contributions from servers14aand14bstill accumulate. When the entries for servers14a-call expire for a given client IP address, the entire record for that client is removed from collated database36.

Referring toFIG. 1c, a block diagram of one of the servers (i.e., server14a) in server farm14is shown. The block diagrams for servers14band14care analogous to that of server14a. Server14aincludes a blocking engine30, a request queue32, and a local log file33. After server14areceives an access request from client12, server14aeither creates a new entry for client12in request queue32or updates an existing entry for client12in request queue32.

Request queue32has a fixed size and is therefore limited in how many IP addresses it can record. Request queue32deletes existing entries of clients based on both the frequency of requests made by the clients to server14aand the amount of time that passes before the clients send requests to server14a. This type of deletion scheme is referred to as a least-recently-frequently used (LRFU) deletion scheme. For example, the collated log file36may delete an entry for client12if client12fails to make a request within a certain time period (e.g. thirty minutes) and if the tally of requests recorded for client12is below a given threshold (e.g., five requests). Request queue32applies the LRFU queuing mechanism so that the most active clients (i.e., the clients making the most requests over a given time period) filter to the top of request queue32. The most active clients are of the most interest as they are the most indicative of suspicious behavior. In aggregating the dynamic tallies from the request queues of all the servers in server farm14, mid-tier server16makes the larger determination of which clients are engaged in wholesale pernicious activity.

Server14aalso stores a local log file33that logs client requests. Local log file33is separate from request queue32. In general, local log file33stores more information about client requests than request queue32. Local log file records a client's identification information (referred to as a “client fingerprint”) along with information that is specific to the client's request. A client fingerprint, for example, may include a client IP address and a user agent string. Information that is specific to the client's request may include the web browser from which a request is made, the web page that is being requesting, the page from which the requests originated, the time and date of the requests, the client ip address, and the “cookies” the client presented with the request.

Blocking engine30determines whether or not to block client12from accessing server14abased on the information contained in configuration file34. After server14areceives a request from a client, the fingerprint of the client is recorded in local log file33. Blocking engine30determines whether any information in the client fingerprint is contained in the configuration file34. If a match is found, blocking engine30determines whether an allow instruction or a blocking instruction is assigned to the client fingerprint. If blocking engine30matches any information in the client fingerprint to a blocking instruction, blocking engine30blocks the client from accessing server14a. If specified in the blocking instruction, blocking engine30may also send a message back to the client (e.g., an HTTP 403 “Permission Denied” message) or redirect the client to another web page.

Referring toFIG. 2, a process50for updating configuration file34is performed at mid-tier server16. Mid-tier server16receives (52) dynamic tallies from server farm14. The dynamic tallies include the IP addresses of clients requesting access to the server farm14and the number of requests sent from each IP address. In some embodiments, the dynamic tallies are sent to mid-tier server16at scheduled times (e.g., 12:00 AM, 1:35 AM, 3:30 AM, etc.). In other embodiments, the dynamic tallies entries are sent to mid-tier server16at delta time intervals (e.g., every ten minutes). Mid-tier server16then collates (54) the log entries and records them in collated database36. Collated database36shows all of the requests sent from a particular IP address to various servers in server farm14.

Analysis engine38analyzes (56) the collated dynamic tallies in collated database36to determine which, if any, IP addresses should be blocked from accessing server farm14or flagged to an operator's attention. From the collated dynamic tallies, the analysis engine38calculates the total number of requests made from each client over a given time period. Based on the frequency of requests calculated for a client, the analysis engine38determines whether to block further access requests from the client's IP address or to flag the client's IP address to an operator's attention. Analysis engine38decides to block an IP address, if within a given time period, the total frequency of requests originating from the IP address exceeds a predefined threshold. If a decision is made to block a client, the analysis engine38assigns a blocking instruction to the client's fingerprint information (e.g., IP address). After receiving a request, if server14adoes not receive anymore requests from the IP address within an expiration period, the subtotal of requests from the IP address that were reported by server14ais subtracted from the total number of requests that is recorded for the IP address in collated database36. As long as the net of new requests from the client IP address to any of the servers less the count of requests that expire continues to be above the threshold, the analysis engine38maintains a block instruction on the client IP address. If the total number of requests recorded for the IP address falls below the threshold, analysis engine38deletes the blocking instruction assigned to the IP address from configuration file34. Therefore, the next time server14adownloads configuration file34, server14awill grant access to the client IP address.

Because some client IP addresses may be cycled or shared between different users (e.g., through an Internet Service Provider), there is a chance that a legitimate user could acquire a blocked IP address that was previously assigned to a malicious user. Thus, the predefined threshold and/or the time period over which requests are recorded and reported to mid-tier server16may be adjusted to reduce the likelihood of blocking legitimate users with recycled IP addresses.

In some embodiments, the analysis procedure (56) determines that a client should be permanently blocked from accessing server farm14. In other embodiments, configuration file34includes blocking instructions that are only executed if particular information is absent from a client's fingerprint. For example, access may be denied to clients whose client fingerprints are missing a user-agent string value.

Analysis engine38stores (58) the client IP address and associated blocking instructions in configuration file34. After configuration file34has been updated (58), mid-tier server16sends (60) a copy of configuration file34to each of the servers14a-cin server farm14. In some embodiments, the mid-tier server16sends configuration file34to server farm14at scheduled times (e.g., 12:00 AM, 1:35 AM, 3:30 AM, etc.). In other embodiments, configuration file34is sent at delta time intervals (e.g., every ten minutes). Delta time intervals are relative to the last (or first) time that a front end server performed a given task. As each of the servers14a-ccan be rebooted or restarted independently, each machine could be performing these operations at different times with delta time configuration. In some embodiments, the configuration file34is manually updated by an operator accessing mid-tier server16either directly or remotely over network22.

Referring toFIG. 3, a process70for identifying and blocking robots is performed at each of the servers14a-c. For ease of explanation, process70is described with respect to server14a. Server14adownloads (72) configuration file34from mid-tier server16and saves it in a data storage device. Older versions of configuration file34stored in server14aare replaced by the new configuration file34that is downloaded (72) from mid-tier server16. After receiving (74) an access request from client12, server14aupdates request queue32and generates a new log entry for client12in local log file33. The log entry includes the client's fingerprint and information that is specific to the client's request (e.g., the web page that is being requesting). After a predetermined time, server14asends (88) the dynamic tallies stored in request queue32to mid-tier server16. In some embodiments, server14asends the dynamic tallies to mid-tier server16at scheduled times (e.g., 12:00 AM, 1:35 AM, 3:30 AM, etc.). In other embodiments, the dynamic tallies are sent at finite delta time intervals (e.g., every ten minutes). As each of the front end servers14a-ccan be rebooted or restarted independently, each of the servers14a-ccould be sending dynamic tallies at different times with the delta time configuration.

Blocking engine30compares (76) the client fingerprint stored in local log file33to the information stored in configuration file34to determine whether any information in the client fingerprint is contained in the configuration file34. Blocking engine30determines (78) whether the configuration file34includes an instruction for allowing client12to connect to server14a. In some embodiments, blocking engine30compares the client's fingerprint to a white list of client fingerprint information. If blocking engine30determines (78) that the configuration file34includes an instruction for allowing client12to connect to server14a(e.g., the client's fingerprint matches an entry in the white list), client12is allowed (82) to connect to server14a.

If blocking engine30does not find an instruction for allowing client12to connect to server14a, blocking engine30determines (80) whether configuration file34includes a static blocking instruction for permanently blocking the client from accessing server14a. In some embodiments, blocking engine30compares the client's fingerprint information to a black list of client fingerprint information. If blocking engine30determines (80) that the configuration file34includes a static blocking instruction for permanently blocking the client from server14a(e.g., the client's fingerprint matches an entry in the black list), blocking engine30blocks (86) client12from accessing server14aand sends a message (e.g., a HTTP 403 “Permission Denied” message) to client12. In some embodiments, a static blocking instruction is based on information included in the local log file33. For example, a static block instruction may instruct blocking engine30to deny access to a client if the client's web browser is known to be that of a robot (or if the client's web browser is unknown).

If blocking engine30does not find a static blocking instruction associated with the client fingerprint, blocking engine30determines (84) whether configuration file34includes a dynamic blocking instruction for temporarily blocking client12from accessing server14a. If blocking engine30determines (84) that the configuration file34includes a dynamic blocking instruction associated with the IP address of client12, blocking engine30blocks (86) client12from accessing server14aand sends a message (e.g., a HTTP 403 “Permission Denied” message) to client12. Blocking engine30will continue to block client12from accessing server14aso long as the total dynamic tally of requests made to server farm14from client12exceeds a threshold. Likewise, the blocking engines in servers14b-c, will block client12from accessing server14aso long as the total dynamic tally of requests made to server farm14from client12exceeds the threshold. If server14adoes not receive any more requests from client12within an expiration period, the subtotal of requests from client12that were reported by server14ais subtracted from the total number of requests that is recorded for client12in collated database36. As long as the net of new requests from client12to any of the servers less the count of requests that expire continues to be above the threshold, the analysis engine38maintains the block instruction on the IP address of client12. If the total tally of requests recorded for client12falls below the threshold, analysis engine38will delete the blocking instruction assigned to the IP address of client12when it updates (58) (FIG. 2) configuration file34. Therefore, the next time servers14a-cdownload configuration file34, client12will be granted access to server farm14. If blocking engine30does not find a dynamic blocking instruction associated with the client fingerprint in configuration file34, blocking engine30allows (82) client12to connect to server14a.

Referring toFIG. 4, an exemplary configuration file34is shown. Configuration file34is expressed in extended markup language (XML). Configuration file34includes SETTINGS instructions100, ALLOW instructions102, static BLOCK instructions104, and DYNAMIC BLOCK instructions106. The SETTINGS instructions100are parsed each time blocking engine30reloads configuration file34. A RELOAD_TIME attribute indicates the time for blocking engine30to reload configuration file34. In the example shown inFIG. 4, blocking engine30reloads configuration file34at delta time intervals of 120 seconds. In some embodiments, blocking engine30reloads configuration file34at scheduled times (e.g., 12:00 AM, 1:35 AM, 3:30 AM, etc.). To accomplish this, the RELOAD_TIME may include a scheduling command, such as the crontab command found in Unix and other similar operating systems.

In some embodiments, the SETTINGS instructions100include a DYNAMIC_TIME attribute that indicates a time for blocking engine30to upload dynamic tallies of request queue32if dynamic blocking is turned on. As described above, request queue32stores dynamic tallies of clients that are most actively requesting access to server farm14. In some embodiments, blocking engine30uploads the dynamic tallies from the server farm14at delta time intervals (e.g., every 3000 seconds). In other embodiments, blocking engine30uploads the dynamic tallies at scheduled times (e.g., 12:00 AM, 1:35 AM, 3:30 AM, etc.). For scheduling reloads at precise times, the DYNAMIC_TIME attribute may include a scheduling command, such as the crontab command found in Unix and other similar operating systems. In some embodiments, if dynamic blocking is being employed, both the RELOAD_TIME and the DYNAMIC_TIME attributes are defined with crontab syntax with the times staggered slightly. Scheduling the reloading of configuration file34and the uploading of the tallies in this way provides a consistent state of the dynamic data. On the other hand, scheduling the reloading and uploading at delta time interval may cause the loading of configuration file34and the dynamic blocking operations to be out of synch and possibly collide.

The TOPN attribute is the number of suspect IP addresses that are being monitored. In the example shown inFIG. 4, TOPN has a value of five, meaning the five most active client IP addresses are being monitored at a given time. In some embodiments, assigning TOPN a value of zero turns off dynamic blocking. If TOPN is negative, it indicates that the dynamic usage information is to be gathered but not used to reject requests. The entries tracked in request queue32may be recorded and analyzed at a later time. In some embodiments, to better detect hard-hitting bots quickly and reduce the likelihood of blocking legitimate users with recycled IP addresses, request queue32retains between approximately one-hundred and two-hundred entries, configuration file34is updated every ten minutes, and suspicious IP address are blocked for no more than two hours. In these embodiments, robots are detected every ten minutes and blocked yet the robot's IP address is blocked temporarily in case the robot's IP address is later assigned to a legitimate user.

The THRESHOLD attribute is the number of access requests that are allowed from client12. If the total number of access requests from client12exceeds the THRESHOLD, client12is blocked from connecting to server14a. In the example shown inFIG. 4, the THRESHOLD is set to four. In some embodiments the THRESHOLD could be on the order of ten to one-thousand.

In some embodiments, the SETTINGS instructions100include a LOCAL_THRESHOLD attribute that indicates the number of requests allowed from a client to a specific server in a server farm. If the total number of requests from client12to the specified server exceeds the LOCAL_THRESHOLD, client12is blocked until its IP address rotates out of request queue32.

In some embodiments, the SETTINGS instructions100include a REPORTING_THRESHOLD attribute that indicates the number of requests from client12to server14athat must be reached before the dynamic tally recorded in request queue32for a particular client is sent to mid-tier server16. Aggregating only the dynamic tallies above REPORTING_THRESHOLD reduces the amount of statistical noise of single requests in the dynamic blocking data. In some embodiments, the REPORTING_THRESHOLD attribute has a default value of two.

The configuration file34includes ALLOW instructions102for granting access to the server. The ALLOW instructions102are applied before static BLOCK instructions104and before DYNAMIC_BLOCK instructions106. The ALLOW instructions102include NAME attributes and STANZA elements. A NAME attribute includes a name assigned to an ALLOW instruction102and a STANZA element includes a set of matching values associated with the ALLOW instruction102. The ALLOW instruction102shown inFIG. 4grants access to client12if the client's fingerprint contains an IP address of 255.255.255.20. In some embodiments, a diagnostic log records the number of times that the ALLOW instruction102allows client12to connect to the server.

Configuration file34includes static BLOCK instructions104for permanently blocking access to server farm14. Static BLOCK instructions104are executed after ALLOW instructions102and before DYNAMIC_BLOCK instructions106. Static BLOCK instructions104include HTTP_CODE and NAME attributes, and STANZA and ADD_HEADER elements. An HTTP_CODE attribute specifies the HTTP code sent back to client12if client12is blocked. Examples of HTTP code include code numbers “301”, “302” (which indicate the client should be redirected to another page), or “403” (which explicitly denies the request). A NAME attribute includes a name assigned to a static BLOCK instruction104and a STANZA element includes a set of matching values associated with the static BLOCK instruction104. The ADD_HEADER element includes a response header that can be sent back to client12. Examples of BLOCK instructions104are shown inFIG. 4.

The first static BLOCK instruction104ashown inFIG. 4is given the name “1st BLOCK”. The static BLOCK instruction104ainstructs blocking engine30to block clients whose fingerprints contain the user-agent string (i.e., HTTP_USER_AGENT) that starts with “go!zilla”. When client12is blocked by static BLOCK instruction104a, blocking engine30sends a “304” (the value of HTTP_CODE attribute) to client12. A second static BLOCK instruction104bis called “2nd BLOCK”. Static BLOCK instruction104bblocks clients whose fingerprints match any one of the user-agent strings: “go!zilla (www.gozilla.com)”, “checkbot/1.59 lwp/5.41”, and “cosmos/0.8_(robot@xyleme.com)”. When client12is blocked by static BLOCK instruction104b, blocking engine30sends a “302” message (i.e., the value of HTTP_CODE attribute) and a “Location: http://www.lycos.com/” message (i.e., the values of the NAME attribute and VALUE attributes) back to client12. A third BLOCK instruction104c(i.e., “3rd BLOCK”) instructs blocking engine30to block clients whose fingerprints contain undefined or empty user-agent strings.

Configuration file34includes DYNAMIC BLOCK instructions106for blocking access to the server. DYNAMIC_BLOCK instructions106are executed after ALLOW instructions102and static BLOCK instructions104are executed. DYNAMIC_BLOCK instructions106may include DYNAMIC_HEADERS elements that define the action to be taken when requests are to be blocked by dynamic blocking. DYNAMIC_HEADERS elements include HTTP_CODE and ADD_HEADER attributes such as those described for static BLOCK instructions104. DYNAMIC_BLOCK instructions106include STANZA elements that specify the conditions under which clients are dynamically blocked. A NAME attribute of a STANZA element specifies which information in the local log entries are to be applied to a particular DYNAMIC BLOCK instruction106. Table 1 lists examples of NAME attributes.

TABLE 1NAMEDescriptionREMOTE_ADDRIP address of the clientREMOTE_HOSTHostname of the clientREMOTE_USERUsername supplied by the client and authen-ticated by the serverSERVER_NAMEServer's hostname (or IP address) as itshould appear in self-referencing URLsSERVER_PORTTCP/IP port on which the request was receivedSERVER_PROTOCOLName and version of the informationretrieval protocol relating to a requestSERVER_SOFTWAREName and version of the web server underwhich the CGI program is running
STANZA elements include VALUE child elements that specify the value of an attribute. VALUEs can include the wildcard pattern matching characters “?” and “*”. STANZA elements may include UNDEFINED child elements that are matched to local log entries in which no specified header is defined. STANZA elements may also include FUNCTION attributes that determine the function that the blocking engine30uses to resolve a NAME attribute.

Examples of DYNAMIC BLOCK instructions106are shown inFIG. 4. A REMOTE_ADDR attribute is assigned a VALUE of a suspicious IP address, i.e., “209.202.241.249”. The SERVER/IP attribute denotes the IP address of a server in server farm14and the SERVER/HITS attribute represents the number of HITS (i.e., access requests) that a particular server has received from the suspected IP address. In this example, a client at IP address “209.202.241.249” has made three access requests to a server at IP address “209.202.241.247” and two access requests to a server at IP address “209.202.241.246”. Thus, the client has sent a total number of five access requests to the server farm. Because the total number of access requests (i.e, five) is greater than the THRESHOLD (i.e., four), the IP address “209.202.241.249” is blocked from accessing server farm14.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made. For example, the procedures of processes50and70may be performed in different orders than are shown inFIGS. 2 and 3. Furthermore, some of the procedures of processes50and70, e.g., receiving procedure (74) of process70, may be performed multiple times in repetition. Accordingly, these and other embodiments are within the scope of the following claims.