Abstract:
Disclosed herein is a computer implemented technique for discovering the information technology resources that are involved in a particular networked business transaction. The system comprises three basic components. A robotic transaction playback client records the various steps in a particular transaction and can repetitively execute these steps to analyze the results. A network record collector observes the traffic throughout the network in response to the repeated instances of the transaction executed by the robotic transaction playback client. A backend processor analyzes the observations of network traffic to determine which ones are possibly tied to the transaction, and from this information determine which network components are part of the transaction being analyzed. Event timing information from a plurality of executions of a particular transaction are used to determine resource usage and paths.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application claims priority under 35 U.S.C. §120 and is a continuation of U.S. patent application Ser. No. 11/153,303, filed Jun. 15, 2005, entitled “NETWORK TRANSACTION DISCOVERY.” The subject matter of this earlier filed application is hereby incorporated by reference. 
    
    
     BACKGROUND 
     Commercial transactions of nearly all sorts have become dependent on networked computing. Because of this, the business performance of many organizations has become tied to the performance of computer networks and various components of the networks. As these networked systems have evolved and businesses have become more dependent on them, tools have developed for network administrators to monitor the performance of the network and of the various network components. However, it has been difficult to transition the technical aspects of network and network component performance monitoring into the business aspects of network performance. 
     What is needed in the art is a way to link the performance of business tasks, i.e., transactions, to the underlying and supporting information technology (“IT”) infrastructure. This linking serves three important functions, which may also be viewed as temporally sequential phases: discovery, diagnosis, and administration/prediction. The discovery function allows both business and technical managers to ascertain what IT components (clients, servers, network links, etc.) are used by a particular transaction, and, conversely, to determine what transactions require the use of a particular IT component. The discovery phase also helps to identify which transactions are affected by outages or other problems. Once this information is known, the second phase, diagnosis, allows business and technical managers to determine the cause of a performance problem with respect to a particular transaction and/or IT component and how to remedy the performance problem. Once the discovery and diagnosis phases are completed, business and technical managers can use this information administer existing resources (e.g., charge IT costs back to individual departments on a usage basis) and predict the need for future IT resources or scheduled maintenance, etc. 
     This need for information can be met by a system disclosed herein, which comprises computer software executable on a machine running on the computer network to discover the components of a particular transaction or service. 
     SUMMARY 
     The present invention relates to a computer implemented technique for discovering the information technology resources that are involved in a particular networked business transaction. The system comprises three basic components. A robotic transaction playback client records the various steps in a particular transaction and can repetitively execute these steps to analyze the results. A network record collector observes the traffic throughout the network in response to the repeated instances of the transaction executed by the robotic transaction playback client. A backend processor analyzes the observations of network traffic to determine which ones are possibly tied to the transaction, and from this information determine which network components are part of the transaction being analyzed. Event timing information from a plurality of executions of a particular transaction are used to determine resource usage and paths. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a typical computer network in which the present invention finds application along with a transaction path of a particular transaction of interest. 
         FIG. 2  illustrates message traffic into and out of a particular node for a group of transactions. 
         FIG. 3  illustrates message traffic into and out of a node and how traffic related to a particular node is identified. 
         FIG. 4  illustrates a temporal sequence of executions of a transaction used to identify what network nodes are part of a given transaction. 
         FIG. 5  illustrates a network in which it cannot be absolutely determined what network nodes are part of a given transaction because more than one path cannot be isolated. 
     
    
    
     DETAILED DESCRIPTION 
     A system for discovering an application transaction and what IT resources are used by the transaction is. described herein. The following embodiments of the invention, described in terms of applications and environments developed by BMC Software of Houston, Tex. are illustrative only and should not be considered limiting in any respect. 
     A typical computer enterprise in which the present invention finds application is schematically illustrated in  FIG. 1 . Network  100  comprises a plurality of network nodes of varying types. For example, a client computer  101  (also denoted “C”) serves as the interaction point for a user. Network  100  also includes a plurality of web servers  103   a - 103   d  (also denoted “WS”). At any given time, client  101  may have one or more interactions  102   a - 102   d  to the web servers  103   a - 102   d . Typically these interactions are implemented over Ethernet using hypertext transfer protocol (HTTP), although other connection types and protocols are also possible. 
     Network  100  also includes a plurality of application servers  105   a - 105   c  (also denoted “AS”). Typically web servers  103   a - 103   d  interact with application servers  105   a - 105   c  through a plurality of interactions  104   a - 104   l . Typically interactions  104   a - 104   l  would also be implemented over Ethernet using HTTP, although other connection types and protocols may also be used. Additionally, it is also possible for a client computer to interact directly with an application server, e.g., client  109  is connected directly to application server  105   c  by connection  108   b . In still another variations, one computer in network  100  could simultaneously function as a web server and as an application server. 
     Finally, network  100  includes a database server.  107  (also denoted “DB”). Typically the application servers  105   a - 105   c  will interact with database server  107  by interactions  106   a - 106   c . Interactions  106   a - 106   c  would typically be implemented over Ethernet using SQL, although other connection types and protocols could also be used. It should also be noted that other computers may also interact directly with database server  107 , e.g., FTP server  111  via connection  110 . 
     The data flow for an exemplary client-server transaction is also depicted in  FIG. 1 . Suppose an client web-based application on client node  101  needs to interact in some way with a database application running on database server  107 . One example might be an order entry system interacting with a sales database. The interaction will take the form of a transaction. Particularly, the client application will send data or a command (a “request”) to the database node  107  and will receive some sort of data in return (a “response”). Several interactions as described in the preceding paragraphs may together make up a single transaction. 
     To process the transaction, client node  101  will send a request to web server  103   a  via connection  102   a . Client  101  will then wait for a response from web server  103   a . To process the transaction, and provide the required response to client  101 , web server  103   a  will need to contact application server  105   c . However, so far as client  101  node knows, it is interacting only with web server  103   a . The transactions taking place on the back end of web server  103   a  are not visible to client computer  101 . 
     In response to the request from client  101  web server  103   a  will issue a request of its own. This request goes, for example, to application server  105   c  via connection  104   c . As with the previous transaction step, so far as web server  103   a  knows, it is only interacting with application server  105   c . Any transactions taking place on the back end of application server  105   c  are not visible to web server  103   a . Finally, application server  105   c  will issue a request to database server  107  over connection  106   c . For purposes of this example, this is the endpoint of the transaction. 
     Database server will process the request it receives and return a response to application server  105   c  over connection  106   c . This response will correspond to the request issued by application server  105   c  in response to the request it received from web server  103   a . Application server will thus process the response received from database server  107  and send a response to web server  103   a  over connection  104   c . This response corresponds to the request sent by web server  103   a , and thus web server will send a response to the initiating client  101 . This completes the entire transaction. It should be noted that the web server may have several requests/responses to one or more application servers, the same as application servers would have with one or more data base servers. 
     As described briefly above, it is often advantageous to be able to track a transaction through the various nodes and to know which nodes are used by a particular transaction. However, because transactions on the backside of a server are not typically visible from the front side of the server, another approach must be used to track a transaction. This approach makes use of a synthetic transaction robot such as PATROL® End-to-End Response Timer (“ETE”) available from BMC Software to define and drive a transaction. Based on information derived from ETE, it can be determined when a transaction truly starts and ends. 
     A first component of the transaction discovery system disclosed herein is the robotic transaction playback client. This component records the various components of a transaction in a fashion similar to a macro recorder in typical office suite type software. Once the transaction has been recorded, the system is able to replay the transaction multiple times and so that the resulting network traffic may be observed. As will be better understood with reference to the discussion below, the robotic transaction playback client may be configured to execute subsequent instances of a transaction at varying times and under varying network traffic conditions to allow the backend processor to uniquely identify specific nodes involved in a particular transaction and the behavior and performance of those nodes. 
     A second component of the transaction discovery system disclosed herein is a network record collector. The network record collector comprises a plurality of network probes that are used to collect discrete event information beginning with network flows. Each discrete event (for example, a request or a response) is recorded with relevant information about the event. Relevant information that is recorded (and later analyzed by the backend processor) includes, for example, originating network address and port, destination address and port, byte counts, and a timestamp, preferably with at least microsecond granularity. All of this information can be obtained by looking at the packet headers on the network, and it is not necessary to know the exact nature of the data being transmitted. This information is logged for each instance of a particular transaction being examined, and the aggregation of historical network flows are examined by the backend processor to determine which nodes a transaction goes through. Other events such as operating system or database events help determine activity within a node that is also tied to a transaction. 
     The third component of the system is the backend processor, operation of which will be described with reference to  FIGS. 2 and 3 . Backend processing generally takes in information from the robotic transaction player and the network probes. This information is analyzed with a goal of correlating network activities to the robotically executed transaction. This process may be better understood with reference to  FIG. 2 , in which a generic server node  200  is illustrated. Server node  200  receives a plurality of incoming requests, Req 1  and Req 2  from one or more nodes over connections  201  and  203 , respectively. These incoming requests are processed by server  200 , and server  200  issues further outgoing requests to another server (or servers), not shown, to complete the transaction. Outgoing request Req 1′  corresponds to incoming request Req 1  and is sent out over connection  204 . Outgoing request Req 2′  corresponds to incoming request Req 2 , and is sent out over connection  204 . 
     Incoming response ResP 1′ , corresponding to Req 1′ , is received over connection  203 , and incoming response Resp 2′ , corresponding to Req 2′ , is received over connection  204 . These responses are processed by server  200 , and outgoing response Resp 1 , corresponding to incoming request Req 1  and incoming response Resp 1′ , is returned over connection  201 . Similarly, outgoing response Resp 2 , corresponding to incoming request Req 2  and incoming response Resp 1′  is returned over connection  202 . 
     From the sequence of these responses and requests, it can ultimately be determined which nodes are involved in a particular transaction. (Sequence information is determined from the timestamps associated with each request and response, not shown, but described below with reference to  FIG. 3 .) 
     Illustrated in  FIG. 3  is a simple example involving one portion of the network  100  illustrated in  FIG. 1 . Client  101  issues a request Req 1  to web server  103   a . This request occurs at time 1:00:00. (Although the times are indicated in a particular format, it should be understood that any timestamp in any format would be usable in conjunction with the teachings herein. Additionally, specific values shown for each time stamp are exemplary only.) Web server  103   a  responds to this request at time 1:03:00 with response Resp 1 . To determine what other nodes are involved in the transaction, a monitoring program monitors the traffic on the backside of web server  103   a  for the time between 1:00:00 and 1:03:00. 
     Suppose that during this period, three request/response pairs having some network traffic occurring during the time period between 1:00:00 and 1:03:00 are identified: Req A /Resp A , Req B /Resp B , and Req c Resp c . Request/response pairs Req A /Resp A  and Req B /Resp B  involve application server  105   c , and request/response pair Req c Resp c  involves application server  105   b . Analysis of these request/response pairs, and their timing, can be used to determine which application server is involved in the transaction initiated by client  101 . 
     For example, it can be determined that request/response pair Req A /Resp A  is not part of the relevant transaction, because request Req A  was made by web server  103   a  at time 0:59:75, which is before the web server received the transaction initiating request Req 1  from client  101  at time 1:00:00. Further because RespC is received at 1:03:25, which is after Resp1 at 1:03:00, it can be determined that request/response pair Req c Resp c  is not part of the relevant transaction. Because request/response pair Req B /Resp B  is the only request/response pair during the relevant time frame, it can be determined that application server  105   c  is part of in the transaction in question. 
     In this simple example, it is clear that request/response pair Req B /Resp B  is communication that is relevant to the monitored transaction and application server  105   c  is the relevant node. By looking at the traffic that occurred on the backside of application server  105   c  during the time period between time 1:01:25 (the time of request Req B ) and 1:02:00 (the time of response Resp B ), it can similarly be determined which nodes downstream of application server  105   c  are involved in the transaction. By recursively analyzing traffic from subsequently further removed layers of the network, it can be determined which nodes are part of the relevant transaction. 
     Of course in any real world network, the traffic patterns will be substantially more complex than that described above. In such a case it is likely that there will be multiple request/response pairs that occur completely within the relevant timeframe. Additionally, it is likely that there may be multiple servers involved with these request response pairs. Many of these multiple servers may actually act as one server, e.g., a server farm or cluster that is addressed separately. The backend process is able to recognize this. In any case, it is likely that each step in a transaction could be uniquely identified by repeated application of the analysis technique described above. A slightly more complicated example is illustrated in  FIG. 4 . 
       FIG. 4  illustrates four successive executions  401 - 404  of the technique described with respect to  FIG. 3 . Each of these executions is a playback of the recorded transaction by the robotic transaction playback client. In the first execution  401 , client  101  initiates a transaction by sending request Req 1  at time 1:00:00 to web server  103   a . The transaction is concluded when client  101  receives response Resp 1  from web server  103   a  at time 1:05:00. The backend traffic from web server  103   a  during the time period between 1:00:00 and 1:00:05 consists of five request response pairs. Request/response pairs Req A /Resp A  and Req B /Resp B  are sent to/received from application server  105   a . Request response pair Req c Resp c  is sent to/received from application server  105   b . Finally, request response pairs Req D /Resp D  and Req E /Resp E  are sent to/received from application server  105   c . From this execution of the probing routine, it cannot be determined which one of application servers  105   a - 105   c  is involved in the transaction. 
     A second execution  402  provides additional information. As in the first execution, the transaction is begun by request Req 1  at time 2:00:00 and is concluded by response Resp 1  at time 2:05:00. However, during the intervening time period there are only three request/response pairs on the backend of web server  103   a . Request/response pair Req A /Resp A  is sent to/received from application server  105   a . Request/response pairs Req D /Resp D  and Req E /Resp E  are sent to/received from application server  105   c . Because there is no request/response pair sent to application server  105   b , it can be determined that application server  105   b  is not part of the transaction. However, it cannot yet be determined whether application server  105   a  or application server  105   c  is part of the relevant transaction. 
     It is thus necessary to monitor a third execution  403  of the transaction. As before, the endpoints of the transaction are request Req 1  sent from client  101  to web server  103   a  at time 3:00:00 and response Resp 1  received by client  101  from web server  103   a  at time 3:05:00. During the relevant time period, there are three request/response pairs on the backend of web server  103   a . Reqeust/response pair Req A /Resp A  is sent to/received from application server  105   a ; request/response pair Req C Resp C  is sent to/received from application server  105   b ; and request/response pair Req D /Resp D  is sent to/received from application server  105   c . However, in the previous execution of the transaction it was determined that application server  105   b  was not part of the transaction, therefore request/response pair Req C Resp C  can be eliminated from consideration, even though it involves a request response pair during the relevant time period. The third execution  403  thus provides no additional information for narrowing down the application server that is part of the transaction. 
     Therefore, a fourth execution  404  is required. Like the previous three executions, the transaction is defined as the time period between the originating request Req 1  sent from client  101  to web server  103   a  at time 4:00:00 and the terminating response Resp 1  received by client  101  from web server  103   a  at time 4:05:00. During the relevant time period, there are two request/response pairs on the backend of web server  103   a . Request/response pair Req c /Resp c  is sent to/received from application server  105   b , and request/response pair Req D /Resp D  is sent to/received from application server  105   c . From previous executions it is known that application server  105   b  is not part of the transaction, and thus it is known that the application server required by the transaction is application server  105   c.    
     In a sufficiently busy and/or complicated network, it may be that there is so much backend traffic on an affected node that it is impossible to identify with 100% certainty which downstream node is part of the relevant transaction, even with a substantial number of executions of the method described herein. Nonetheless, in these cases, it is still possible to establish a probability for each downstream node being the relevant node of the transaction. 
     Various heuristic methods are possible to determine the probability that a given node is part of a particular transaction. One approach to establishing such probabilities may be better understood with respect to  FIG. 5 . Network  500  is of substantially similar topology to network  100  discussed above with respect to  FIG. 1 . It is desired to know what network resources are used by a particular transaction executed by client  501 . Repeated executions of the transaction by the robotic transaction playback client. However, unlike the case illustrated in  FIG. 4 , the data collected from the network record collectors may not identify a single unique transaction path. 
     For example, it can clearly be determined that the transaction executed by client  501  requires interaction with web server  503  over communication link  502 . However, it may not be possible to derive from the traffic patterns whether the transaction ends at web server  503  or whether the transaction further requires interaction between web server  503  and one of application servers  505   a ,  505   b , or  505   c  over network links  504   a ,  504   b , or  504   c , respectively. On busy networks, there is likely to be additional traffic between these nodes that is temporally coincident with the request response pair from client  501  corresponding to the transaction of interest. Furthermore, the traffic patterns may not clearly indicate whether the transaction ends with one of the application servers  505   a - 505   c  or whether further interaction is required with database server  507  over one of communication links  506   a ,  506   b , or  506   c . Thus, for purposes of this example, assume that seven possible transaction paths have been identified: (1)  501 - 503 , (2)  501 - 503 - 505   a , (3)  501 - 503 - 505   b , (4)  501 - 503 - 505   c , (5)  501 - 503 - 505   a - 507 , (6)  501 - 503 - 505   a - 507 , and (7)  501 - 503 - 505   a - 507 . 
     Initially, equal probabilities are assigned to each of the transaction paths. Thus with seven possible paths, there is a 14.285% chance that one of these paths is the correct transaction path. The backend processor thus assigns a 14.285% probability to each of the identified paths. However, as noted above, more than timing information of a request response pair is monitored by the network record collector. Other information—for example, byte counts—are also monitored. If there is additional correlation of one of these additional parameters between one or more of the identified potential transaction paths, it is flagged as being higher priority. 
     Once this additional information has been identified for each path, the paths that have been flagged as being a higher priority are assigned an adjusted probability to account for the higher likelihood that it is the correct path. This adjusted probability may be determined by various methods. One method is to adjust the probability by adding 10% of the base probability of the path times the number of nodes by which the particular endpoint is removed from client. So, for purposes of the example depicted in  FIG. 5 , assume that only the path  501 - 503 - 505   b  were flagged as higher probability. Server  505   b  is two endpoints removed from client  501 , so the adjusted probability is 14.285+(1.4285*2)=17.142. However, because the probability of this node has been increased without decreasing the probability of the other nodes, it is necessary to rescale the probabilities, which is done by summing the adjusted probabilities and using this sum as a divisor against the individual probabilities. With only one path adjusted, the sum would be (14.285*6)+17.142=102.852. The probability for the adjusted node is then 17.142/102.852=16.667%, and the probability for the remaining nodes is 14.285/102.852=13.889%. As would be apparent to one skilled in the art, recursive application of this algorithm, or any similar probability adjustment algorithm, would ultimately allow the node involved in a transaction to be identified with relative certainty. 
     A method and system for discovering information technology resources involved in a particular network transaction have been disclosed herein. While the invention has been disclosed with respect to a limited number of embodiments, numerous modifications and variations will be appreciated by those skilled in the art. It is intended that all such variations and modifications fall with in the scope of the following claims.