Abstract:
An apparatus and associated method is provided comprising a software system resident in a memory space configured to execute a service request which results in a plurality of events carried out by the software system to complete the service request, and creates a plurality of entries associated with the events, wherein each entry defines an operational state of the software system.

Description:
FIELD OF THE INVENTION  
       [0001]     The claimed invention relates generally to the field of computer software and more particularly, but not by way of limitation, to an apparatus and method for real time logging and analyzing event flows associated with execution of a service request in a software system.  
       BACKGROUND  
       [0002]     Computer-based systems enable a wide variety of data processing tasks to be accomplished in a fast and efficient manner. From hand-held consumer products to geographically distributed storage area networks with multi-device data storage arrays, such systems continue to proliferate into all areas of society and commerce.  
         [0003]     Software is provided to direct the operation of such systems. Software (including firmware) can take a number of forms such as application programs, operating systems, interface and controller routines, and maintenance and housekeeping modules.  
         [0004]     Each time a process is initiated by the software, a number of additional processes, handshakes, links, calculations, and other events can be carried out by the various layers of software in order to service and complete the service request. Generally, with increased complexity in the overall software system, including additional layers of software applications and operating systems, distributed processing and fault tolerant redundancy, it becomes increasingly difficult to assess the extent to which system resources are utilized to carry out a given process.  
         [0005]     In some solutions these resultant events are logged for purposes of analysis in the event an execution error is identified. However, typically the error is identified at some time after the execution step causing the error, making it painstakingly difficult, if not impossible, to trace back through the logged events to ascertain a root cause. What is needed is a solution providing a real time analysis of system execution errors. It is to these benefits and advantages that the embodiments of the present invention are directed.  
       SUMMARY OF THE INVENTION  
       [0006]     Embodiments of the present invention are generally directed to a method and an apparatus for logging and analyzing event flows associated with execution of a service request in a software system, such as a computer-based system comprising a multi-disc data storage array.  
         [0007]     In some embodiments a method is provided comprising: executing a service request in a software system resulting in a plurality of events performed by the software system; accumulating in a selected memory space a plurality of entries associated with the events, wherein each entry defines a software system operational state; and analyzing the entries to determine whether an error occurred during the accumulating step.  
         [0008]     In some embodiments an apparatus is provided comprising a software system resident in a memory space configured to execute a service request which results in a plurality of events carried out by the software system in completing the service request. The software system also creates a plurality of entries associated with the events, wherein each entry defines an operational state of the software system.  
         [0009]     In some embodiments an apparatus is provided comprising a software system resident in a memory space configured to execute a service request which results in a plurality of events carried out by the software system to complete the service request, and means for determining a software system execution error by associating the state of the software system with the events.  
         [0010]     These and various other features and advantages which characterize the claimed invention will become apparent upon reading the following detailed description and upon reviewing the associated drawings. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0011]      FIG. 1  is a top level functional block depiction of a computer-based system characterized as a storage area network utilizing mass storage.  
         [0012]      FIG. 2  provides a functional block diagram to illustrate a selected one of the controllers of  FIG. 1 .  
         [0013]      FIG. 3  illustrates a number of different software layers utilized by the system of  FIG. 1 .  
         [0014]      FIG. 4  provides a preferred arrangement for an event log and analysis module utilized by the software of  FIG. 3 .  
         [0015]      FIG. 5  shows a resource index stack to identify system resources available to the software of  FIG. 3 .  
         [0016]      FIG. 6  depicts a generalized format for the global event log of  FIG. 4 .  
         [0017]      FIG. 7  generally illustrates a partial chain of events carried out in response to issuance of an operation request.  
         [0018]      FIG. 8  is a flowchart of illustrative steps of an error detect method in accordance with embodiments of the present invention.  
         [0019]      FIG. 9  represents an illustrative output of the analysis tool in tracking the entries from the global data log of  FIG. 6 . 
     
    
     DETAILED DESCRIPTION  
       [0020]     To illustrate an exemplary environment in which presently preferred embodiments of the present invention can be advantageously practiced,  FIG. 1  shows a computer-based system  100  characterized as a storage area network (SAN) utilizing mass storage.  
         [0021]     The system  100  includes a number of host computers  102 , respectively identified as hosts A, B and C. The host computers  102  interact with each other as well as with a pair of data storage arrays  104  (denoted A and B, respectively) via a fabric  106 . The fabric  106  is preferably characterized as a fibre-channel based switching network, although other configurations can be utilized as well, including the Internet.  
         [0022]     Each array  104  includes a pair of controllers  108  (denoted A 1 , A 2  and B 1 , B 2  respectively) and a set of data storage devices  110  preferably characterized as hard disc drives operated as a RAID (redundant array of independent discs). The controllers  108  and devices  110  preferably utilize a fault tolerant arrangement so that the various controllers  108  utilize parallel, redundant links and at least some of the user data stored by the system  100  is mirrored on both sets of devices  110 .  
         [0023]     It is further contemplated that the A host computer  102  and the A data storage array  104  are physically located at a first site, the B host computer  102  and B storage array  104  are physically located at a second site, and the C host computer  102  is at yet a third site, although such is merely illustrative and not limiting.  
         [0024]      FIG. 2  illustrates a selected one of the controllers  108  in greater detail. The controller  108  can be embodied in a single integrated circuit, or distributed among a number of discrete circuits as desired.  
         [0025]     A main processor  112 , preferably characterized as a programmable, general purpose computer processor, provides top level control in accordance with programming steps and processing data stored in non-volatile memory  114  (such as flash memory or similar) and in dynamic random access memory (DRAM)  116 .  
         [0026]     A fabric interface (I/F) circuit  118  communicates with the other controllers  108  and the host computers  102  via the fabric  106 , and a device I/F circuit  120  communicates with the storage devices  110 . The I/F circuits  118 ,  120  and a path controller  122  form a communication path to pass commands and data between the storage array and the host using cache memory  124 .  
         [0027]      FIG. 3  provides a generalized depiction of a preferred arrangement for software utilized by the system  100  of  FIG. 1 . In  FIG. 3 , the software is viewed as being arranged in a number of hierarchical layers (in this case Layer  1  through Layer  7 ), with Layer  1  representing a top-most layer in farthest proximity to physical media  126  of the devices  110 , and Layer  7  representing a bottom-most layer most closely associated with the media  126 . Other suitable arrangements can be used as desired, including different numbers and types of layers. For purposes of the present discussion, it is contemplated that Layers  1 - 3  generally represent application routines and operating systems utilized by the host computers  102 , Layers  4 - 5  represent routines and modules at the controller  108  level, and layers  6 - 7  represent routines and modules at the device  110  level.  
         [0028]     As will be recognized, a data transfer request from a selected host computer  102  to write data to the devices  110  will invoke one or more host level command processes. Such processes in turn invoke various other processes down and back up through the various layers to ultimately result in the completion of the desired task. Along the way, various system resources are allocated, utilized and released.  
         [0029]      FIG. 4  represents a preferred arrangement for an event log and analysis module  130  used to collect entries associated with each event carried out by the software of the system  100 , at all levels, and then to generate an analysis of software system resource utilization from the sequence of entries. The module  130  preferably includes a log manager block  132 , a global event log  134 , an analysis tool  136  and a graphical user interface (GUI)  138 . Each of these blocks will be discussed in greater detail below.  
         [0030]      FIG. 5  provides a resource index stack  140  to illustrate various types of system resources utilized by the system software. Because the formats and nomenclatures for the resources can vary widely depending upon the architecture of a given application, for simplicity the resources will be generically referred to as Resource  0  through Resource N. Resource  1 , for example, can represent an operation request which is used to identify various variables, values or attributes associated with a given host level command process, and so on.  
         [0031]     During operation, a service request will generally allocate one or more of the resources to perform a desired operation, and may invoke or link to one or more lower level processes until the service request is completed. Preferably, upon the operation of every event, each of the software layers operates to provide an entry into the global event log  134 , as depicted by  FIG. 6 . The log  134  is preferably written to a selected disc location on the media  126  and represents a sequential listing of all events carried out by the software.  
         [0032]     Each entry is associated with a respective event, and defines the operational state of the software system at a time corresponding to the event. The event entries also include indexing information that identifies the initiating process (such as Process( 1 ), Process( 2 ), and Process( 3 ) as generally identified in  FIG. 6 ). Each event is broadly defined by the operation of a particular layer or module to obtain a result, such as the retrieval of or writing of data to a memory space, performing a calculation, linking to or delinking from another layer, etc. Preferably, all layers write to the same log and the log reflects all activity in a time-sequential manner.  
         [0033]     The size of the log (i.e., the amount of memory allocated therefor) can be selectively varied by the log manager and by inputs from the user. A larger allocated space will track a greater history window, but will also consume greater system resources. It is contemplated that the log is preferably sized to reflect several hours of continued operation of the system  100 . The log  134  is further preferably characterized as having a buffer for storing a predetermined plurality of the entries, which can be batch processed by the analysis tool  136  for reducing the processing overhead burden on the system  100 . A counter  142  ( FIG. 4 ) of the log manager  132  can be used for collection and/or retention of the predetermined number of entries.  
         [0034]      FIG. 7  represents initial operations carried out in response to a selected initiating process. For purposes of illustration, it is contemplated that this process is an operation command associated with the writing of user data by a selected host computer  102  to the devices  110 .  
         [0035]     As shown by  FIG. 7 , the top level command invokes the operation of a first module  144  in Layer  1  which allocates a first resource (Resource  1 ) and links to a second module  146  in Layer  2 . The second module  146  allocates a second resource (Resource  2 ) and performs one or more additional links to lower level processes (denoted at  148 ). This process will continue until at some point the user data are transferred to cache memory  124  ( FIG. 2 ), a write-back command is issued to the initiating host computer  102 , the data are transferred to the appropriate physical media  126 , and the responsible controller is notified that the data have been successfully written. It will be noted that the global event log  134  preferably captures all events relating to the service request, including events that are carried out after the top level process has been identified as being completed (as in, for example, the case of write-back data).  
         [0036]     The log  134  can be updated in a real-time fashion, but process efficiencies can be generally obtained by caching, at either a layer level and/or at a device level, a short listing of the recent events which is then transferred to the log manager at an appropriate time. In this way, the global collection of event data as described herein can have a reduced, if not wholly transparent, effect upon system  100  availability.  
         [0037]     The analysis tool  136  of  FIG. 4  preferably operates in response to manual user requests to poll the global event log  134  and retrieve all entries relating to a particular initiating process. The analysis tool uses the addresses for link and delink events to expand and contract its search lists, thereby accounting for all of the events associated with the service request through the various layers.  
         [0038]      FIG. 8  is a flowchart illustrating steps for an error detect method  150  performed by the logging and analysis module  130  in accordance with embodiments of the present invention. The method begins with executing the service request in the software system resulting in the plurality of events performed by the software system, and managing the entries by the software system associated with the events in block  152 . In block  154  the entries are accumulated until the predetermined number are obtained. In determination block  156  it is determined whether the sample collected, “n,” is equal to the predetermined number desired, “N.” If no, then processing returns to block  154  and more entries are obtained. If yes, then control passes to block  158  where the analysis tool  136  batch processes the entries to determine whether any execution error is evident. The results of the analysis are reported in block  160 . The counter is reset in block  162  and control returns to block  152  where the next set of entries is obtained.  
         [0039]     The resulting analysis can be presented by the GUI  138  in any suitable fashion, such as the system resource utilization diagram in  FIG. 9 , or in time sequence order to provide the desired system allocation information desired in relation to the events that resulted from the initial service request. In the embodiments of  FIG. 9 a  threshold utilization  170  provides a visual comparison against individual utilization indications  172 - 180 . Variance bands (not shown) can be constructed around the threshold  170  to identify potential error conditions. For example, it would appear that the observed utilization  176  is substantially below the expected level, while the observed utilization  178  is substantially above the expected level. Variances such as these can be used to investigate the root cause for error conditions.  
         [0040]     It is contemplated that the GUI  138  is accessible via the host computers  102  using system administrator access, although other suitable arrangements including a dedicated computer can be utilized as well.  
         [0041]     The analysis tool  136  can also be operated in an automated fashion in response to certain pre-set or user selectable occurrences, such as command timeouts, error conditions and the like, in order to provide an indication of system utilization in the moments leading up to such occurrences.  
         [0042]     The global event log  134  is preferably physically arranged as a single structure, but in other preferred embodiments the log can be physically distributed among different locations so long as the log correctly reflects all of the entries associated with each event. The logging and analysis module  130  thus provides valuable information regarding system resource utilization during system design efforts as well as during field maintenance performance evaluation and failure analysis.  
         [0043]     In view of the foregoing discussion, it will now be understood that preferred embodiments of the present invention are generally directed to a method and apparatus for logging and analyzing event flows associated with execution of a service request in a software system, such as a computer-based system (such as  100 ) comprising a multi-disc data storage array (such as  104 ).  
         [0044]     Execution of the service request preferably results in a plurality of events carried out by multiple layers of the software system. A global event log (such as  134 ) accumulates a corresponding plurality of entries, each defining a current software system state.  
         [0045]     An analysis tool (such as  136 ) preferably accesses the global event log to provide a real time indication of execution errors, such as in a graphical display by a graphical user interface (such as  138 ). A log manager preferably controls operation of the global event log (such as  132 ).  
         [0046]     Entries in the global event log are preferably entered sequentially for each ongoing process, and each entry preferably includes an identifier that identifies the associated service request.  
         [0047]     For purposes of the appended claims, the recited “means for determining” will be understood to correspond to the disclosed module  130  of  FIG. 4 . Systems that provide localized logging of some events at certain layers without providing a centralized logging of all events associated with a particular service request fail to carry out the recited function and are excluded from the definition of an equivalent.  
         [0048]     It is to be understood that even though numerous characteristics and advantages of various embodiments of the present invention have been set forth in the foregoing description, together with details of the structure and function of various embodiments of the invention, this detailed description is illustrative only, and changes may be made in detail, especially in matters of structure and arrangements of parts within the principles of the present invention to the full extent indicated by the broad general meaning of the terms in which the appended claims are expressed. For example, the particular elements may vary depending on the particular processing environment without departing from the spirit and scope of the present invention.  
         [0049]     In addition, although the embodiments described herein are directed to a data storage array, it will be appreciated by those skilled in the art that the claimed subject matter is not so limited and various other processing systems can be utilized without departing from the spirit and scope of the claimed invention.