Abstract:
An improved method of analyzing software issues may include retrieving and storing selected data elements from the operating system kernel data prior to performing a memory dump. The method of retrieving the selected kernel data may include creating a thread dedicated to collecting the data and storing it in a memory location for analysis after the memory dump. The operating system kernel data may be analyzed in conjunction with the prior art dump data to identify a root cause of the software issue.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This Patent Application claims priority to Chinese Patent Application No. 201310755993.9 filed on Dec. 30, 2013, entitled “INCLUDING KERNEL OBJECT INFORMATION IN A USER DUMP,” the contents and teachings of which are hereby incorporated by reference in their entirety. 
       BACKGROUND 
       [0002]    Storage arrays in data storage systems are controlled by storage processors that process data store and data access requests from host computers. Storage processors are computers that have software designed to handle input and output requests from other computers or electronic devices needing access to large memory capacity. 
         [0003]    In a storage processor, a computer software program may not always perform exactly as intended, and there may be problems or glitches in the software that need to be corrected or fixed. Along these lines, consider handle leaks, which happen because the pool of available file handles gets exhausted when file handles are not properly returned to the pool after usage. The storage processor adds a file handle to a file to simplify handling and tracking of the file while it is in the storage processor. When the file is deleted from the storage processor, software in the storage processor is supposed to cause the storage processor to return the file handle to the pool for reassignment to another file. If the storage processor does not always return the file handle of a deleted file to the pool, then eventually the number of available file handles will be exhausted. 
         [0004]    To determine where the issue is located in the software operation, a failure analyst will look at what the computer software program, known as a thread, was doing at the time of the failure. The data used by the failure analyst may be found in what is known as a memory dump file. A dump is essentially a snapshot of the thread status. A memory dump does not have to be a system crash, but may be used for data gathering. 
         [0005]    The computer may generate a memory dump file before the program reboots, in order to provide some information for the failure analyst to use to identify where in the program code the fault maybe found and a root cause. Conventional software issue identification methods involve examining the memory dump file. 
       SUMMARY 
       [0006]    Unfortunately, there are deficiencies with the above-described conventional software issue identification methods. For example, the memory dump file may not always provide sufficient data for an analyst to determine or identify a root cause of the software issue. For example, in the file handle leakage situation discussed above where the program handling the files fails to return the file handle to the pool when the file is deleted, the dump file alone may not provide enough information to easily identify a root cause of the software issue. This is because the file handle data in the memory may not have a file name associated with each of the file handles, and the particular thread of the software producing the software issue may be difficult to identify without knowing the file names associated with the issue. 
         [0007]    In contrast to the above described conventional software issue identification methods, improved techniques involve retrieving and storing selected data elements from the operating system (OS) kernel prior to performing a memory dump. Retrieving OS kernel data may include creating a thread dedicated to collecting the data and storing it in a memory location for analysis after the memory dump. The operating system kernel data will be analyzed either alone or in conjunction with the memory dump data to identify a root cause of the software issue. The selected data elements from the operating system kernel can include file names associated with outstanding file handles, and sync objects such as mutex, critical section, event, and semaphore data. Operating system kernel data can be used to more easily identify the location in the software that is having handle leak issues, deadlock issues, process hanging issues and thread hanging issues. 
         [0008]    In an arrangement, a system for identifying a software issue in a computer includes a communication interface, a memory and a processing circuit with a controller. The processing circuit can execute a software process to store OS kernel data in a memory location prior to a memory dump caused by a software issue. The OS kernel data can be analyzed either alone or in conjunction with memory dump data to identify a root cause, and generate an alert when a root cause is identified. 
         [0009]    In an arrangement, a computer program product with a non-transitory computer readable medium stores a set of instructions performing a method of identifying a software issue, by storing and analyzing operating system kernel data for identifying a root cause of the software issue. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0010]      FIG. 1  is a block diagram of exemplary components of an electronic device that is enabled to identify a software issue. 
           [0011]      FIG. 2  is a flow chart of a procedure performed by the electronic device of  FIG. 1 . 
       
    
    
       [0012]    The foregoing and other objects, features and advantages will be apparent from the following description of particular embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of various embodiments of the invention. 
       DETAILED DESCRIPTION 
       [0013]      FIG. 1  is a block diagram of exemplary components of an electronic device  100  that is enabled to identify a software issue. The electronic device may be a computer, for example, a storage processor such as  102  or  132 , and may receive and store data from host servers such as hosts  128  and  158 , and recover and transmit data to host servers  128  and  158 . 
         [0014]    The storage processor  102  has a memory  104 , a processor  112 , a logic circuit  120  and an input/output (I/O) unit  126 . The memory  104  has a set of memory locations, some of which are used to store data obtained from the memory dump operation in a memory section  106 . A memory section  108  is used to store data obtained from the OS kernel  118 , and a memory section  110  is used to store the integrated data from both the memory dump  106  and the OS kernel memory  108 . The integrated data in memory section  110  may be the addition of the two memory sections, or it may be calculated values from various forms of analysis such as may be created by the logic circuit  120 . There may be portions of memory  104  which are used for storing results from the root cause analysis, or historical trend data. 
         [0015]    The processor  112  may operate a set of individual program steps, which may be known as a thread. For example, thread  1  may be located at a portion of the processor shown as  114 , and may be handling data store requests and data retrieve requests for a portion of a set of hosts, such as  128  or  158  via bus  130 . Thread  2  at location  116  may be handling requests from a different set of hosts or users, or for a different storage processor, such as storage processor  132 , via buses  130 ,  160  and  162 . 
         [0016]    Processor  112  may be the only processor in storage processor  102 , or it may be one of a set of individual processors. Processor  112  has an operating system (OS) program that controls its actions, and an OS kernel  118  has information stored in a memory location in processor  112 . The OS kernel will include tables of pointers and maps that associate a data file with the logical and physical location of the file in the storage processor  102 , and shortcuts such as file handles that are attached to the files. 
         [0017]    The logic circuit  120  may include circuitry  122  for monitoring the status of operation of the various threads  114  and  116 . For example, if the number of file handles available in the pool is less than a threshold, the monitor  122  may look at the recent past trend in the number of file handles in the pool to determine that a handle leak software problem exists. The monitor  122  may initiate a thread to collect selected elements of information from the OS kernel, such as the file names associated with the file handles that have not been returned to the pool even though the files have been deleted, and send the data to the analyzer  124 , and store the data in memory location  108 . With such an arrangement, data needed for proper analysis may be obtained if a file dump occurs, or the root cause of the handle leakage may be discovered prior to a failure. 
         [0018]    The I/O unit  126  controls input and output communications between the various portions of storage processor  102  and other storage processors, for example,  132 , and hosts  128  and  158 . I/O unit  126  may also communicate with a storage processor user or manager, and with other devices. 
         [0019]      FIG. 2  is a flow chart of a procedure  200  performed by the electronic device  100  of  FIG. 1 , where at step  202  it is determined that a memory dump may be needed. For example, the monitor  122  of  FIG. 1  may note that a number of file handles available has been steadily dropping, or a connection between thread  114  and a host  128  or another thread  116  has not responded for more than a selected time period, i.e., the thread has hung up. In either of these situations, or with many other possible issue warning signs, the logic circuit  120  may determine that a memory dump may occur and initiate action. A memory dump may be considered a snapshot of the state of a program operation or thread status, and does not necessarily imply an overall system crash. The memory dump process may simply include storing the status of the thread at a known time point, and rebooting the part of the system at issue. 
         [0020]    At step  204  the logic circuit  120  accesses selected data from the OS kernel  118 , and at step  206  stores the selected data in a memory location  108  in memory  104 . Examples of the type of OS kernel data  108  that may be useful to a failure analyst includes file names for the file and event handles, mutex, semaphore, event, critical section, process thread hang and critical timeout data. 
         [0021]    At step  208  either an analyst looks at the OS kernel data, or the analyzer  124  of the logic circuit  120  examines the stored OS kernel data, to identify a root cause for the software issue. If a root cause can be identified at step  208 , then at step  210  the root cause is transmitted at step  212  via I/O  126  to a user of the storage processor, such as a manager of the storage processor, or the user of host  128  if that user has management authority. 
         [0022]    If no root cause is identified at step  208  then at step  210  the method moves to step  214  where the logic circuit  120  performs a memory dump, and at step  216  stores the dump data in memory location  106  of memory  104 . At step  218  either an analyst looks at the dump data, or the analyzer  124  of the logic circuit  120  examines the dump data, to identify a root cause for the software issue. If a root cause can be identified at step  220 , then at step  222  the root cause is transmitted to the user of the storage processor, such as a manager of the storage processor  102 . 
         [0023]    If no root cause can be identified, then the method moves to step  224  where the logic circuit  120  combines the OS kernel data  108  and the dump data  106  to form combined data  110  in the memory  104 . The combined data may be the addition of the OS kernel data to the dump data, or it may be mathematically transformed by the logic  120 , or it may involve historical trend data or extrapolated data as well. Also at step  224  the combined data is analyzed, by either an analyst or the analyzer  124 , to identify a root cause of the software issue. 
         [0024]    At step  226 , if a root cause has been identified at step  224  the root cause is transmitted by I/O  126  to the user. If no root cause has been identified the method ends, and the analyst will perform test procedures on the storage processor to reproduce the software issue and attempt to identify a root cause. 
         [0025]    With such an arrangement many root causes of software issues in electronic devices can be automatically identified, or the task of the failure analyst is improved and resolved more quickly. 
         [0026]    As used throughout this document, the words “comprising,” “including,” and “having” are intended to set forth certain items, steps, elements, or aspects of something in in that these are provided by way of example only and the invention is not limited to these particular embodiments. In addition, the word “set” as used herein indicates one or more of something, unless a statement is made to the contrary. 
         [0027]    Having described certain embodiments, numerous alternative embodiments or variations can be made. For example, the above discussion has used storage processors directly connected by bus communication lines to other storage processors and host servers, however, the storage processors do not need to use bus lines for communication, and wireless, RF, infrared, communication lines may be used. The communications may also use network connections, such as the internet or cloud and need not be directly wired. Furthermore, the arrangements discussed may be used by any electronic device and not simply limited to storage processors, and any computer may use the described arrangement or equivalent apparatus and instructions to automatically obtain operating system information that may be relevant to failure analysis. 
         [0028]    The improvements or portions thereof described herein may be embodied as a non-transient computer-readable storage medium, such as a magnetic disk, magnetic tape, compact disk, DVD, optical disk, flash memory, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), and the like. Multiple computer-readable media may be used. The medium (or media) may be encoded with instructions which, when executed on one or more computers or other processors, perform methods that implement the various processes described herein. Such medium (or media) may be considered an article of manufacture or a machine, and may be transportable from one machine to another. 
         [0029]    Further, although features are shown and described with reference to particular embodiments hereof, such features may be included in any of the disclosed embodiments and their variants. Thus, it is understood that features disclosed in connection with any embodiment can be included as variants of any other embodiment, whether such inclusion is made explicit herein or not. 
         [0030]    While various embodiments of the invention have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.