Method and system for problem determination using probe collections and problem classification for the technical support services

A system and method for problem determination using probe collections and problem classification for the technical support services monitor and collect data associated with a computer system, raise an alarm based on the monitored and collected data, probe the computer system for additional information, filter the monitored and collected data based on the additional information established from probing, and use the filtered data to label a problem associated with the raised alarm.

BACKGROUND

The present disclosure relates generally to computer systems and service technologies, and more particularly to problem determination using probe collections and problem classification for the technical support services.

An effective problem determination and resolution (PDR) process can contribute to a substantial reduction in technical support services costs. PDR is the process of detecting anomalies in a monitored system, locating the problems responsible for the issue, determining the root cause and fixing the cause of the problem. Thus, once the user (customer or technical personnel) detects a problem, he first tries to identify the type of problem in order to search for the relevant fix. Especially in case of software problems in multi-tier information technology (IT) environment with complex system dependencies, the user may experience a front-end issue caused by a back-end problem. Thus, the problem may be only the effect of an underlying issue within the IT environment and, on one hand, the fixes found may not address the root cause, while on the other hand, the root cause may be buried in large amounts of logs, traces, and monitoring data from healthy resources involved in the propagated failure. Analyzing all the available logs and monitoring data is time consuming and error prone, therefore the PDR process would benefit from filtering the data related to the failing resource.

An example of a multi-tier environment is an e-business system which is supported by an infrastructure including for example the following subsystems connected by local and wide area networks: web based presentation services, access services, application business logic, messaging services, database services and storage subsystems. The existing solutions that provide problem determination are limited in that they are problem specific and as such lack the potential of being applied to wider type of issues. Other known methodologies use only one type of the available information, overlooking or ignoring information that may be relevant to the problem at hand. Yet other known methods provide particular approaches to the PDR technology that is applicable in specific scenarios only.

BRIEF SUMMARY

A method and system for problem determination using probe collections and problem classification for the technical support services are provided. The method, in one aspect, may include monitoring and collecting, by a processor, data associated with a running computer system and raising an alarm, automatically by the processor, based on the monitored and collected data. The method may further include probing the computer system for additional information associated with the alarm; and filtering the monitored and collected data based on the additional information established from probing. The method may also include using the filtered data to label a problem associated with the raised alarm.

A system for problem determination using probe collections and problem classification for the technical support services, in one aspect, may include a monitoring and data collection processing module operable to monitor and collect data associated with a computer system, the monitoring and data collection module further operable to raise an alarm based on the monitored and collected data. The system may also include a probe platform operable to probe the computer system for additional information, the probe platform further operable to filter the monitored and collected data based on the additional information established from probing. The system may further include a classifier module operable to use the filtered data and automatically label a problem associated with the raised alarm.

A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform methods described herein may be also provided.

DETAILED DESCRIPTION

In one embodiment, the system and method of the present disclosure effectively localize IT problems by appropriately probing the failing environment and categorizing the IT resources in view of filtering the available data such as, but not limited to, performance data, resources consumption data, logs data, for more focused or targeted problem determination.

The system and method of the present disclosure may include processes and algorithms to:Probe the failing environment thus localizing the problem and filtering out the logs and monitoring data relevant to the failing resource from the data generated by resources under failure propagation;Label the problem (e.g., the problem ticket) and the related log and monitoring data with the real underlying problem that occurred when the data was collected;Learn from historical labeled data the patterns specific to the existing problem taxonomy;Recognize problems when given a new set of log and monitoring data based on the patterns learned at c).Enrich the probing repository with adding executable probing algorithms and/or plans by the community to share the experience across customer environment and problem resolutions.

The system and method of the present disclosure may 1) enhance the problem determination efficiency by combing appropriately different problem determination technologies instead of focusing on a particular one; and, 2) enable the user to identify the root cause so that the fix is searched based on the cause rather than on its propagated effects. This may result in operational cost savings. A more focused, filtered set of data increases the accuracy of the problem identification. If the problem results in a service request submission to the technical support, the problem routing can benefit from this initial problem classification instead of symptom based routing. This reduces the risk of re-routing the request when the symptom is a remote effect of a cause located in a completely different resource.

In one embodiment, we build a framework that brings the principles of Service Oriented Architecture (SOA) and information integration to automate problem determination activities. The framework may reduce the time to localize a problem. Following SOA principles, various reusable activities are modeled as services that are further composed to offer higher level services. Examples of such services may include:Probe execution services (e.g., a script execution service whose inputs are a script, the values of associated parameters and the execution environment);Output analysis services (e.g., a service that parses the text output of a probe execution service and creates output in the form of structured data, given mapping rules, possibly based on regular expressions);Information integration services (e.g., a service that provides values to variables given the mapping from query to information stores such as CMDB).

A diagram of problem determination process using probe collections and problem classification according to a method in one embodiment of the present disclosure is shown inFIG. 1. Each item and the flow of information between them during the operational phase of the system are described below.

The customer or user102may be experiencing a problem in their IT environment, for example an IT multi-tier distributed environment104. An example of such a multi-tier environment is an e-business system which is supported by an infrastructure including the subsystems for example that may be connected by local and wide area networks, such as the following: web based presentation services, access services, application business logic, messaging services, database services and storage subsystems.

A monitor and data collector module108may monitor the customer's IT environment104, for example, by running a process106, and collecting periodically data such as, but not limited to, performance, resource consumption, logs data. A monitor and data collector module108may be a monitoring tool deployed for the customer's IT environment, for example, including monitoring server, agents, probes, data warehouses, etc. Such monitoring may employ tools such as NetSol™, ITM™, Director™.

A problem notification or an alarm114may originate from the user. A problem notification or an alarm110,112also may be generated from the monitoring tool, for example, monitor and data collector108. For instance, if a monitored data exceeds a predetermined criteria or threshold, the monitoring tool108may raise an alarm. The monitoring tool108also may provide data associated with monitoring shown at116and124. This data may include, but is not limited to, performance data, resource utilization data, inventory, logs, traces. The type of data provided may depend on the type of monitoring tool employed in the monitoring tool108. Data is provided for example, to End-to-end Probing Platform (EPP) also referred to as a probing plan execution platform118, Labeled Data Repository120and Classifier module122.

The monitored data116may be filtered, for example, by the monitor and data collector module108, and sent to Problem Determination Module128as shown at126, to Labeled Data Repository120shown at124, and to Classifier module112as shown at150. This data includes data filtered from the data at116, i.e., data collected by the monitoring tool108, by using the information on the failing resource(s) also collected at the monitoring tool108. All the data related to well behaving resources is filtered out.

A probing plan execution platform118may include, for example, a workflow engine such as a Business Process Execution Language (BPEL) engine, a probe descriptor interpreter, connectors to the configuration and monitoring databases such as configuration management database (CMDB)132, connectors to the managed systems such as ssh client, http client, java rmi client, and web services client, various types of probe launchers and result analyzers including script executer, text parsers for script output, logs, traces data, etc., and web service response analyzer. Based on the received alarm type110and monitoring data116, logs, etc., the platform118decides on and executes the workflow in the probing plan.

Examples of probes may include Telnet and Ping. Telnetting to a database server port or invoking an operation of a Java Management Extensions (JMX) MBean of an Application Server are examples of system probes. For instance, the Telnet probe can be used to check whether a database server is running, while the JMX probe can be used to check whether the connections held by the Application Server are usable.

Low level services may be grouped in a collection of probes based on different ways of launching the probes and analyzing the outputs, such as invoking a web service, executing a database query, enabling ARM or monitoring transaction paths. For instance, the following illustrates a low level service template that executes an OS “command” and produces “textOutput” that contains the output of the command:OSCommand Execution ServiceInput—String: commandOutput—String: textOutput

This service executes the command on its hosting machine in one embodiment. An advanced service for similar purpose may use host, sshUser and sshPassword as other possible inputs for executing the command remotely via ssh.

As another example the following illustrates a low level service template that creates an instance of a java class, namely javaClassName and populates its attributes using the regular expression based functions applied on text. attribute_RegEx, which contains the mapping of attribute names to such functions:Simple Text Analysis ServiceInput—String: textString: javaClassNameMap: attribute_REgEXOuput—Object: output

A simple system/network level probe template, namely Telnet, may be composed of lower level services. The probe can take two input parameters ip and port and produce a Boolean output, namely serviceRunning, that shows whether a process is listening on the given port. An example flow for such probe is shown inFIG. 4.

Another probe may be built that abstracts the Telnet probe at higher level for easily using it to check whether a database server is running. For example, a high level probe template namely, IsDBServerUP, may be composed of Telnet Probe and CMDB Query Service. This probe may provide a higher level of abstraction by taking DBServerID (unique ID of a DBServer in the CMDB) as input and producing boolean output namely isDBServerUP that shows whether the DBServer is running fine.FIG. 5illustrates an example of this probe. Other advance implementations of similar service may involve more rigorous probing e.g., issuing a database query.

A composition of probes that includes control flow logic as well may provide the higher level probe referred to as a probing or probe plan. For instance, a simple probe template may operate at the application level and encapsulate a probing plan. A probing plan may include composite probes/services and control flow, data transformations and flow, with decision control flow.FIG. 6illustrates an example of a probe template that operates at an application level and encapsulates a probing plan. For instance, services shown inFIG. 6, e.g., Service1, Service2, Service3, Service4, Service5, Service6represent probes incorporated in the control flow ofFIG. 6. The flow shown inFIG. 6omits the details of data transformations and flow, for simplicity. The decision box614illustrates the handling of variation in the topology of the solution, based on information available in CMDB.

The low level services described above can be grouped in a collection of probes based on different ways of launching the probes and analyzing the outputs such as invoking a web service, executing a database query, enabling ARM or monitoring transaction paths.FIG. 4illustrates a simple system/network level probe template, namely Telnet, whileFIG. 5illustrates another probe that abstracts the Telnet probe at higher level for easily using it to check whether a database server is running. When a composition of probes includes a control flow logic as well, the higher level probe is referred to as a probe plan. One such probe plan is illustrated inFIG. 6. The probe plans may be generated in that manner in the present disclosure to create repetitive problem determination processes.

InFIG. 1, the operational architecture for probes may comprise probe descriptors140, probing134, probe platform118, CMDB dependencies132, PD plans136, Query130. The community138can contribute with various low level services, simple probes, composite probes or probing plans to a shared repository called PD Plans136. The framework of the present disclosure also may enable experts who develop various PD plans in their labs, to contribute and share the PD plans for reuse in any data center or remote management services environment (e.g., IBM Remote Infrastructure Management Services (RIMS)) without requiring any change to the PD plan template. The Probe Platform118may provide the run-time services for executing probe templates in a given context. In the Probe Platform118, the PD plan136gets customized with the CMDB information.

During the workflow execution, when the platform118needs to execute a probe expressed by a generic descriptor (e.g., probe template), the platform118instantiates the values of the execution host and the probe parameters by querying (as shown at130) the CMDB132related to that particular IT environment104(e.g., specified in the probe descriptor). Based on the probe results134received back from the IT environment104, the probing plan execution platform118filters out the anomalous replies from the flawless ones and thus better localizes the failing resource(s). The data126related to the targeted failing resource(s) is sent to the problem classifier150for further analysis.

A database, a file or a collection of file132(e.g., CMDB) contains the information about the environment being managed. This information includes, but is not limited to, details of managed entities and structural relationships among those. Another database, a file or a collection of file136may store the probing algorithms and/or plans. The probing plans range from general purpose plans to services for specific plans. In one embodiment, the plans are environment independent.

The probe platform118gathers data by querying (130) the probe descriptor (PD) plan136and CMDB132. The probe platform118uses the PD plan data and data from CMDB132to instantiates its probes.

The probing algorithms and/or plans may be developed and/or updated by a community of users. For instance, the community138may develop and consume various probing plans using authoring tooling shown at140. The community138may be users or experts that utilize various systems and/or IT environment. The community shares in this way the experience gained across customer environments and problem resolutions, and populates at140the probing algorithms and/or plans repository136.

As discussed above, authoring toolings or tools140may be used for the creation of probing plans. The tooling140may provide the environment to express the flow of a probing plan with its specific steps. These steps include but are not limited to, (i) invoking another probing plan, (ii) gathering environment configuration data from CMDB like repositories for just-in-time customization of the probing plan, as well as of related tools and scripts, (iii) data gathering from these tools and running scripts. The semantics of how to launch a probe and how to interpret its results are specified by an expert. This semantic description is referred to as a probe descriptor. Experts associate a probe descriptor with a particular resource or relationship type defined in CMDB schema. Few items in the probe descriptor are concrete, e.g., what script to execute and what parameters it has. Further, few of the description items are environment specific and hence, cannot be specified concretely in the probe descriptor. These items are specified in terms of graph paths on the CMDB object model starting from the associated model element e.g., how to derive a host machine, where to execute the script specified in the probe descriptor, how to compute the parameter values, etc.

Shown at134is an example of the process of probing the managed environment by Probing plan execution platform118. A probe descriptor added by the community or expert via the tool at140is interpreted by the probing platform118and probing results are produced. Thus, the community may provide instructions by way of probe descriptor to query the environment for certain data. Under specific monitoring event types, predefined probes are applied to the environment to gather more detailed data. A particular example of probe collection is the one leading to the end-to-end transaction response time decomposition into the intervals spent at each resource involved in the transaction. This decomposition can be used to pinpoint the resource that, due to failure, has an increased response time compared to its normal behavior. Other mechanisms may be used in conjunction with the embodiments of the present invention. Based on the probe results received back from the IT environment, Probing plan execution platform18performs farther analysis.

A classifier module122receives the filtered or focused data150and automatically generates the root cause label142. The classifier module122may be populated with patterns relevant to the problem at hand and a label is sent to Problem Determination module128based on the current problem monitoring data150. The classifier module122, for example, categorize any problem a user experiences by recognizing the problem specificity leveraging all available data such as, but not limited to, performance data, resources consumption data, logs data. As shown at146, historical labeled monitoring data120is used to learn the patterns specific for an existing problem taxonomy; then problems may be recognized when given a new set of monitoring and log data based on the patterns previously learned.

The Problem Determination processes and tools128are used to fix the customer's incident. It may take the form of a self-assist tour (e.g., IBM Support Assistant (ISA)) or of a cycle through the technical support process. The root cause label144is manually given or attached to a problem by the individual who solved and closed the problem ticket opened for the customer's issue. The root cause label144may be stored in the labeled data repository120. This label may be different from the type of problem initially inferred based on the problem symptoms in the Alarms110,112and114.

A database, a file or a collection of files120stores the monitoring data together with the corresponding label, i.e., the type of problem during which that data was generated by the monitored systems. In one embodiment, technical personnel or an operator or the like may designate a “problem label” to the monitored data manually. A learning module or process146learns from the labeled data120the patterns stored in problem data patterns-classifier122. Any known or will be known techniques or algorithms for detecting patterns and classifying data may be used in conjunction with the embodiments of the present invention.

The following is a pseudo algorithm of the classifier122:

Input M classifiers {fl(x), ..., fM(x)}, a test instance xfor each classifier fm(x) (m = 1...M)if fm(x) classifies x to be negative (0)assign ym = 0else if fm(x) classifies x to be positive (1)if yparent(m) = 1 then assign ym = 1 else assign ym = 0end forOutput predicted class label y for x

The resolution148to the problem at hand, either found by the user or provided by the technical support service based on the data from the probe platform118and problem label at142may be also provided.

The integration of the technologies presented inFIG. 1, e.g., Problem Classification, Probes, ISA, on top of the monitoring platform, may lead to cost savings and PDR duration decrease, for example, by:Saving labor in building and maintaining environment specific scripts/probes;Sharing the new PD plans across environments, developed out of experiences from one account;Providing the abstraction of probe knowledge for other people to compose higher level probes; andEmpowering the service desk operators to execute relevant PD plans to gather better failure targeted monitoring data. The monitoring DATA is filtered based on the PD plans results into DATA′ for ISA for further analysis before forwarding the problem tickets to technical support personnel.

FIG. 2is a flow diagram illustrating a method of the present disclosure in one embodiment. The steps described below need not be performed in the sequential order as described below. Rather some of the steps may occur asynchronously or simultaneously with other steps. At202, monitor and data collector module monitors and collects data from the computer system or IT environment. The data may include but is not limited to data such as performance data, resources consumption data, and log data. At204, the monitor and data collector module may raise an alarm based on anomaly in the monitored data. In addition, at206, a user may raise an alarm, for example, if the user using the computer system or IT environment detects a problem or error. At208, the monitor and data collector module sends the alarm and associated data that caused the alarm to a probe platform. User raised alarm and associated data may also be sent to the probe platform For instance, a customer may call or send an email to the technical support service, which in turn may collect the necessary data and send the alarm and associated data that caused the alarm to a probe platform.

At210, the probe platform automatically further probes the computer system or IT environment for additional information related to the problem raised as an alarm. For instance, a PD plan is used and CMDB is queried to create a customized probe for the client.

At212, based on the probed information or results of step210, the probe platform filters the data received from the monitor and data collector module at step208to focus the data to the problem. For instance, in case of problems in complex IT environment with multi-tier system dependencies, the same issue may cause various failures, and hence generate several notifications, at different levels of the multi-tier system. In such cases, the user would greatly benefit of embedded mechanisms to filter the monitoring data related to the failing resource, from all the generated data in the system. An example of such a multi-tier environment is an e-business system supported by an infrastructure that may have the following subsystems connected by local and wide area networks: web based presentation services, access services, application business logic, messaging services, database services and storage subsystems. A complex probe would generate testing probes to see what works and what does not, thus informing on what to focus the monitoring data collection.

At214, the filtered data is sent to labeled data repository and problem data classifier. The problem data classifier matches the filtered data against the labeled patterns generated by the classification process in order to label the received monitoring data. At216, the problem data classifier automatically determines the root cause label associated with the problem that caused the alarm to be raised at step202.

At218, a problem determination module may also manually determine the root cause label by using the filtered data from the probe platform. For instance, a technical support service many manually find the root cause and/or ISA may guide the problem determination process by suggesting possible tasks to perform for detecting the root cause. The determined root cause label may be stored in the labeled data repository for future references.

At220, the solution or fix to the problem is sent to the user. For instance, once the problem is identified, known solutions may be sent to the user.

Referring now toFIG. 3, the systems and methodologies of the present disclosure may be carried out or executed in a computer system that includes a processing unit302, which houses one or more processors and/or cores, memory and other systems components (not shown expressly in the drawing) that implement a computer processing system, or computer that may execute a computer program product. The computer program product may comprise media, for example a hard disk, a compact storage medium such as a compact disc, or other storage devices, which may be read by the processing unit302by any techniques known or will be known to the skilled artisan for providing the computer program product to the processing system for execution.

The computer processing system that carries out the system and method of the present disclosure may also include a display device such as a monitor or display screen304for presenting output displays and providing a display through which the user may input data and interact with the processing system, for instance, in cooperation with input devices such as the keyboard306and mouse device308or pointing device. The computer processing system may be also connected or coupled to one or more peripheral devices such as the printer310, scanner (not shown), speaker, and any other devices, directly or via remote connections. The computer processing system may be connected or coupled to one or more other processing systems such as a server310, other remote computer processing system314, network storage devices312, via any one or more of a local Ethernet, WAN connection, Internet, etc. or via any other networking methodologies that connect different computing systems and allow them to communicate with one another. The various functionalities and modules of the systems and methods of the present disclosure may be implemented or carried out distributedly on different processing systems (e.g.,302,314,318), or on any single platform, for instance, accessing data stored locally or distributedly on the network.