Incremental problem determination and resolution in cloud environments

Installation files are annotated, which annotations may trigger system snapshots to be taken at a plurality of points during the execution of the installation files and/or collected. During a test run, the generated snapshots are examined incrementally to determine whether the installation is success or failure at that point. Checkpoint snapshots are stored, and those indicating failure are recorded with description of the error and/or remediation that suggest how the errors may be resolved or fixed. During a production run, the annotated installation files may be executed and the checkpoint snapshots generated during the production run may be compared with those stored of the test run to incrementally identify and resolve potential problems in the production run.

FIELD

The present application generally relates to computer systems, and more particularly to incremental problem determination and resolution in cloud environments.

BACKGROUND

A multitude of studies on the Total Cost of Operation (TCO) show that almost half of TCO, which in turn is five to ten times the purchase price of the system hardware and software, is spent in resolving problems or preparing for imminent problems in the system. See, for example, David A. Wheeler, “Why Open Source Software/Free Software (OSS/FS, FLOSS, or FOSS)? Look at the Numbers!”, available at http://www.dwheeler.com/oss_fs_why. html#tco, Revised as of Apr. 12, 2007; and Gillen A., Kusnetzky, McLaron S., The role of Linux in reducing cost of enterprise computing, IDC white paper, January 2002. Hence, the cost of problem determination and resolution (PDR) represents a substantial part of operational costs.

Making PDR cost effective, for example, through standardization and asset reuse has not worked in traditional information technology (IT) environments. See, for example, “WebSphere Application Server V6 Problem Determination for Distributed Platforms”, SG24-6798-00, Redbook, 20 Nov. 2005; and “DB2 Warehouse Management: High Availability and Problem Determination Guide”, SG24-6544-00, Redbook, 22 Mar. 2002. The IT resources being dedicated to a particular customer and their applications, leads to a diversity of configuration among IT environments and applications that makes it difficult to programmatically reuse scripts, workflows, lessons learned from one environment to another.

Existing art in the area of problem determination and resolution provide methodology restricted to particular products, such as in “WebSphere Application Server V6 Problem Determination for Distributed Platforms”, SG24-6798-00, Redbook, 20 Nov. 2005; and “DB2 Warehouse Management: High Availability and Problem Determination Guide”, SG24-6544-00, Redbook, 22 Mar. 2002, which provide problem troubleshooting guidance from the developer perspective for WebSphere™ and DB2™, respectively. These guides, although very informative, address only potential problems that have been identified in the product pre-production phase and have been categorized in error codes integrated in the product. They do not consider the historical troubleshooting knowledge related to fixing uncategorized failures in production environment at a customer's site.

Oren Laadan, Ricardo A. Baratto, Dan B. Phung, Shaya Potter, and Jason Nieh, DejaView: A Personal Virtual Computer Recorder provides a virtual computer recorder that captures the user's computing experience, which a user can play back, search, and browse. The tool records the visual output, the corresponding checkpoint and the file system state, and allows the user to annotate particular screenshots and system snapshots for future search. The system checkpoints are related to visual changes (e.g., no checkpoint is taken if the screen does not change), rather than to system changes.

In Helen J. Wang, John C. Platt, Yu Chen, Ruyun Zhang, and Yi-Min Wang, “Peer Pressure for Automatic Troubleshooting,” Association for Computing Machinery, Inc. June 2004, PeerPressure troubleshooting system uses statistics from a large sample set of machine configuration snapshots to identify a healthy machine configuration to compare a failing one and evaluate its performance. It leverages only the Windows Registry configuration data. The success rate may be reduced due to heterogeneity issues and false positives on healthy snapshots. The present disclosure, on the other hand, uses for system snapshots or the like, for comparing system checkpoints.

These existing techniques suffer from one or more of the following shortcomings: (i) address the detection of problems for a particular application or product, they are not applicable to the case when the application or product is a part of a complex distributed system; (ii) do not provide problem resolution, they do not attempt to go beyond problem determination; (iii) focus on runtime problem determination only while the error may be an installation error; (iv) look for changes in a limited set of data, not in the whole checkpoint snapshot.

BRIEF SUMMARY

A method and system for incremental problem determination and resolution in cloud environment are provided. The method, in one aspect, may include annotating checkpoints in one or more installation files, the annotating checkpoints including inserting triggers to have one or more system snapshots taken and/or collected at one or more points during the execution of the installation files on the system. The method may further include executing the one or more installation files as a test run and storing a plurality of checkpoint snapshots generated during the test run. The method may also include storing error description associated with one or more of the checkpoint snapshots if it is determined that the test run failed at one or more points corresponding to the one or more of the checkpoint snapshots.

In another aspect, the method may further include executing the one or more installation files as production run and comparing a checkpoint snapshot of the production run with the plurality of checkpoint snapshots generated during the test run. The method may also include, in response to determining that the checkpoint snapshot of the production run matches with one or more of the plurality of checkpoint snapshots generated during the test run, and the matching one or more of the plurality of checkpoint snapshots generated during the test run includes the stored error description, notifying with the stored error description.

A system for incremental problem determination and resolution in cloud environment, in one aspect, may include a processor and a storage device. The processor may be operable to execute one or more installation files as a test run, the one or more installation files having checkpoint annotations that trigger the processor to take and/or collect one or more system snapshots at one or more points during the execution of the installation files on the system. The processor further may be operable to store a plurality of checkpoint snapshots generated during the test run, and store error description associated with one or more of the checkpoint snapshots if it is determined that the test run failed at one or more points corresponding to the one or more of the checkpoint snapshots.

In another aspect, the system may include a processor operable to execute the one or more installation files as production run and compare a checkpoint snapshot of the production run with the plurality of checkpoint snapshots generated during the test run. In response to determining that the checkpoint snapshot of the production run matches with one or more of the plurality of checkpoint snapshots generated during the test run, and that the matching one or more of the plurality of checkpoint snapshots generated during the test run includes the stored error description, the processor may be further operable to present notification with the stored error description.

DETAILED DESCRIPTION

The emergence of IT Cloud environments brings with it a shift from the traditional heterogeneous environment towards more homogenous ones through resource virtualization, dynamic allocation, image technology. The IT Cloud model enables opportunities for making the PDR efficient and cost effective though new processes and methodologies. The present disclosure proposes an incremental problem determination and resolution (PDR) process that covers the operations for detecting anomalies during product installation in a virtual system and providing assistance to automatically recover from the cause of the problem into a stable state. The processes and algorithms of the present disclosure may include one or more of the following capabilities: (1) to annotate an installation bundle with synchronization points that trigger virtual system checkpoints to be taken; (2) create repository of virtual system checkpoints labeled “success” or “failure” by the technical support personnel, and in case of failure suggest repair action; (3) compare the system checkpoints taken during a product installation at the points described in (1), to the correspondent checkpoints in the repository of (2); (4) update the repository of virtual system checkpoints with new failure cases or repair actions; (5) compare the system checkpoints taken during product post-installation tests to the correspondent checkpoints in the repository of (2).

The aspects of the present disclosure may provide (i) control over the IT resources that leads to a systematic approach to PDR that was not possible prior to virtual image technology; (ii) method to select the synchronization points; (iii) method and system to store and label virtual system checkpoints for PDR.

The benefits involved by the usage of the aspects of the present disclosure may be realized by the decrease of the cost of the current incident and problem management methodology, through systematizing existent data, knowledge, and expertise for reusability, as well as the avoidance of cost associated with problem determination by allowing for proactive problem resolution (fix before break) through knowledge based early notification (e.g., avoiding waiting for the end of the installation to realize it failed).

The incremental PDR system in one embodiment proposed by the present disclosure is shown inFIG. 1. Items, operations, and procedures involved in the system are described below.

Item A (102) represents the original installation package (e.g., exe, rpm, zip, scripts) of a software product such as an operating system, a middleware product, an application. Installation packages are also referred to as deployment packages and/or installation bundle in this disclosure. Installation packages may include multiple components that go through multi-phase set up process. For example, an installation package for a data center automation framework may include middleware (e.g., web server, application server, database management server, and directory server), the core services (a common application framework), the core components of the automation software, and the web frontend components for the application. Through the Procedure1, Checkpoint Annotation (114), the development and test teams or others transform Item A (102) into Item B (104) by adding annotations to the installation process (e.g., installation programs or software), which will trigger at the install time the hypervisor or another system operating system or the like to take system checkpoint snapshots.

Checkpointing refers to taking snapshots of the current computer system or current application state. In computer systems, a snapshot is a copy of the state of a system at a particular point, for example, system configuration of information of a program or machine at a particular point. Checkpoint snapshot in the present disclosure refers to a snapshot taken at a point in time. A checkpoint may be also considered as being a copy of the computer's memory that is saved periodically on disk or another storage device along with the current register settings. In the event of a failure, the last saved checkpoint may be restored as a recovery point.

Annotations refer to additional codes in the installation script or the like. Such annotations tell the installation procedure or process or program to stop and wait for the snapshot to be taken. Annotating, for instance, may include editing the one or more installation files such as executables, scripts and/or others, to include system calls or application program interface calls that trigger snapshots to be taken of the system when the installation files are executed or run, and for the installation procedure (e.g., process, program and/or software) to stop and wait for the snapshot to be taken. In another aspect, for instance, for installations on systems that already have automatic and periodic checkpointing mechanism configured and enabled, annotating may include editing the installation files to include commands or the like for the installation procedure (e.g., process, program and/or software) to stop and wait for the periodic system snapshot to be taken. Annotating may be manually performed. For example, a programmer or the like person may add the annotating code in the installation script or the like. In another aspect, annotating may be performed automatically. For instance, a tool or a program may read the installation script and add code into places in the installation script. An example would be automatically detecting “delete temp files” execution in the installation script and adding automatically “take snapshot” after that execution line.

Annotations may be inserted at different points in the installation. Annotations may be inserted or placed at defined deployment (installation) phases, for example, so that the state of a given deployment phase may be identified before proceeding to the next phase or step in the installation. Annotations also may be inserted after installing a component of a multiple component product or package.

Item B (104) is the processed installation package that includes annotations added in the Procedure1, Checkpoint Annotation (114). Additionally to the installation package annotation, the development and test teams (or another) may generate into Item B (104), via Procedure2, Functionality Test Generation (116), a set of product functionality tests checkpoint annotated such that after the product installation, the user can validate the functionality of the product beyond the successful installation. That is, checkpoint annotations may be added into the product functionality test scripts or code or the like. For instance, after the installation of software or application, its functionality may be validated by running one or more specific tests. The tests may be running functions provided by the software or application. Those tests may be annotated so that snapshots are generated as in the installation script. Similar to the above checkpoints of the annotated installation package, the checkpoints of the annotated functionality tests will trigger the hypervisor or the like to take system checkpoint snapshots and/or wait for the snapshot to be taken.

Item C (106) represents a cloud environment with virtual machines (Items D (108), E (110)) residing on hypervisors that expose application program interfaces (APIs) allowing system checkpoint snapshots to be triggered programmatically. On such an environment the development and test teams or the like install (test) during the Procedure3, Test (118), the checkpoint annotated products (e.g., operating system (OS), applications), as a test run. The systematically generated system checkpoint snapshots are stored in the repository Item F (112).

Each snapshot may be labeled “success” or “failure” in Procedure4, Generate Labeled Checkpoint Snapshots (120), depending on the installation (test) status at that point, for instance, as decided by the technical personnel performing the installation (test). If a failure occurs, the technical personnel or the like may attach the corresponding error description and suggested remediation to the snapshot. Potential remediations may include, but are not limited to, restart installation, load the latest checkpoint, command scripts, and others.

In Item C (106), the customer or the like, who may be the product consumer, installs the annotated products tested above in the Procedure5, Production Installation (122). This installation is also referred to herein as production run. Item C (106) may be the same cloud as the one the development and test teams used for their tests or can be a different one that has a copy of or link to the checkpoint snapshot repository Item F (112). As during the tests (Procedure3at118), the checkpoint annotated products installed by the customer (Procedure5at122) trigger checkpoint snapshots to be taken.

Each customer snapshot may be compared in Procedure6, Check Snapshot (124), to the corresponding test snapshot stored in the repository Item F (112). Existing tools such as CVS found in “http://www.nongnu.org/cvs/” and GIT found in “http://git-scm.com/” may be used for this comparison functionality.

In Procedure6′, Validate/Correct Installation & Test (126), in case of matching a “success” labeled snapshot, the installation (test) continues. Otherwise, the failure description and remediation are presented to the customer or another to support his or her next actions. For example, the customer or another is notified of the failure status in the stored snapshot that matches the current snapshot of the production run.

Procedures6and6′, in one aspect, find stored file system differences to determine where problems have occurred in a given process.

In Procedure7, Update Repository (128), failing installation (test) checkpoint snapshots, which for example caused expert level technical personnel to fix the problem are added by the technical personnel or another to the repository Item F (112) together with the corresponding error description and suggested remediation. Due to components involving complex interactions, some of which may only appear under specific circumstances, as well as to customer deployment environments of a large variety, not all aspects that may lead to failures may be detected in the test labs. Thus, for instance, the snapshots generated during the test run may be augmented and/or updated with additional information identified during the production run.

Item D (108), E (110) are virtual machines in a cloud. They contain an operating system and one or more applications. Item F (112) is a database, a collection of files, or a file resource that stores the labeled system checkpoint snapshots and any related attachment such as, but not limited to, error description and suggested remediation.

A cloud represents infrastructure that can efficiently provide IT services by taking advantage of virtualization and distributed computing. The cloud environment106may be private computing cloud that is contained within the enterprise, but not limited to only a private cloud. The cloud environment106may include enterprise level applications that provide the infrastructure enabling cloud computing services built by, for example, integrating multiple interrelated software component. The method and system described herein may be utilized when deploying a cloud computing management infrastructure.

The cloud environment106may include components of a computer system that may include any computing node that is able to load and execute programmatic code, for instance, running the operating system and server application suite. The system logic may be embodied as computer executable code that is loaded from a remote source (e.g., from a network file system), local permanent optical (CD-ROM), magnetic storage (such as disk), or storage into memory for execution by a central processing unit, a processor and/or cores. A network interface may be used for communicating to other network accessible services. Such an interface may include, but is not limited to a hardwired device, e.g., Ethernet over coax cable, wireless IP, and telephone to IP (VoIP), or others. The memory may include computer readable instructions, data structures, program modules and application interfaces providing logic or functional components.

It should be understood, however, that the system and method of the present disclosure is not limited to only that configuration. Thus, the components may be distributed over a network on different platforms in a distributed environment, and/or configured in a cloud computing environment. The system may include multiple processors and/or cores.

FIG. 2illustrates control flow or logic in one embodiment of the present disclosure to incrementally determine and resolve problem in cloud environment. At202, checkpoint annotations are added to installation packages. As described above, annotations may be additional code in the installation script or like that tells the installation procedure or process trigger checkpointing and/or to stop and wait for the snapshot to be taken. Thus, installation packages are annotated for specific checkpoints at defined deployment phases so as to trigger a clone to be taken and used to get the deployment health status.

At204, test generation functionality may be also added to the installation packages. That is, the test codes or programs that validate the installed software or application may be annotated similarly, for instance, by adding additional code that tells the test code to stop and wait for a snapshot to be taken at one or more points in the test. In addition to the deployment package annotations for image capturing, if functionality tests are available as part of the package, for example, a similar approach can be applied to validating these tests.

The installation packages may be annotated such that it triggers the system state to be captured at precise intermediary phases of the deployment and functionality tests, if any. Particular intermediary steps of the deployments may be relevant in terms of overall system status compared to others. For instance, the beginning and end of the involved components deployment may be checked for prerequisites and installation success, respectively.

In one aspect, the system state may be captured at file system level, for instance, in terms of memory used, time elapsed and practicability. The file system images may be used to characterize the condition of a particular deployment intermediary step.

At206, the annotated installation packages or products are installed or run on the cloud as a test. For example, operating system (OS) and the applications may be installed in test environments with a goal of capturing and fixing errors as they occur, including deployment errors. The mechanism of the present disclosure triggers specific data collection at explicit deployment phases and/or installation points. For example, during the installation, the added annotations trigger the system to automatically generate system checkpoint snapshots and/or stop and wait until a snapshot is taken. The checkpoint snapshots are stored in a repository.

At208, labels are generated for the snapshots. The labels indicate the status of the installation during the test run, for example “success” or “failure” at the point the snapshot is taken. The snapshot is labeled “success” if installation at that point succeeded. The snapshot is labeled “failure” if the installation at that point failed, for example, an error message is received that the installation failed. If a failure is detected at this point in test installation, additional information such as error description, suggested recovery or remediation or correction mechanism, may be attached to or associated with the corresponding snapshots and stored together in the repository.

In one aspect, there may be two classes of failure: deterministic and non-deterministic. Deterministic failures are those that are caused by a setting, a particular configuration of inputs to the software and items in the environment, including those in the operating system as well as in the network. The setting associated with a deterministic failure is its failure setting. Running an installation in a replica of a failure setting will reliably reproduce the associated failure. Deterministic failures may be caused by, e.g., inappropriate user input, missing operating system components, or software firewalls that block required network connections. Non-deterministic failures may or may not be reproducible. They may be triggered by performance problems that lead to problems if the software has race conditions, by unreliable hardware or device drivers, or by other environmental situations that lead to the execution of untested code. While software should be built and tested to handle such circumstances reliably, this is not always the case. Therefore, in production deployments, most of the time when failures occur, the technical support may be involved for troubleshooting, either because the issues arise for the first time, or because there is no prior knowledge captured to allow for an effective remediation reuse. An inventory of discovered potential issues that have occurred in test environments and have known remediation plans would be beneficial by itself for production deployments. Adding to this inventory the failures fixed during production deployments, would be of help also to the deployment administrator who is now able to leverage past knowledge when facing known failures.

At210, a customer or the like installs the annotated packages in a production system for use. For instance, once a repository has been populated with sufficient labeled snapshots, the production deployments can start taking benefit of an automated deployment status validation and failure assistance. As in the test run at206, the annotations trigger the system to generate snapshots at points of installation.

At212, the snapshots of the production installation at defined phases or points of installation are compared to the snapshots of the test run that were generated at the corresponding defined phases or points of the test run and previously stored in the repository at each checkpoint. Relevant patterns in the snapshots may be identified. For instance, the names of the files, their sizes, content and permissions, last modification time, and others may be compared. If the matching snapshots are found, a notification may be sent to the deployment application. Based on that notification, the deployment may continue with the next step in case of success or may proceed with presenting the user the remediation suggestions in case of failure.

At214, for matching snapshots, if the test version indicates a failure status, the associated error description and/or remediation steps are suggested to the customer or the like performing the installation.

At216, the repository is updated. For instance, new failure type, remediation and snapshots identified under the production run are added to the repository. In another aspect, the new snapshot may be used to update the corresponding snapshot in the repository. Snapshots that do not match any of the existing patterns may be troubleshooted, for example, by the technical personnel, labeled and added to the repository.

At218, functionality validation test may be performed. For instance, after the installation of the software or application is complete, tests may be run on the software or application to validate the workings of the software or application on the installed environment. During this functionality validation test, snapshots are generated as a result of the added annotations at step204. If errors occur during this functionality validation test, the snapshots then may be compared with the previously stored snapshots, for instance, of a working installation. The previously stored snapshot of a working installation may be used to determine the problems in the failed installation. If previously stored snapshots that also failed exist, the remediation or resolution associated with those previously stored snapshots may be used to correct the problems that caused the errors of this functionality validation test.

The above description referred to system snapshots as the information or data being captured and collected at each point or phase in the installation or deployment. It should be noted that any other data may be captured and collected. Variety of approaches may be used to collect this type of information. For example, one or more or combination of revision control software, backup system, versioning filesystems, hypervisor-directed shanpshots, and hypervisor-directed cloning may be utilized to capture and collect system image, configuration data, or the like at a point in installation step. Revision control systems (RCS), such as GIT™, CVS™, Subversion™, may provide snapshots in time of a tree of files along with direct comparisons between arbitrary snapshots. A distributed RCS, such as Git™, Bazaar™, or Mercurial™ supports a serverless mode, where all versions are recorded locally.

Versioning filesystems, susch as CopyFS™, NILFS™, and BTRFS™, and some automated snapshotting filesystems such as Apple™ Time Machine™ or Microsoft™ Shadow COpy™ provide previous versions of files (or the entire filesystem) at points in time in the past. In hypervisor-level snapshotting, a copy-on-write is created from the disk image when the snapshot is taken. All future writes by the virtual machine (VM) to its disk are captured there, leaving the previous disk image untouched. The state of the filesystem may be captured with the VM shutdown. Hypervisors may also create a clone of a virtual machine, which contains an exact copy of all the data on the virtual machine data.

Mirage™ provides a set of APIs that enable sophisticated operations directly on dormant images within an image repository, without having to first retrieve the image or instantiate and run it. It builds on semantic information about the disk image. Mirage searchability enables disk comparisons at file-level granularity.

In the virtual technology, it may be possible to have an incremental live clone capturing to back up a remote system that allows an efficient clone generation and storage.

The mechanism of the present disclosure may provide an enhanced deployment methodology and may provide for, among other, (i) early detection of anomalies during the installation of complex software solutions, such as the cloud computing services infrastructure; (ii) capture and storage of problems' distinctiveness together with label and potential remediation for future retrieval; (iii) problem identification and assistance to automatically recover from a known problem into a stable state; (iv) problem repository update. Intermediary health status of, for example, lengthy deployments like the ones of the cloud computing services' underlining infrastructure may be identified and resolved, thus for example allowing for control of the deployment validation and troubleshooting in virtual environments.

Referring now toFIG. 3, the systems and methodologies of the present disclosure may be carried out or executed in a computer system that includes a processing unit2, which houses one or more processors and/or cores, memory and other systems components (not shown expressly in the drawing) that implement a computer processing system, or computer that may execute a computer program product. The computer program product may comprise media, for example a hard disk, a compact storage medium such as a compact disc, or other storage devices, which may be read by the processing unit2by any techniques known or will be known to the skilled artisan for providing the computer program product to the processing system for execution.

The computer processing system that carries out the system and method of the present disclosure may also include a display device such as a monitor or display screen4for presenting output displays and providing a display through which the user may input data and interact with the processing system, for instance, in cooperation with input devices such as the keyboard6and mouse device8or pointing device. The computer processing system may be also connected or coupled to one or more peripheral devices such as the printer10, scanner (not shown), speaker, and any other devices, directly or via remote connections. The computer processing system may be connected or coupled to one or more other processing systems such as a server10, other remote computer processing system14, network storage devices12, via any one or more of a local Ethernet, WAN connection, Internet, etc. or via any other networking methodologies that connect different computing systems and allow them to communicate with one another. The various functionalities and modules of the systems and methods of the present disclosure may be implemented or carried out distributedly on different processing systems (e.g., 2, 14, 16), or on any single platform, for instance, accessing data stored locally or distributedly on the network.