Recovery execution system using programmatic generation of actionable workflows

Programmatic generation of an actionable recovery workflow from data stored inside a Configuration Management Database which may be primarily populated through automated discovery. The programmatic workflow can be sent to an orchestration engine for execution, leveraging underlying automation components.

BACKGROUND

Background Information

Information technology (IT) service providers are quite often challenged to find ways to effectively manage and maintain very large-scale infrastructures. An example enterprise environment may have many thousands of devices and hundreds of installed software applications to support. The typical enterprise also uses many different types of central data processors, networking devices, operating systems, storage services, data backup solutions, cloud services, and other resources.

There is certainly a push to migrate to automated solutions for management of such a diverse environment. In some implementations point-to-point integration can be accomplished with Run Book Automation (RBA) solutions. But even coordination of automated solutions can become quite complex as that often involves integration of multiple products and services including provisioning engines, functional level products, and security concerns.

Disaster Recovery (DR) operations are for the most part a primarily manual operation. For highly virtualized environments and cloud-based applications, there are some available tools that leverage automation. But a large portion of enterprise IT is still not virtualized or placed in the cloud. For such environments, the only option is to manually codify recovery processes for each and every application and each and every data center scenario. That is typically a very labor intensive and time-consuming process.

Some implementations do leverage “task level” automation tools, freeing human operators to focus on coding the overall “orchestration”. One such tool is Hewlett-Packard's (HP's) Operations Orchestration (HPOO), which permits automation of enterprise-scale IT processes and solutions. But even a solution based on HPOO still requires a human programmer to write a set of procedures that determine the appropriate tasks and the correct sequence in which to execute them.

SUMMARY

What is needed is a way to programmatically automate the creation of an actionable workflow to support automated task execution, such as might be used in Disaster Recovery (DR). The approach should automatically generate a master workflow containing subflows with automated decisions to further activate available automation components. The master workflow may be generated from data available in a Configuration Management DataBase (CMDB) and stored as a markup language (XML) format file. The CMDB can be initially populated through automated discovery of a production data center's Configurable Items (CI's). The automated discovery of CI's may have a scope, such as the particular application(s) for which recovery is desired. The CMDB data and resulting workflow file can then be sent to an orchestration engine to execute an actionable workflow, leveraging the underlying automation components. This approach also expedites the substitution of different automation components as they become available.

One distinction with the approach from a high-level perspective is that the orchestration of a series of recovery tasks may now be completely automated.

The master workflow can be specified as a set of instructions to an orchestration engine. In one example implementation, the master workflow is dynamically and programmatically created by extracting information from the CMBD and storing it as a specified Recovery Markup Language (RML) file. The RML file may contain specialized tags and may be formatted on, for example, an extensible markup language (XML) file. The CMDB contains information about the configuration of each Configurable Item (CI) in the IT infrastructure. The CMDB also maintains an Application Map that defines not only the in-scope servers, storage devices, and network devices in the production environment but also the relationship of each given application to each of these in-scope Configurable Items.

A Recovery Execution System (RES) then leverages these elements. The RES includes several components including an RML generator, one or more RML models, an orchestration engine, automation components, and access to a list of assets available in the recovery site. The RML generator may for example be a Java program that reads the CMDB to obtain information about the configurable items related to the application scope. The RML model generator then automatically creates the XML-based schema with specialized tags and attributes to specify the high-level master flow. The orchestration engine then reads the RML file to execute the master flow, which in turn instantiates subflows to execute corresponding automation components.

The RML file specifies the workflow as a sequence of tasks or “phases” needed at a high-level rather than being involved with exactly specifying how to implement each task. As one example, the RML file may specify “build an Operating System” as one phase in the master workflow rather than specifying an exact sequence of steps for how to actually build the particular OS. As a result, the master workflow is not tightly coupled to any one particular recovery technology, and the RML master flow can remain the same as different recovery technologies become available. This approach also does not require the recovery solution to virtualize the physical elements of the data center, or to change recovery strategies as different technologies become available.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

FIG. 1is a high-level diagram of an environment where a Recovery Execution System (RES) may be used to programmatically create and maintain a recovery procedure. In this end use of the RES, the goal is to recover one or more Configurable Items (CI's) existing in a production data center101, recovering them to a hot site120. The implementation may be used as a managed recovery program for Configurable Items (CI's) that include data processors, servers, storage subsystems, firewalls, security appliances, networking devices, and many other types of information technology (IT) infrastructure elements.

The illustrated elements include the production data center101, production configuration information102, a discovery tool103, a production environment Configuration Management DataBase (CMDB)104, a Disaster Recovery CMDB105, a portal106, workflow and project management tools110, a Recovery Execution System (RES)111including a Recovery Markup Language (RML) generator209, recovery workflow orchestration210, recovery automation component subsystems122, and a hot site120. In addition, an event scheduling subsystem107accessable as a Worldwide Web Scheduling Service (WWSS) may include a scheduler108and a hot site inventory database109.

A typical process to automate disaster recovery proceeds as follows. In a first step201administrative personnel or configuration engineers install application probes in the customer production data center101. The probes are used by the discovery tool103to discover installed applications and the dependencies those applications have on particular Configurable Items (CI's) in the data center101. The discovery tool103may be Hewlett-Packard (HP) Discovery and Dependency Mapping Advanced (DDMA), BMC Atrium Discovery Dependency Mapping (ADDM) or other tools that can automatically discover physical and virtual IT assets, applications, and the dependencies between them.

Configuration data for these Configurable Items (CI's) may be discovered by these probes (or in other ways) and stored in the Production CMDB104. The Production CMDB may, for example, be an HP Universal CMDB (uCMDB).

In a next step203, the discovered configuration information is also copied over a secure unidirectional connection (such as an HTTPX connection), and cloned to a Disaster Recovery (DR) CMDB105.

The cloned DR CMDB105may then be used to create, maintain and specify a recovery configuration without impacting the dependency of the operating production environment on the Production CMDB104. Subsequent exchanges between the two configuration databases104and105can keep the configuration information in DR CMDB105up to date.

In a next step204, when a customer wishes to schedule a test or declare a disaster, they access the portal106. Based on a presented customer identification (ID) credential, such as a login ID, a specific recovery configuration is retrieved from the DR configuration database105. The customer can then enter additional details, such as the scope of the test or disaster recovery. The scope may specify, for example, one or more applications to be recovered and a time at which to perform the recovery.

In a next step205, using the provided scope and a requested time (which may be a future date in the case of the test or immediate execution in the case of disaster recovery) a scheduling service107is contacted. The scheduling service107uses the recovery configuration to assess availability of matching resources in the hot site inventory109.

In a next step206, based on resource availability, the scheduling service reserves the needed resources for use at the scheduled time and provides a confirmation report to the customer via portal106.

In step207, the recovery event occurs at the scheduled time and the Recovery Execution System111is launched in step208.

Next in step209, the RES111extracts the recovery configuration information from the CMDB105for the specific scope. The RML generator209then automatically creates an XML document using a proprietary schema referred to in herein as the Recovery Markup Language (RML) file. The RML file contains a specification for a master workflow that contains a set of logical steps to recover the in-scope application(s). The RML generator209is discussed in greater detail below.

In a next step211, the recovery workflow orchestration engine210then translates the logical workflow as specified in the RML file into a set of physical set of workflow-enabled tasks. These workflows may specify using particular automation subsystems as “subflows” that automate tasks to be performed on the servers, network devices, storage devices and other configurable items on the hot site120.

In step211, the RES111then communicates with event scheduling system107to acquire access to the assets that were reserved for the event.

In state212, the RES111then invokes the various automation subsystems122to perform the workflow tasks on the reserved components. As seen, typical automation subsystems122may include SunGard Recovery Automation, HP Server Automation, HP Network Automation, HP Storage Automation, and other OEM automation components depending upon the configurable items with the scope of the recovery event.

FIG. 2is a more detailed view of components of the RES111. An RML process engine125reads the RML file and engages various subsystems to implement actions. A pluggable driver interface is provided to the various subsystem functions including backup and restore subsystem131, server provisioning systems132, network provisioning systems133, and storage provisioning systems134. In general, other types of subsystems135useful in provisioning or arranging configurable items may also be accessible to the RML process engine125.

The RML process engine125may for example be implemented as a Java program that can read the RML file and the contents of the DR CMDB105and then invoke corresponding systems of action140as indicated by workflows in the RML file. Input is also provided to the RML process engine125from the scheduler107to indicate which assets are currently at its disposal to instantiate a particular work flow at hot site120.

The RML process engine125may also invoke recovery automation functions to implement actions. In one example environment, these automation functions may include a Net backup141, Tivoli storage manager (TSM)142, HP Server Automation143, HP Network Automation144, HP Storage Essentials145, HP Database Middleware Automation (DMA)146, Commvault147, Specialized Recovery Automation (RA) services such as SunGard RA145and/or manual tasks to be performed by humans149.

FIG. 3depicts an example sequence of steps performed by the RES111to implement recovery of data center101at hot site120.

In a first step301, a user indicates application(s) to be restored via portal106. In a next step302, the RES111acquires asset information the DR CMDB105, scheduler107and inventory database109concerning the needed target hardware configuration.

In a next step303, the RES111processes the RML file in an order as specified in the RML file itself. Any error conditions may be handled the specified in the RML file.

More particularly, in a next step304, the RES111can then process the master workflow in many different phases, as was determined from the application dependencies.

Finally, in step305for each phase in the RML file, the RES communicates with the corresponding system of action to execute one or more subflows. For example, a given system of action140(or subflow) can execute action(s). When the subflow is complete, it can reply back to the RES111to execute the next phase in the master workflow.

Turning attention now toFIG. 4, a simple example of a master workflow and actionable subflows will be described in more detail. A Recover Applications Map401is retrieved together with the elements of a RML file that indicate a master workflow402. In this example shown here, the master workflow402includes two phases—an Operating System (OS) build phase411and an application install phase412.

The Application Map401, described in more detail below, includes data representing all configurable items that are connected to the application(s) within the recovery scope. This may include a list of servers, storage devices and network devices, and their associated configuration information.

OS build phase411invokes a sequence of steps including “detecting which operating system to build451” and “detecting available automation components452.”

Step451can detect which operating system needs to be built, for example, by reading the CMDB105and/or Application Map information. Automation component detection452then uses the OS information to determine the available automation components that can be used to build that operating system. The RES111thus specifies that an operating system needs to be built, but when it comes to the “hows” of actually building the OS, the available automation components are leveraged. In the present example, available subtasks for building an operating system include Recovery Automation (RA)453, HP Server Automation (SA)454, Altris455, and a manual build456.

In one example, assume that the CMDB indicates that the OS to build is an implementation of Red Hat Linux. Assume also that this is the first time that the RES111has been asked to build this particular instance of Red Hat Linux. Here the automated OS build options453,454,455are not available, and the OS must be built manually456. However at some later time an administrative user may determine how to build the particular Red Hat Linux instantiation using HP SA. An automated HP SA Red Hat install subflow can then be linked to the detect automation component452via input at portal106. On subsequent iterations, without altering the high-level master workflow402, this new automation component454for building Red Hat Linux is then available to be automatically invoked by the corresponding detect automation component.

Thus when a new automation component becomes available, another subflow can be added to the available options in the detect automation component452, and related aspect(s) of the master workflow need not be modified.

It is also possible that the subflows are specific to each type of automation component and also to each service provider or customer. For example, building Red Hat Linux for one customer's data center may involve a different automation component than for another customer.

Note that the master flow402can remain the same regardless of how the OS build phase411and application install phase412are actually implemented. Thus, in a more general sense, a recovery workflow tends to provision infrastructure, configure infrastructure, restore data to the configured infrastructure, and then start the application(s). Also, in a general sense, the subflows called by the master flow indicate how to carry out particular tasks running on individual task automation systems.

It is also now understood that one essential aspect herein is eliminating hand coding of Operations Orchestration (OO) by detecting what to do (build an OS, or install an application) and automation components452which in turn enable programmatic execution of available automation tasks.

FIG. 5is a more complicated example of a master workflow for restoring an entire data center. In this example, the master workflow402includes an OS build phase411and application install phase412but also includes a number of other phases such as a virtual machine build phase413, application restore from backup phase414, operating system restore phase415, network configuration phase416and storage configuration phase417. Each phase type in the master flow has a detection piece and an automation component piece, and the automation components links to the available subflows to carry out automated tasks. In one example, for “application data restore from backup414,” the detect automation components414-1may provide links to automation components including Commvault414-2, TSM414-3, NetBackup414-4or manual restore414-5. A network configuration phase416may have links to subflows that include performing network configuration through HPNA and Cisco configurator or manual configuration. Similar sets of subflows for storage configuration417OS restore415, and VM build413options are also illustrated.

FIG. 6is a rendition of an example application map600that maintains the relationship for one single application to a number of in-scope Configurable Items (CI's) as previously mentioned. In the particular instance shown, the application602is an Electronic Data Interchange (EDI) application named “edi-si-gis”. Six servers are within the recovery scope here, including four UNIX servers604(the hosts named wgisprd01, wgisprd02, winxprd02, and wodbprd04) and two Windows servers606(the hosts named wfldedi03and wfdedi01). The digit displayed next to each item type in the hierarchy indicates how many configurable items (CI's) are associated with that particular configurable item type. Graphical User Interface options convenient for displaying the application map600(such as + and − click boxes to expand or collapse the application map hierarchy), may be provided.

Also show as part of the application map600are additional resources used by the application602including Internet Protocol addresses608(of which there are fourteen (14) in use, interfaces610(of which there are thirteen (13) in use, and which may for example be interfaces to other systems and databases), CPU cores612(of which there are eighteen (18) in use) and filesystems614. InFIG. 6, the view of these resources is collapsed and thus the details are not visible; the details can be accessed via the expand click box (+) displayed next to each configurable item type.

Each of the items in the example application map600hierarchy ofFIG. 6are connected to the single EDI application602. It should be understood that, more generally, a particular recovery scope may include multiple applications, each having their own application map600.

FIGS. 7A, 7B, and 7Care excerpts from an example RML file700generated for the EDI application shown inFIG. 6. It is but one possible example of a generated XML-based schema with specialized tags and attributes to specify a high-level master flow with subflows as previously described above.

A first excerpt of the RML file700shown inFIG. 7Acontains server definition info that pertains to one of the Linux hosts (wgisprd02). This is identified by a first tag701and the following tags that specify that host's platform type, operating system type, and needed random access memory (RAM). Several classes of subflows are associated with each configurable item, including Provision702, Configuration704, Restoration706, and Cleanup708. The end of the server info definition for Linux host (wgisprd02) is indicated by another tag701. It should be understood thatFIG. 7Ais meant to be a non-limiting example and that there may be action types in each class of automation actions other than those illustrated.

Within each class are included one or more callable automation components (each identified by an <Action ID> tag) and one or more parameters (identified by <param name> tags). Within the <Provision> class for this host, a first Action ID718is an automation component for deploying a server, with parameters including EVENT_ID, SERVER_ASSET, RAID_LEVEL, HOSTNAME, BACKUP AGENT, OS_NAME, OS_SP, and OS_ARCH. A second Action ID720enables an automation component that uses HP SA for installing a Linux distribution at a particular MAC address.

Within the Configuration class of subflow, a third Action ID722is for enabling an automation component that disables a Linux firewall. Yet another available action is Action ID724that is a manual component that waits for user input until migration to a customer network to complete.

The example Restoration class may include an Action ID730that invokes an HP SA script to perform a Netbackup action and Action ID732that runs another HP SA script to restore Linux.

An example Cleanup class is not shown in detail but would include other Action IDs that might be necessary for orderly conclusion of subflows.

It should be understood that the RML associated with each configurable item type may have different Action ID types.FIG. 7Bis an example RML excerpt738for Provisioning, Configuration, and Restoration of AIX for the host named “wodbprd04” in the application map ofFIG. 6. As such, the automation components associated with each action type will be tailored for an AIX installation and thus differ from those shown for the Linux installation inFIG. 7A.

FIG. 7Cis an RML excerpt740for the host names “wfldedi01” that is running Windows 2003. The associated Action IDs are thus appropriate for this environment, including HP SA automation components which install Windows740, disable a Windows firewall742, and configure and partition a Windows disk744.