Method and system for deploying multiple distributed application stacks on a target machine

Method, system, and programs for automatically deploying a plurality of distributed application stacks on a target machine. A plurality of virtual environments isolated from each other are first created on the target machine through which a user has access to a plurality of clusters. A distributed application stack is deployed from the plurality of distributed application stacks in a respective virtual environment created on the target machine. The plurality of distributed application stacks are deployed in the plurality of clusters. Based on the deployed distributed application stack, each of the virtual environments is then associated with a corresponding cluster.

BACKGROUND

1. Technical Field

The present teaching relates to methods, systems, and programming for distributed computing. Particularly, the present teaching is directed to methods, systems, and programming for deploying a plurality of distributed application stacks on a target machine.

2. Discussion of Technical Background

Distributed computing is a field of computer science that studies distributed systems, which include multiple autonomous computers or parallel virtual machines that communicate through a computer network, such as a computer cluster having multiple nodes. The machines in a distributed system interact with each other in order to achieve a common goal. A computer program that runs in the distributed system is called a distributed application. In distributed computing, a problem is divided into many tasks, each of which is solved by one or more computers, such as the nodes of a computer cluster. Distributed systems and applications may be applied as various paradigms, including grid computing, utility computing, edge computing, and cloud computing by which users may access the server resources using a computer, netbook, tablet, smart phone, or other device through the Internet.

For instance, APACHE HADOOP is a software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of nodes and petabytes of data. Rather than rely on hardware to deliver high-availability, HADOOP is designed to detect and handle failures at the application layer, thereby delivering a highly-available service. HADOOP is deployed on a computer cluster in the form of a HADOOP stack, which includes a set of software artifacts (HADOOP components), such as HADOOP software, configuration files, libraries, links, source code, documentations, miscellaneous, etc. The deployment of HADOOP on a cluster of machines usually involves hardware installation, operating system installation, update, and configuration, JAVA installation and configuration, and HADOOP stack installation, configuration, and diagnostic.

When deploying a distributed application stack, such as HADOOP, server-side software artifacts of the distributed application stack need to be deployed on regular nodes of the cluster while certain client-side software artifacts need to be installed and configured on one or more special target machines (e.g., lightweight HADOOP clients) from which a user has access to the cluster and the deployed distributed application stack. The target machines include, for example, the users' client-side machines outside the cluster and some special nodes, such as a gateway node, of the cluster. Moreover, certain applications on the clusters, e.g., development and Quality Engineering (QE), may require deploying multiple (e.g., tens or hundreds of) distributed application stacks on the same physical target machine and isolating the deployed distributed application stacks from each other on the target machine. Therefore, there is a need to provide a solution for automated deployment of multiple independent distributed application stacks in isolated environments on a target machine, such that a user can access and interact with multiple clusters through the target machine.

SUMMARY

The present teaching relates to methods, systems, and programming for distributed computing. Particularly, the present teaching is directed to methods, systems, and programming for deploying a plurality of distributed application stacks on a target machine.

In one example, a method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for automatically deploying a plurality of distributed application stacks on a target machine. A plurality of virtual environments isolated from each other are first created on the target machine through which a user has access to a plurality of clusters. A distributed application stack is deployed from the plurality of distributed application stacks in a respective virtual environment created on the target machine. The plurality of distributed application stacks are deployed in the plurality of clusters. Based on the deployed distributed application stack, each of the virtual environments is then associated with a corresponding cluster.

In a different example, a system for automatically deploying a plurality of distributed application stacks on a target machine is presented, which includes a deployment module and a virtual environment manager. The deployment module is configured to deploy a distributed application stack from the plurality of distributed application stacks in a virtual environment created on the target machine. The plurality of distributed application stacks are deployed in the plurality of clusters. The virtual environment manager is operatively coupled to the deployment module and is configured to create a plurality of virtual environments isolated from each other on the target machine through which a user has access to a plurality of clusters. The virtual environment manager is further configured to associate each of the virtual environments with a corresponding cluster based on the deployed distributed application stack.

Other concepts relate to software for automatically deploying a plurality of distributed application stacks on a target machine. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data regarding parameters in association with a request or operational parameters, such as information related to a user, a request, or a social group, etc.

In one example, a machine readable and non-transitory medium having information recorded thereon for automatically deploying a plurality of distributed application stacks on a target machine recorded thereon, wherein the information, when read by the machine, causes the machine to perform a series of steps. A plurality of virtual environments isolated from each other are first created on the target machine through which a user has access to a plurality of clusters. A distributed application stack is deployed from the plurality of distributed application stacks in a respective virtual environment created on the target machine. The plurality of distributed application stacks are deployed in the plurality of clusters. Based on the deployed distributed application stack, each of the virtual environments is then associated with a corresponding cluster.

DETAILED DESCRIPTION

The present disclosure describes method, system, and programming aspects of deploying a plurality of distributed application stacks on a target machine in a fully automated manner. The deployment method and system as disclosed herein aim at eliminating manual interventions in distributed application stack deployment in order to make the deployment more efficient and effective. Such method and system benefit users in several ways: for example, the isolated target machine deployment allows hundreds of independent HADOOP clients to coexist on a single physical machine, thereby greatly aiding the tasks of HADOOP stack usage, testing, and certification; the ability to share clusters via a separate target machine installation reduces the need for multiple development and QE clusters, since a single cluster may sometimes be shared by multiple users, whereas a target machine environment usually cannot; the sharing of the gateway node greatly reduces the number of machines that need to be excluded from the general node pool to be deployed as dedicated gateways and also reduces the demand for other supporting resources, such as IP addresses.

FIGS. 1(a)-1(c) depict high level exemplary system configurations in which multiple distributed application stacks can be automatically deployed on a target machine, according to an embodiment of the present teaching. InFIG. 1(a), the exemplary system100includes a host102, a plurality of clusters104, a user106, a network108, one or more client-side machines109, and data sources110, including a software package repository110-a, a configuration repository110-b, and a record repository110-c. The network108may be a single network or a combination of different networks. For example, the network108may be a local area network (LAN), a wide area network (WAN), a public network, a private network, a proprietary network, a Public Telephone Switched Network (PSTN), the Internet, a wireless network, a virtual network, or any combination thereof. The network108may also include various network access points, e.g., wired or wireless access points such as base stations or Internet exchange points108-a, . . . ,108-b, through which a data source may connect to the network in order to transmit information via the network.

Each of the clusters104in which one of multiple distributed application stacks is to be deployed may include a plurality of regular nodes104-a,104-b, . . .104-h,104-i, and at least one gateway node111, which communicate through the network108or through an internal network (not shown). The gateway node111of the cluster104may be a special node that serves as an entry point and/or proxy when a user accesses the cluster104. In this example, each node of the cluster104may be an autonomous physical machine, such as a server, a workstation, a desktop or laptop computer, a netbook, a tablet, a smart phone, a game console, or any other suitable machine. In another example, some or all of the nodes may be parallel virtual machines implemented by either software emulation or hardware virtualization. Each cluster104may be a set of machines owned by one entity, e.g., enterprise, and maintained as a server farm or server cluster where the servers are mounted on racks in a server room or data center. A cluster104may also be a collection of machines owned by different entities and that are physically separate from each other at a distance.

The data sources110include one or more databases on one or more servers for providing and storing information related to the multiple distributed application stacks deployment. For example, the software package repository110-ais a storage location from which software packages may be retrieved and installed on the machines of the cluster104and the client-side machines109. The configuration repository110-bmay keep the configuration information necessary for the multiple distributed application stacks deployment including, for example, the version of each software artifact of the distributed application stack and the type of each node of the cluster104. The record repository110-cmay include a database storing all the records related to the multiple distributed application stacks deployment, including, for example, the status of each cluster node104-a,104-b, . . .104-h,104-i, and client-side machine109during the deployment (e.g., the version of each software artifact to be installed on a particular machine) and the result of the deployment for each cluster node104-a,104-b, . . .104-h,104-i, and client-side machine109(e.g., whether each software artifact has been properly installed on a particular machine and whether a diagnostic procedure has been performed and passed). It is understood that the data sources110may be maintained by the same entity that owns the cluster104or by a third-party, and may be implemented either on a single server having one or more databases or on multiple connected or unconnected servers.

In this exemplary system100, a user106, who may be a network administrator, operator, developer, or customer of the cluster104, may send a request to the host102via an internal or proprietary network connection to automatically deploy multiple distributed application stacks on a target machine, e.g., a client-side machine109or a gateway node111. For each distributed application stack, the host102, in response to the request, may retrieve configuration information from the configuration repository110-bof the data sources110and create an instruction for regular nodes of the cluster104and an instruction for the target machine based on the retrieved configuration information. Each regular node of the cluster104and the target machine, in accordance with the respective instruction received from the host102, may fetch the appropriate software artifacts from the software package repository110-aof the data sources110and install them onto the machine. In addition, the status and result of the deployment may be fully and automatically recorded and stored in the record repository110-cof the data sources110during and/or after the deployment by the host102in conjunction with the cluster104and the client-side machines109.

In this exemplary system100, each client-side machine109may be, for example, a server, a workstation, a desktop or laptop computer, a netbook, a tablet, a smart phone, a game console, or any other suitable physical machine, which is outside the clusters104and can be customized and configured by the user. A user may access and interact with one or more clusters104through the client-side machine109. A plurality of virtual environments isolated from each other may be created on a client-side machine109or a gateway node111such that a distributed application stack from the plurality of distributed application stacks may be deployed in a respective virtual environment and associated with a corresponding cluster104based on the deployed distributed application stack.

FIG. 1(b) presents a similar system configuration as what is shown inFIG. 1(a) except that the host102is now configured as two separate machines: a deployment host102-aand an admin host102-b. In this exemplary system112, only the admin host102-bhas the super-user or administrator access privileges to all the machines of the cluster104. The deployment host102-ain this example may be any computing device serving as an interface for the user106to input the request of deployment.

FIG. 1(c) presents another similar system configuration as what is shown inFIG. 1(a) except that the host102is one of the client-side machines109. In this exemplary system114, the user106may be a customer who subscribes to the usage of the cluster104and intends to deploy multiple distributed application stacks on the clusters104and his/her own computer (i.e., a client-side machine109) through his/her computer. It is noted that different configurations as illustrated inFIGS. 1(a)-1(c) can also be mixed in any manner that is appropriate for a particular application scenario.

FIG. 2is a depiction of multiple distributed application stacks deployment on a target machine, according to an embodiment of the present teaching. In this example, the target machines200(e.g., lightweight HADOOP clients) include client-side machines109and gateway nodes111. Each cluster104includes a plurality of regular nodes and at least one gateway node111, which serves as an entry point and/or proxy for a client-side machine109to access the cluster104. In this example, a plurality of distributed application stacks may be deployed on the regular nodes of the plurality of cluster104, respectively, in a one-to-one mapping relationship. On each target machine200, the plurality of distributed application stacks may be deployed on isolated environments such that a user can access and interact with any of the clusters104through a target machine200.

FIG. 3(a) shows a more detailed high level exemplary system diagram of the system100shown inFIG. 1(a), according to a first application embodiment of the present teaching. In this exemplary embodiment, the system100includes a host102, a target machine200, at least one regular cluster node104-afor each of multiple clusters104, a software package repository110-a, a configuration repository110-b, and a record repository110-c. The host102may receive a request202from a user in the form of, for example, a deployment form or a single-click submission. The request202may include a version of each distributed application stack to be deployed on a respective cluster104, a description of each cluster104, and an identifier of the target machine200. In response to receiving the request202, the host102may be responsible for retrieving configuration information204from the configuration repository110-bbased on the version of the distributed application stack and the description of the cluster104in the request202. For example, the configuration information204may include a software configuration having the version of each software artifact of the distributed application stack and a cluster configuration having the type of each regular node104-aof the cluster104. The configuration information204may be stored and retrieved in various forms, such as but not limited to a XML file or a text file. Based on the retrieved configuration information204, the host102may create an instruction206-afor the regular cluster nodes104-aand an instruction206-bfor the target machine200. The instruction206-amay be used to direct the regular cluster nodes104-ato deploy at least one cluster-side software artifact in the distributed application stack from the software package repository110-aonto the nodes; the instruction206-bmay be used to direct the target machine200to deploy at least one target-side software artifact from the software package repository110-ain a virtual environment on the target machine200. The instructions206may include executable code such as a script that can be executed by the machines104-a,200and other information necessary for deployment. For example, such information may include association information between one or more software artifacts and each type of the machines104-a,200and version information of the one or more software artifacts for each type of the machines104-a,200. The instruction206-bmay also include the identifier of the target machine200, such as, but not limited to an IP address, a URL, a formal name, an alias, etc. The instructions206may be transmitted to the regular cluster nodes104-aand the target machine200. To deploy multiple distributed application stacks, the host102in one example, may create multiple instructions each of which is directed to deploy one of the multiple distributed application stacks. In another example, a single instruction206may be created for deploying more than one distributed application stacks.

In this example, each regular cluster node104-ain the same cluster104may receive the same generic instruction206-afrom the host102. That is, it is unnecessary for the host102to create a customized instruction for each individual regular cluster node104-ain the same cluster104. Based on the received instruction206-a, each regular cluster node104-amay be responsible for identifying at least one cluster-side software artifacts208-afrom all the software artifacts of the distributed application stack for the respective regular cluster node104-a. For example, for HADOOP deployment, the instruction206-amay include cluster configuration indicating that one of the regular cluster nodes104-ais a Name node and also include association information indicating a set of cluster-side software artifacts associated with a Name node. Accordingly, by executing the script in the instruction206-a, the regular cluster node104-amay identify its cluster-side software artifacts208-a. Similarly, in another example, the regular cluster node104-amay identify a different set of cluster-side software artifacts208-aif the regular cluster node104-ais a different type of regular node. There may be more than one version of each software artifact stored in the software package repository110-aand thus, the version of each cluster-side software artifact208-ato be deployed may also be determined by each regular cluster node104-abased on the received instruction206-a, for example, the version information as noted above. As shown inFIG. 3(a), for each distributed application stack, the cluster-side software artifacts208-ain the determined versions may be fetched from the software package repository110-aand installed onto the regular cluster nodes104-aof a corresponding cluster104. As noted above, in this example, all the operations on the regular cluster nodes104-amay be automatically performed by executing the script in the received instruction206-awith reference to the information in the instruction206-a.

As to the target machine200, the instruction206-bmay include an identifier of the target machine200and deployment information for directing deployment of one or more distributed application stacks on the same target machine200. Similarly, for each distributed application stack, the deployment information may include association information indicating a set of target-side software artifacts associated with the target machine200and the version information of the set of target-side software artifacts. As shown inFIG. 3(a), for each distributed application stack, the target-side software artifacts208-bin the determined versions may be fetched from the software package repository110-aand installed onto the target machine200.

In this example, the regular cluster nodes104-aand the target machine200may be also responsible for transmitting records210-a,210-bto the host102, which indicate the status and result of the software artifacts fetching and installation. The host102then may collect all the records210-a,210-band store the records210into the record repository110-c. The records210may be stored in the form of, for example, a log file or a TWiki page.

FIG. 3(b) shows a more detailed high level exemplary system diagram of the system112shown inFIG. 1(b), according to a second application embodiment of the present teaching.FIG. 3(b) presents a similar system configuration as what is shown inFIG. 3(a) except that the host102is now configured as two separate machines: a deployment host102-aand an admin host102-b. In this example, the deployment host102-amay be responsible for receiving the deployment request202from a user. Since the deployment host102-amay not have the access privilege to the cluster104and/or the data sources110, in this example, the deployment host102-amay generate an instruction212, including executable code such as one or more scripts and information in the received request202. The first script may log into the admin host102-band cause the admin host102-bto execute the second script with reference to the information in the received request202. The admin host102-bthen may perform all other functions of the host102, as described inFIG. 3(a).

FIG. 3(c) shows a more detailed high level exemplary system diagram of the system114shown inFIG. 1(c), according to a third application embodiment of the present teaching.FIG. 3(c) presents a similar system configuration as what is shown inFIG. 3(a) except that the host is also the target machine200. In this example, since the host is also the target machine200, a set of target-side software artifacts208-bmay need to be deployed on the host. Different from the regular cluster nodes104-a, the instruction206-band the record210-bof the target machine200do not need to be transmitted through an outside network.

FIG. 4depicts an exemplary system diagram of the host102, according to an embodiment of the present teaching. In this exemplary embodiment, the host102acts as both a deployment host and an admin host. In this example, the host102includes a user interface302, a configuration retrieval unit304, an instruction generator306, a communication platform308, a recording unit310, and a target machine manager312. The user interface302is configured to receive a request from a user to deploy multiple distributed application stacks on a target machine200. In one example, the user interface302is in the form of a deployment form, including a target machine field, a cluster field, a distributed application stack field, and a record field. The target machine field may receive the identifier of the target machine200, for example, IP address “252.0.0.1.” The cluster field may receive the description of one or more clusters in the form of, for example, an alias, for example, “hit2”. The distributed application stack field may receive the version of one or more distributed application stacks, for example, “hit—0—20—203—0—11042713292—2.” The record field may receive the location of the record repository110-c, for example, at “hadoop-hit@yahoo-inc.com.” It is understood that the record field may not be necessary in other examples and that the location of the record repository110-cmay be identified in other forms, such as but not limited to alias, ID, URL, etc. Since multiple distributed application stacks may be deployed on multiple clusters, the deployment form may also include the mapping between each received cluster description and the respective distributed application stack version. For example, the distributed application stack version “hit—0—20—203—0—1104271329_t2” may be specified in the deployment form to be deployed on cluster “hit2” and target machine “252.0.0.1”, and another distributed application stack version “hit—0—30—203—0—1104271329_t1” may be deployed on another cluster “hit3” and on the same target machine “252.0.0.1.” It is understood that the user interface302may be in any other suitable form, such as an interactive user interface, a single-click submission (for a default cluster/distributed application stack version), command lines, etc.

In this example, the configuration retrieval unit304is operatively coupled to the user interface302and the configuration repository110-b. The configuration retrieval unit304is configured to, by executing executable code such as a script generated by the user interface302based on the received request, retrieve configuration information from the configuration repository110-b. As noted above, the configuration information may include the version of each software artifact of the distributed application stack and the type of each regular cluster node.

In this example, the type of each regular cluster node may be obtained based on the received description of the cluster104from the user interface302. For example, the cluster configuration may contain a full list of regular nodes participating in the cluster104. For example, a node “gsb190949.blue.ygrid.yahoo.com” may be defined as a Name node, and a node “gsb190947.blue.ygrid.yahoo.com” may be defined as a Secondary node. It is understood that any other suitable form, for example IP address and alias, may be applied in other examples for identifying a cluster node. As noted above, the cluster configuration is associated with a specific description of a cluster (e.g., “hit2”) and stored in the configuration repository110-b. All the cluster configurations may have already been predefined by a user or a third-party in advance to the deployment. Depending on the specific descriptions of clusters, various cluster configurations may be retrieved by the configuration retrieval unit304.

In this example, the version of each software artifact may also be obtained based on the received version of the distributed application stack from the user interface302. As noted above, the version of each software artifact in a HADOOP stack may be determined based on the received version of the HADOOP stack in the distributed application stack field in the deployment form (e.g., “hit—0—20—203—0—1104271329_t2”). For example, the version of HADOOP Distributed File System (HDFS) Proxy in the specified version of HADOOP stack is 3.0.0.0.1104072227, and the version of Data Acquisition (DAQ) is 0.1.1104262128. Since some components of a HADOOP stack may include more than one software artifact, the version of each software artifact in those components may be separately defined. For example, a Hive component may include a client-side software artifact and a server-side software artifact having the same version of 0.0.7—6. In this example, all the software artifacts are in the version-controlled package formats, such as but not limited to Tape Archive (TAR), General Pubic License ZIP Archive (GZ), Cabinet (CAB), ZIP Archive (ZIP), Roshal Archive (RAR), etc. As noted above, the exemplary software configuration is associated with a specific version of the HADOOP stack (“hit—0—20—203—0—1104271329_t2”) and stored in the configuration repository110-b. All the software configurations may be automatically updated or manually input by a user or a third-party in advance to the deployment. Depending on the specific versions of distributed application stacks, various software configurations may be retrieved by the configuration retrieval unit304. In another example, instead of receiving the version of the distributed application stack, some or all of the versions of the software artifacts in the distributed application stacks to be deployed may be specified directly through the user interface302. In this situation, it may not be necessary to retrieve the full software configuration or may not be necessary to retrieve the software configuration at all from the configuration repository110-b.

The version of the distributed application stack is orthogonal to the description of the cluster104. That is, the description of the cluster104determines which conceptual role (type) each node plays (e.g. a Name node, a Secondary node, etc.), and the version of the distributed application stack (i.e. a set of software artifacts versions) specifies which specific versions of software artifacts will be deployed on each particular node, so that the node can carry out its role.

In this example, the instruction generator306is operatively coupled to the configuration retrieval unit304and is configured to create deployment instructions for the regular nodes of the cluster104and the target machine200based on the retrieved configuration information. For example, the instruction generator306, by executing executable code such as a script, may read the cluster configuration and software configuration in the retrieved configuration information to generate association information between at least one software artifact and each type of the regular nodes. That is, each type of regular cluster nodes is mapped to a set of software artifacts that are necessary for the particular type of cluster nodes to act properly. Although it is usually not necessary to further map each individual cluster node to a set of software artifacts at the host side, in some examples, such mapping may be performed by the instruction generator306of the host102.

The instruction generator306may also generate version information of the at least one cluster-side software artifact for each type of regular nodes. That is, the version of each software artifact for each type of regular nodes may be identified based on the software configuration. Optionally, in generating such version information, the instruction generator306may further check the latest version of each software artifact from the software package repository110-aor from any other suitable source to determine if the current version specified in the retrieved software configuration should be updated. For example, the instruction generator306may determine that the version of a Nova component (1.1.1.2) retrieved from the configuration repository110-bmay be not up to date and that a latest version 1.1.2.1 of Nova may be available in the software package repository110-a. In this situation, the instruction generator306may further determine whether the latest version of Nova should replace the retrieved version in the version information by considering various factors, such as but not limited to compatibility, reliability, completeness, audit, certification, and compliance. The instruction generator306may optionally request the confirmation from the user or the cluster administrator to update one or more software artifacts to the latest versions in the version information in the instruction and/or the software configuration in the configuration information.

The instruction generator306is further configured to provide the instruction by including the association information and the version information into the instruction. The cluster configuration, as part of the configuration information retrieved by the configuration retrieval unit304, may be incorporated into the association information or separately included into the instruction by the instruction generator306. Optionally, as noted above, the instruction generator306may also provide executable code, such as a script, as part of the instruction. The instruction in this example may be created by the instruction generator306in the form of a compressed or non-compressed software package. In this example, the instruction generator306is also configured to cause the communication platform308to transmit the instruction to each regular node of the cluster104.

In this example, the target machine manager312is operatively coupled to the user interface302, the configuration retrieval unit304, and the instruction generator306. The target machine manager312is configured to facilitate the configuration retrieval unit304and the instruction generator306to generate an instruction for directing the deployment of multiple distributed application stacks on the target machine200. In one example, based on the identifier of the target machine200in the request, the target machine manager312may assist the configuration retrieval unit304to retrieve version information for the specific target machine and assist the instruction generator306to incorporate the retrieved information into the instruction. Each distributed application stack may include a plurality of cluster-side software artifacts (e.g., server-side software artifacts) to be deployed on regular nodes of the clusters104and a plurality of target-side software artifacts (e.g., client-side software artifacts) to be deployed on the target machine200. In one example, some software artifacts may only need to be deployed on the regular cluster nodes or only on the target machine. In another example, some software artifacts may have both a cluster-side version and a target-side version. In an example noted above, a Hive component in a HADOOP stack may include a cluster-side software artifact and a target-side artifact having the same version of 0.0.7—6. Accordingly, the target machine manager312may assist the configuration retrieval unit304to retrieve the version information for the target-side software artifacts to be deployed on the target machine200.

As noted above, since the deployment of multiple distributed application stacks on multiple clusters104may have a one-to-one mapping relationship while the deployment of multiple distributed application stacks on a target machine200may have a multiple-to-one mapping relationship, the target machine manager312may be also responsible for recording all the mapping relationship between each distributed application stack and a respective cluster and causing the instruction generator306to include such information in the instruction to be sent to the target machine200. For example, distributed application stacks 1-n may be requested to be deployed on cluster 1-n, respectively, in a single request or multiple requests, and such information may be recorded by the target machine manager312. Some distributed application stacks 1-m (m≦n) may be also requested to be deployed on a target machine200. In this situation, the target machine manager312may include the mapping relationship between distributed application stacks 1-m and the cluster 1-m in the deployment instruction for the target machine200. In addition, the target machine manager312may add location of the target machine200into the instruction. If the deployment request includes more than one target machine200, the target machine manager312may be also responsible for managing the deployment on different target machines200. It is noted that since the gateway node111of a cluster104is considered as a target machine in the present teaching, the deployment instruction for the gateway node111is different from other regular nodes of the cluster104and may be also created by the target machine manager312in conjunction with the configuration retrieval unit304and the instruction generator306.

In this example, the recording unit310is operatively coupled to the communication platform308and the record repository110-c. The recording unit310is configured to receive, through the communication platform308, the record from each regular node of the cluster104and the target machine200indicating the status during the deployment and/or the result after the deployment for the respective machine. In this example, the recording unit310is also configured to consolidate all the received records and store them in the record repository110-c. It is understood that although the record repository110-cinFIG. 4is remote from the host102, in other examples, the record repository110-cmay be part of the host102.

FIG. 5depicts an exemplary system diagram of the regular cluster node104-a, according to an embodiment of the present teaching. In this exemplary embodiment, the regular cluster node104-aincludes a communication platform402, a node deployment unit404, a fetching unit406, an installation unit408, and a record generator410. The communication platform402receives the deployment instruction from the host102and forwards the instruction to the node deployment unit404. If the instruction is received in the form of a software package, the instruction may be first unpackaged into the local file system of the regular cluster node104-a. In this example, the node deployment unit404is configured to identify at least one cluster-side software artifact from the plurality of software artifacts for the specific regular cluster node104-abased on the instruction, for example, the association information as noted above. For example, the association information in the instruction may associate the Secondary node type of cluster nodes to a particular set of software artifacts in the distributed application stack. The script in the instruction may be executed by the node deployment unit404to identify that the regular cluster node104-ais a Secondary type cluster node according to the cluster configuration, which may be part of the association information. The node deployment unit404may further identify the set of cluster-side software artifacts to be installed on the regular cluster node104-aaccording to the mapping in the association information. In this example, the node deployment unit404is also configured to determine the version of each identified cluster-side software artifact based on the instruction, for example, the version information.

In this example, the fetching unit406is operatively coupled to the software package repository110-aand the node deployment unit404. The fetching unit406is configured to fetch the identified at least one cluster-side software artifact in the determined version from the software package repository110-a. The fetching may be performed by the fetching unit406under any suitable protocols known in the art, such as but not limited to File Transfer Protocol (FTP), Secure Copy Protocol (SCP), Secure Shell (SSH), Peer-to-Peer (P2P), etc. In this example, each software artifact is stored in a version-controlled package format in the software package repository110-a. A version-controlled package format may be, for example, a compressed TAR file or any other suitable file containing all files for a given software artifact. The package may also contain several manifest files describing component versions and other component-related information. An integral feature of this package system is a server-based repository of all the packages. Once a package is created, it is uploaded to the software package repository110-a. From this point on, the package may be permanently kept in the software package repository110-aand associated with a version that was given to it at its creation time. This guarantees that the association between the software package and the version is permanent and immutable. Accordingly, each software artifact may be stored in the software package repository110-aunder various versions, and the fetching unit406may fetch the version that is determined by the node deployment unit404based on the instruction from the host102.

In this example, the installation unit408is operatively coupled to the fetching unit406and configured to install the fetched cluster-side software artifacts onto the machine. If the cluster-side software artifacts are in the version-controlled package format, the installation unit408may unpack all the files into the local file system of the regular cluster node104-abefore installation. The installation unit408may further perform any suitable operation specific to the particular type of node to complete the installation. The record generator410is operatively coupled to the installation unit408and is configured to cause the communication platform402to transmit the deployment record to the host102. For example, the record generator410may record all the information related to the deployment, such as the name, version, installation time, and size of each software artifact installed on the machine and whether the installation of each software artifact was successful. The record may be generated in the form of, for example, a log file or a TWiki page. The record generator410may optionally run a node-type specific test to verify whether the installation has been successful and record the test result as well.

FIG. 6depicts an exemplary system diagram of the target machine200, according to an embodiment of the present teaching. In this exemplary embodiment, the target machine200includes a communication platform402, a virtual environment manager602, a deployment module604, and a user interface606. The communication platform402receives the deployment instruction from the host102and forwards the instruction to the virtual environment manager602. If the instruction is received in the form of a software package, the instruction may be first unpackaged into the local file system of the target machine200. The user interface606in this example is an interface between the virtual environment manager602and the user of the target machine200. In one example, the user interface606may present a selection of multiple distributed application stacks deployed on the target machine200to the user and receive a request from the user to access one of clusters that is associated with the corresponding selected distributed application stack. The request may be forwarded to the virtual environment manager602.

In this example, the virtual environment manager602is configured to create a plurality of virtual environments608isolated from each other on the target machine200through which a user has access to the plurality of clusters104. Each virtual environment608may be labeled with an identifier. A virtual environment608may be a means for delivering resources that have been decoupled from physical machinery, thereby more efficiently utilizing resources while avoiding costly over-provisioning. The virtual environments608in this example may be process or application virtual environments that run as normal applications inside a host operating systems and support a single process. In other examples, the virtual environments608may be system virtual environments that provide complete system platforms which support the execution of a complete operating system. Nevertheless, one characteristic of a virtual environment608is that the software running inside is limited to the resources and abstractions provided by the virtual environments608. In other words, multiple virtual environments608created on the same physical machine are isolated from each other. A virtual environment608may enable the running of virtual desktops, servers or other virtual machines. For example, a virtual machine is a completely isolated guest operating system installation within a normal host operating system. Virtual environments may be implemented with either software emulation or hardware virtualization or both together, such as emulation of the underlying raw hardware, emulation of a non-native system, and operating system-level virtualization. In this example, at least one resource (e.g., computing resource, storage resource, communication resource, etc.) associated with the target machine200is allocated to each of the virtual environments608, and the plurality of target-side software artifacts deployed in each virtual environment608are capable of accessing the at least one resource allocated to the virtual environment608.

In this example, the deployment module604is operatively coupled to the virtual environment manager602and is configured to deploy a distributed application stack from the plurality of distributed application stacks in a virtual environment608created on the target machine200. As noted above, the plurality of distributed application stacks are deployed in the plurality of clusters104(e.g., the regular nodes104-aof each cluster104). In this example, the deployments of the same distributed application stack (e.g., the same version of HADOOP stack) on the regular nodes104-aand on the target machine200may occur at substantially the same time. That is, the host102transmits the deployment instructions to the regular nodes104-aand the target machines200at substantially the same time. In order to deploy a distributed application stack on the target machine200, one or more virtual environments608need to be created prior to the deployment. In one example, the virtual environment manager602may create a new virtual environment608once the deployment instruction is received. In another example, the virtual environment manager602may have created multiple virtual environments608in advance and assign one virtual environment608to each deployment upon receiving the deployment instruction. In deploying a distributed application stack, the deployment module604is further configured to receive an instruction to direct the deployment of at least one target-side software artifact from the software package repository110-ainto the virtual environment608. The instruction may be created by the virtual environment manager602and includes an identifier for a specific virtual environment608. As noted above, each distributed application stack may be identified by the version of the distributed application stack and thus, the instruction may be generated based on the version of the distributed application stack to be deployed on the specific virtual environment608. In other words, the version of a distributed application stack may be associated with the identifier of a virtual environment608, and such association information may be included in the instruction.

In this example, the deployment module604includes a node deployment unit404, a fetching unit406, an installation unit408, and a record generator410. Each unit of the deployment module604may perform the similar function of a counterpart unit inFIG. 5. For example, the node deployment unit404is configured to identify at least one target-side software artifact from the plurality of target-side software artifacts for the virtual environment608based on the instruction and determine a version of the at least one target-side software artifact for the virtual environment608based on the instruction; the fetching unit406is configured to fetch the at least one target-side software artifact in the determined version from the software package repository110-ato the target machine200; the installation unit408is configured to install the at least one target-side software artifact in the determined version in the virtual environment608of the target machine200that is identified by the identifier in the instruction; the record generator410is configured to run test and recording procedures to verify whether the installation has been successful and record the test result.

In this example, the virtual environment manager602is further configured to associate each of the virtual environments608with a corresponding cluster104based on the deployed distributed application stack. In one example, the association is achieved based on the version of the deployed distributed application stack. Referring now toFIG. 8, for example, the target-side software artifacts of HADOOP 1 are deployed in virtual environment 1 on the target machine200, and the cluster-side software artifacts of the same HADOOP 1 are deployed on the regular nodes of cluster 1. Thus, virtual environment 1 is then associated with cluster 1 by the virtual environment manager602in this example. Similarly, virtual environment 2 may be associated with cluster 2 because they are deployed with the same version of HADOOP 2.

Referring back toFIG. 6, the virtual environment manager602is further configured to receive a request from the user to access one of the plurality of clusters104through the target machine200. As noted above, the request may be received by the virtual environment manager602via the user interface606. The virtual environment manager602is then responsible for identifying one of the plurality of virtual environments608created on the target machine200that is associated with the cluster104in accordance with the request. In this example, the identification may be achieved by finding the same version of a distributed application stack deployed both on a cluster104and in a virtual environment608created on the target machine200. In this example, the virtual environment manager602is further configured to invoke the distributed application stack deployed in the identified virtual environment608such that the user may access and interact with the cluster104through the invoked distributed application stack.

FIG. 7depicts an exemplary system diagram of the host/target machine200according to an embodiment of the present teaching. In this exemplary embodiment, most of the units perform the similar functions of their counterpart units inFIGS. 5 and 6. The instruction generator702in this example, in addition to causing the communication platform308to transmit instructions to the regular nodes of the cluster104, may directly forward an instruction to the node deployment unit404in the same machine. The record generator704in this example may directly forward the record to the recording unit310.

FIG. 9is a flowchart of an exemplary process in which a plurality of distributed application stacks is automatically deployed on a target machine, according to an embodiment of the present teaching. Beginning at block902, a plurality of virtual environments isolated from each other are created on the target machine through which a user has access to a plurality of clusters. As described above, this may be performed by the virtual environment manager602of the target machine200. At block904, processing may continue where a distributed application stack from the plurality of distributed application stacks is deployed in a virtual environment created on the target machine. The plurality of distributed application stacks are deployed in the plurality of clusters. As described above, this may be performed by the deployment module604of the target machine200. At block906, each of the virtual environments is associated with a corresponding cluster based on the deployed distributed application stack. As described above, this may be performed by the virtual environment manager602of the target machine200.

FIG. 10is a more detailed flowchart of an exemplary process in which a plurality of distributed application stacks is automatically deployed on a target machine, according to an embodiment of the present teaching. Blocks902,904,906may be performed as described above with reference toFIG. 9. Proceeding to block1002, a request is received from a user to access one of the plurality of clusters through the target machine. At block1004, one of the plurality of virtual environments created on the target machine is identified as being associated with the cluster in accordance with the request. At block1006, the distributed application stack deployed in the identified virtual environment is invoked. As described above, blocks1002,1004,1006may be performed by the virtual environment manager602of the target machine200.

FIG. 11is a flowchart of another exemplary process in which a plurality of distributed application stacks is automatically deployed on a target machine, according to an embodiment of the present teaching. Beginning at block902, a plurality of virtual environments isolated from each other are created on the target machine through which a user has access to a plurality of clusters. As described above, this may be performed by the virtual environment manager602of the target machine200. At block1102, processing may continue where an instruction is received to be used to direct the target machine to deploy at least one of the plurality of target-side software artifacts from a software package repository in the virtual environment. The instruction includes an identifier for the virtual environment. As described above, this may be performed by the deployment module604of the target machine200. At block1104, at least one target-side software artifact is identified from the plurality of target-side software artifacts for the virtual environment based on the instruction. At block1106, a version of the at least one target-side software artifact for the virtual environment is determined based on the instruction. As described above, blocks1104and1106may be performed by the node deployment unit404of the target machine200. Proceeding to block1108, the at least one target-side software artifact is fetched in the determined version from the software package repository to the target machine. As described above, this may be performed by the fetching unit406of the target machine200. At block1110, the at least one target-side software artifact is installed in the determined version in the virtual environment of the target machine. As described above, this may be performed by the installation unit408of the target machine200. At block906, each of the virtual environments is associated with a corresponding cluster based on the deployed distributed application stack. As described above, this may be performed by the virtual environment manager602of the target machine200.

FIG. 12depicts a general computer architecture on which the present teaching can be implemented and has a functional block diagram illustration of a computer hardware platform that includes user interface elements. The computer may be a general-purpose computer or a special purpose computer. This computer1200can be used to implement any components of the architecture as described herein. Different components of the system100,112,114, e.g., as depicted inFIGS. 1(a),1(b),1(c),3(a),3(b), and3(c), can all be implemented on one or more computers such as computer1200, via its hardware, software program, firmware, or a combination thereof. Although only one such computer is shown, for convenience, the computer functions relating to dynamic relation and event detection may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load.

The computer1200, for example, includes COM ports1202connected to and from a network connected thereto to facilitate data communications. The computer1200also includes a central processing unit (CPU)1204, in the form of one or more processors, for executing program instructions. The exemplary computer platform includes an internal communication bus1206, program storage and data storage of different forms, e.g., disk1208, read only memory (ROM)1210, or random access memory (RAM)1212, for various data files to be processed and/or communicated by the computer, as well as possibly program instructions to be executed by the CPU. The computer1200also includes an I/O component1214, supporting input/output flows between the computer and other components therein such as user interface elements1216. The computer1200may also receive programming and data via network communications.

Those skilled in the art will recognize that the present teachings are amenable to a variety of modifications and/or enhancements. For example, although the implementation of various components described above may be embodied in a hardware device, it can also be implemented as a software only solution—e.g., an installation on an existing server. In addition, the units of the host and the client nodes as disclosed herein can be implemented as a firmware, firmware/software combination, firmware/hardware combination, or a hardware/firmware/software combination.