Patent Publication Number: US-2021176328-A1

Title: Management Services for Distributed Computing Architectures Using Rolling Changes

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a Continuation of, and claims the priority benefit of, U.S. patent application Ser. No. 15/907,072 filed on Feb. 27, 2018, entitled “MANAGEMENT SERVICES FOR DISTRIBUTED COMPUTING ARCHITECTURES USING ROLLING CHANGES,” which is related to application Ser. No. 15/907,011 (Attorney Docket No. PA9051US) filed Feb. 27, 2018, the disclosures of which are hereby incorporated by reference in their entirety for all purposes. 
    
    
     FIELD OF THE PRESENT TECHNOLOGY 
     The present technology relates generally to distributed computing, and more specifically, but not by limitation, to management services for distributed computing architectures that provide self-replication of management services, as well as constructor management of nodes including rolling changes grouped by node attributes. Other changes performed by the constructor can include dynamic creation and/or deletion of nodes in a cluster. 
     SUMMARY 
     Various embodiments of the present technology include a system comprising: clusters of nodes providing services; and a plurality of management servers, each of the plurality of management servers comprising: at least a distributed coordination service for the clusters of nodes, the distributed coordination service being a datastore; a director that manages the distributed coordination service and promotes at least one of nodes of the clusters to being one of the plurality of management servers, wherein promoting comprises replicating the distributed coordination service; and a constructor that manages allocation and life cycle deployments of the one or more nodes, the constructor further configured to implement rolling attribute changes for the clusters of nodes, the rolling attribute changes defining a processing order in which the nodes in a cluster are modified. 
     Various embodiments of the present technology include a method comprising: providing clusters of nodes providing services; providing a plurality of management servers, each of the plurality of management servers comprising at least a distributed coordination service for the clusters of nodes, the distributed coordination service being a datastore; managing access to the distributed coordination service by a director; promoting at least one node of the clusters to being one of the plurality of management servers, wherein promoting comprises replicating the distributed coordination service; and managing allocation and life cycle deployments of the one or more nodes using a constructor that is configured to implement rolling attribute changes for the clusters of the nodes, the rolling attribute changes defining a processing order in which the nodes of the clusters are modified. 
     Various embodiments of the present technology include a system comprising: clusters of nodes providing services; and a plurality of management servers, each of the plurality of management servers comprising: at least a distributed coordination service for the clusters of nodes, the distributed coordination service being a datastore; and a constructor that manages allocation and life cycle deployments of the nodes of the clusters, the constructor further configured to manage topological changes to nodes of the clusters by implement rolling attribute changes for the nodes. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Certain embodiments of the present technology are illustrated by the accompanying figures. It will be understood that the figures are not necessarily to scale and that details not necessary for an understanding of the technology or that render other details difficult to perceive may be omitted. It will be understood that the technology is not necessarily limited to the particular embodiments illustrated herein. 
         FIG. 1  is a high level schematic diagram of computing architecture for practicing aspects of the present technology. 
         FIG. 2  illustrates a self-replicating node promotion process within a distributed computing architecture. 
         FIG. 3  is a flowchart of an example method of the present disclosure. 
         FIG. 4  is a flowchart of an additional example method of the present disclosure. 
         FIG. 5  is a diagrammatic representation of an example machine in the form of a computer system. 
     
    
    
     DETAILED DESCRIPTION 
     The present disclosure is directed to various embodiments of systems and methods that comprise self-replicating management services for distributed computing architectures. In some instances, these management services can self-replicate using nodes operating in the computing architecture in order to maintain redundancy and data integrity. 
     Prior to additional discussion, some definitions are provided for context. A datastore as referred to herein is a distributed coordination service that acts as a datastore. The datastore in the coordination service is what is used to configure and govern other services. Its ability for its management to be automated and its resiliency in the face of failures is what makes a datastore useful as a coordination service. ZooKeeper™, and specifically Apache ZooKeeper™, as used herein, is an example of a coordination service. The present disclosure provides a director that manages a coordination service, such as ZooKeeper™. A Tunnel or Stunnel is a particular product used to establish cryptographic tunnels (e.g., encryption infrastructure). A cryptographic tunnel is useful when the product that needs securing either has no or poor support for cryptographic features. 
     Operations performed by a director/client forwarder of the present disclosure happen to be interacting with a stunnel, but a “tunnel” product is not required or necessary if the underlying system possesses acceptable cryptographic support. 
     Ports, as referred to herein, include client ports of ZooKeeper™. The systems of the present disclosure adapt to ZooKeeper™ in this context. If a different coordination service (e.g., not ZooKeeper™) is utilized that has a better separation of privileged users, separate ports may not be required. 
     A Blueprinter is a distributed workload scheduler and schedules workloads based on roles that can be dynamically managed at runtime. A function of the Blueprinter in context of the director is to act as a gatekeeper that verifies tokens, providing a means of automatically providing credentials such as certificates, private keys, and authentication without human approval. 
     In some embodiments, a distributed computing architecture, such as the Elastic Cloud Enterprise™, is managed and implemented with self-replicating management services. Elastic Cloud Enterprise™ is a distributed computing architecture that is located on premises at a customer site or hosted remotely. 
     In general, the self-replicating management services are mediated. An example coordination service that can be replicated includes, but is not limited to, Apache ZooKeeper™ (referred to herein as ZooKeeper). The coordination service implements a database that stores data for the distributed computing architecture. Operations and access to the coordination service is mediated through a director. The self-replicating nature of these systems is also facilitated through use of the director. 
     In some embodiments, the coordination service coordinates a state of the distributed computing architecture and a state of all clusters (e.g., nodes or allocators) executing within the distributed computing architecture. In some instances, a coordination service is managed by one or more directors. 
     In various embodiments, an instance of coordination service and director runs on a management server (which could be physical or virtual). In one or more embodiments, the management server can be self-replicating through recruitment or promotion of nodes within the distributed computing architecture. According to some embodiments, the directors ensure that a quorum of management servers is available. In some embodiments a quorum is at least two out of three management servers. Other quorums and totals are also contemplated. 
     The management servers are all synchronized and interconnected, in some embodiments. That is, the database (e.g., coordination service) is replicated and each management server is connected to every other management server in the architecture. In sum, the directors are involved in establishing a quorum when new coordination service nodes are created. New directors are also instantiated on the new nodes. 
     In some embodiments, the directors can be configured to establish not only a quorum of management servers, but also promote and establish a pool of inactive but pre-configured management servers. Upon detection of a failure in one of the management servers currently in a quorum, the directors can automatically select from these pre-configured management servers rather than having to wait until a quorum failure is detected in order to promote a new node. Failure to rectify a lack of quorum in management servers leads to deleterious downstream effects such as data unavailability, which results in the nodes being unable to provide services to their end users. 
     In some embodiments, the directors sign certificate signing requests (CSRs) for nodes that want to communicate with a coordination service instance. The directors also maintain the encryption infrastructure used by coordination services for communication inside the architecture. These and other advantages of the present disclosure are provided herein. 
       FIG. 1  is a schematic diagram of an example system that includes a plurality of management servers  101 , such as management server  102 , and one or more nodes, such as node  104  (e.g., allocator). The management server  102  and node  104  are communicatively coupled over an encryption infrastructure  106  such as an S-Tunnel or other secure channel. In general, administrators can access the functions of the management server  102  and end users can access services executing on the node  104 . 
     In more detail, each of the management servers comprises a distributed coordination service  108  (e.g., coordination service) and director  110 , each of which are disclosed in greater detail with respect to  FIG. 2 . 
     Turning to  FIG. 2 , the management server  102  (of a plurality of management servers) is illustrated as comprising the distributed coordination service  108  and the director  110 . The distributed coordination service  108  is implemented as a datastore, in some embodiments. The distributed coordination service  108  maintains the data necessary to allow the node  104  to provide various services to end users. For example, node  104  can provide one or more services to end users such as a visualization tool, data shippers, data log pipeline, and so forth. 
     In general, the director  110  is configured to manage requests for data on the datastore from the node  104  and promote one or more nodes (such as node  104 , for example) to being one of the plurality of management servers. Generally speaking, promoting comprises replicating the distributed coordination service  108  and its data stored thereon. A new director is also spun up on the node  104  during promotion. 
     For context, the director  110  is a sidecar to the distributed coordination service  108  that performs several functions. In some instances, the director  110  maintains a lifecycle of the distributed coordination service  108 . The director  110  can write distributed coordination service  108  configuration files (as well as guarantee uniqueness of the management server identifiers) and schedule a management server to start locally (if no nodes are available to promote). The director  110  can also handle promotion of a node from observer to participant once all the data in the distributed coordination service of one management server is replicated to the node and has been guaranteed to be synchronized from the existing ensemble of management servers. 
     The director  110  also maintains an encryption infrastructure (e.g., secure communications channel) configuration for distributed coordination service  108  server ports (inbound and outbound, referred to as a management interface) and client ports (all discussed in more detail below), using a server-specific and a client-specific schema. This provides transparent (from the distributed coordination service of view) encryption of traffic between the management servers. 
     In  FIG. 2 , an initial set of management servers in a quorum  112  is present. The director  110  of management server  102  begins a process of promoting the node  104 . This can occur when the node  104  requests access to the distributed coordination service  108 . For example, in order to execute a service  114  of the node  104 , the node  104  requests data from the distributed coordination service  108 . In some embodiments, this is effectuated by the node  104  connecting to a node port  116  on the management server  102 . Initially, the node  104  is a non-trusted server/entity. 
     When the node  104  requests access to the distributed coordination service  108 , the node  104  will provide the director with a CSR request. If approved, the director  110  will sign the CSR request and set up encryption infrastructure  106  (e.g., tunnel, etc.) between the management server  102  and the node  104  on a node interface  111  of the management server  102 . The node  104  is now trusted and can be allowed to request data from the distributed coordination service  108 . In some embodiments, all connections between management servers and nodes include encryption infrastructures, as well as connections between management servers and other management servers. 
     If the director  110  determines that a new management server is required, the director  110  can initiate a process of promoting the node  104  to management server status. This preemptive promotion can be based on the fact that a quorum of management servers is present, but that loss of one of these management servers would result in a quorum failure. For example, management servers  102 ,  122 , and  124  are present and form the quorum  112 . Again, while not illustrated for purposes of clarity, each of the management servers  102 ,  122 , and  124  are interconnected to one another. These management servers  102 ,  122 , and  124  are connected on a dedicated management interface  128  that allows unrestricted access to the underlying distributed coordination service  108 . Thus, while the node facing ports provide only limited access to data, the dedicated management interface provides complete access to the data on the distributed coordination service  108 . 
     In addition to interconnection between instances of distributed coordination services on different management servers through their respective dedicated management interfaces, the director of each management server is coupled with the directors and distributed coordination services of all other management servers in the quorum. 
     Due to the fact that three management servers are needed for a quorum (in some embodiments it may be fewer or more), the director of the management server  102  automatically initiates promotion of node  104  to management status. In other words, three management servers are desired so as to allow for surviving the failure of one management server. 
     The selection of node  104  is undertaken because node  104  is already a trusted node. Thus, the automatic promotion of nodes can be based on threshold criteria related to the quorum. It will also be understood that any of the management servers are capable of automatically promoting a node, although directors on the various management servers can coordinate so as to not promote additional management servers if another director has already initiated a promotion process. An expanded quorum  126  is illustrated in  FIG. 2 , as now including node  104 . 
     The node  104  now includes a distributed coordination service  130  and a director  132 . In some embodiments, the director  110  of the management server  102  writes distributed coordination service configuration files onto the node  104  in such a way that a uniqueness of an identification of the distributed coordination service is maintained. Thus, the distributed coordination service of the management server  102  is replicated onto the node  104 . The director  110  of the management server  102  can also verify that replication of the distributed coordination service and the data on the node  104  has resulted in synchronization with the distributed coordination service of each of the plurality of management servers in the quorum. 
     Once the node  104  is promoted to the management server status, the node  104  is capable, through its director  132 , of promoting other nodes, if needed. 
     In one or more embodiments, a director can facilitate promotion of a node by issuing certificates to a node that desires to connect to any of the plurality of management servers and granting access to certificate bearing nodes on the one or more client ports. 
     In some embodiments, the director can implement a blueprinter function that assigns nodes in the cloud to the management servers based on a role associated with the one or more nodes. That is, a particular management server can be dedicated to service a specific type of node/client. The management server is thus assigned this type or role. Nodes requiring data from the management server can be assigned based on a role. 
     The blueprinter can also dynamically reassign a node to a different management server when the role of the node changes or a role of a currently assigned management server is changed. For example, if a node is performing a visualization role, the management server is associated with a visualization role. 
     In some embodiments, the blueprinter is configured to ensure that automatically assigned management server relationships do not conflict with manually assigned management server relationships (for example, if an administrator has manually set up a relationship between a node or nodes and a management server or management servers). 
     In sum, given a set of containers associated with a role, the blueprinter ensures these containers are assigned to servers based on the servers&#39; roles. The container assignments are updated dynamically if either the role definitions change or the roles of the server change. The roles associated with a server can be authenticated either through an administrator explicitly validating the server for the roles or through a token-based system where the server supplies a token that the blueprinter can cryptographically verify as valid and untampered. It will be understood that care is taken to ensure that the automatically assigned containers do not conflict with manually assigned containers in such a way that a manually assigned container will not be removed from a server as part of the blueprinter operations. 
     Additional enhancements to the processes described above are also disclosed. For example, when a client forwarder  118  is implemented on the management server  102  to cooperate with the director  110  to forward a known set of local ports (e.g., node or client ports) and transparently handle encryption of traffic and automatically update the forwarded ports to new management servers when any of the directors of the management servers in a quorum are unavailable. 
     In various embodiments, the client forwarder  118  is configured to cooperate with the director  110  and forward a known set of local ports on each server in an installation to potentially remote distributed coordination service APIs, transparently handle encryption over the network, and automatically update forwarded ports when directors are added/removed from an installation. 
     Another advantage includes the use of tokens for security and rapid integration of new servers. For example, the replication process of creating a new node and promoting the node to management server status can be optimized through use of a token. For example, if a node needs to be terminated, the end user of the node can request a token that includes the configurations of that particular node, which include trusted status or permissions for that node. When a new node is created, the node can be provisioned with the token. The new node can present the token to a director of a currently existing management node in order to automate the certificate exchange processes and trust verification steps disclosed above. 
     Also, the token can identify the new node as a management node. That is, the prior role of the node was a management server. These configurations are set forth in the token such that the new node can present the token to a director of a currently existing management node and be promoted without having to complete the certificate exchange processes and trust verification steps. In some embodiments, the certificate is still issued and the token is used to automate the certificate provisioning without having to await approval. 
       FIG. 3  is a flowchart of an example method that is executed in accordance with the present disclosure. The method includes a step  302  of providing one or more nodes providing services. For example, the network includes one or more nodes that provide services such as data visualization, data transfer, and data logging to end users. These node(s) are maintained by management servers in the architecture. Thus, the method also includes a step  304  of maintaining a quorum of a plurality of management servers serving the nodes. These management servers mediate the flow of data within the architecture to the node(s). The management servers also function to promote nodes if needed to ensure that a quorum of management servers is present. 
     The method can also include a step  306  of providing at least a distributed coordination service for the one or more nodes on each of the plurality of management servers. This distributed coordination service is a datastore that stores the data needed for the nodes to provide their respective services to end users. 
     In some embodiments, the method includes a step  308  of managing, via a director, requests for data on the distributed coordination service from the one or more nodes. This includes the director mediating interactions for data from nodes into the distributed coordination service, ensuring that access to the distributed coordination service is efficient and simple. 
     In one or more embodiments, the method includes a step  310  of promoting at least one of the one or more nodes to being one of the plurality of management servers. This approach is taken to ensure a promoting process comprises replicating the distributed coordination service in the node being promoted. When the replication has been validated and the newly promoted management server has been synchronized to all other management nodes in the quorum, the node is considered to be part of the quorum. 
     Referring briefly back to  FIG. 1 , in some embodiments, the management server  102  can also comprise a constructor  134  that manages allocation and life cycle of “deployments” (e.g., services such as Elasticsearch™ or Kibana™) on allocators, which are the servers that provide their capacity to a “capacity pool.” In some embodiments, the allocators herein have been referred to as nodes and each node or group of nodes has one or more services running thereon. In one example, a cluster  136  of nodes or allocators is provided that includes a plurality of nodes that includes node  104 . 
     In some embodiments, the constructor  134  can effect changes including, but not limited to, dynamic creation and/or deletion of nodes in a cluster. Thus, in addition to causing rolling attribute changes for nodes in a cluster, such as increasing compute resources, memory, roles, behaviors, and so forth, the constructor  134  can also be configured to delete and create nodes as required. In general, a change to an attribute or a change to a node can include any of the node altering processes described herein, such as creation, deletion, or attribute change. 
     Thus, as illustrated, the system comprises several clusters of nodes providing services. As noted above, the system also includes a plurality of management servers, with each of the plurality of management servers comprising at least a distributed coordination service for the clusters of nodes. The distributed coordination service is a datastore in some instances. Also, as mentioned above, the management server includes a director that manages the distributed coordination service and promotes at least one of the nodes of the clusters to being one of the plurality of management servers. Details regarding promotion of nodes are discussed in greater detail above. 
     In various embodiments, the constructor  134  manages allocation and life cycle deployments of the one or more nodes. The constructor  134  also implements rolling attribute changes for the clusters of nodes. That is, the constructor  134  implements rolling attribute changes for nodes inside a cluster. In some embodiments, the rolling attribute changes define a processing order in which the nodes in a cluster are modified. Also, the constructor  134  is utilized to manage risk when executing rolling attribute changes in a group or cluster of nodes, as will be described in greater detail infra. 
     In one non-limiting example, the processing order can declare groups of nodes that should be updated simultaneously in order to perform updates based on any attribute, such as their logical and/or physical location, node role, or configuration, just to name a few. 
     Attribute changes with respect to nodes can involve many forms such as operational attributes of how a node operates or functions, what computing resources are dedicated to the node, topological changes, and so forth. In more detail, in an example, configuration change includes enabling and/or disabling a plugin used by the node that requires the restart of a process. Another example change is a “topology change,” when a number of and/or size of nodes change in a cluster. In yet another example, an attribute change can include changing to/from a high-availability configuration of one or more nodes. 
     Other changes include, but are not limited to, adding or removing nodes, resizing nodes, and changing to or from dedicated master nodes. Of particular note, a master node in a cluster is a selected node that coordinates additional nodes in a cluster. In general, some topology changes are relatively simple, whereas others (e.g., those touching master nodes) require considerable care. In general, a master node is a node that is a node that has been or is being promoted to being a management server. The master node can also include a node that has been given some control over subordinate nodes, but is not explicitly a management server. 
     There are different strategies to implementing a change. Different strategies have different characteristics. One such characteristic is speed, which refers to a time to completion. Another characteristic involves availability that involves an extent to which a cluster is available or not available or overloaded during attribute changes. 
     There are also risk considerations such whether it is possible to recover back to a known-good state for that node when an attribute change may result in a deleterious effect to a node. The system can also determine the complexity of an attribute change, how stable intermediate states are, and if the attribute changes can be retried. 
     Other considerations include resources required to affect the attribute changes. One example method of performing attribute changes includes a “grow and shrink” method by the constructor  134  where new nodes are added. In these instances, existing nodes are moved over and then old undesired nodes are spun down. These methods can be slow and resource hungry as the process requires twice the capacity during changes, but this method provides low risk and good availability. 
     Another example process includes a full cluster restart where the constructor  134  stops all nodes in a cluster, causes the attribute changes in place, and then the constructor  134  restarts everything. These methods are quick, requiring no extra resources, but the cluster is unavailable throughout the change and risk is medium as attribute changes may not necessarily be available for roll back after the change. 
     The constructor  134  of the present disclosure is configured to allow for implementation of rolling attribute changes. In more detail, a “rolling attribute” is chosen by the constructor  134  providing groups/batches of nodes to process in order. Stated otherwise, the constructor  134  manages allocation and life cycle deployments of the nodes of the clusters and manages topological changes to nodes of a cluster by implementing rolling attribute changes for the nodes. 
     If the rolling attribute change is the same for every node in a cluster (or all nodes in more than one cluster), the process is similar to a full cluster restart event. If the rolling attribute change is different for every node in a cluster, the rolling attribute change “rolls” or iterates through every node one by one causing the attribute change. 
     The constructor  134  can also implement attribute changes that are rolling with respect to an availability zone or deployment group, such as different racks on different power supplies. This process is faster than processes that operate per node and is safer and more available than full cluster restart. 
     The constructor  134  can also ensure, in some embodiments that attribute changes to master nodes occur last. A change of master nodes may briefly disrupt the cluster, so the constructor  134  can optimize its rolling attribute change for the least amount of master elections. 
     In some embodiments, the rolling attribute change can be combined with the grow and shrink methods disclosed above. For example, this can include a rolling grow and shrink method. 
     The need to perform a rolling attribute change can occur under various conditions or triggers. For example, the constructor  134  can be triggered based on user request. For example, a customer desires to upgrade or change their services. In these instances, the constructor  134  can weigh risk, cost, availability, and speed in order to select the proper attribute change process. 
     In another example, a change in infrastructure is desired such as planned or unplanned maintenance. In these instances, administrators are less willing to impose unavailability on the customer. 
     Another trigger can be based on a system component failure. For example, underlying hardware of a node/allocator has failed and the constructor  134  implements a rolling change to restore availability. In these instances, brief complete unavailability might be preferable to prolonged periods of impairment. 
     During operation, a cluster can be in different states, such as working, where the nodes of the cluster are performing in an acceptable manner. In these instances, rolling changes are implemented for situations such as upgrading or in order to cause a service level change, as examples. 
     Another state of a node or cluster is broken, which involves instances such as missing data, not having enough memory, running out of disk, and so forth. Yet another state of a node or cluster is overloaded. This overloaded state can refer to compute resources such as memory, central processing unit (CPU), or both. 
     In addition to the constructor  134  picking an attribute change strategy based on triggers and or conditions, an attribute change can be influenced in other ways. For example, the constructor  134  can stop all traffic in or out of a node or cluster (e.g., to reduce load on a node/cluster while attributes are changed). 
     In some embodiments, the constructor  134  may not desire to wait for data migration to finish because there are enough replicas and a node being replaced is unavailable anyway. In other instances, the constructor  134  may not wait for a backup of a node/cluster to finish because a very recent backup is available or making one is impossible in the current state. 
     In various embodiments, the constructor  134  can also dynamically allocate additional temporary compute resources such as CPU/power or memory to a node/cluster when a repair or upgrade is underway. 
     Referring yet again to  FIG. 1 , in some embodiments, the constructor  134  works in tandem with proxies  138  to control where traffic is sent during an attribute change. This ensures that a new node does not receive traffic until the node is ready and checked by the constructor as consistent. The constructor  134  can stop sending traffic to an old node right before it is stopped and/or removed. 
       FIG. 4  is a flowchart of an example method that is performed in accordance with embodiments of the present disclosure. For context, a system that performs the method of  FIG. 4  includes clusters of nodes providing services, a plurality of management servers that each include a distributed coordination service, a director, and a constructor. 
     The method comprises a step  402  of providing clusters of nodes providing services. Again, this can include providing nodes in groups, where the nodes provide services such as Elasticsearch™, and so forth. 
     Next, the method includes a step  404  of providing a plurality of management servers. In some embodiments, each of the plurality of management servers comprises at least a distributed coordination service for the clusters of nodes. Again, the distributed coordination service is a datastore. 
     It will be understood that steps  402  and  404  can be executed in any order or could also be performed in parallel. 
     In various embodiments, the method includes a step  406  of managing access to the distributed coordination service through a director. Also, the method includes an optional step of promoting at least one node of the clusters to being one of the plurality of management servers. As noted above, promoting of a node can include replicating the distributed coordination service thereon. 
     In accordance with the present disclosure, the method can also include a step  408  of managing allocation and life cycle deployments of the one or more nodes using a constructor. In some embodiments, the method includes the constructor being configured to perform a step  410  of implementing rolling attribute changes for the clusters of the nodes. Rolling attribute changes can define a processing order in which the nodes of the clusters are modified. 
     As noted above, a full cluster restart is performed when the processing order is identical for each node in one of the cluster of nodes. In some embodiments, the processing order allows attribute changes to each node in one of the cluster of nodes in a sequential manner. In one or more embodiments, the processing order allows attribute changes to availability zones or deployment groups of the cluster of nodes. 
     As noted above, at least a portion of the cluster of nodes comprises master nodes. These master nodes control other subordinate nodes in a cluster. In these instances, the processing order specifies that master nodes have attribute changes performed last. 
     In some embodiments, the rolling attribute changes are executed upon any of a failure of any of the nodes in a cluster and infrastructure changes or customer request. 
       FIG. 5  is a diagrammatic representation of an example machine in the form of a computer system  1 , within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In various example embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a portable music player (e.g., a portable hard drive audio device such as an Moving Picture Experts Group Audio Layer 3 (MP3) player), a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. 
     The example computer system  1  includes a processor or multiple processor(s)  5  (e.g., a CPU, a graphics processing unit (GPU), or both), and a main memory  10  and static memory  15 , which communicate with each other via a bus  20 . The computer system  1  may further include a video display  35  (e.g., a liquid crystal display (LCD)). The computer system  1  may also include input device(s)  30  (also referred to as alpha-numeric input device(s), e.g., a keyboard), a cursor control device (e.g., a mouse), a voice recognition or biometric verification unit (not shown), a drive unit  37  (also referred to as disk drive unit), a signal generation device  40  (e.g., a speaker), and a network interface device  45 . The computer system  1  may further include a data encryption module (not shown) to encrypt data. 
     The drive unit  37  includes a machine-readable medium  50  (which may be a computer readable medium) on which is stored one or more sets of instructions and data structures (e.g., instructions  55 ) embodying or utilizing any one or more of the methodologies or functions described herein. The instructions  55  may also reside, completely or at least partially, within the main memory  10  and/or within the processor(s)  5  during execution thereof by the computer system  1 . The main memory  10  and the processor(s)  5  may also constitute machine-readable media. 
     The instructions  55  may further be transmitted or received over a network  520  via the network interface device  45  utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP)). While the machine-readable medium  50  is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals. Such media may also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, random access memory (RAM), read only memory (ROM), and the like. The example embodiments described herein may be implemented in an operating environment comprising software installed on a computer, in hardware, or in a combination of software and hardware. 
     One skilled in the art will recognize that the Internet service may be configured to provide Internet access to one or more computing devices that are coupled to the Internet service, and that the computing devices may include one or more processors, buses, memory devices, display devices, input/output devices, and the like. Furthermore, those skilled in the art may appreciate that the Internet service may be coupled to one or more databases, repositories, servers, and the like, which may be utilized in order to implement any of the embodiments of the disclosure as described herein. 
     The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present technology has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the present technology in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present technology. Exemplary embodiments were chosen and described in order to best explain the principles of the present technology and its practical application, and to enable others of ordinary skill in the art to understand the present technology for various embodiments with various modifications as are suited to the particular use contemplated. 
     Aspects of the present technology are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present technology. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. 
     These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks. 
     The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. 
     The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present technology. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. 
     In this detailed description, for purposes of explanation and not limitation, specific details are set forth, such as particular embodiments, procedures, techniques, etc. in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. 
     Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment,” “in an embodiment,” or “according to one embodiment” (or other phrases having similar import) at various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Furthermore, depending on the context of discussion herein, a singular term may include its plural forms and a plural term may include its singular form. Similarly, a hyphenated term (e.g., “on-demand”) may be occasionally interchangeably used with its non-hyphenated version (e.g., “on demand”), a capitalized entry (e.g., “Software”) may be interchangeably used with its non-capitalized version (e.g., “software”), a plural term may be indicated with or without an apostrophe (e.g., PE&#39;s or PEs), and an italicized term (e.g., “N+1”) may be interchangeably used with its non-italicized version (e.g., “N+1”). Such occasional interchangeable uses shall not be considered inconsistent with each other. 
     Also, some embodiments may be described in terms of “means for” performing a task or set of tasks. It will be understood that a “means for” may be expressed herein in terms of a structure, such as a processor, a memory, an input/output (I/O) device such as a camera, or combinations thereof. Alternatively, the “means for” may include an algorithm that is descriptive of a function or method step, while in yet other embodiments, the “means for” is expressed in terms of a mathematical formula, prose, or as a flow chart or signal diagram. 
     The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. 
     It is noted at the outset that the terms “coupled,” “connected,” “connecting,” “electrically connected,” and so forth, are used interchangeably herein to generally refer to the condition of being electrically/electronically connected. Similarly, a first entity is considered to be in “communication” with a second entity (or entities) when the first entity electrically sends and/or receives (whether through wireline or wireless means) information signals (whether containing data information or non-data/control information) to the second entity regardless of the type (analog or digital) of those signals. It is further noted that various figures (including component diagrams) shown and discussed herein are for illustrative purpose only, and are not drawn to scale. 
     While specific embodiments of, and examples for, the system are described above for illustrative purposes, various equivalent modifications are possible within the scope of the system, as those skilled in the relevant art will recognize. For example, while processes or steps are presented in a given order, alternative embodiments may perform routines having steps in a different order, and some processes or steps may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or sub-combinations. Each of these processes or steps may be implemented in a variety of different ways. Also, while processes or steps are at times shown as being performed in series, these processes or steps may instead be performed in parallel, or may be performed at different times. 
     While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. The descriptions are not intended to limit the scope of the invention to the particular forms set forth herein. To the contrary, the present descriptions are intended to cover such alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims and otherwise appreciated by one of ordinary skill in the art. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments.