Source: https://patents.google.com/patent/EP1654649B1/en
Timestamp: 2019-11-13 06:00:04
Document Index: 799993420

Matched Legal Cases: ['Application No. 60', 'Application No. 60', 'Application No. 60', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No.10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No.10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10']

EP1654649B1 - On demand node and server instance allocation and de-allocation - Google Patents
EP1654649B1
EP1654649B1 EP04781141.9A EP04781141A EP1654649B1 EP 1654649 B1 EP1654649 B1 EP 1654649B1 EP 04781141 A EP04781141 A EP 04781141A EP 1654649 B1 EP1654649 B1 EP 1654649B1
EP04781141.9A
EP1654649A2 (en
2004-08-13 Priority to PCT/US2004/026405 priority patent/WO2005017745A2/en
2006-05-10 Publication of EP1654649A2 publication Critical patent/EP1654649A2/en
2015-12-23 Publication of EP1654649B1 publication Critical patent/EP1654649B1/en
The present application claims priority to U.S. Provisional Application No. 60/495,368 , Computer Resource Provisioning, filed on August 14, 2003; the present application claims priority to U.S. Provisional Application No. 60/500,096 , Service Based Workload Management and Measurement in a Distributed System, filed on September 3, 2003; the present application claims priority to U.S. Provisional Application No. 60/500,050 , Automatic And Dynamic Provisioning Of Databases, filed on September 3, 2003.
U.S. Application No. 10/917,687 , On Demand Node and Server Instance Allocation and De-Allocation (Attorney Docket No. 50277-2413), filed by Lakshminarayan Chidambaran, et al. on August 12, 2004;
U.S. Application No. 10/917,873 , Hierarchical Management of the Dynamic Allocation of Resources in a Multi-Node System (Attorney Docket No. 50277-2382), filed by Benny Souder, et al. on August 12, 2004;
U.S. Application No. 10/917,661 ,Calculation of Service Performance Grades in a Multi - Node Environment That Hosts the Services (Attorney Docket No. 50277-2410), filed by Lakshminarayanan Chidambaran, et al. on August 12, 2004;
U.S. Application No. 10/918,055 , Incremental Run-Time Session Balancing in a Multi - Node System (Attorney Docket No. 50277-2411) filed by Lakshminarayanan Chidambaran, et al. on August 12, 2004;
U.S. Application No.10/918,056 , Service Placement for Enforcing Performance and Availability Levels in a Multi-Node System (Attorney Docket No. 50277-2412), filed by Lakshminarayanan Chidambaran, et al. on August 12, 2004;
U.S. Application No. 10/918,054 Recoverable Asynchronous Message Driven Processing in a Multi-Node System (Attorney Docket No. 50277-2414), filed by Lakshminarayanan Chidambaran, et al. on August 12, 2004; and
U.S. Application No. 10/917,715 , Managing Workload by Service (Attorney Docket No. 50277-2337), filed by Carol Colrain, et al. on August 12, 2004.
International Application No. PCT/US2004/026389 , Hierarchical Management of the Dynamic Allocation of Resources in a Multi-Node System (Attorney Docket No. 50277-2592), filed on August 13, 2004 by Oracle International Corporation;
International Application No. PCT/US2004/026445 Transparent Session Migration Across Servers (Attorney Docket No. 50277-2593), filed on August 13, 2004 by Oracle International Corporation;
INTERNATIONAL Application No. PCT/US2004/026570 , Transparent Migration of Stateless Sessions Across Servers (Attorney Docket No. 50277-2594), filed on August 13, 2004 by Oracle International Corporation.
One such mechanism is the system described in U.S. Application No. 10/917,873 , Hierarchical Management of the Dynamic Allocation of Resources in a Multi-Node System (50277-2382), which uses a hierarchy of directors to manage resources at different levels. One type of director, a database director, manages resources allocated to a database among users of the database. For example, a grid may host a group of database servers for a database. Each database server in the group is referred to as a database instance. Each database instance hosts a number of database sessions for users and one or more services. The database director manages the allocation of resources available to a database among users and services.
WO 02/07037 A1 discloses a hosted service provider for the Internet that is operated so as to provide dynamic management of hosted services. A plurality of individual servers are allocated to a common administrative group. The system automatically monitors the performance and health of the servers in each administrative group. At least one server from a first administrative group is automatically and dynamically reallocated to a second administrative group in response to the automatic monitoring. Each administrative group includes a local decision software program that communicates with a master decision software program that determines when and how to dynamically reallocate servers to different administrative groups in response to usage demands, available resources and service level agreements for different customer accounts.
WO 03/014928 A2 discloses an ASP server farm in which requests to use an application are directed to a particular executing instance of the application, namely the least loaded of the available such instances. A global decision maker takes the current system state and the client service level agreements as input and generates a new resource allocation plan as its output. One of the inputs of the global decision maker is a number indicating the cost of starting a new instance or shutting down an existing instance.
A defining characteristic of a cluster in a cluster farm is that the cluster's nodes may be automatically transferred between clusters within the farm through software control without the need to physically reconnect the node from one cluster to another. A cluster farm is controlled and managed by software utilities referred to herein as clusterware. Clusterware may be executed to remove a node from a cluster and to provision the node to a cluster. Clusterware provides a command line interface that accepts requests from a human administrator, allowing the administrator to enter commands to provision and remove a node from a cluster. The interfaces may also take the form of Application Program Interfaces ("APIs"), which may be called by other software being executed within the cluster farm. Clusterware uses and maintains metadata that defines the configuration of a cluster within a farm, including cluster configuration metadata, which defines the topology of a cluster in a farm, including which particular nodes are in the cluster. The metadata is modified to reflect changes made to a cluster in a cluster farm by the clusterware. An example of clusterware is software developed by Oracle™, such as Oracle9i Real Application Clusters or Oracle Real Application Clusters 10g. Oracle9i Real Application Clusters is described in Oracle9i RAC: Oracle Real Application Clusters Configuration and Internals, by Mike Ault and Madhu Tumma, 2nd edition (August 2, 2003).
Resources from multiple nodes in a multi-node computer system can be allocated to running a particular server's software. Each combination of the software and allocation of the resources from a node is a server that is referred to herein as a "server instance" or "instance". Thus, a multi-node server comprises multiple server instances that can run on multiple nodes. Several instances of a multi-node server can even run on the same node. A multi-node database server comprises multiple "database instances", each database instance running on a node, and governing and facilitating access to a particular database. Database instances 123, 125, and 127 are instances of the same multi-node database server.
According to an embodiment of the present invention, a workload monitor is hosted on each database instance and generates "performance metrics". Performance metrics is data that indicates the level of performance for one or more resources or services based on performance measures. Approaches for performing these functions are described in U.S. Application No. 10/917,715 , Managing Workload by Service (50277-2337). The information generated is accessible by various components within multi-node database server 222 that are responsible for managing the allocation of resources to meet service-level agreements, as shall be described in greater detail later.
CPU aval This is the available unused CPU utilization on a node, which is defined as the difference between Xcpu and Ncpu (i.e. Xcpu - Ncpu).
A hierarchy of directors, such as that described in U.S Application No. 10/917,873 Hierarchical Management of the Dynamic Allocation of Resounces in a Multi-Node System (50277-2382), is used to dynamically adjust the allocation of resources within cluster farm 101 to meet service-level agreements. Cluster farm 101 includes a database director for each database managed by a database server on cluster farm 101, a cluster director for each cluster within cluster farm 101, and a farm director for cluster farm 101.
A database director, such as database director 152 and 162, dynamically manages and adjusts the allocation of resources of a database between services hosted by the database instances of the database. One measure a database director uses to perform this responsibility is to perform incremental runtime session balancing between the database instances of a database as described in U.S. Application No. 10/918,055 , Incremental Run-Time Session Balancing in a Multi-Node System (50277-2411). Incremental runtime session balancing migrates the database sessions of a service between databases instances hosting the service. Another measure that can be undertaken by a database director is service expansion. Under service expansion, another database instance is allocated to host a service, as described in U.S. Application No.10/918,056 , Service Placement for Enforcing Performance and Availability Levels in a Multi-Node System (50277-2412).
FIG. 2 shows a procedure that may be used to manage the escalation of remedies employed to resolve service-level violations. The procedure is initiated by a database director in response to detecting a resource alert for a service on a "source" database instance of a "source" database. A resource alert is the detection of a condition or event that triggers attention to resource allocation within a multi-node computer system. Resource alerts include, in particular, detection of service-level violations. For example, a resource alert can be detecting that the average transaction time for service FIN on source database instance 125 exceeds the service-level agreement for this measure.
Referring to FIG. 2, at step 210, the database director first determines whether to attempt run-time session balancing. This determination is made by determining whether there are any candidate services on the source database instance for which session balancing can be performed. If it is determined that session balancing should be performed, then at step 215 session balancing is performed, such as that described in U.S. Application No. 10/918,055 , Incremental Run-Time Session Balancing In A Multi-Node System (50277-2411).
If, at step 210, the determination is that run-time session balancing should not be attempted, then the procedure determines at step 220 whether service expansion should be attempted. This determination is made by determining whether there are any candidate services on a source database instance that can be expanded, as explained in greater detail in U.S. Application No. 10/918,056 Service Placement for Enforcing Performance and Availability Levels in a Multi-Node System (50277-2412). If the determination is that service expansion should be performed, then at step 225, service expansion is performed, and the procedure returns to step 205.
At step 230, the database director determines whether there is a pending "NEED-INSTANCE" request being processed for the database director by the cluster director. A NEED-INSTANCE request is a request for another database instance on another node. A pending NEED-INSTANCE request is a NEED-INSTANCE request that is currently being processed by, for example, the cluster director. Such processing can include (as described in later detail) finding a node for a database instance or removing a database instance from a node so that another database instance may be provisioned to the node. If there is a pending NEED-INSTANCE request, then at step 235 the database director waits and then proceeds to step 205. If the resource alert persists, then the database director performs a subsequent iteration. A new database instance may have been allocated to the database as a result of the pending NEED-INSTANCE request. In the subsequent iteration, the database director may expand a service to the newly allocated database instance and alleviate or remedy the resource alert.
While performing the various steps in the procedure, the database director delegates various actions to an asynchronous job and then waits for the results of various actions performed by the job. For example, the database director may be waiting for the result of an asynchronous job it issued to expand a service (see U.S. Application No. 10/918,056 Service Placement for Enforcing Performance and Availability Levels in a Multi-Node System (50277-2412)) or to migrate sessions (see U.S. Application No. 10/918,055 ,Incremental Run-Time Session Balancing in a Multi-Node System (50277-2411)). In addition, the database director may wait for the result of a NEED-INSTANCE request issued to a cluster director. While waiting in these cases, the database director does not in fact lay idle. Instead, it concurrently processes other resource alerts by performing the procedure depicted in FIG. 2.
FIG. 3 shows a procedure followed by a cluster director to initiate allocation of a node to a database in response to processing a NEED-INSTANCE request made by a "requesting database director."
If, at step 305, the cluster director determines that a node is not free, then the cluster director attempts to obtain a node that is allocated to another database ("database victim") within the cluster.
The cluster director may poll the database directors of the candidates in the victim candidate list one-by-one in order. To poll a database director of a candidate, the cluster director transmits a "QUIESCE-VOLUNTEER" request to the database director, which responds by transmitting a message specifying whether the database director volunteers to quiesce. If the polled candidate volunteers, the cluster director selects the candidate as the victim.
At step 335, the cluster director issues an asynchronous job. The job transmits to the victim's database director a "QUIESCE-INITIATE" request to request that the database director quiesce a database instance.
Subsequently, the cluster director receives an "IDLE-INSTANCE" message from the victim's database director, to which the cluster director responds by shutting down the quiesced instance. Meanwhile, the cluster director processes other requests, including other NEED-INSTANCE requests.
FIG. 4 shows a procedure followed by the cluster director to respond to an IDLE-INSTANCE request according to an embodiment of the present invention. At step 405, the cluster director receives an IDLE-INSTANCE message. The IDLE-INSTANCE message includes the identity of the "victim database instance" quiesced by the victim's database director and the node on which the victim database instance is hosted.
FIG. 5 shows a procedure for determining whether a database director may volunteer to quiesce a database instance for the database of the database director. The procedure is performed in response to a "QUIESCE-VOLUNTEER" request sent by a "requesting cluster director."
At step 610, the database director sends a "BLOCK-INSTANCE" message to the candidate victim's listener. The message instructs the listener to stop directing database connection requests to the selected candidate. Accordingly, the listener directs database connection requests for any service to another database instance hosting the service.
At step 615, the database director determines whether the database sessions are migratable, i.e., whether the database sessions can be migrated to another database instance. For example, a database session may not be migratable when the session state includes a file descriptor of an open file. The file descriptor contains information that is only valid for a session on the source instance but not the destination instance. Other ways of determining whether the database sessions can be migrated are described in U.S. Application No. 10/917,953 , Transparent Session Migration Across Servers (50277-2383). If the sessions are not migratable, an "UNBLOCK-INSTANCE" message is sent to the listener at step 630, informing the listener that it may direct connection requests to the candidate victim. Another candidate victim is selected at step 605.
At step 620, the sessions are migrated from the victim to the other database instances of the database. The sessions maybe migrated in a way similar to that described in U.S. Application No. 10/918,055 Incremental Run-Time Session Balancing in a Multi-Node System (50277-2411). As described therein, sessions are migrated to other database instances in response to a resource alert caused by a service-level violation. Sessions on the database instance for a service are migrated to other database instances until the resource alert no longer persists.
A method for managing resources in a multiple node system, the method comprising the steps of:
said multiple node system detecting a violation of a service-level agreement on a multi-node server, wherein said multi-node server includes a set of server instances (123, 125, 127, 133, 135, 137) that run on a first set of nodes (122, 124, 126, 132, 134, 136);
in response to detecting said violation, said multiple node system determining whether to resolve said violation by using a node (122, 124, 126, 132, 134, 136) that is not a member of said first set of nodes to host another server instance (123, 125, 127, 133, 135, 137) of said multi-node server; and
said multiple node system finding another node (122, 124, 126, 132, 134, 136) to use to host a second server instance (123, 125, 127, 133, 135, 137) of said multi-node server, said another node hosting a third server instance (123, 125, 127, 133, 135, 137) when found;
said multiple node system migrating (620) at least one session established for one or more clients on said third server instance to another server instance hosted on said multiple node system; and
said multiple node system starting said second server instance of said multi-node server on said another node.
The method of claim 1, wherein the step of attempting to resolve said violation further includes said multiple node system shutting down said third server instance before starting said second server instance.
The method of claim 1, wherein the step of finding another node further includes:
generating a set of candidates that correspond to another multi-node server hosted by another set of nodes (122, 124, 126, 132, 134, 136) different than said first set of nodes; and
the steps further include transmitting a request to a multi-node server corresponding to a candidate of said set of candidates to volunteer to quiesce a server instance on a node (122, 124, 126, 132, 134, 136); and
The method of claim 4, wherein the step of transmitting a request to a multi-node server includes broadcasting the request to the set of candidates.
The method of claim 4, wherein the step of selecting a victim from a candidate of a plurality of multi-node servers includes selecting a victim based on one or more factors that include at least one of:
The method of claim 2, wherein the step of attempting to resolve said violation includes:
transmitting a request to a second multi-node server to prepare a server instance (123, 125, 127, 133, 135, 137) to shut down; and
An apparatus, namely a multiple node system, that is adapted to perform an operation of managing resources, the apparatus comprising means for executing the steps recited in any of claims 1-8.
A computer readable medium containing program code that, when executed by one or more processors (704), performs the method recited in any of claims 1-8.
EP04781141.9A 2003-08-14 2004-08-13 On demand node and server instance allocation and de-allocation Active EP1654649B1 (en)
EP1654649A2 EP1654649A2 (en) 2006-05-10
EP1654649B1 true EP1654649B1 (en) 2015-12-23
JP (2) JP4970939B2 (en)
JP6094272B2 (en) 2013-03-06 2017-03-15 富士通株式会社 Management system, management method, management program, and management apparatus
JP6206496B2 (en) 2013-07-17 2017-10-04 富士通株式会社 Cluster system, control device, control method, control program, and computer-readable recording medium recording the program
2006-08-23 R17D Search report (correction)
Ref document number: 1086899
Ref document number: 602004048400