Fair decentralized throttling in distributed cloud-based systems

A cloud availability manager configured to execute a recovery workflow that fails over one or more virtual machines (VMs) to and from a cloud computing system. In doing so, the cloud availability manager typically performs multiple operations for each VMs. The operations involve making several application programming interface (API) calls to component APIs of management components within the cloud computing system. To avoid bringing down the entire cloud infrastructure, the cloud availability manager throttles the API calls to other components while executing a recovery workflow. The throttling spans multiple instances (nodes) of the cloud availability manager and involves cooperation from the other management components to ensure the throttling is fair across all tenants of the cloud computing system.

BACKGROUND

Cloud architectures are used in cloud computing and cloud storage systems for offering infrastructure-as-a-service (IaaS) cloud services. Examples of cloud architectures include the VMware vCloud Director® cloud architecture software, Amazon EC2™ web service, and OpenStack™ open source cloud computing service. IaaS cloud service is a type of cloud service that provides access to physical and/or virtual resources in a cloud environment. These services provide a tenant application programming interface (API) that supports operations for manipulating IaaS constructs, such as virtual machines (VMs) and logical networks.

SUMMARY

One or more embodiments provide techniques for managing a virtualized computing instance. The method includes registering with a central application programming interface (API) gateway a plurality of component APIs of management components within a cloud computing system. The plurality of component APIs includes a first component API. The method further includes, generating, for each tenant of the cloud computing system, a first queue for synchronous requests to the plurality of component APIs and a second queue for asynchronous requests to the plurality of component APIs. The method includes receiving, at the central API gateway, an API request to a virtualized computing instance in the cloud computing system. The request is associated with a first tenant. The method includes forwarding the request to the first component API based on a state of the first queue and the second queue associated with the first tenant.

Further embodiments include a non-transitory computer-readable storage medium comprising instructions that cause a computer system to carry out the above method above, as well as a computer system configured to carry out the above method.

DETAILED DESCRIPTION

Embodiments of the present disclosure provide a cloud availability manager configured to execute a recovery workflow that fails over one or more virtual machines (VMs) to and from a cloud computing system. In doing so, the cloud availability manager typically performs multiple operations for each VM. The operations involve making several application programming interface (API) calls to component APIs of management components within the cloud computing system. To avoid bringing down the entire cloud infrastructure, the cloud availability manager is configured to throttle the API calls to other components while executing a recovery workflow. The throttling spans multiple instances (nodes) of the cloud availability manager and involves cooperation from the other management components to ensure the throttling is fair across all tenants of the cloud computing system. Embodiments of the present disclosure provide a throttling scheme that achieves fairness across tenants without any cross-node or cross-component locking (e.g., to count API requests in the system).

FIG. 1is a block diagram of a hybrid cloud computing system100in which one or more embodiments of the present disclosure may be utilized. Hybrid cloud computing system100includes a first virtualized computing system implementing an on-premise datacenter102communicatively coupled, via a network140, to a second virtualized computing system implementing a cloud computing system150. Hybrid cloud computing system100is configured to provide a common platform for managing and executing virtual workloads seamlessly between on-premise datacenter102and cloud computing system150. In one embodiment, on-premise datacenter102may be a data center controlled and administrated by a particular enterprise or business organization, while cloud computing system150may be operated by a cloud computing service provider and exposed as a service available to account holders, such as the particular enterprise in addition to other enterprises. As such, on-premise datacenter102may sometimes be referred to as a “private” cloud, and cloud computing system150may be referred to as a “public” cloud.

As used herein, an internal cloud or “private” cloud is a cloud in which a tenant and a cloud service provider are part of the same organization, while an external or “public” cloud is a cloud that is provided by an organization that is separate from a tenant that accesses the external cloud. For example, the tenant may be part of an enterprise, and the external cloud may be part of a cloud service provider that is separate from the enterprise of the tenant and that provides cloud services to different enterprises and/or individuals. In embodiments disclosed herein, a hybrid cloud is a cloud architecture in which a tenant is provided with seamless access to both private cloud resources and public cloud resources.

On-premise datacenter102includes one or more host computer systems (“hosts104”). Hosts104may be constructed on a server grade hardware platform106, such as an x86 architecture platform. As shown, hardware platform106of each host104may include conventional components of a computing device, such as one or more processors (CPUs)108, system memory110, a network interface112, storage system114, and other I/O devices such as, for example, a mouse and keyboard (not shown). CPU108is configured to execute instructions, for example, executable instructions that perform one or more operations described herein and may be stored in memory110and in local storage. Memory110is a device allowing information, such as executable instructions, cryptographic keys, virtual disks, configurations, and other data, to be stored and retrieved. Memory110may include, for example, one or more random access memory (RAM) modules. Network interface112enables host104to communicate with another device via a communication medium, such as a network interconnecting hosts104within on-premise datacenter102. Network interface112may be one or more network adapters, also referred to as a Network Interface Card (NIC). Storage system114represents local storage devices (e.g., one or more hard disks, flash memory modules, solid state disks, and optical disks) and/or a storage interface that enables host104to communicate with one or more network data storage systems. Examples of a storage interface are a host bus adapter (HBA) that couples host104to one or more storage arrays, such as a storage area network (SAN) or a network-attached storage (NAS), as well as other network data storage systems.

Each host104is configured to provide a virtualization layer that abstracts processor, memory, storage, and networking resources of hardware platform106into multiple virtual machines1201to120N(collectively referred to as VMs120) that run concurrently on the same hosts. VMs120run on top of a software interface layer, referred to herein as a hypervisor116, that enables sharing of the hardware resources of host104by VMs120. One example of hypervisor116that may be used in an embodiment described herein is a VMware ESXi™ hypervisor provided as part of the VMware vSphere® solution made commercially available from VMware, Inc. of Palo Alto, Calif. Hypervisor116may run on top of the operating system of host104or directly on hardware components of host104.

On-premise datacenter102includes a virtualization management component (depicted inFIG. 1as virtualization manager130) that may communicate to the plurality of hosts104via a network, sometimes referred to as a management network. In one embodiment, virtualization manager130is a computer program that resides and executes in a central server, which may reside in on-premise datacenter102, or alternatively, running as a VM in one of hosts104. One example of a virtualization manager is the vCenter Server™ product made available from VMware, Inc. Virtualization manager130is configured to carry out administrative tasks for computing system102, including managing hosts104, managing VMs120running within each host104, provisioning VMs, migrating VMs from one host to another host, and load balancing between hosts104.

In one embodiment, virtualization manager130includes a hybrid cloud management module (depicted as hybrid cloud manager132) configured to manage and integrate virtualized computing resources provided by cloud computing system150with virtualized computing resources of computing system102to form a unified “hybrid” computing platform. Hybrid cloud manager132is configured to deploy VMs in cloud computing system150, transfer VMs from virtualized computing system102to cloud computing system150, and perform other “cross-cloud” administrative tasks, as described in greater detail later. In one implementation, hybrid cloud manager132is a module or plug-in complement to virtualization manager130, although other implementations may be used, such as a separate computer program executing in a central server or running in a VM in one of hosts104. One example of hybrid cloud manager132is the VMware vCloud Connector® product made available from VMware, Inc.

In one or more embodiments, cloud computing system150is configured to dynamically provide an enterprise (or users of an enterprise) with one or more virtual data centers in which a user may provision VMs164, deploy multi-tier applications on VMs164, and/or execute workloads. Cloud computing system150includes an infrastructure platform154upon which a cloud computing environment may be executed. In the particular embodiment ofFIG. 1, infrastructure platform154includes hardware resources having computing resources (e.g., hosts162), storage resources (e.g., storage array systems), and networking resources, which are configured in a manner to provide a virtualization environment156that supports the execution of a plurality of virtual machines164across hosts162. It is recognized that hardware resources160of cloud computing system150may in fact be distributed across multiple data centers in different locations.

Each cloud computing environment is associated with a particular tenant of cloud computing system150, such as the enterprise providing virtualized computing system102. In one embodiment, the cloud computing environment is configured as a dedicated cloud service for a single tenant comprised of dedicated hardware resources (i.e., physically isolated from hardware resources used by other users of cloud computing system150). In other embodiments, the cloud computing environment is configured as part of a multi-tenant cloud service with logically isolated virtualized computing resources on a shared physical infrastructure. Cloud computing system150may support multiple cloud computing environments, available to multiple enterprises in single-tenant and multi-tenant configurations.

In one embodiment, virtualization environment156includes an orchestration component158(e.g., implemented as a process running in a VM) that provides infrastructure resources to a cloud computing environment responsive to provisioning requests. For example, if a tenant required a specified number of virtual machines to deploy a web application or to modify (e.g., scale) a currently running web application to support peak demands, orchestration component158can initiate and manage the instantiation of virtual machines (e.g., VMs164) on hosts162to support such requests. In one embodiment, orchestration component158instantiates virtual machines according to a requested template that defines one or more virtual machines having specified virtual computing resources (e.g., compute, networking, storage resources). Further, orchestration component158monitors the infrastructure resource consumption levels and requirements of the cloud computing environment and provides additional infrastructure resources to the cloud computing environment as needed or desired. In one example, similar to on-premise datacenter102, virtualization environment156may be implemented by running on hosts162VMware ESXi™-based hypervisor technologies provided by VMware, Inc. (although it should be recognized that any other virtualization technologies, including Xen® and Microsoft Hyper-V® virtualization technologies may be utilized consistent with the teachings herein).

In one embodiment, cloud computing system150may include a cloud director152(e.g., run in one or more virtual machines) that manages allocation of virtual computing resources to a tenant for deploying applications. Cloud director152may be accessible to users via a REST (Representational State Transfer) API (Application Programming Interface) or any other client-server communication protocol. Cloud director152may authenticate connection attempts from the tenant using credentials issued by the cloud computing provider. Cloud director152maintains and publishes a catalog166of available virtual machine templates and packaged virtual machine applications that represent virtual machines that may be provisioned in a cloud computing environment. A virtual machine template is a virtual machine image that is loaded with a pre-installed guest operating system, applications, and data, and is typically used to repeatedly create a VM having the pre-defined configuration. A packaged virtual machine application is a logical container of pre-configured virtual machines having software components and parameters that define operational details of the packaged application. An example of a packaged VM application is vApp Template technology made available by VMware, Inc., although other technologies may be utilized. Cloud director152receives provisioning requests submitted (e.g., via REST API calls) and may propagate such requests to orchestration component158to instantiate the requested virtual machines (e.g., VMs164). One example of cloud director152is the VMware vCloud Director® produced by VMware, Inc.

In the embodiment ofFIG. 1, the cloud computing environment supports the creation of a virtual data center having a plurality of virtual machines164instantiated to, for example, host deployed multi-tier applications. A virtual data center is a logical construct that provides compute, network, and storage resources to an organization. Virtual data centers provide an environment where VMs164can be created, stored, and operated, enabling complete abstraction between the consumption of infrastructure service and underlying resources. VMs164may be configured similarly to VMs120, as abstractions of processor, memory, storage, and networking resources of hosts162and other hardware resources.

In one or more embodiments, cloud computing system150includes a cloud availability manager165configured to orchestrate disaster recovery workflows from the on-premise data center to cloud computing system150and back. That is, cloud availability manager165enables tenants to use cloud computing system150as disaster recovery site. In some embodiments, cloud availability manager165uses host-based replication technology to replicate running VMs to and from cloud computing system150, although in other embodiments, array-based data replication may be used as well. An example cloud availability manager165configured to perform the described operations is the Site Recovery Manager Air™ made available by VMware, Inc. In one embodiment, cloud availability manager165uses a separate service (replication service170) configured to perform host-based replication on infrastructure platform154. An example host-based replication techniques may be provided by vSphere™ Replication technology made available by VMware, Inc.

In one scenario, a tenant user specifies recovery plans and configures replication to cloud computing system150for a number of VMs that need disaster recovery protection. A recovery plan refers to a collection of VMs replicated in the same direction (e.g., from on-premise to cloud). The user can configure dependencies between VMs, and for each VM specify Internet Protocol (IP) customization parameters to be used during a failover, assign scripts to each VM, and so forth. Once the recovery plan is configured, the user can run various recovery workflows for the recovery plan (e.g., planned failover, forced failover, test failover, etc.) During the execution of a workflow for a recovery plan, cloud availability manager165can perform all steps for each VM with as much parallelism as allowed by the constraints configured by the user (i.e., VM dependencies, priority tiers). Executing a workflow typically involves making multiple API calls (e.g., REST API calls) to a component API168(hereinafter simply referred to as an API) of cloud director152or to an API172of a replication service170. Cloud director152and replication service170are examples of management components or services within cloud computing system150, and other management components and services are generically depicted inFIG. 1as other management components176having APIs178within cloud computing system150.

A typical recovery plan workflow may contain five to ten tasks to be performed for each VM. For example, one recovery plan may include one task to synchronize the state of a VM with a recovery VM, another task to power off the original VM, another task to re-synchronize the state again, a task to fail over to the recovery VM, a task to customize the recovery VM, and a task to power on the recovery VM. Consequently, a recovery workflow for a recovery plan with 1,000 VMs would contain 5,000-10,000 tasks. Again, to execute each task, cloud availability manager165would have to make several REST API calls to the management components of cloud computing system150(e.g., cloud director152, replication service170, etc.)

In some embodiments, component APIs are implemented as synchronous calls, i.e., the caller is blocked while the management component is processing the call. Synchronous calls are generally able to be processed and completed by the server relatively quickly. In contrast, APIs that have a substantial processing time are implemented as asynchronous calls. For asynchronous APIs, when a client makes a call, the management component first creates a temporary tracking object referred herein to as a “task,” then starts processing the call. It is noted that the management component completes the call without waiting for the processing to complete. Instead, the management component returns the task back to the client. The client starts polling on the task properties such as progress and completion status while the management component is processing the request and updates these properties. Once the management component completes processing the request, the management component marks the task as completed and assigns the results of the processing to the task. When the client notices that the task has been marked as completed, the client retrieves the results of the operation (if any) from the task object.

If cloud availability manager165were to execute a recovery workflow without applying any throttling, the result might be thousands or even millions of REST API requests being made to cloud director152and replication service170(which might itself make calls to the API of cloud director152). This cascading behavior may bring the entire cloud infrastructure to a halt, making cloud computing system150inaccessible to all other clients. One approach to this problem may be to implement request throttling in each server or management component However, in this approach, if the number of requests in the queue is large, the management component may take a significant amount of time even to start processing a request from the queue. By that time, a client may have already timed out of the request from the client-side. Increasing client side request timeouts would not help either. Setting the timeouts to be longer in duration (to accommodate potentially large throttling queues) can result in the client taking too much time to detect if the other components have legitimately failed and report the problem to the user. For example, if a client were to use a 24-hour request timeout to give enough time for other management components to process requests, 24 hours may elapse before the client is able to detect that the other component has failed. Such long delays may be unacceptable in disaster recovery scenarios in which a user is trying to recover from a disaster as quickly as possible.

Furthermore, in some embodiments, each management component within cloud computing system150is implemented in a multi-node cluster. For example, multiple instances of cloud director152are depicted inFIG. 1. In order to achieve a balanced load on a cluster of management components, a successful throttling scheme has to be applied in a load balancer that fronts the cluster and distributes requests to each node in the cluster. Load balancers are typically implemented as generic components without knowledge about the contents of each request. Thus, a load balancer may have difficulty in understanding which requests are easy and quick to process and which request may take a significant amount of time to process. While one simple approach may be to maintain a counter at the load balancer that counts how many requests each node is processing to determining to which node to send a new request. However, under this approach, the load balancer may overload one node by sending that node too many heavy requests.

Asynchronous (i.e., task-based) requests can present a particular problem for throttling. Typically, a management component needs to return a task object to a client as soon as possible to allow the client to track the progress of the request. After that, in some approaches, the management component places the task into a queue. At this point, the task is said to be in a queued state. The management component then starts processing tasks from the queue. Once the management component starts processing a task, the management component marks the task as running and starts reporting progress for the task. In some approaches, there can be different throttling limits for the number of queued tasks and the number of running tasks for each management component. If the management component is overloaded, tasks may stay in the queued state for such a long time that the client may time out the tasks on the client's side before the management component even starts processing the task. As described above, simply using longer timeout durations for tasks may not be a tenable solution because of the effect on detecting component failures in a timely manner.

Another approach is to throttle outgoing requests on the client-side of cloud availability manager165. If cloud availability manager165is implemented as a multi-node cluster, then the throttling would require some form of a distributed request queue spanning all nodes in the cluster, possibly needing some form of distributed locking or a transactionally consistent database, which are inefficient from a performance point of view. Further, client side throttling may suffer from fairness problems across tenants of cloud computing system150. Using a client side throttling scheme, cloud availability manager165may not know how many requests are being processed for each tenant by other management components.

Accordingly, in one or more embodiments, cloud computing system150includes a central API gateway180configured to maintain a certain allotment of REST API calls that each tenant of cloud computing system150is allowed to make against the cloud infrastructure. Central API gateway180maintains the allotments between requests initiated by management components themselves and requests initiated directly by a tenant client. In some embodiments, central API gateway180interacts with components APIs of each management component within cloud computing system150, such as API168of cloud director152, API167of cloud availability manager165, API172of replication service170, API(s)178of management components176, as well as APIs (not shown) of infrastructure platform154and orchestration component158.

In the described use case, the throttling scheme prevents one tenant from interfering with other tenants in a cloud computing system. However, while embodiments of the present disclosure are described in the context of virtualized computing instances and a cloud computing system, it should be noted that the described techniques may be used to throttle API calls across any multiple components that are not tightly coupled with each other. That is, a collection of services which provide some operations to come clients, it can be desirable to make the services reliable enough to withstand attacks from clients that try to overload the system. In one example, parts of a same data center may employ the described throttling scheme to guard against other parts of the same center misbehaving and overloading the system. In other words, in one alternative embodiment, the system150does not have to be a fully independent cloud computing system, but can be a private data center that owns the systems102. In yet other embodiments, systems150and102can be parts of the same data center controlled by the same organization.

FIG. 2Ais a sequence diagram for operations200of a throttling scheme that manages requests in a cloud computing system150, according to an embodiment. WhileFIG. 2Ais described in conjunction with the systems ofFIG. 1, it should be understood that embodiments of the present disclosure may be performed using other similar systems and structures.

Operations200begin at202, where central API gateway180generates and maintains a first queue182for synchronous requests and a second queue184for asynchronous requests for each tenant of cloud computing system150. In some embodiments, synchronous queue182may be made of a sub-queue for queued tasks for synchronous requests and another sub-queue for running tasks for synchronous requests for a given tenant. Similarly, asynchronous queue184may be comprised of one sub-queue for queued tasks for asynchronous requests and another sub-queue for running takes for asynchronous requests for a given tenant. Queues182,184may be considered “global” as each queue182,184pertains to all requests associated with a given tenant within cloud computing system150regardless of which management component the request is for.

In some embodiments, the size of each queue182,184is controlled by tenant-specific limits. Central API gateway180determines the size of queues182,184based on how many synchronous requests can be processed at the same time, how many synchronous requests can be queued at the same time, how many asynchronous requests (i.e., tasks) can be queued at the same time, and how many asynchronous requests can be running at the same time. In some embodiments, the limits for queued and running tasks can be implemented as a global per-tenant limit. For example, central API gateway180may set for a particular tenant a limit of 1,000 queued synchronous requests and 100 running synchronous requests. In some embodiments, the limits for queued and running requests can be implemented as API specific per-tenant limits, which enables cloud computing system150to implement different throttling limits for time-consuming and time-light requests (synchronous and asynchronous).

At204, each management component within cloud computing system150(e.g., cloud director152, cloud availability manager165, etc.) registers some or all of its component APIs with central API gateway180. For each REST API, the management component may specify whether a request to the component API is a synchronous request or an asynchronous (i.e., task-based) request. The registrations for components APIs may further specify a type of the API request.

In one implementation, a management component may provide a resource address or path (e.g., URL) for a component API, a REST method (e.g., GET, PUT, POST, etc.), an indication of whether the request is synchronous or asynchronous, a type of API, and one or more network address for the cluster of management components. For example, cloud availability manager165registers an API at the URL “/cam/rp/{id}”, which has a GET request, which is a synchronous request, and which may be found at the network addresses “IP1”, “IP2”, and “IP3”.

It is noted that operations202and204may be preferred when the components of cloud computing system150start. That is, operations202and204may be performed once upon initialization and startup of cloud computing system150, while subsequent operations (operations206and onward) inFIG. 2A-Bmay be repeated, at least once for each REST API request received.

At206, central API gateway180receives a request from a client associated with a particular tenant (“tenant client”). Responsive to receiving a request to create a task, central API gateway180requests creation of a task object205corresponding to the request from the management component and returns a task ID back to the tenant client. For synchronous requests, central API gateway180may proceed directly to block208.

At blocks208, central API gateway180may schedule and execute the request based on a state of the queues182,184associated with the tenant. At block210, at the scheduled time, central API gateway180directs the management component to the executed the particular request, passing parameters and a task identifier. The management component communicates with task205to update the progress of the task and set the completed status and return results. Details for describing when the request is processed and handled are described in greater detail in conjunction withFIGS. 3A and 3B.

FIG. 2Bis a sequence diagram for operations250of a throttling scheme that manages requests in a cloud computing system150, according to an embodiment. Operations250are a variation of the operations depicted inFIG. 2B, except that central API gateway180first makes a validation request, at block252, to the management component. Central API gateway180receives a task type and other information about the request from the management component, and then creates a task object. Operations250may continue as depicted inFIG. 2A.

FIGS. 3A and 3Bare flow diagrams illustrating a method300for throttling API requests in a cloud computing system, according to one embodiment of the present disclosure. While method300is described in conjunction with the systems shown inFIG. 1, it should be recognized that other systems may be used to perform the described methods.

Method300begins at step302, where central API gateway180receives an API request associated with a particular tenant. For example, the tenant client may be a hybrid cloud manager132or virtualization manager130of an on-premise data center. In other cases, the tenant client may be a management component within cloud computing system150acting at the direction of a particular tenant. Such cases might arise where one management component requests the actions of other management components within cloud computing system150. Nonetheless, these requests still get funneled through central API gateway180to ensure fair throttling is applied to all request activity in cloud computing system150. In some embodiments, when a management component makes a call to another management component (via central API gateway180), the calling management component makes the call within the context of the tenant to place the call into the appropriate queue. In other embodiments, the API request may be received from other sources besides a tenant client, for example, the tenant may have a script performing tasks through the API.

At step304, central API gateway180determines whether the request from the tenant client is a synchronous request or an asynchronous request. In some embodiments, central API gateway180performs a lookup of registered component APIs (as described inFIG. 2A-Babove), for any APIs matching received request. Central API gateway180may then determine whether the component API is registered as a synchronous request or an asynchronous request.

At step306, central API gateway180checks whether the number of outstanding (i.e., currently running) synchronous requests is below the tenant-specific limit. If so, at step308, responsive to determining that a number of running tasks in queue182for synchronous requests associated with the particular tenant does not exceed a limit, central API gateway180forward the request to the API of the corresponding management component for processing. The term “tenant-specific limit” as used herein refers to both limits associated with a particular tenant and limits associated with a particular type of request, and both.

Otherwise, at step310, central API gateway180checks whether the number of queued synchronous requests is below the tenant-specific limit. If so, at step312, responsive to determining that a number of queued tasks in queue182for synchronous requests associated with the particular tenant does not exceed the limit associated with the tenant, central API gateway180places the request into the synchronous request queue182. Once the number of outstanding requests synchronous requests drops below the limit (subsequent to being placed in the queue182), central API gateway180may start sending queued synchronous requests to the corresponding management component for processing.

At step314, responsive to determining that the number of queued tasks has exceeded the tenant-specific limit, central API gateway180rejects the request from the tenant. In some embodiments, central API gateway180returns a response message indicating an “Service Unavailable” error has occurred. In one implementation, central API gateway may use an HTTP status code (e.g., “503”) indicating the corresponding management component is unavailable, and further indicating that the tenant has exceeds the number of requests assigned to that tenant.

Referring back to step304, responsive to determining that the received request is an asynchronous request, central API gateway180may proceed with method350depicted inFIG. 3B. Method350begins at step351, where central API gateway180optionally makes a validation call to the management component and retrieves task type form the management component. At step352, central API gateway180checks whether the number of running tasks is below the tenant-specific limit. If so, at step354, responsive to determining that the number of running tasks in queue184for asynchronous requests does not exceed the limit associated with the tenant, central API gateway180generates a task object for the request. The task object is marked such that the object indicates that the task is running and is assigned a task ID. Central API gateway180immediately transmits the task object in a response back to the tenant client. At step356, central API gateway180proceeds to forward the asynchronous request to the component API of the corresponding management component for processing. In some embodiments, central API gateway180passes the ID of the task object to the management component, allowing the management component to update the task with progress and results. The term “tenant-specific limit” as used herein refers to both limits associated with a particular tenant and limits associated with a particular type of request, and both.

At step358, responsive to determining that the number of running tasks has reached a tenant-specific limit, central API gateway180checks whether the number of queued tasks has reached the tenant-specific limit. If so, at step360, responsive to determining that the number of queued tasks is below the tenant-specific limit, central API gateway180still generates a task object for the request, except this time the task object is marked such that the object indicates that the task is queued. Central API gateway180returns the task object (having a task identifier) in a response back to the tenant client. At step362, central API gateway180places the asynchronous request in queue184for asynchronous requests associated with the tenant. Subsequent to placing this request in the asynchronous queue, central API gateway180may continue to monitor the queue and once the number of running tasks drops below the tenant-specific limit, central API gateway180examines queue184for pending asynchronous requests and forwards the request for which the per-task type limits are satisfied to the component API of the corresponding management component, together with the corresponding task ID.

At step362, responsive to determining that number of queued tasks associated with the tenant exceeds the tenant-specific limit, central API gateway180rejects the request from the tenant client. In some embodiments, central API gateway180returns a “Service Unavailable” error response (e.g., HTTP status code503) to the tenant client indicating that the management request has been rejected due to exceeding the tenant-specific limit.

In one or more embodiments, tenant clients and management components176of cloud computing system150are configured to receive a “Service Unavailable” error message for any request. Each management component176may be configured to wait a predefined period of time and retry the request. As discussed above, the management component may safely retry the request as it is determined that the original request has been rejected fully (as opposed to being processed or queued for an inordinate amount of time.)

FIG. 4is a block diagram depicting an example of a computer system400in which one or more embodiments of the present disclosure may be utilized. Computer system400can be used as a host to implement hybrid cloud manager132, central API gateway180, or other component described above. Computer system400includes one or more central processing units (CPUs)402, memory404, input/output (JO) circuits406, and various support circuits408. Each of CPUs402can include any microprocessor known in the art and can execute instructions stored on computer readable storage, such as memory404. Memory404can include various volatile and/or non-volatile memory devices, such as random access memory (RAM), read only memory (ROM), and the like. Instructions and data410for performing the various methods and techniques described above can be stored in memory404for execution by CPUs402. That is, memory404can store instructions executable by CPUs402to perform one or more steps/sub-steps described above inFIGS. 3 and 4. Support circuits408include various circuits used to support operation of a computer system as known in the art.