Patent ID: 11880726
Assignee: AMAZON TECHNOLOGIES, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 14:
15. A host computing device in a provider network, the host computing device comprising:
a set of one or more processors;
a main queue;
an overload queue;
a thread pool comprising a plurality of threads; and
a set of instructions which when executed cause the host to:
receive a request to execute a generative operation from another host computing device in the provider network;
execute the generative operation over a time period to yield an operation result;
send the operation result to the other host computing device in the provider network;
dequeue, during the period of time, each request task of a first set of request tasks spawned by executing the generative operation from the main queue;
send, during the period of time, each request task of the first set of request tasks to the thread pool for execution;
determine, during the period of time, a request cost index reflecting an amount of computing resources used to execute the first set of request tasks;
determine, during the period of time, based on the request cost index, to queue, in the overload queue, subsequent request tasks spawned by executing the generative operation;
queue, during the period of time, a second set of request tasks spawned by executing the generative operation in the overload queue; and
dequeue, during the period of time, each request task of the second set of request tasks from the overload queue; and
send, during the period of time, each request task of the second set of request tasks to the thread pool for execution.