Package: rate-limited
rate_limitedA lightweight package meant to enableefficient parallel use of slow, rate-limited APIs- like
those of Large Language Models (LLMs).Featuresparallel execution - be as quick as possible within the rate limit you haveretry failed requestsvalidate the response against your criteria and retry if not valid (for non-deterministic APIs)if interrupted (KeyboardInterrupt) e.g. in a notebook:returns partial resultsif ran again, continues from where it left off - and returns full resultsrate limits of some popular APIs already described - custom ones easy to addno dependencies (will use tqdm progress bars if installed)Assumptions:the client:handles timeouts (requests will not hang forever)raises an exception if the request fails (or the server returns an error / an "invalid" response)can be any type of a callable - function, method, coroutine function, partial, etcrequests are independent from each other, can be retried if failedwe want a standard, simple interface for the user - working the same way in a script and in a
notebook (+ most data scientists do not want to deal with asyncio). Therefore,Runner.run()is
a blocking call, with the same behavior regardless of the context.async use is also supported - viarun_coro()- but it will not supportKeyboardInterrupthandling.Installationpipinstallrate_limitedUsageIn short:wrap your function with aRunner(), describing the rate limits you havecallRunner.schedule()to schedule a request - with the same arguments as you would call the
original functioncallRunner.run()to run the scheduled requests, get the results and any exceptions raised
(results are returned in the order of scheduling)Creating a RunnerThe following arguments are required:function- the callable of the API clientresources- a list ofResourceobjects, describing the rate limits you have (see examples below)Important optional arguments:max_retries- the maximum number of retries for a single request (default: 5)max_concurrent- the maximum number of requests to be executed in parallel (default: 64)validation_function- a function that validates the response and returnsTrueif it is valid
(e.g. conforms to the schema you expect). If not valid, the request will be retried.OpenAI exampleimportopenaifromrate_limited.runnerimportRunnerfromrate_limited.apis.openaiimportchatopenai.api_key="YOUR_API_KEY"model="gpt-3.5-turbo"# describe your rate limits - values based on https://platform.openai.com/account/rate-limitsresources=chat.openai_chat_resources(requests_per_minute=3_500,tokens_per_minute=90_000,model_max_len=4096,# property of the model version in use - max sequence length)runner=Runner(openai.ChatCompletion.create,resources)topics=["honey badgers","llamas","pandas"]fortopicintopics:messages=[{"role":"user","content":f"Please write a poem about{topic}"}]# call runner.schedule exactly like you would call openai.ChatCompletion.createrunner.schedule(model=model,messages=messages,max_tokens=256,request_timeout=60)results,exceptions=runner.run()Validating the responseWe can provide custom validation logic to the Runner - to retry the request if the response
does not meet our criteria - for example, if it does not conform to the schema we expect. This
assumes that the API is non-deterministic.Example above continued:defcharacter_number_is_even(response):poem=response["choices"][0]["message"]["content"]returnlen([chforchinpoemifch.isalpha()])%2==0validating_runner=Runner(openai.ChatCompletion.create,resources,validation_function=character_number_is_even,)fortopicintopics:messages=[{"role":"user","content":f"Please write a short poem about{topic}, containing an even number of letters"}]validating_runner.schedule(model=model,messages=messages,max_tokens=256,request_timeout=60)results,exceptions=validating_runner.run()Custom server with a "requests per minute" limitproceed as above - replaceopenai.ChatCompletion.createwith your own function, and
describe resources as follows:fromrate_limited.resourcesimportResourceresources=[Resource("requests_per_minute",quota=100,time_window_seconds=60,arguments_usage_extractor=lambda_:1,),]More complex resource descriptionsOverall, the core of an API description is a list ofResourceobjects, each describing a single
resource and its limit - e.g. "requests per minute", "tokens per minute", "images per hour", etc.Each resource has:a name (just for logging/debugging)a quota (e.g. 100)a time window (e.g. 60 seconds)functions that extract the "billing" information:arguments_usage_extractorresults_usage_extractormax_results_usage_estimatorTwo distinct "billing" models are supported:Before the call - we register usage before making the call, based on the arguments of the call.In these cases, justarguments_usage_extractoris needed.After the call - we register usage after making the call, based on the results of the call,
and possibly the arguments (if needed for some complex cases).To avoid sending a flood or requests before the first ones complete (and we register any usage)
we need to estimate the maximum usage of each call, based on its arguments. We
"pre-allocate" this usage, then register the actual usage after the call completes.In these cases,results_usage_extractorandmax_results_usage_estimatorare needed.Both approaches can be used in the same Resource description if needed.Note: it is assumed that resource usage "expires" fully after the time window elapses, without a
"gradual" decline. Some APIs (OpenAI) might use the "gradual" approach, and differ in other details,
but this approach seems sufficient to get "close enough" to the actual rate limits, without running
into them.Limiting the number of concurrent requestsThemax_concurrentargument ofRunnercontrols the number of concurrent requests.More advanced usageSeeapis.openai.chatfor an example of a more complex API
description with multiple resources.Implementation detailsConcurrency modelThe package uses a single thread with an asyncio event loop to kick off requests and keep track of
the resources used.However, threads are also in use:each call is actually executed in a thread pool, to avoid blocking the Runner
(if the client passed to the Runner is synchronous/blocking)Runner.run(), spawns a new thread and a new event loop to run the main Runner logic in
(run_coro()). This serves two objectives:having the same sync entrypoint for both sync and async contexts (e.g. run the same code
in a notebook and in a script)gracefully handlingKeyboardInterruptDevelopment and testingInstall the package in editable mode withpip install -e .[test, lint]To quickly run the whole test suite, run the following in the root directory of the project:pytest-n8# or another number of workersThis uses pytest-xdist to run tests in parallel, but will not support--pdbor verbose output.To debug tests in detail, examine the usage of resources etc, run:pytest--log-cli-level="DEBUG"--pdb-kname_of_testLinting and formattingFormat:isort.&&black.Check:flake8&&black--check.&&mypy.TODOs:make it easier to import things; perhaps dedicated runner classes? (OpenAIChatRunner etc)more ready-made API descriptions - incl. batched ones?examples of using each pre-made API descriptionfix the "interrupt and resume" test in Python 3.11Nice to have:(optional) slow start feature - pace the initial requests, instead of sending them all at oncebetter utilization of APIs with a "gradual" quota growth, like OpenAItext-based logging if tqdm is not installedif/where possible, detect RateLimitExceeded - notify the user, slow downsupport "streaming" and/or continuous operation:enable scheduling calls while running and/or getting inputs from generatorssupport "streaming" results - perhaps similar to "as_completed" in asyncio?support async calls (for people who prefer openai's acreate etc)add timeouts option? (for now, the user is responsible for handling timeouts)OpenAI shares information about rate limits in http response headers - could it be used without
coupling too tightly with their API?tests (and explicit support?) for different ways of registering usage (time of request
vs time of completion vs gradual)more robust wrapper-like behavior of schedule() - more complete support of VS Code