Package: openai-ratelimiter
openai-ratelimiteropenai-ratelimiter is a simple and efficient rate limiter for the OpenAI API. It is designed to help prevent the API rate limit from being reached when using the OpenAI library. Currently, it supports only Redis as the caching service.Note: This package has been tested with Python 3.11.4.InstallationTo install theopenai-ratelimiterlibrary, use pip:pipinstallopenai-ratelimiterRedis SetupThis library uses Redis for caching. If you don't have a Redis server setup, you can pull the Redis Docker image and run a container as follows:# Pull the Redis imagedockerpullredis# Run the Redis containerdockerrun--namesome-redis-p6379:6379-dredisThis will set up a Redis server accessible atlocalhoston port6379.UsageThe library provides two classes,ChatCompletionLimiterandTextCompletionLimiter, for limiting the rate of API calls.ChatCompletionLimiterfromopenai_ratelimiterimportChatCompletionLimiterimportopenaiopenai.api_key="{your API key}"model_name="gpt-3.5-turbo-16k"messages=[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is the capital of Morocco."},]max_tokens=200chatlimiter=ChatCompletionLimiter(model_name=model_name,RPM=3_000,TPM=250_000,redis_host="localhost",redis_port=6379,)withchatlimiter.limit(messages=messages,max_tokens=max_tokens):response=openai.ChatCompletion.create(model=model_name,messages=messages,max_tokens=max_tokens)...TextCompletionLimiterfromopenai_ratelimiterimportTextCompletionLimiterimportopenaiopenai.api_key="{your API key}"model_name="text-davinci-003"prompt="What is the capital of Morocco."max_tokens=200textlimiter=TextCompletionLimiter(model_name=model_name,RPM=3_000,TPM=250_000,redis_host="localhost",redis_port=6379,)withtextlimiter.limit(prompt=prompt,max_tokens=max_tokens):response=openai.Completion.create(model=model_name,prompt=prompt,max_tokens=max_tokens)...Note: The rate limits (RPM and TPM) and the Redis host and port provided in the examples are not universal and should be tailored to your specific use case. Please adjust these parameters in accordance with the selected model and your account's rate limits. To find your specific rate limits, please refer to your OpenAIaccount settings at OpenAI Rate Limits.Available Methods for Limiter ClassesThe following methods are available in all Limiter classes includingChatCompletionLimiter,AsyncChatCompletionLimiter,BaseAPILimiterRedis, andAsyncBaseAPILimiterRedis:clear_locks()Theclear_locks()method is used to remove all locks that are currently associated with the model. This method is useful when you need to manually reset the lock state.Usage:limiter=AsyncChatCompletionLimiter(model_name=model_name,RPM=3_000,TPM=180_000,redis_host="localhost",redis_port=6379,)success=limiter.clear_locks()In this example,successwill beTrueif the locks were cleared successfully, otherwise it will beFalse.is_locked(messages: List[Dict[str, str]], max_tokens: int)Theis_locked()method is used to check if the request would be locked given the specified messages and max tokens.Usage:limiter=AsyncChatCompletionLimiter(model_name=model_name,RPM=3_000,TPM=180_000,redis_host="localhost",redis_port=6379,)messages=[{"role":"system","content":"You are a helpful assistant."}]max_tokens=200lock_state=limiter.is_locked(messages=messages,max_tokens=max_tokens)In this example,lock_statewill beTrueif the request would be locked given the messages and max tokens, otherwise it will beFalse.This should provide users with a clear understanding of how to use theclear_locksandis_lockedmethods with any of the Limiter classes.Asynchronous Programming SupportThis library also provides support for asynchronous programming with two classes:AsyncChatCompletionLimiterandAsyncTextCompletionLimiter. You can import these classes as follows:fromopenai_ratelimiter.asyncioimportAsyncChatCompletionLimiter,AsyncTextCompletionLimiterThe methods for these classes are the same as their synchronous counterparts. However, the context managers for these classes are asynchronous and must run within an async function.Here are some examples of how to use these classes:AsyncChatCompletionLimiterimportasyncioimportopenaifromopenai_ratelimiter.asyncioimportAsyncChatCompletionLimiteropenai.api_key="{Openai API key}"model_name="gpt-3.5-turbo-16k"messages=[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is the capital of Morocco."},]max_tokens=175chatlimiter=AsyncChatCompletionLimiter(model_name=model_name,RPM=3_500,TPM=180_000,redis_host="localhost",redis_port=6379,)asyncdefsend_request():asyncwithchatlimiter.limit(messages=messages,max_tokens=max_tokens):response=awaitopenai.ChatCompletion.acreate(model=model_name,messages=messages,max_tokens=max_tokens)# process response here...asyncdefmain():asyncwithasyncio.TaskGroup()astg:for_inrange(100):tg.create_task(send_request())asyncio.run(main())AsyncTextCompletionLimiterimportasyncioimportopenaifromopenai_ratelimiter.asyncioimportAsyncTextCompletionLimiteropenai.api_key="{OpenAI API key}"model_name="text-davinci-003"prompt="What is the capital of Morocco."max_tokens=200textlimiter=AsyncTextCompletionLimiter(model_name=model_name,RPM=3_500,TPM=180_000,redis_host="localhost",redis_port=6379,)asyncdefsend_request(_):asyncwithtextlimiter.limit(prompt=prompt,max_tokens=max_tokens):print(_)response=awaitopenai.Completion.acreate(model=model_name,prompt=prompt,max_tokens=max_tokens)# process response here...asyncdefmain():asyncwithasyncio.TaskGroup()astg:for_inrange(100):tg.create_task(send_request(_))asyncio.run(main())Note: The rate limits (RPM and TPM) and the Redis host and port provided in the examples are not universal and should be tailored to your specific use case. Please adjust these parameters in accordance with the selected model and your account's rate limits. To find your specific rate limits, please refer to yourOpenAI account settings at OpenAI Rate Limits.Future PlansIn-memory cachingLimiting for embeddingsLimiting for DALL·E image modelImplementing more functions that provide information about the current stateImplement limiting for the organization level.ContributingContributions are welcome! Please feel free to submit a pull request or open an issue on the GitHub repository. Before contributing, make sure to read through any contributing guidelines and adhere to the code of conduct.AuthorThis library is created and maintained by Youssef Benhammouda. If you have any questions or feedback, you can reach out to him through the GitHub repository.