Package: openai-manager
OpenAI-ManagerSpeed up your OpenAI requests by balancing prompts to multiple API keys.Quite useful if you are playing withcode-davinci-002endpoint.Update on 2023/03/24: OpenAI terminated allCODEXendpoint access today. An immediate migration togpt-3.5-turboor other endpoints is needed to ensure the stability of your service.DisclaimerBefore using this tool, you are required to read the EULA and ToS of OpenAI L.P. carefully. Actions that violate the OpenAI user agreement may result in the API Key and associated account being suspended. The author shall not be held liable for any consequential damages.Caution: do not deploy this tool directly in Mainland China, Hong Kong SAR or any other locations where OpenAI disallows API usage. UseOPENAI_API_PROXYenvironmental variable to set a proxy (e.g. Japan) for connectting OpenAI API. Failure to do so will bring quick termination of your account.DesignTL;DR: this package helps you manage rate limit (both request-level and token-level) for each api_key for maximum number of requests to OpenAI API.This is extremely helpful ifyou useCODEXendpoint oryou have a handful offree-trial accountsdue to limited budget. Free-trial accounts applystrictrate limit.QuickstartInstall openai-manager on PyPI. Notice we need Python 3.8+ for maximum compatibility ofasyncioandtiktoken.pipinstall-Uopenai-managerPrepare your OpenAI credentials in:Environment VariablesAny envvars beginning with `OPENAI_API_KEY` will be used to initialized the manager. Best practice to load your api keys is to prepare a `.env` file like:OPENAI_API_KEY_1=sk-Nxo******OPENAI_API_KEY_2=sk-TG2******OPENAI_API_KEY_3=sk-Kpt******# You can set a global proxy for all api_keysOPENAI_API_PROXY=http://127.0.0.1:7890# You can also append proxy to each api_key.# Make sure the indices match.OPENAI_API_PROXY_1=http://127.0.0.1:7890OPENAI_API_PROXY_2=http://127.0.0.1:7890OPENAI_API_PROXY_3=http://127.0.0.1:7890openai-managerwill try to read the.envfile in your current working directory. You can also load environmental varibles manually by:export$(grep-v'^#'.env|xargs)YAML config fileYou can add more fine-grained restrictions on each API key if you know the ratelimit for each key in advance. See [example_config.yml](/example_config.yml) for details.importopenai_manageropenai_manager.append_auth_from_config(config_path='example_config.yml')Two ways to useopenai_manager:Use it just like how you use officialopenaipackage. We implement exact the same call signature as officialopenaipackage.importopenaiasofficial_openaiimportopenai_managerfromopenai_manager.utilsimporttimeit@timeitdeftest_official_separate():foriinrange(10):prompt="Once upon a time, "response=official_openai.Completion.create(model="text-davinci-003",prompt=prompt,max_tokens=20,)print("Answer{}:{}".format(i,response["choices"][0]["text"]))@timeitdeftest_manager():prompt="Once upon a time, "prompts=[prompt]*10responses=openai_manager.Completion.create(model="text-davinci-003",prompt=prompts,max_tokens=20,)assertlen(responses)==10fori,responseinenumerate(responses):print("Answer{}:{}".format(i,response["choices"][0]["text"]))Use it as a proxy server between you and OpenAI endpoint. First, runpython -m openai_manager.serving --port 8000 --host localhost --api_key [your custom key]. Then set up the official pythonopenaipackage:importopenaiopenai.api_base="http://localhost:8000/v1"openai.api_key="[your custom key]"# run like normalprompt=["Once upon a time, "]*10response=openai.Completion.create(model="text-davinci-003",prompt=prompt,max_tokens=20,)print(response["choices"][0]["text"])ConfigurationMost configurations are manupulated by environmental variables.GLOBAL_NUM_REQUEST_LIMIT: aiohttp connection limit, default is500;REQUESTS_PER_MIN_LIMIT: number of requests per minute, default is10; config file will overwrite this;TOKENS_PER_MIN_LIMIT: number of tokens per minute, default is40000; config file will overwrite this;COROTINE_PER_AUTH: number of corotine per api_key, default is3; decrease it to 1 if ratelimit errors are triggered too often;ATTEMPTS_PER_PROMPT: number of attempts per prompt, default is5;RATELIMIT_AFTER_SUBMISSION: whether to track ratelimit after submission, default isTrue; keep it enabled if response takes a long time;OPENAI_LOG_LEVEL: default log level is WARNING, 10-DEBUG, 20-INFO, 30-WARNING, 40-ERROR, 50-CRITICAL; set to 10 if you are getting stuck and want to do some diagnose;Rate limit triggers will be visible underlogging.WARNING. Runexport OPENAI_LOG_LEVEL=40to ignore rate limit warnings if you believe current setting is stable enough.Performance AssessmentAfter ChatCompletion release, thecode-davinci-002endpoint becomes slow. Using 10 API keys, running 100 completions withmax_tokens=20and other hyperparameters left default took 90 seconds on average. Using official API, it took 10 seconds per completion, thus 1000 in total.Theroticallly, the throughputincreases linearlywith the number of API keys.Frequently Asked QuestionsQ: Why don't we just use official batching function?prompt="Once upon a time, "prompts=[prompt]*10response=openai.Completion.create(model="code-davinci-002",prompt=prompts,# official batching allows multiple prompts in one requestmax_tokens=20,)assertlen(response["choices"])==10fori,answerinenumerate(response["choices"]):print("Answer{}:{}".format(i,answer["text"]))A: Some OpenAI endpoints (likecode-davinci-002) apply strict token-level rate limit, even if you upgrade to pay-as-you-go user. Simple batching would not solve this.Q: Why don't we just use server-less service (e.g.Cloudflare Workers,Tencent Cloud Functions) to do the same thing?A: First, I usually write in Python, and most cloud services do not support Python server-less function. Second, I am not sure server-less solutions are capable of handlingrate limit controlsgiven their status-less nature. Tracking usage of each API key would be difficult (practical but not elegant) for server-less solutions.Acknowledgementopenai-cookbook: Best practice when dealing with official APIs.openai-python: Official Python version of OpenAI.TODOFeaturesSupport all functions in OpenAI Python API.CompletionsEmbeddingsGenerationsChatCompletionsBetter back-off strategy for maximum throughput.Properly handling exceptions raised by OpenAI API.Serving as a reverse proxy to balance official requests.Proxy-only mode to bypass blocks in Mainland China.Advance FunctionsAutomatic batching prompts to reduce the number of requests.Now a request only carries a single prompt. We can implementa splitting strategy to let it carry multiple promptswhile keeping token consumption not exceeding quotas.Automatic rotation of tons of OpenAI API Keys. (removing invaild, adding new, etc.)Distributed serving mode for concurrent requests viapython -m openai_manager.serving.DonationIf this package helps your research, consider making a donation via GitHub!