Package: oaib
OpenAI Batcher (oaib)A Python library for making rate-limited, async batch requests to the OpenAI
API.UseBatchto make batch requests as quickly as possible given TPM/RPM
limits.UseAutoto automatically read your rate limits from OpenAI's response
headers, and run the job as fast as possible.This notebook is available atREADME.ipynb.UsageSet OpenAI API Keyimportosos.environ['OPENAI_API_KEY']=input()Using theBatchclassYou can mix and match endpoints as needed for regularBatch, but it's not
recommended as the rate limits for different endpoints/models will differ. For
maximum efficiency, useAuto.fromoaibimportBatch# Use low rate limits for this example.batch=Batch(rpm=100,tpm=1_000,workers=5)# Creating a batch with 20 chat completions.foriinrange(20):awaitbatch.add("chat.completions.create",model="gpt-3.5-turbo",messages=[{"role":"user","content":"say hello"}])awaitbatch.run()✅ DONE: 100%|█████████████| 20/20 [00:22<00:00,  1.12s/req]
RPM:  53%|█████████████████████▏                  | 53.0/100
TPM:  93%|███████████████████████████████████▎  | 928.0/1000

Run took 20.02s.endpointmodelmessagesresult0chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzMf7oXHlLWpsISVpAaUPxps0g5...1chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzNeFY9TLN71o2FMEqGssHIrOQq...2chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzNDiD3ikFtBZ4hHXWEsLONHUeS...3chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzP9dccsvrGsOR3X5HgmHqsR2fm...4chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzQV7ZnIoXccx9R8dIfS4rdPd0U...5chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzRVQvp3wwmEvbFzNPtrXBmcOhR...6chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzRSw7iTCLs0uu8fWZwDcaPGB0s...7chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzS6D1gACsJW6JXvuS42N4lQLh7...8chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzUFmpIzWjKsNGnlLvZW3DhF752...9chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzU6Mg6Zk4BC5uelndmHjmGAQ0I...10chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzV5K4OEk80dDuSwohiTualLOoO...11chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzXCcmbcuy1EQPskJrfN5po1Ix9...12chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzX4UfaBPDO3fF8vMO1dsQ6tfiT...13chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzZme2VBNhckfItEZRJqmpC3E53...14chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzaYddjxrFpmYUDjMHXlDPgS7G4...15chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzcTLFClvwjtPNNyv3KJ2xvZ8dQ...16chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzcuWlMUwqXj0AeMQkZqzNIQvJo...17chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzepfAhFnZj1AlnVSyNHHMyZ0mK...18chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzfdULvZx6OyUZpARpTMNKYzfKx...19chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVzfyEru2f67JyunYCdPMaC0fD2H...Using theAutoclassAutomatically use the given TPM/RPM rate limits provided by OpenAI API
responses.fromoaibimportAuto# Automatically set rate limits.batch=Auto(workers=8)# Fetch 1,000 chat completions as quickly as possible, setting rate limits# automatically from OpenAI's response headers.foriinrange(1000):awaitbatch.add("chat.completions.create",model="gpt-3.5-turbo",messages=[{"role":"user","content":"say hello"}])awaitbatch.run()✅ DONE: 100%|█████████| 1000/1000 [00:10<00:00, 92.98req/s]
RPM:  56%|████████████████████                | 5573.0/10000
TPM:   5%|█▌                               | 94401.0/2000000

Run took 12.58s.endpointmodelmessagesresult0chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVxz211nsiQQSY2k54r4r141UX83...1chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVy0TrnGax3XqkDOlEiImosPnvIL...2chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVxzMdsF5v1je6iAmvjK7BCZKQna...3chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVy0KiRWYPtQ099p3b1k1HfYBYwT...4chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVy0M7l6Fg0mvWCpOXomVSqV6Pow..................995chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVyBemEPgb6lV5Opnu8X9UQ7T9iZ...996chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVyBBlh4PMbI9qtca80UyMbrOGAF...997chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVyBWveGsMeSLS5SzUbXIiMLvGaS...998chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVyBbBnX5SleJWSDIHqZ8lS0y15V...999chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'say hello'}]{'id': 'chatcmpl-8qVyBYTizFbBIu3RQ9XlvKDlnSuEG...1000 rows × 4 columnsNotesIt is not possible to perfectly guarantee the tokens per minute limit is not
breached because we cannot know the total token usage until the response
comes back.Use thesafetyparam to set the rate limit tolerance.  By default it is set
to 10%, and will wait until the predicted TPM (the current TPM plus the
average number of tokens per request) drops below 90% of the limit.By default, important logs are stored atoaib.txt.  This can be disabled
usingloglevel=0.There's an error with TPM/RPM progress bar display in Jupyter Notebooks for
theAutoclass only. This is caused by atqdm.notebookbug where only the
initial totals (here, our limits) are used to calculate the width of the bar,
and theAutoclass updates these values only after the first request. The
text percentage displays are accurate.Metadata and IndexYou can add custom metadata to your observations withadd(metadata={...}, and
set the index for the DataFrame withBatch(index=[...])andAuto(index=[...]).fromoaibimportBatchn=5batch=Batch(rpm=1000,tpm=10000,workers=5,index=["difficulty","i"])difficulties=["easy","medium","hard"]fordifficultyindifficulties:foriinrange(n):awaitbatch.add("chat.completions.create",metadata={"difficulty":difficulty,"i":i},model="gpt-3.5-turbo",messages=[{"role":"user","content":f"difficulty:{difficulty}. write a math problem."}])awaitbatch.run()✅ DONE: 100%|█████████| 15/15 [00:01<00:00, 10.52req/s]
RPM:  56%|████████████████████                | 631.0/1000
TPM:     |                                    | 10781.0/?

Run took 1.43s.endpointmodelmessagesresultdifficultyieasy0chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: easy...{'id': 'chatcmpl-8rdiFkLTnjs4LbX2ZUKyb1UaRcPwH...1chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: easy...{'id': 'chatcmpl-8rdiFtEcz6CEvO8K9jZpAdbFaaJ3o...2chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: easy...{'id': 'chatcmpl-8rdiFEmy4TfO4iR3aJbh4u9lgAD1W...3chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: easy...{'id': 'chatcmpl-8rdiFVw5YuHY8WhuNqJjyJcFO9Mhs...4chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: easy...{'id': 'chatcmpl-8rdiGoYqWA5wH3xoFKVl8pRGGTuDZ...hard0chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: hard...{'id': 'chatcmpl-8rdiG8ZAUhcsBOtgtPuOjblzEo14a...1chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: hard...{'id': 'chatcmpl-8rdiGgl9uEe4ASQgt5uzwMpeJhnU6...2chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: hard...{'id': 'chatcmpl-8rdiGfVkiZqrE1p2TxeKnCxc2zzUb...3chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: hard...{'id': 'chatcmpl-8rdiGdKxmYC6mS4QSuiW3HjsMRW05...4chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: hard...{'id': 'chatcmpl-8rdiGqTe3MVy8FGJ6qtVAjxmMODy6...medium0chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: medi...{'id': 'chatcmpl-8rdiGsk7ohvIVwzuFMfO3cH9zNLW4...1chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: medi...{'id': 'chatcmpl-8rdiG64Y66W8CZZ9MI4xiVNHheHiF...2chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: medi...{'id': 'chatcmpl-8rdiGGmXfXx0uRuKafeKODGQe42vz...3chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: medi...{'id': 'chatcmpl-8rdiG95XCozUiFXY7ryA4rSfzbwRk...4chat.completions.creategpt-3.5-turbo[{'role': 'user', 'content': 'difficulty: medi...{'id': 'chatcmpl-8rdiGi6nF5FLMdPIYrkrYgI15Yfcw...