Package: openai-multi-client
openai-multi-clientopenai-multi-clientis a Python library that allows you to easily make multiple concurrent requests to the OpenAI API,
either in order or unordered, with built-in retries for failed requests. It keeps your application code synchronous and
easy to understand, without you having to reason about concurrency and deadlocks. This library is particularly useful
when working with the OpenAI API for tasks that require a large number of requests.Table of ContentsMotivationFeaturesInstallationUsage ExampleAPI ReferenceExtended Example Adapted from Real-World UseContributingLicenseAuthorship DisclosureMotivationImagine you're sitting at your desk, eager to run a state-of-the-art language model analysis on your extensive database
of collected articles. You're excited about discovering all the hidden gems and insights your data has to offer.
However, there's one tiny problem – if you send requests to the API in a serial manner, it would take a lifetime to
complete.This is where the OpenAI Multi-Client comes in. This library was created to help you fully utilize OpenAI's API without
having to wait too long for results. It's designed to manage concurrent API requests so you can focus on analyzing the
insights provided by the language model.No longer do you have to wait for one request to finish before starting the next. With OpenAI Multi-Client, you can now
send multiple requests simultaneously, while also ensuring that any failed requests will be retried automatically.
Furthermore, the library can be configured to maintain the order of requests and responses.So, the next time you find yourself with a collection of articles waiting to be analyzed, remember that OpenAI
Multi-Client is here to help. Happy analyzing!FeaturesConcurrently send multiple requests to the OpenAI APISupport for ordered and unordered request handlingBuilt-in retries for failed requests with customizable retry settingsCustomizable API client for easy testing and mockingSimple, easy-to-use interfaceInstallationTo installopenai-multi-client, run:pipinstallopenai-multi-clientUsage ExampleHere is an example of how to use theopenai-multi-clientlibrary.fromopenai_multi_clientimportOpenAIMultiClient# Remember to set the OPENAI_API_KEY environment variable to your API keyapi=OpenAIMultiClient(endpoint="chats",data_template={"model":"gpt-3.5-turbo"})defmake_requests():fornuminrange(1,10):api.request(data={"messages":[{"role":"user","content":f"Can you tell me what is{num}*{num}?"}]},metadata={'num':num})api.run_request_function(make_requests)forresultinapi:num=result.metadata['num']response=result.response['choices'][0]['message']['content']print(f"{num}*{num}:",response)If you want the answers to be in the same order as the requests are sent, import and useOpenAIMultiOrderedClientinstead ofOpenAIMultiClient.If you prefer, you can use callback style programming instead of the iterating over the results:fromopenai_multi_clientimportOpenAIMultiClient,Payload# Remember to set the OPENAI_API_KEY environment variable to your API keyapi=OpenAIMultiClient(endpoint="chats",data_template={"model":"gpt-3.5-turbo"})defon_result(result:Payload):num=result.metadata['num']response=result.response['choices'][0]['message']['content']print(f"{num}*{num}:",response)defmake_requests():fornuminrange(1,10):api.request(data={"messages":[{"role":"user","content":f"Can you tell me what is{num}*{num}?"}]},metadata={'num':num},callback=on_result)api.run_request_function(make_requests)api.pull_all()You can find more complete exampleshereandhere.API ReferenceIn theOpenAIMultiClientandOpenAIMultiOrderedClientclasses, theendpointanddataparameters correspond to
the endpoints and parameters expected by the official OpenAI API clients.Configuring API Keys and EndpointsSetting up OpenAI Multi-Client is straightforward. Since it utilizes the official OpenAI client under the hood, all you
need to do is import theopenailibrary and configure it as you usually would.To set up your API key, simply import theopenaimodule and configure the API key using the following code:importopenaiopenai.api_key="your_api_key_here"Setting the environmental variableOPENAI_API_KEYalso works as expected.You can also configure the API endpoint if needed:openai.api_base="azure_openai_api_base_here"Once you've configured theopenailibrary with your API key and endpoint, OpenAI Multi-Client will automatically use
these settings when sending requests to the API. This makes it easy to integrate OpenAI Multi-Client into your existing
projects without having to worry about separate configurations.EndpointsTheendpointparameter in therequest()method or during the initialization of the classes specifies which OpenAI
API endpoint to use for the requests. The available endpoints are:"completions": For text completion requests using theCompletionendpoint."chats"or"chat.completions": For chat completion requests using theChatCompletionendpoint."embeddings": For embedding requests using theEmbeddingendpoint."edits": For edit requests using theEditendpoint."images": For image requests using theImageendpoint."fine-tunes": For fine-tuning requests using theFineTuneendpoint.DataThedataparameter in therequest()method specifies the request data sent to the OpenAI API. The data should be a
dictionary containing the required and optional parameters for the specified endpoint. For example:For the"completions"endpoint, thedatadictionary may include
themodel,prompt,temperature,max_tokens, and other parameters expected by theCompletionendpoint.For the"chats"endpoint, thedatadictionary may include themodel,messages,temperature,max_tokens,
and other parameters expected by theChatCompletionendpoint.When using theOpenAIMultiClientorOpenAIMultiOrderedClient, make sure to provide the appropriateendpointand
the correspondingdataas required by the official OpenAI API clients.For more details, see theofficial documentation.OpenAIMultiClientOpenAIMultiClientis the primary class for making unordered concurrent requests to the OpenAI API.InitializationYou initialize multiple instances of the clients and each will behave independently, with their own queue of requests
and responses. TheOpenAIMultiClientclass can be initialized with the following parameters:OpenAIMultiClient(concurrency=10,max_retries=10,wait_interval=0,retry_multiplier=1,retry_max=60,endpoint=None,data_template=None,metadata_template=None,custom_api=None)concurrency: (Optional) The number of concurrent requests. Default is 10.max_retries: (Optional) The maximum number of retries for failed requests. Default is 10.wait_interval: (Optional) The waiting time between retries. Default is 0.retry_multiplier: (Optional) The multiplier for the waiting time between retries. Default is 1.retry_max: (Optional) The maximum waiting time between retries. Default is 60.endpoint: (Optional) The OpenAI API endpoint to be used, e.g.,"chats"or"completions".data_template: (Optional) A template for the data sent with each request. The request data will be merged with this
template.metadata_template: (Optional) A template for the metadata associated with each request. The request metadata will be
merged with this template.custom_api: (Optional) A custom API function that can be used for testing or mocking the OpenAI API. You can also
use this to connect to models other than LLMs.You should setconcurrencyto a sensible value based on your API rate limit. For paid customers usinggpt-3.5-turbo,
you have 3,500 requests per minute. Let's be generous and assume a request completes in one second. To avoid hitting the
rate limit while maintaining a high throughput, you could setconcurrencyto a value like 50 or 100. Since you are not
charged for failed requests and since exponential backoff is in effect, your requests will eventually complete even if
you setconcurrencyto a high value, but as failed requests take up your rate limit, throughput will be lower.It is recommended to test your code with a mock API first when you are developing, because using the real API, high
concurrency burns money fast.This examplemight be helpful.You can check your limitshere.MethodsImportant Note: Callingrequestmay block the thread if the input queue is full. It is recommended to put the
requesting logic into a function and call that function usingrun_request_functionas done in the example, as it
ensures that the requesting logic runs concurrently without blocking the main thread. The blocking of the input queue is
akey featureof this library to ensure that your memory will not be flooded with a petabyte of pending requests
streaming from your database.request(data, endpoint=None, metadata=None, max_retries=None, retry_multiplier=None, retry_max=None): Adds a request
to the queue.data: The data (as a dict) to be sent with the request.endpoint: (Optional) The API endpoint to be used for this specific request.metadata: (Optional) Metadata associated with the request.callback: (Optional) A callback function that will be called with the payload object.max_retries: (Optional) The maximum number of retries for failed requests. Default is the value set during
initialization.retry_multiplier: (Optional) The multiplier for the waiting time between retries. Default is the value set
during initialization.retry_max: (Optional) The maximum waiting time between retries. Default is the value set during initialization.run_request_function(input_function, *args, stop_at_end=True, **kwargs): Executes the input function in a separate
thread, allowing it to add requests to the queue without blocking.input_function: A function that adds requests to the queue using therequest()method.*args: (Optional) Additional arguments passed to the input function.stop_at_end: (Optional) Whether to stop the event loop after executing the input function. Default is True.**kwargs: (Optional) Additional keyword arguments passed to the input function.To retrieve results, use the client as an iterator. The following is also available:pull_all(): Pulls all responses from the queue and discard them. Useful to get the queue started if you are using
callbacks.OpenAIMultiOrderedClientOpenAIMultiOrderedClientis a subclass ofOpenAIMultiClientfor making ordered concurrent requests to the OpenAI
API. The usage is the same asOpenAIMultiClient, but the responses will be returned in the order they were added to
the queue.LoggingThe library uses the standard Python logging module. You can set the logging level
usinglogging.basicConfig(level=logging.INFO)orlogging.basicConfig(level=logging.DEBUG).Extended Example Adapted from Real-World UseThe scenario in the Motivation section is genuinely real. Although the OpenAI cookbook provides arecipefor making parallel requests, it relies on files for data persistence and can be quite challenging to adapt and use. We were surprised that a library like this didn't exist already, so we decided to create one ourselves.Here's a brief overview of the system. External and internal articles are continuously streamed from scrapers and other sources into a database. For each article, we let ChatGPT answer a few questions about it, and then we store the text and embeddings of the answers back into the database. The embeddings are indexed for nearest-neighbor search. At peak hours, we expect a high volume of articles, making the use ofOpenAIMultiClientessential.We'll be usingCozoDBto store everything. CozoDB is a transactional hybrid relational-graph-vector database that uses Datalog for queries, which we developed for our work. In the real scenario that this example is based on, article deduplication, full-text search, and recursive, agentic behavior are also integrated using the functionalities provided by the database. However, these details have been omitted in the example for simplicity.The setup of the database and schema can be found in the code snippet below, where we create tables for articles and embeddings and index the embeddings for nearest-neighbor search:frompycozo.clientimportClient# We use a global variable for the db for simplicitydb=Client('rocksdb','analysis.db')# Using the RocksDB backend for persistence# Create the schema# The articlesdb.run(''':create article {id: Uuid default rand_uuid_v4() # The key=>title: String,                  # Title of the articletext: String,                   # Text of the articleanalysis: Json default{}# Analyses will be stored in the JSON object}''')# The embeddings are stored separately from the articles# Not strictly necessary, but cleanerdb.run(''':create embedding {id: Uuid,                 # The key consists of the article ID and the analysis keyanalysis_key: String      # The analysis key=>embedding: <F32; 1536>    # The embedding of the article}''')# Put the embeddings into an index for nearest-neighbor searchdb.run('''::hnsw create embedding:idx {dim: 1536,fields: [embedding],ef_construction: 200,m: 50,}''')The streaming service calls a function to insert articles into the database, and the database's ID is auto-generated:definsert(title,text):db.run('''?[title, text] <- [[$title, $text]]:put article {=> title, text}''',{'title':title,'text':text})In another file, we set up the OpenAI multi-client and define the questions we want to ask.fromopenai_clientimportOpenAIMultiClient# We could use separate clients for embedding and analysis to achieve more concurrency# But we'll just use a single client for simplicityclient=OpenAIMultiClient()QUESTIONS={'economic':'''You are the economics advisor to company X.Read the article from the user and summarize the potential economic impact for company X implied by the article from your perspective.Write clearly and concisely, in one paragraph. If there is no economic impact, write "No economic impact."''','political':'''You are the political advisor to company X.Read the article from the user and summarize the potential political impact for company X implied by the article from your perspective.Write clearly and concisely, in one paragraph. If there is no political impact, write "No political impact."''','technological':'''You are the technology advisor to company X.Read the article from the user and summarize the potential technological impact for company X implied by the article from your perspective.Write clearly and concisely, in one paragraph. If there is no technological impact, write "No technological impact."'''}The core of the system involves using the client to ask questions and store the results back into the database. When the answers are inserted into the database, we use the client again to get the embeddings and store them in the database as well. We use database callbacks to achieve this:# Callback function to be called when the database is updateddefinsert_callback(op_name,new_rows,old_rows):# We only handle puts. Logic for handling deletes omitted for simplicityifop_name=='Put':# Make a dict from the old rows replaced by the updateexisting={id:analysisfor[id,_title,_text,analysis]inold_rows}# Iterate over the inserted or updated rowsforid,title,text,analysisinnew_rows:# If the article already exists, we request embeddings for any newly added analysesifidinexisting:old_analyses=existing[id]forkey,valueinanalysis.items():ifkeynotinold_analyses:# Make a request for the embedding# The metadata are used for inserting the results back into the DBclient.request(endpoint="embeddings",data={'model':"text-embedding-ada-002",'input':f'{key}impact:\n{value}'},metadata={'type':'embedding','id':id,'key':key},callback=on_embedding_result)else:# Otherwise, we ask for analyses for all questionscontext_text=f'{title}\n\n{text}'forkey,questioninQUESTIONS.items():# Make a request for the analysisclient.request(endpoint="chats",data={# You can use GPT-4 if you are rich!"model":"gpt-3.5-turbo","messages":[{"role":"system","content":question},{"role":"user","content":context_text}]},metadata={'type':'analysis','id':id,'key':key},callback=on_analysis_result)# Register the callback, this will be called whenever an article is inserted or deleted# As the database already runs callbacks on a separate thread,# we don't need to use run_request_functioncallback_handle=db.register_callback('article',insert_callback)Callback functions handle the results from the OpenAI client and start the system:defon_embedding_result(result):ifresult.failed:print(f'Failed to get embedding for article{result.metadata["id"]}')returnembedding=result.response['data'][0]['embedding']db.run('''?[id, analysis_key, embedding] <- [[$id, $analysis_key, $embedding]]:put embedding {id, analysis_key => embedding}''',{'id':result.metadata['id'],'analysis_key':result.metadata['key'],'embedding':embedding})defon_analysis_result(result):ifresult.failed:print(f'Failed to get embedding for article{result.metadata["id"]}')returnanalysis={result.metadata['key']:result.response['choices'][0]['message']['content']}db.run('''?[id, analysis] := *analysis{id, analysis: old}, analysis = old ++ $analysis:update article {id => analysis}''',{'id':result.metadata['id'],'analysis':analysis})Finally, we start the system:client.pull_all()With this system in place, we can easily access the top news we're interested in by running nearest-neighbor vector searches:forquestionin['Red-alert level innovation from competitors','Disruption to our supply chain','Dangers to our overseas personnel in the Middle East']:# the `embed` function also uses the OpenAI multi-client, but its definition is omitted.embedding=embed(question)res=db.run('''?[id, title, text] := ~article:idx{id, title, text | query: $q, k: 3}''',{'q':embedding})print(res)As you can see from the example questions, it would be challenging to find relevant news articles in the database without the analysis. But with the analysis, we can get the relevant articles very quickly. This is the power of vector indexing. We can also set up more agents to monitor the analysis and notify the CEO when something important comes up, or even have a community of agents using the database to collaborate. Thanks to the OpenAI multi-client, these agents can move at the speed of thought!ContributingContributions toopenai-multi-clientare welcome! Feel free to submit a pull request or open an issue on GitHub.Licenseopenai-multi-clientis released under the MIT License. See theLICENSEfile for details.Authorship DisclosureGPT-4 wrote most of this README by analyzing code, incorporating the techniques outlined before.