Package: inference-providers
Inference providersDisclaimer: I made this for my own use, so there arelotsof gaps and omissions, which I hope can be filled over time.Popular models have been given short canonicalnames, likecodellama-70borgpt-4.inference-providers.jsoncontains a list of known commercial inferenceproviders, the models they offer, and their magic internal names.Environment variables contain API keys for one or more of those providers.These things go together to allowauto-selection of model sourcesat runtime. The user will be connected to whatever service(s) they happen to subscribe to, and which have the model requested.ProviderKeyAnyscaleANYSCALE_API_KEYdeepinfraDEEPINFRA_API_KEYfireworks.aiFIREWORKS_API_KEYGroqGROQ_API_KEYLepton AILEPTON_API_KEYMistral AIMISTRAL_API_KEYOcto AIOCTOAI_TOKENOpenAIOPENAI_API_KEYOpenRouter AIOPENROUTER_API_KEYPerplexityPERPLEXITYAI_API_KEYtogether.aiTOGETHERAI_API_KEYSupported using OpenAI compatibility proxyGeminivia endpointhttp://localhost:6006/v1Google doesn't directly support the OpenAI API, but shims are available likethis one.
If you run a proxy server on port6006, it will be automatically recognized as Gemini. The easiest way to do this is Docker:docker run --restart=always -it -d -p 6006:8080 --name gemini zhu327/gemini-openai-proxy:latestAlmost but not quite supportedReplicateReplicate servers require non-standardheaderAuthentication: Token <key>instead ofAuthentication: Bearer <key>, which breaks OpenAI compatibility.
This is also something a local proxy could fix by rewriting the headers on outgoing requests.AnthropicI don't have an API key to test a direct connection yet, butclaude-2is available via OpenRouter if you have anOPENROUTER_API_KEY. The first-party connection will be used by default once it is available.LimitationsThe scope of this library is to find a working connection without requiring configuration, that's all.No attempt is made to find thecheapestorfastestprovider of a given model.First party providers take priority over resellers (like OpenRouter), and connections are internally sorted very roughly by the speeds I saw while testing, but of course that changes constantly. Your results could be quite different.Usagefrominference_providersimportProviderListproviders=ProviderList(auto_update=True)client,name=providers.connect_to_model("mixtral-8x7b")print(providers.get_response(client,name,"What's your sign?"))If you are not too picky about exactly which model you use, you can submit a list of models that are known to be "good enough". If there is no source for the first model, it looks for the next, and so on.client,name=providers.connect_to_first_available_model(["mistral-7b","llama-2-7b","gpt-3.5","gemini-nano"])It's fine to include models from paid subscription services in the list. They will be skipped if the user does not have access, but used opportunistically if keys are found.If you are even less picky,connect_to_ai()will connect you to the "best" model available, which is determined by a list of popular models inauto_model_priority, at the bottom of the JSON file.client,name=providers.connect_to_ai()Connections are tried in the order they appear ininference_providers.json, and the first one that works is returned. There is also a flagchoose_randomly=Truethat does what it sounds like. If you'd rather sort through the options yourself,find_model_providers()will give you a list of all services that are compatible with a given model.Handling updatesThe JSON file bundled with the package will slowly get out of date, as providers and models come and go over time. There are a couple of ways to deal with that:settingauto_update=Trueon theProviderListconstructor (like the example code) will download and use the latest version of the JSON checked in to GitHubyou can also maintain your own up-to-date or edited copy, and use it by specifyingjson_overrideat startupyou can fetch the latest JSON with yourself usingProviderList.get_updated_provider_list()and do as you willYou can also usejson_mergein the constructor to patch user settings on top of the embedded file. Use the same schema, but sparse: only include items to merge/override. For example, to support a new model without updating the package:{"providers":{"openai":{"model_names":{"gpt-5":"gpt-5-instruct-preview"}}}}This is the most robust solution. Look insideinference_providers.jsonfor details.frominference_providersimportProviderListproviders=ProviderList(json_merge=open('my_custom_config.json').read())client,name=providers.connect_to_model("gpt-5")print(providers.get_response(client,name,"How long do we have?"))NotesAPI cheat sheetA model is normally considered "working" if we can connect to the server, but the methods also have atestoption that runs an inference sample to verify the API key is authorized and a valid response comes back. It is disabled by default because the extra check makes initialization slower, but enable it if you can for more reliable connections.classProviderList(verbose=False,json_override=None,json_merge=None,auto_update=False)# Get a list of unique model names availableget_canonical_model_names()# Scan ports on the local machine for servers/proxiesdetect_local_connections(self,refresh_cache=False)# Get a list of provider connection options for a modelfind_model_providers(canonical_name)# Send a query to an existing client, blocking until the response arrivesget_response(client,model_name,prompt)# Connect to a specific model using a provider with known supportconnect_to_model(canonical_name,choose_randomly=False,test=False)# Connect to a model from a (prioritized) list of acceptable onesconnect_to_first_available_model(model_names,test=False)# Connect to the "best" model available given the user's API keysconnect_to_ai(test=True)Canonical model names recognizedgpt-3.5llama-2-7bmistral-7bpplx-7bfalcon-40bgpt-4llama-2-13bmistral-tinypplx-70bdolphin-8x7bgpt-4-32kllama-2-70bmixtral-8x7bdeepseek-34bwizard-70bgpt-4-turbocodellama-7bmistral-smallphind-34bopenchat-7bgemini-procodellama-13bmistral-mediumyi-34bopenhermes-7bclaude-2codellama-34bmistral-nextqwen-72bcodellama-70bThis list is arbitrary and basically represents the models I happen to have tested.The "instruct" versions of models will be used if they exist. Failing that, any "chat" versions. The-instructand-chatsuffixes on model names are left off the canonical names, but the presence of one of them is always implied.No base/completion modelsare specified, only ones you can talk to.