Package: litellm
🚅 LiteLLMCall all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, etc.]OpenAI Proxy Server|Enterprise TierLiteLLM manages:Translate inputs to provider'scompletion,embedding, andimage_generationendpointsConsistent output, text responses will always be available at['choices'][0]['message']['content']Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) -RouterSet Budgets & Rate limits per project, api key, modelOpenAI Proxy ServerJump to OpenAI Proxy DocsJump to Supported LLM ProvidersSupport for more providers. Missing a provider or LLM Platform, raise afeature request.Usage (Docs)[!IMPORTANT]
LiteLLM v1.0.0 now requiresopenai>=1.0.0. Migration guideherepipinstalllitellmfromlitellmimportcompletionimportos## set ENV variablesos.environ["OPENAI_API_KEY"]="your-openai-key"os.environ["COHERE_API_KEY"]="your-cohere-key"messages=[{"content":"Hello, how are you?","role":"user"}]# openai callresponse=completion(model="gpt-3.5-turbo",messages=messages)# cohere callresponse=completion(model="command-nightly",messages=messages)print(response)Call any model supported by a provider, withmodel=<provider_name>/<model_name>. There might be provider-specific details here, so refer toprovider docs for more informationAsync (Docs)fromlitellmimportacompletionimportasyncioasyncdeftest_get_response():user_message="Hello, how are you?"messages=[{"content":user_message,"role":"user"}]response=awaitacompletion(model="gpt-3.5-turbo",messages=messages)returnresponseresponse=asyncio.run(test_get_response())print(response)Streaming (Docs)liteLLM supports streaming the model response back, passstream=Trueto get a streaming iterator in response.Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)fromlitellmimportcompletionresponse=completion(model="gpt-3.5-turbo",messages=messages,stream=True)forpartinresponse:print(part.choices[0].delta.contentor"")# claude 2response=completion('claude-2',messages,stream=True)forpartinresponse:print(part.choices[0].delta.contentor"")Logging Observability (Docs)LiteLLM exposes pre defined callbacks to send data to Langfuse, DynamoDB, s3 Buckets, LLMonitor, Helicone, Promptlayer, Traceloop, Athina, Slackfromlitellmimportcompletion## set env variables for logging toolsos.environ["LANGFUSE_PUBLIC_KEY"]=""os.environ["LANGFUSE_SECRET_KEY"]=""os.environ["LLMONITOR_APP_ID"]="your-llmonitor-app-id"os.environ["ATHINA_API_KEY"]="your-athina-api-key"os.environ["OPENAI_API_KEY"]# set callbackslitellm.success_callback=["langfuse","llmonitor","athina"]# log input/output to langfuse, llmonitor, supabase, athina etc#openai callresponse=completion(model="gpt-3.5-turbo",messages=[{"role":"user","content":"Hi 👋 - i'm openai"}])OpenAI Proxy - (Docs)Set Budgets & Rate limits across multiple projectsThe proxy provides:Hooks for authHooks for loggingCost trackingRate Limiting📖 Proxy Endpoints -Swagger DocsQuick Start Proxy - CLIpipinstall'litellm[proxy]'Step 1: Start litellm proxy$litellm--modelhuggingface/bigcode/starcoder#INFO: Proxy running on http://0.0.0.0:8000Step 2: Make ChatCompletions Request to Proxyimportopenai# openai v1.0.0+client=openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:8000")# set proxy to base_url# request sent to model set on litellm proxy, `litellm --model`response=client.chat.completions.create(model="gpt-3.5-turbo",messages=[{"role":"user","content":"this is a test request, write a short poem"}])print(response)Proxy Key Management (Docs)UI on/uion your proxy serverSet budgets and rate limits across multiple projectsPOST /key/generateRequestcurl'http://0.0.0.0:8000/key/generate'\--header'Authorization: Bearer sk-1234'\--header'Content-Type: application/json'\--data-raw'{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "ishaan@berri.ai", "team": "core-infra"}}'Expected Response{"key":"sk-kdEXbIqZRwEeEiHwdg7sFA",# Bearer token"expires":"2023-11-19T01:38:25.838000+00:00"# datetime object}Supported Providers (Docs)ProviderCompletionStreamingAsync CompletionAsync StreamingAsync EmbeddingAsync Image Generationopenai✅✅✅✅✅✅azure✅✅✅✅✅✅aws - sagemaker✅✅✅✅✅aws - bedrock✅✅✅✅✅google - vertex_ai [Gemini]✅✅✅✅google - palm✅✅✅✅google AI Studio - gemini✅✅mistral ai api✅✅✅✅✅cloudflare AI Workers✅✅✅✅cohere✅✅✅✅✅anthropic✅✅✅✅huggingface✅✅✅✅✅replicate✅✅✅✅together_ai✅✅✅✅openrouter✅✅✅✅ai21✅✅✅✅baseten✅✅✅✅vllm✅✅✅✅nlp_cloud✅✅✅✅aleph alpha✅✅✅✅petals✅✅✅✅ollama✅✅✅✅deepinfra✅✅✅✅perplexity-ai✅✅✅✅Groq AI✅✅✅✅anyscale✅✅✅✅voyage ai✅xinference [Xorbits Inference]✅Read the DocsContributingTo contribute: Clone the repo locally -> Make a change -> Submit a PR with the change.Here's how to modify the repo locally:
Step 1: Clone the repogit clone https://github.com/BerriAI/litellm.gitStep 2: Navigate into the project, and install dependencies:cd litellm
poetry installStep 3: Test your change:cd litellm/tests # pwd: Documents/litellm/litellm/tests
poetry run flake8
poetry run pytest .Step 4: Submit a PR with your changes! 🚀push your fork to your GitHub reposubmit a PR from thereSupport / talk with foundersSchedule Demo 👋Community Discord 💭Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬Our emails ✉️ishaan@berri.ai/krrish@berri.aiWhy did we build thisNeed for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.Contributors