Repository: BerriAI/litellm

Issue:
[Feature]: Flag to always send queries via the lite_proxy
### The Feature

Hi all, 

we can use the proxy via various ways but it's never transparent. 
I would like to add a new variable that would always send queries via the proxy without changing code or anything else:

proposal:
```python
os.environ["LITELLM_PROXY_ALWAYS_USE"] = 'True' # new
os.environ["LITELLM_PROXY_BASE"] = 'https://'    # exists already
os.environ["LITELLM_PROXY_KEY"] = 'virtualkey'  # exists alreay

response = await litellm.acompletion(
    model="gemini/gemini-2.5-pro-preview-03-25",
     messages=[ {"role": "user", "content": "Hello, how are you?"} ])
```

By switch this flag on/off,  without changing the code it would either reach to provider or the proxy. 
The code and model name doesn't have to change and it won't be using the pass-through endpoints. 


The current other ways to configure the proxy are not transparent and switching between using the proxy or not nearly always require code change. The closest way to do it transparently is to use passthrough, but then it requires to configure every single providers ENV vars, which defeat a little the centralized proxy benefits. 
Examples of current state:

1. configure the LITE_PROXY_API_BASE and use the model:
```python
response = await litellm.acompletion(
    model="litellm_proxy/gemini/gemini-2.5-pro-preview-03-25",
     messages=[ {"role": "user", "content": "Hello, how are you?"} ])
```
2. configure the lite_proxy Client:
```python
client = Client(
base_url="https://litellm.great.project",
api_key=mykey
)
```
3. using the passthrough endpoints and configure the individual ENVS configurations
```python
os.environ['GEMINI_API_BASE']= "https://litellm.great.project/gemini'
os.environ['GEMINI_API_KEY']= "thevirtualkey"

response = await litellm.acompletion(
    model="gemini/gemini-2.5-pro-preview-03-25",
     messages=[ {"role": "user", "content": "Hello, how are you?"} ])
```  

Happy to contribute to it eventually if approved.



### Motivation, pitch

I'm working on integrating litellm proxy and openai-agents-sdk in our services. 
I would like to use the proxy in the most transparent way. 
1. Many other tooling (e.g tracing) are using the model name (e.g for costs or token count etc), by using "litellm_proxy/$model_name"   breaks it. 
2. using passthrough, requires to be more dependent on the model provider configuration on the client side (set many individual envs, and sometimes it's not 100% compatible or missing few features)
3.  using the llm_proxy client is not always on option when integrating with 3rd party sdk such as openai-agents-sdk or aider-chat etc

The proposal would help us a lot. 

(love the project btw :) )

### Are you a ML Ops Team?

Yes

### Twitter / LinkedIn details

@ant_legrand / https://www.linkedin.com/in/antoine-legrand-76739039/

Discussion:
@ant31:  This PR works for me:  https://github.com/BerriAI/litellm/pull/10633 

@krrishdholakia: Hi @ant31 curious - why are you trying to use passthrough via litellm sdk / litellm proxy client within your project? 

Trying to understand the use-case better @ant31: I don't want to use passthrough and I don't have control over the code so what are my option to use the proxy? 

for example:
using https://github.com/openai/openai-agents-python/blob/main/docs/models/litellm.md

Let's say one developer wrote this code:
```
from agents.extensions.models.litellm_model import LitellmModel

async def main(api_key: str):
    agent = Agent(
        name="Assistant",
        instructions="You only respond in haikus.",
        model=LitellmModel(model="gemini/gemini-1.5-pro", api_key=api_key),
    )

    result = await Runner.run(agent, "What's the weather in Tokyo?")
    print(result.final_output)
```
The Agent code is using `litellm.acompletion(model="gemini/gemini-1.5.pro")`

How can I configure it so it's using the litellm proxy without changing the code and without using passthrough ? 
It's not possible as far as I Know. 


Currently the only options are to change the code and update the model with litellm_proxy/:
```...
        model=LitellmModel(model="litellm_proxy/gemini/gemini-1.5-pro", api_key=api_key),
   ...
```
or to configure passthrough (which I'd prefer not):
```
GEMINI_API_BASE = $litellmproxy/gemini/
GEMINI_API_KEY = $virtualkey
```
 


The proposal solve that by saying "regardless of the model pick or configuration, always use the litellm_proxy".
It's easy to turn on and off. 


Let me know
 
@ishaan-jaff: Spoke to a user about this today. The use case is when they use Google ADK or OpenAI agents SDK they want a flag to switch between sending their request to litellm proxy or going direct to Gemini. 

Since Google ADK uses litellm python SDK we can support this like this 


```python
litellm.use_litellm_proxy=True
```

```
USE_LITELLM_PROXY=True
```@ishaan-jaff: Added here @ant31 and @mattcalt https://github.com/BerriAI/litellm/pull/10773 @ant31: Thank you @ishaan-jaff for picking it up the PR and merging. 
I tested main and the usecase described is covered. 



Solution patch:
diff --git a/litellm/litellm_core_utils/get_llm_provider_logic.py b/litellm/litellm_core_utils/get_llm_provider_logic.py
index 29f6b2d3f..a49f3a4c9 100644
--- a/litellm/litellm_core_utils/get_llm_provider_logic.py
+++ b/litellm/litellm_core_utils/get_llm_provider_logic.py
@@ -4,7 +4,7 @@ import httpx
 
 import litellm
 from litellm.constants import REPLICATE_MODEL_NAME_WITH_ID_LENGTH
-from litellm.secret_managers.main import get_secret, get_secret_str
+from litellm.secret_managers.main import get_secret, get_secret_bool, get_secret_str
 
 from ..types.router import LiteLLM_Params
 
@@ -85,12 +85,51 @@ def handle_anthropic_text_model_custom_llm_provider(
     return model, custom_llm_provider
 
 
+def force_use_litellm_proxy(
+        model: str,
+        api_base: Optional[str] = None,
+        api_key: Optional[str] = None) -> Tuple[str, str, Optional[str], Optional[str]]:
+    """
+    Force use litellm proxy for all models
+    it forces the llm_provider to be 'litellm_proxy'
+    LITTELLM_PROXY_API_BASE and LITELLM_PROXY_API_KEY will be used
+    if api_base and api_key are not provided
+
+    Allow the model name to be passed in original format and still use litellm proxy:
+    "gemini/gemini-1.5-pro", "openai/gpt-4", "mistral/llama-2-70b-chat" etc.
+
+    Raises Error - if api_base or api_key are not set
+
+    Return model, custom_llm_provider, dynamic_api_key, api_base
+    """
+    custom_llm_provider = "litellm_proxy"
+    api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
+    dynamic_api_key = api_key or get_secret_str("LITELLM_PROXY_API_KEY")
+    if model.startswith("litellm_proxy/"):
+        model = model.split("/", 1)[1]
+
+    if api_base is not None and not isinstance(api_base, str) or not api_base:
+        raise Exception(
+            "api base needs to be a string. api_base={}".format(api_base)
+        )
+
+    if dynamic_api_key is not None and not isinstance(dynamic_api_key, str) or not dynamic_api_key:
+        raise Exception(
+            "dynamic_api_key needs to be a string. dynamic_api_key={}".format(
+                dynamic_api_key
+            )
+        )
+
+    return model, custom_llm_provider, dynamic_api_key, api_base
+
+
 def get_llm_provider(  # noqa: PLR0915
     model: str,
     custom_llm_provider: Optional[str] = None,
     api_base: Optional[str] = None,
     api_key: Optional[str] = None,
     litellm_params: Optional[LiteLLM_Params] = None,
+    use_proxy: bool = False,
 ) -> Tuple[str, str, Optional[str], Optional[str]]:
     """
     Returns the provider for a given model name - e.g. 'azure/chatgpt-v-2' -> 'azure'
@@ -111,6 +150,9 @@ def get_llm_provider(  # noqa: PLR0915
             api_base = litellm_params.api_base
             api_key = litellm_params.api_key
 
+        if get_secret_bool("LITELLM_PROXY_ALWAYS") or use_proxy:
+            return force_use_litellm_proxy(model=model, api_base=api_base, api_key=api_key)
+
         dynamic_api_key = None
         # check if llm provider provided
         # AZURE AI-Studio Logic - Azure AI Studio supports AZURE/Cohere


Test patch:
diff --git a/tests/local_testing/test_get_llm_provider.py b/tests/local_testing/test_get_llm_provider.py
index b5ea12897..83dd052bb 100644
--- a/tests/local_testing/test_get_llm_provider.py
+++ b/tests/local_testing/test_get_llm_provider.py
@@ -16,6 +16,8 @@ import pytest
 
 import litellm
 
+from litellm.litellm_core_utils.get_llm_provider_logic import force_use_litellm_proxy
+
 
 def test_get_llm_provider():
     _, response, _, _ = litellm.get_llm_provider(model="anthropic.claude-v2:1")
@@ -244,3 +246,194 @@ def test_xai_api_base(model):
     assert model == "grok-2-vision-latest"
     assert api_base == "https://api.x.ai/v1"
     assert dynamic_api_key == "xai-my-specialkey"
+
+# -------- Tests for force_use_litellm_proxy ---------
+
+def test_force_use_litellm_proxy_with_env_vars():
+    """
+    Tests force_use_litellm_proxy uses LITELLM_PROXY_API_BASE and LITELLM_PROXY_API_KEY from env.
+    """
+    test_model = "gpt-3.5-turbo"
+    expected_api_base = "http://localhost:8000"
+    expected_api_key = "test_proxy_key"
+
+    with patch.dict(os.environ, {
+        "LITELLM_PROXY_API_BASE": expected_api_base,
+        "LITELLM_PROXY_API_KEY": expected_api_key
+    }, clear=True):
+        model, provider, key, base = force_use_litellm_proxy(model=test_model)
+
+    assert model == test_model
+    assert provider == "litellm_proxy"
+    assert key == expected_api_key
+    assert base == expected_api_base
+
+def test_force_use_litellm_proxy_with_args_override_env_vars():
+    """
+    Tests force_use_litellm_proxy uses api_base and api_key args over environment variables.
+    """
+    test_model = "gpt-4"
+    arg_api_base = "http://custom-proxy.com"
+    arg_api_key = "custom_key_from_arg"
+    
+    env_api_base = "http://env-proxy.com"
+    env_api_key = "env_key"
+
+    with patch.dict(os.environ, {
+        "LITELLM_PROXY_API_BASE": env_api_base,
+        "LITELLM_PROXY_API_KEY": env_api_key
+    }, clear=True):
+        model, provider, key, base = force_use_litellm_proxy(
+            model=test_model,
+            api_base=arg_api_base,
+            api_key=arg_api_key
+        )
+
+    assert model == test_model
+    assert provider == "litellm_proxy"
+    assert key == arg_api_key
+    assert base == arg_api_base
+
+def test_force_use_litellm_proxy_model_prefix_stripping():
+    """
+    Tests force_use_litellm_proxy strips 'litellm_proxy/' prefix from model name.
+    """
+    original_model = "litellm_proxy/claude-2"
+    expected_model = "claude-2"
+    expected_api_base = "http://localhost:4000"
+    expected_api_key = "proxy_secret_key"
+
+    with patch.dict(os.environ, {
+        "LITELLM_PROXY_API_BASE": expected_api_base,
+        "LITELLM_PROXY_API_KEY": expected_api_key
+    }, clear=True):
+        model, provider, key, base = force_use_litellm_proxy(model=original_model)
+
+    assert model == expected_model
+    assert provider == "litellm_proxy"
+    assert key == expected_api_key
+    assert base == expected_api_base
+
+def test_force_use_litellm_proxy_env_vars_not_set():
+    """
+    Tests force_use_litellm_proxy raises Exception for base/key if env vars are not set and no args provided.
+    """
+    test_model = "gemini-pro"
+
+    with patch.dict(os.environ, {}, clear=True):
+        with pytest.raises(Exception) as excinfo:# Ensure env vars are cleared
+            model, provider, key, base = force_use_litellm_proxy(model=test_model)
+
+def test_force_use_litellm_proxy_invalid_api_base_type():
+    """
+    Tests force_use_litellm_proxy raises Exception for non-string api_base.
+    """
+    with pytest.raises(Exception) as excinfo:
+        force_use_litellm_proxy(model="any-model", api_base=12345, api_key="valid")
+    assert "api base needs to be a string" in str(excinfo.value)
+
+def test_force_use_litellm_proxy_invalid_api_key_type():
+    """
+    Tests force_use_litellm_proxy raises Exception for non-string api_key.
+    """
+    with pytest.raises(Exception) as excinfo:
+        force_use_litellm_proxy(model="any-model", api_base="https://something", api_key=12345)
+    assert "dynamic_api_key needs to be a string" in str(excinfo.value)
+
+
+# -------- Tests for get_llm_provider triggering force_use_litellm_proxy ---------
+
+def test_get_llm_provider_LITELLM_PROXY_ALWAYS_true():
+    """
+    Tests get_llm_provider uses litellm_proxy when LITELLM_PROXY_ALWAYS is "True".
+    """
+    test_model_input = "openai/gpt-4"
+    expected_model_output = "openai/gpt-4"
+    proxy_api_base = "http://my-global-proxy.com"
+    proxy_api_key = "global_proxy_key"
+
+    with patch.dict(os.environ, {
+        "LITELLM_PROXY_ALWAYS": "True",
+        "LITELLM_PROXY_API_BASE": proxy_api_base,
+        "LITELLM_PROXY_API_KEY": proxy_api_key
+    }, clear=True):
+        model, provider, key, base = litellm.get_llm_provider(model=test_model_input)
+
+    assert model == expected_model_output
+    assert provider == "litellm_proxy"
+    assert key == proxy_api_key
+    assert base == proxy_api_base
+
+def test_get_llm_provider_LITELLM_PROXY_ALWAYS_true_model_prefix():
+    """
+    Tests get_llm_provider with LITELLM_PROXY_ALWAYS="True" and model prefix "litellm_proxy/".
+    """
+    test_model_input = "litellm_proxy/gpt-4-turbo"
+    expected_model_output = "gpt-4-turbo"
+    proxy_api_base = "http://another-proxy.net"
+    proxy_api_key = "another_key"
+
+    with patch.dict(os.environ, {
+        "LITELLM_PROXY_ALWAYS": "True",
+        "LITELLM_PROXY_API_BASE": proxy_api_base,
+        "LITELLM_PROXY_API_KEY": proxy_api_key
+    }, clear=True):
+        model, provider, key, base = litellm.get_llm_provider(model=test_model_input)
+
+    assert model == expected_model_output
+    assert provider == "litellm_proxy"
+    assert key == proxy_api_key
+    assert base == proxy_api_base
+
+
+def test_get_llm_provider_use_proxy_arg_true():
+    """
+    Tests get_llm_provider uses litellm_proxy when use_proxy=True argument is passed.
+    """
+    test_model_input = "mistral/mistral-large"
+    expected_model_output = "mistral/mistral-large" # force_use_litellm_proxy keep the model name
+    proxy_api_base = "http://my-arg-proxy.com"
+    proxy_api_key = "arg_proxy_key"
+    
+    # Ensure LITELLM_PROXY_ALWAYS is not set or False
+    with patch.dict(os.environ, {
+        "LITELLM_PROXY_API_BASE": proxy_api_base,
+        "LITELLM_PROXY_API_KEY": proxy_api_key
+    }, clear=True): # clear=True removes LITELLM_PROXY_ALWAYS if it was set by other tests
+        model, provider, key, base = litellm.get_llm_provider(model=test_model_input, use_proxy=True)
+
+    assert model == expected_model_output
+    assert provider == "litellm_proxy"
+    assert key == proxy_api_key
+    assert base == proxy_api_base
+
+def test_get_llm_provider_use_proxy_arg_true_with_direct_args():
+    """
+    Tests get_llm_provider with use_proxy=True and explicit api_base/api_key args.
+    These args should be passed to force_use_litellm_proxy and override env vars.
+    """
+    test_model_input = "anthropic/claude-3-opus"
+    expected_model_output = "anthropic/claude-3-opus"
+    
+    arg_api_base = "http://specific-proxy-endpoint.org"
+    arg_api_key = "specific_key_for_call"
+
+    # Set some env vars to ensure they are overridden
+    env_proxy_api_base = "http://env-default-proxy.com"
+    env_proxy_api_key = "env_default_key"
+
+    with patch.dict(os.environ, {
+        "LITELLM_PROXY_API_BASE": env_proxy_api_base,
+        "LITELLM_PROXY_API_KEY": env_proxy_api_key
+    }, clear=True):
+        model, provider, key, base = litellm.get_llm_provider(
+            model=test_model_input, 
+            use_proxy=True,
+            api_base=arg_api_base,
+            api_key=arg_api_key
+        )
+
+    assert model == expected_model_output
+    assert provider == "litellm_proxy"
+    assert key == arg_api_key  # Should use the argument key
+    assert base == arg_api_base # Should use the argument base