Repository: BerriAI/litellm

Issue:
[Feature]: Implement /api/generate for Continue.dev FIM / autocompletion with Ollama?
### The Feature

I am using **Ollama** as a Backend for my models.
In **Continue.dev** I want to use Qwen2.5 1.5B to **autocomplete** my code.
This works perfectly if set up the config to directly talk to the Ollama API under http://ollamahostip:11434**/api/generate**.

I never got it to work with directly talking to the LiteLLM-API (using mistral api or openai api) so I tried the **pass-through** function and that finally worked. I have two PCs running the same model as redundancy, so if I set up a _pass-through, only one server would be utilized_.

I also use **Langfuse** for Monitoring the requests, and when using pass-through the API **User is not visible**. 
**My questions, are there any plans to implement /api/generate ?**

Thank you very much!
Best regards, Robert

### Motivation, pitch

I want to always use LiteLLM for all my AI-API-Requests, it would be great if the endpoint /api/generate can be implemented.


### Twitter / LinkedIn details

_No response_

Hints from issue comments:
@krrishdholakia: ollama `/api/generate` is already supported - https://github.com/BerriAI/litellm/blob/8673f2541e88331ff576df36b21cb55ceecd0330/litellm/llms/ollama.py#L284

can you share a sample request to repro the issue? for FIM tasks we recommend using the `/completions` endpoint, not `/chat/completions` @deliciousbob: Hi Krish, thank you for the quick reply.
I was not able to find the /api/generate in the Swagger of Litellm (https://litellm-api.up.railway.app/)
Contiune.dev tries to directly contact url:port**/api/generate** when selecting ollama as provider. (i added the LiteLLM url:4000 as a baseurl to handle the requests)
They do not support OpenAI API as OpenAI does not support FIM they mentioned, so only Ollama API or Mistral is supported.
(see: https://docs.continue.dev/autocomplete/model-setup).@universam1: We have made some test setups and can confirm that LiteLLM breaks the FIM access to Ollama API.

Note following test results using latest Continue.dev plugin (also pre-release):

- VS Code (continue.dev) --> Open WebUI --> Ollama: works!
- VS Code (continue.dev) --> Open WebUI --> **LiteLLM** --> Ollama: **fail**!
- VS Code (continue.dev) --> **LiteLLM** --> Ollama: **fail**!
- VS Code (continue.dev) -->  Ollama: works!

This is unfortunate as it prevents us using LiteLLM as the enterprise central AI gateway@wizche: Can confirm the same behavior, `VS Code -> LiteLLM -> Ollama` fails, wheres `VS Code -> Ollama` works.
Looking at the prompt output while failing it seems that the answer received is more of an instruct-model answer (e.g. `The given snippet is a python code that try to perform ....`.@raihan0824: any workaround for this?@JakubCerven: similar issue with vllm 

only receiving 
 as response (using qwen coder)@ggqshr: We can use Pass Through Endpoints to do this.
https://docs.litellm.ai/docs/proxy/pass_through

Here is an litellm example config file:
```yaml
general_settings:
  pass_through_endpoints:
    - path: "/api/generate"
      target: "http://localhost:11434/api/generate"
      forward_headers: True
```
And in continue config file, the corresponding provider should be `ollama`, not the  `openai`.


But this way doesn't feel elegant enough, because it bypasses a lot of the features provided by litellm.
@krrishdholakia: Hey all, what is the required change on our end, to support this? @krrishdholakia: Our `ollama/` route already called `/api/generate` @ggqshr: I guess the cause of the problem is that the prompt is different, but i don't know how to fix this, adding a `ollama_fim` model prefix ?
https://github.com/BerriAI/litellm/blob/97ed4d3a16414d22764e95b2cf39ed8672206734/litellm/litellm_core_utils/prompt_templates/factory.py#L215

When continue's provider set to openai, among the parameters received by the /completions interface of litellm, the beginning of the prompt is as follows：

![Image](https://github.com/user-attachments/assets/882663d7-c502-4ff7-8e3b-009031549a64)

But the prompt field recv by  ollama /api/generate endpoint is like this:

![Image](https://github.com/user-attachments/assets/13a24ba0-22af-4c95-abd5-4c35f09ff429)

That cause follow issue.
> Can confirm the same behavior, `VS Code -> LiteLLM -> Ollama` fails, wheres `VS Code -> Ollama` works. Looking at the prompt output while failing it seems that the answer received is more of an instruct-model answer (e.g. `The given snippet is a python code that try to perform ....`.

For comparison, when using the passthrough mode, the prompt received by ollama does not have the `### User` prefix, so it's working.
@krrishdholakia @krrishdholakia: Oh - i think i know the fix - we can just support the non-templated call via `/completions` 


It makes sense to do this on `/chat/completions` since that's what the route is asking for. 

Are you able to call `/completions` on vs code? @ggqshr: > When continue's provider set to openai, among the parameters received by the /completions interface of litellm, the beginning of the prompt is as follows：

The openai provider in `continue` will call litellm `/completions` endpoint (now vscode is just calling `/completions` directly.), but then litellm will call ollama `/api/generate` endpoint   with `### User:` prefix in prompt field.@krrishdholakia: That's fixable - thanks for confirming that@deliciousbob: Thx for the fix, I've tested it with the lates stable version.
It is forwarding the prompt to ollama to **<server-ip>:11434/api/generate**
But it directly crashes my Ollama instance with Qwen2.5-Code-3B and restarts Ollama again.

I get that warning before it crashed:
time=2025-04-09T11:16:22.074+02:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.key_length default=128
time=2025-04-09T11:16:22.076+02:00 level=WARN source=ggml.go:149 msg="key not found" key=qwen2.attention.value_length default=128

**Do i need to configure some additional parameter manually on LiteLLM ?**
On Continue.dev i have simply configured it the following way:

  - name: autocomplete-prod
    provider: openai
    model: autocomplete-prod
    apiBase: http://<litellm-ip>:4000
    apiKey: <api-key>
    defaultCompletionOptions:
      contextLength: 8000
      maxTokens: 1500
    roles:
      - autocomplete

In the LiteLLM-logs I see that the prompt is forwarded to Ollama and was successful but it generated no response (Token usage 592(591+1) )

On LiteLLM:
- model_name: autocomplete-prod
  litellm_params:
    api_base: http://<ollama-ip>:11434
    api_key: ollama
    model: ollama/qwen2.5-coder:3b
    drop_params: true

Thx for your help!
best regards, Robert@krrishdholakia: Doesn't that sound like an ollama issue at this point? @deliciousbob 

Please let me know if you see something we can do better here@aleccai8: 


> additional

@krrishdholakia 
same issue, after debugging, same the model output first word is "```" and continue shutdown the stream?@krrishdholakia: this sounds like an ollama issue - am i missing something? 

You can see the request being sent by litellm to ollama by enabling debug logs - `--detailed_debug` and looking for the "POST Request Sent from LiteLLM:" string -  https://docs.litellm.ai/docs/proxy/debugging#debug-logs@mlibre: > this sounds like an ollama issue - am i missing something?
> 
> You can see the request being sent by litellm to ollama by enabling debug logs - `--detailed_debug` and looking for the "POST Request Sent from LiteLLM:" string - https://docs.litellm.ai/docs/proxy/debugging#debug-logs

Something doesn't add up here. I don't know where the problem is either — maybe [this](https://github.com/continuedev/continue/issues/4542)?

Maybe Lite LLM is using the chat template instead of `completion`.  
Or maybe Ollama itself is doing it, because LiteLLM is changing `/generate` to `/chat`?  

---
@krrishdholakia @krrishdholakia: @mlibre whats the config and request to proxy when you see the /generate changing to /chat?

@mlibre: > [@mlibre](https://github.com/mlibre) whats the config and request to proxy when you see the /generate changing to /chat?

Sorry for the confusion — my mistake. I thought I saw it in the `LiteLLM` documentation, but it’s not actually there!

Still, last month I tried everything to get it working. However, the responses came back as `chat-like`, not as completions. I'm pretty sure the model was interpreting the request as a chat request.

When I changed the settings to send the request directly to the `ollama` server, everything worked perfectly!

@krrishdholakia 

Solution patch:
diff --git a/litellm/litellm_core_utils/prompt_templates/common_utils.py b/litellm/litellm_core_utils/prompt_templates/common_utils.py
index 6ce8faa5c..c8745f511 100644
--- a/litellm/litellm_core_utils/prompt_templates/common_utils.py
+++ b/litellm/litellm_core_utils/prompt_templates/common_utils.py
@@ -77,6 +77,16 @@ def convert_content_list_to_str(message: AllMessageValues) -> str:
     return texts
 
 
+def get_str_from_messages(messages: List[AllMessageValues]) -> str:
+    """
+    Converts a list of messages to a string
+    """
+    text = ""
+    for message in messages:
+        text += convert_content_list_to_str(message=message)
+    return text
+
+
 def is_non_content_values_set(message: AllMessageValues) -> bool:
     ignore_keys = ["content", "role", "name"]
     return any(
diff --git a/litellm/litellm_core_utils/prompt_templates/factory.py b/litellm/litellm_core_utils/prompt_templates/factory.py
index ff366b239..28e09d7ac 100644
--- a/litellm/litellm_core_utils/prompt_templates/factory.py
+++ b/litellm/litellm_core_utils/prompt_templates/factory.py
@@ -181,7 +181,7 @@ def _handle_ollama_system_message(
 
 
 def ollama_pt(
-    model, messages
+    model: str, messages: list
 ) -> Union[
     str, OllamaVisionModelObject
 ]:  # https://github.com/ollama/ollama/blob/af4cf55884ac54b9e637cd71dadfe9b7a5685877/docs/modelfile.md#template
diff --git a/litellm/llms/ollama/completion/transformation.py b/litellm/llms/ollama/completion/transformation.py
index 4a7a3556a..b4db95cfa 100644
--- a/litellm/llms/ollama/completion/transformation.py
+++ b/litellm/llms/ollama/completion/transformation.py
@@ -6,6 +6,9 @@ from typing import TYPE_CHECKING, Any, AsyncIterator, Iterator, List, Optional,
 from httpx._models import Headers, Response
 
 import litellm
+from litellm.litellm_core_utils.prompt_templates.common_utils import (
+    get_str_from_messages,
+)
 from litellm.litellm_core_utils.prompt_templates.factory import (
     convert_to_ollama_image,
     custom_prompt,
@@ -302,6 +305,8 @@ class OllamaConfig(BaseConfig):
         custom_prompt_dict = (
             litellm_params.get("custom_prompt_dict") or litellm.custom_prompt_dict
         )
+
+        text_completion_request = litellm_params.get("text_completion")
         if model in custom_prompt_dict:
             # check if the model has a registered custom prompt
             model_prompt_details = custom_prompt_dict[model]
@@ -311,7 +316,9 @@ class OllamaConfig(BaseConfig):
                 final_prompt_value=model_prompt_details["final_prompt_value"],
                 messages=messages,
             )
-        else:
+        elif text_completion_request:  # handle `/completions` requests
+            ollama_prompt = get_str_from_messages(messages=messages)
+        else:  # handle `/chat/completions` requests
             modified_prompt = ollama_pt(model=model, messages=messages)
             if isinstance(modified_prompt, dict):
                 ollama_prompt, images = (


Test patch:
diff --git a/tests/litellm/litellm_core_utils/test_streaming_handler.py b/tests/litellm/litellm_core_utils/test_streaming_handler.py
index 31d541330..75c4fc103 100644
--- a/tests/litellm/litellm_core_utils/test_streaming_handler.py
+++ b/tests/litellm/litellm_core_utils/test_streaming_handler.py
@@ -485,17 +485,21 @@ async def test_streaming_with_usage_and_logging(sync_mode: bool):
             time.sleep(1)
             mock_log_success_event.assert_called_once()
             # mock_log_stream_event.assert_called()
+            assert (
+                mock_log_success_event.call_args.kwargs["response_obj"].usage
+                == final_usage_block
+            )
         else:
             await asyncio.sleep(1)
             mock_async_log_success_event.assert_called_once()
             # mock_async_log_stream_event.assert_called()
+            assert (
+                mock_async_log_success_event.call_args.kwargs["response_obj"].usage
+                == final_usage_block
+            )
 
         print(mock_log_success_event.call_args.kwargs.keys())
 
-        mock_log_success_event.call_args.kwargs[
-            "response_obj"
-        ].usage == final_usage_block
-
 
 def test_streaming_handler_with_stop_chunk(
     initialized_custom_stream_wrapper: CustomStreamWrapper,
diff --git a/tests/local_testing/test_text_completion.py b/tests/local_testing/test_text_completion.py
index 11c43de2c..c2cee5386 100644
--- a/tests/local_testing/test_text_completion.py
+++ b/tests/local_testing/test_text_completion.py
@@ -1,4 +1,5 @@
 import asyncio
+import json
 import os
 import sys
 import traceback
@@ -4285,3 +4286,25 @@ def test_text_completion_with_echo(stream):
             print(chunk)
     else:
         assert isinstance(response, TextCompletionResponse)
+
+
+def test_text_completion_ollama():
+    from litellm.llms.custom_httpx.http_handler import HTTPHandler
+
+    client = HTTPHandler()
+
+    with patch.object(client, "post") as mock_call:
+        try:
+            response = litellm.text_completion(
+                model="ollama/llama3.1:8b",
+                prompt="hello",
+                client=client,
+            )
+            print(response)
+        except Exception as e:
+            print(e)
+
+        mock_call.assert_called_once()
+        print(mock_call.call_args.kwargs)
+        json_data = json.loads(mock_call.call_args.kwargs["data"])
+        assert json_data["prompt"] == "hello"