Repository: Significant-Gravitas/AutoGPT
Pull request: feat(forge/llm): Add `LlamafileProvider`
### Background

<!-- Clearly explain the need for these changes: -->

This draft PR is a step toward enabling the use of local models in AutoGPT by adding [llamafile](https://github.com/Mozilla-Ocho/llamafile) as an LLM provider.

Implementation notes are included in `forge/forge/llm/providers/llamafile/README.md`

Related issues:
* https://github.com/Significant-Gravitas/AutoGPT/issues/6336
* https://github.com/Significant-Gravitas/AutoGPT/issues/6947

Depends on:
* #7178
* #7183

### Changes 🏗️

<!-- Concisely describe all of the changes made in this pull request: -->

* Add minimal implementation of `LlamafileProvider`, a new `ChatModelProvider` for llamafiles. It extends `BaseOpenAIProvider` and only overrides methods that are necessary to get the system to work at a basic level.

* Add support for `mistral-7b-instruct-v0.2`. This is the only model currently supported by `LlamafileProvider` because this is the only model I tested anything with.

* ~~Misc changes to app configuration to enable switching between openai/llamafile providers. In particular, added config field `LLM_PROVIDER` that, when set to 'llamafile', will use `LllamafileProvider` in agents rather than `OpenAIProvider`.~~

* Add instructions to use AutoGPT with llamafile in the docs at `autogpt/setup/index.md`

Limitations:

* Only tested with (quantized) [Mistral-7B-Instruct-v0.2](https://huggingface.co/jartine/Mistral-7B-Instruct-v0.2-llamafile)
* Only tested with a single AutoGPT 'task' ("Tell me about Roman dodecahedrons")
* Did not attempt extensive refactoring of existing components; I just added special cases as necessary
* Haven't added any tests for new classes/methods

### PR Quality Scorecard ✨

<!--
Check out our contribution guide:
https://github.com/Significant-Gravitas/Nexus/wiki/Contributing

1. Avoid duplicate work, issues, PRs etc.
2. Also consider contributing something other than code; see the [contribution guide]
   for options.
3. Clearly explain your changes.
4. Avoid making unnecessary changes, especially if they're purely based on personal
   preferences. Doing so is the maintainers' job. ;-)
-->

- [x] Have you used the PR description template? &ensp; `+2 pts`
- [x] Is your pull request atomic, focusing on a single change? &ensp; `+5 pts`
- [x] Have you linked the GitHub issue(s) that this PR addresses? &ensp; `+5 pts`
- [x] Have you documented your changes clearly and comprehensively? &ensp; `+5 pts`
- [x] Have you changed or added a feature? &ensp; `-4 pts`
  - [ ] Have you added/updated corresponding documentation? &ensp; `+4 pts`
  - [ ] Have you added/updated corresponding integration tests? &ensp; `+5 pts`
- [ ] Have you changed the behavior of AutoGPT? &ensp; `-5 pts`
  - [ ] Have you also run `agbenchmark` to verify that these changes do not regress performance? &ensp; `+10 pts`


Problem statement:
Support using other/local LLMs
You can modify the code to accept a config file as input, and read the Chosen_Model flag to select the appropriate AI model. Here's an example of how to achieve this:

Create a sample config file named config.ini:

[AI]
Chosen_Model = gpt-4

Offload the call_ai_fuction from the ai_functions.py to a separate library. Modify the call_ai_function function to read the model from the config file:

```
import configparser

def call_ai_function(function, args, description, config_path="config.ini"):
    # Load the configuration file
    config = configparser.ConfigParser()
    config.read(config_path)

    # Get the chosen model from the config file
    model = config.get("AI", "Chosen_Model", fallback="gpt-4")

    # Parse args to comma separated string
    args = ", ".join(args)
    messages = [
        {
            "role": "system",
            "content": f"You are now the following python function: ```# {description}\n{function}```\n\nOnly respond with your `return` value.",
        },
        {"role": "user", "content": args},
    ]

  # Use different AI APIs based on the chosen model
    if model == "gpt-4":
        response = openai.ChatCompletion.create(
            model=model, messages=messages, temperature=0
        )
    elif model == "some_other_api":
        # Add code to call another AI API with the appropriate parameters
        response = some_other_api_call(parameters)
    else:
        raise ValueError(f"Unsupported model: {model}")

    return response.choices[0].message["content"]

```
In this modified version, the call_ai_function takes an additional parameter config_path which defaults to "config.ini". The function reads the config file, retrieves the Chosen_Model value, and uses it as the model for the OpenAI API call. If the Chosen_Model flag is not found in the config file, it defaults to "gpt-4".

 the if/elif structure is used to call different AI APIs based on the chosen model from the configuration file. Replace some_other_api with the name of the API you'd like to use, and replace parameters with the appropriate parameters required by that API. You can extend the if/elif structure to include more AI APIs as needed.

Instructions for local models
Are there any instructions for using local models rather than GPT-3 or 4?  Is there a way to set the basepath to 127.0.0.1:11435 to use ollama or to 1234/v2 for LM Studio?  Is there a configuration file or environment variables to set for this?  Thank you for sharing your wonderful software with the AI community.

Hints from issue discussion:
Excellent work. Lots of people are asking for this submit a pull request!

In order to fully support gpt3.5 (and other models) we need to harden the prompt.

@Koobah had some success by adding this line to the end of prompt.txt:
```
Before submitting the response, simulate parsing the response with Python json.loads. Don't submit unless it can be parsed.
```

This would also help out #21 

I see you looking over my shoulder. Thoughts?

I've got a friend who is going to clone the branch and test for me. (hopefully) I don't have a working environment atm.
I also changed the user prompt from NEXT COMMAND to GENERATE NEXT COMMAND JSON. Basically reminding it to use JSON whenever possible
It's still not 100% working, though. 
https://github.com/DataBassGit/Auto-GPT/blob/master/scripts/ai_function_lib.py

@Koobah This is basically what I'm working with atm. I think we can probably add a verify_json function to a gpt-3.5 segment of that function.
Pull request on this is submitted. I'm going to start looking into more models and platforms that can be incorporated.
Would be huge if it can run llama.cpp locally.
I might be able to swing that. Let's see if this merge gets approved. I'm also looking at implementing GPT4all.
@DataBassGit I see that PR got closed. What's the status of your fork?
They moved the API call to GPT-4 to an external library in main.py, but there are still some scripts that call openai directly, like chat.py, browse.py etc. 

GPT4all doesn't support x64 architecture. I also tried some APIs on hugging face, but it seems that it truncates responses on the free API endpoints.

I just converted BabyAGI to Oobabooga, but it's untested. Should be getting to that tonight. If it works, I will start working on porting AutoGPT to Oobabooga as well. The nice thing about this method is that it allows for local or remote hosting, and can handle many different language models without much issue.

Hugging face should work, though. I need to review Microsoft/Jarvis. They make heavy use of HF APIs.
GPT4all supports x64 and every architecture llama.cpp supports, which is every architecture (even non-POSIX, and webassemly). Their moto is "Can it run ~Doom~ LLaMA" for a reason.

Ooga supports GPT4all (and all llama.cpp ggml models), since it packages llama.cpp and the llamacpp python bindings library. So porting it to ooba would effectively resolve this.
This issue has automatically been marked as _stale_ because it has not had any activity in the last 50 days. You can _unstale_ it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.
Please, any news here?
I am also looking for a solution to this problem.
I was trying to get ollama running with AutoGPT.

curl works:
``` bash
curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "mistral:instruct",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'
{"id":"chatcmpl-447","object":"chat.completion","created":1707528048,"model":"mistral:instruct","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":" Hello there! I'm here to help answer any questions you might have or assist with tasks you may need assistance with. What can I help you with today?\n\nHere are some things I can do:\n\n1. Answer general knowledge questions\n2. Help with math problems\n3. Set reminders and alarms\n4. Create to-do lists and manage tasks\n5. Provide weather updates\n6. Tell jokes or share interesting facts\n7. Assist with email and calendar management\n8. Play music, set timers for cooking, and more!\n\nLet me know what you need help with and I'll do my best to assist!"},"finish_reason":"stop"}],"usage":{"prompt_tokens":16,"completion_tokens":140,"total_tokens":156}}
```

but with this AutoGPT config:

``` bash
## OPENAI_API_KEY - OpenAI API Key (Example: my-openai-api-key)
OPENAI_API_KEY=ollama

## OPENAI_API_BASE_URL - Custom url for the OpenAI API, useful for connecting to custom backends. No effect if USE_AZURE is true, leave blank to keep the default url
# the following is an example:
OPENAI_API_BASE_URL= http://localhost:11434/v1/chat/completions

## SMART_LLM - Smart language model (Default: gpt-4-0314)
SMART_LLM=mixtral:8x7b-instruct-v0.1-q2_K

## FAST_LLM - Fast language model (Default: gpt-3.5-turbo-16k)
FAST_LLM=mistral:instruct
```

I can't get the connection:

```
File "/venv/agpt-9TtSrW0h-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 919, in _request
    raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.
```

maybe someone will figure it out and can post an update here
Connection is solvable via proxy, but then you will get pydantic errors everywhere because Mistral is producing wrong json
If an openai-ready api is needed, I think you can go through litellm to make a bridge to your ollama instance: https://docs.litellm.ai/docs/providers/ollama
Hi!
I am still fighting with ollama to try proxying an agent on my own, but one important thing I want to mention is regarding this:

> Connection is solvable via proxy, but then you will get pydantic errors everywhere because Mistral is producing wrong json

Can't we theoretically code an agent that uses GBNF grammar files for forcing Mistral or other local LLMs to produce correct JSON?

A simple example for correct JSON is viewable in the llama.cpp repo: https://github.com/ggerganov/llama.cpp/blob/master/grammars/json.gbnf
Then you include the correct GBNF in your llama.cpp command (I figure it would be a problem if the Ollama API doesn't support it though).

There are even programs now that generate correct GBNF files based on JSON definitions: https://github.com/richardanaya/gbnf
> If an openai-ready api is needed, I think you can go through litellm to make a bridge to your ollama instance: https://docs.litellm.ai/docs/providers/ollama

@qwertyuu, I thought ollama supports OpenAI compatible API without LiteLLM (https://ollama.com/blog/openai-compatibility). Am I missing something?


I got it to run with Mistral 7B AWQ, neural chat v3 AWQ and a few other models.
Only thing is I had to write my own Auto-GPT from scratch as the prompts from Auto-GPT are too long and confusing for the local llms.
They return correct prompts sometimes, but other times they concentrate so much on the system Prompt by Auto-GPT that they respond with "Hello, I am using the command ask_user to talk to the user, is this correct?" and then it says "Hello, how can I help you?" like 100 times until I cancel it.

My current use case of using oobabooga text-generation-webui works best when I add the JSON grammar to it. It then works with very basic prompts only and only a few commands, otherwise it kept making up new commands and started halucinating and responding with multiple commands at once etc.
I got it to make calls to a [llamafile](https://github.com/Mozilla-Ocho/llamafile/tree/main) server running locally (which has an OpenAI-compatible API) by just setting `OPENAI_API_BASE_URL=http://localhost:8080/v1` in my .env.  I know the requests are getting through based on the debug logs (plus I can see the calls coming into my llamafile server).

However, since the model I'm using doesn't support function calling, the json it returns has null for the `tool_calls` field which results in `ValueError: LLM did not call create_agent function; agent profile creation failed` coming from here: https://github.com/Significant-Gravitas/AutoGPT/blob/a9b7b175fff94ab4a9c5d6c91537089199b27a09/autogpts/autogpt/autogpt/agent_factory/profile_generator.py#L202

Also, setting `OPENAI_FUNCTIONS=False` does not seem to do anything.

If anyone knows of an open source gguf or lllamafile-format model that supports function calling, let me know. That *might* fix this issue?

Commit message: combine llamafile setup.sh and serve.sh into single cross-platform serve.py

Patch to review:
diff --git a/autogpt/scripts/llamafile/.gitignore b/autogpt/scripts/llamafile/.gitignore
new file mode 100644
index 000000000000..55e65ce9706e
--- /dev/null
+++ b/autogpt/scripts/llamafile/.gitignore
@@ -0,0 +1,2 @@
+*.llamafile
+*.llamafile.exe
diff --git a/autogpt/scripts/llamafile/serve.py b/autogpt/scripts/llamafile/serve.py
new file mode 100755
index 000000000000..59867dea8856
--- /dev/null
+++ b/autogpt/scripts/llamafile/serve.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+"""
+Use llamafile to serve a (quantized) mistral-7b-instruct-v0.2 model
+
+Usage:
+  cd <repo-root>/autogpt
+  ./scripts/llamafile/serve.py
+"""
+
+import os
+import platform
+import subprocess
+from pathlib import Path
+
+LLAMAFILE = Path(
+    "mistral-7b-instruct-v0.2.Q5_K_M.llamafile"
+    + (".exe" if platform.system() == "Windows" else "")
+)
+
+
+def report_download_progress(chunk_number: int, chunk_size: int, total_size: int):
+    if total_size != -1:
+        downloaded_size = chunk_number * chunk_size
+        percent = min(1, downloaded_size / total_size)
+        bar = "#" * int(40 * percent)
+        print(
+            f"\rDownloading: [{bar:<40}] {percent:.0%}"
+            f" - {downloaded_size/1e6:.1f}/{total_size/1e6:.1f} MB",
+            end="",
+        )
+
+
+def download_llamafile():
+    print(f"Downloading {LLAMAFILE.name}...")
+    import urllib.request
+
+    url = "https://huggingface.co/jartine/Mistral-7B-Instruct-v0.2-llamafile/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.llamafile"  # noqa
+
+    urllib.request.urlretrieve(url, LLAMAFILE.name, reporthook=report_download_progress)
+    print()
+
+    LLAMAFILE.chmod(0o755)
+    subprocess.run([LLAMAFILE, "--version"], check=True)
+
+    print(
+        "\n"
+        "NOTE: To use other models besides mistral-7b-instruct-v0.2, "
+        "download them into autogpt/scripts/llamafile/"
+    )
+
+
+# Go to autogpt/scripts/llamafile/
+os.chdir(Path(__file__).resolve().parent)
+
+if not LLAMAFILE.is_file():
+    download_llamafile()
+
+subprocess.run(
+    [LLAMAFILE, "--server", "--nobrowser", "--ctx-size", "0", "--n-predict", "1024"],
+    check=True,
+)
+
+# note: --ctx-size 0 means the prompt context size will be set directly from the
+# underlying model configuration. This may cause slow response times or consume
+# a lot of memory.
diff --git a/docs/content/AutoGPT/setup/index.md b/docs/content/AutoGPT/setup/index.md
index 43aba6474774..becc808ed2e3 100644
--- a/docs/content/AutoGPT/setup/index.md
+++ b/docs/content/AutoGPT/setup/index.md
@@ -190,3 +190,35 @@ If you don't know which to choose, you can safely go with OpenAI*.
 
 [groq/api-keys]: https://console.groq.com/keys
 [groq/models]: https://console.groq.com/docs/models
+
+
+### Llamafile
+
+With llamafile you can run models locally, which means no need to set up billing,
+and guaranteed data privacy.
+
+!!! warning
+    At the moment, llamafile only serves one model at a time. This means you can not
+    set `SMART_LLM` and `FAST_LLM` to two different llamafile models.
+
+!!! note
+    These instructions will download and use `mistral-7b-instruct-v0.2.Q5_K_M.llamafile`.
+    `mistral-7b-instruct-v0.2` is currently the only tested and supported model.
+    If you want to try other models, you'll have to add them to `LlamafileModelName` in
+    [`llamafile.py`][forge/llamafile.py].
+    For optimal results, you may also have to add some logic to adapt the message format,
+    like `LlamafileProvider._adapt_chat_messages_for_mistral_instruct(..)` does.
+
+1. Run the llamafile serve script:
+   ```shell
+   python3 ./scripts/llamafile/serve.py
+   ```
+   The first time this is run, it will download a file containing the model + runtime,
+   which may take a while and a few gigabytes of disk space.
+
+3. In `.env`, set `SMART_LLM`/`FAST_LLM` or both to `mistral-7b-instruct-v0.2`
+
+4. If the server is running on different address than `http://localhost:8080/v1`,
+   set `LLAMAFILE_API_BASE` in `.env` to the right base URL
+
+[forge/llamafile.py]: https://github.com/Significant-Gravitas/AutoGPT/blob/master/forge/forge/llm/providers/llamafile/llamafile.py
diff --git a/forge/forge/llm/providers/llamafile/README.md b/forge/forge/llm/providers/llamafile/README.md
new file mode 100644
index 000000000000..e4276ec1a941
--- /dev/null
+++ b/forge/forge/llm/providers/llamafile/README.md
@@ -0,0 +1,36 @@
+# Llamafile Integration Notes
+
+Tested with:
+* Python 3.11
+* Apple M2 Pro (32 GB), macOS 14.2.1
+* quantized mistral-7b-instruct-v0.2
+
+## Setup
+
+Download a `mistral-7b-instruct-v0.2` llamafile:
+```shell
+wget -nc https://huggingface.co/jartine/Mistral-7B-Instruct-v0.2-llamafile/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.llamafile
+chmod +x mistral-7b-instruct-v0.2.Q5_K_M.llamafile
+./mistral-7b-instruct-v0.2.Q5_K_M.llamafile --version
+```
+
+Run the llamafile server:
+```shell
+LLAMAFILE="./mistral-7b-instruct-v0.2.Q5_K_M.llamafile"
+
+"${LLAMAFILE}" \
+--server \
+--nobrowser \
+--ctx-size 0 \
+--n-predict 1024
+
+# note: ctx-size=0 means the prompt context size will be set directly from the
+# underlying model configuration. This may cause slow response times or consume
+# a lot of memory.
+```
+
+## TODOs
+
+* `SMART_LLM`/`FAST_LLM` configuration: Currently, the llamafile server only serves one model at a time. However, there's no reason you can't start multiple llamafile servers on different ports. To support using different models for `smart_llm` and `fast_llm`, you could implement config vars like `LLAMAFILE_SMART_LLM_URL` and `LLAMAFILE_FAST_LLM_URL` that point to different llamafile servers (one serving a 'big model' and one serving a 'fast model'). 
+* Authorization: the `serve.sh` script does not set up any authorization for the llamafile server; this can be turned on by adding arg `--api-key <some-key>` to the server startup command. However I haven't attempted to test whether the integration with autogpt works when this feature is turned on.
+* Test with other models
diff --git a/forge/forge/llm/providers/llamafile/__init__.py b/forge/forge/llm/providers/llamafile/__init__.py
new file mode 100644
index 000000000000..23706b102be6
--- /dev/null
+++ b/forge/forge/llm/providers/llamafile/__init__.py
@@ -0,0 +1,17 @@
+from .llamafile import (
+    LLAMAFILE_CHAT_MODELS,
+    LLAMAFILE_EMBEDDING_MODELS,
+    LlamafileCredentials,
+    LlamafileModelName,
+    LlamafileProvider,
+    LlamafileSettings,
+)
+
+__all__ = [
+    "LLAMAFILE_CHAT_MODELS",
+    "LLAMAFILE_EMBEDDING_MODELS",
+    "LlamafileCredentials",
+    "LlamafileModelName",
+    "LlamafileProvider",
+    "LlamafileSettings",
+]
diff --git a/forge/forge/llm/providers/llamafile/llamafile.py b/forge/forge/llm/providers/llamafile/llamafile.py
new file mode 100644
index 000000000000..e5753bc45c87
--- /dev/null
+++ b/forge/forge/llm/providers/llamafile/llamafile.py
@@ -0,0 +1,317 @@
+import enum
+import logging
+import re
+from pathlib import Path
+from typing import Any, Iterator, Optional, Sequence
+
+import requests
+from openai.types.chat import (
+    ChatCompletionMessage,
+    ChatCompletionMessageParam,
+    CompletionCreateParams,
+)
+from pydantic import SecretStr
+
+from forge.json.parsing import json_loads
+from forge.models.config import UserConfigurable
+
+from .._openai_base import BaseOpenAIChatProvider
+from ..schema import (
+    AssistantToolCall,
+    AssistantToolCallDict,
+    ChatMessage,
+    ChatModelInfo,
+    CompletionModelFunction,
+    ModelProviderConfiguration,
+    ModelProviderCredentials,
+    ModelProviderName,
+    ModelProviderSettings,
+    ModelTokenizer,
+)
+
+
+class LlamafileModelName(str, enum.Enum):
+    MISTRAL_7B_INSTRUCT = "mistral-7b-instruct-v0.2"
+
+
+LLAMAFILE_CHAT_MODELS = {
+    info.name: info
+    for info in [
+        ChatModelInfo(
+            name=LlamafileModelName.MISTRAL_7B_INSTRUCT,
+            provider_name=ModelProviderName.LLAMAFILE,
+            prompt_token_cost=0.0,
+            completion_token_cost=0.0,
+            max_tokens=32768,
+            has_function_call_api=False,
+        ),
+    ]
+}
+
+LLAMAFILE_EMBEDDING_MODELS = {}
+
+
+class LlamafileCredentials(ModelProviderCredentials):
+    api_key = SecretStr("sk-no-key-required")
+    api_base: SecretStr = UserConfigurable(  # type: ignore
+        default=SecretStr("http://localhost:8080/v1"), from_env="LLAMAFILE_API_BASE"
+    )
+
+
+class LlamafileTokenizer(ModelTokenizer[int]):
+    def __init__(self, credentials: LlamafileCredentials):
+        self._credentials = credentials
+
+    @property
+    def _tokenizer_base_url(self):
+        # The OpenAI-chat-compatible base url should look something like
+        # 'http://localhost:8080/v1' but the tokenizer endpoint is
+        # 'http://localhost:8080/tokenize'. So here we just strip off the '/v1'.
+        api_base = self._credentials.api_base.get_secret_value()
+        return api_base.strip("/v1")
+
+    def encode(self, text: str) -> list[int]:
+        response = requests.post(
+            url=f"{self._tokenizer_base_url}/tokenize", json={"content": text}
+        )
+        response.raise_for_status()
+        return response.json()["tokens"]
+
+    def decode(self, tokens: list[int]) -> str:
+        response = requests.post(
+            url=f"{self._tokenizer_base_url}/detokenize", json={"tokens": tokens}
+        )
+        response.raise_for_status()
+        return response.json()["content"]
+
+
+class LlamafileSettings(ModelProviderSettings):
+    credentials: Optional[LlamafileCredentials]  # type: ignore
+
+
+class LlamafileProvider(
+    BaseOpenAIChatProvider[LlamafileModelName, LlamafileSettings],
+    # TODO: add and test support for embedding models
+    # BaseOpenAIEmbeddingProvider[LlamafileModelName, LlamafileSettings],
+):
+    EMBEDDING_MODELS = LLAMAFILE_EMBEDDING_MODELS
+    CHAT_MODELS = LLAMAFILE_CHAT_MODELS
+    MODELS = {**CHAT_MODELS, **EMBEDDING_MODELS}
+
+    default_settings = LlamafileSettings(
+        name="llamafile_provider",
+        description=(
+            "Provides chat completion and embedding services "
+            "through a llamafile instance"
+        ),
+        configuration=ModelProviderConfiguration(),
+    )
+
+    _settings: LlamafileSettings  # type: ignore
+    _credentials: LlamafileCredentials  # type: ignore
+
+    async def get_available_models(self) -> Sequence[ChatModelInfo[LlamafileModelName]]:
+        _models = (await self._client.models.list()).data
+        # note: at the moment, llamafile only serves one model at a time (so this
+        # list will only ever have one value). however, in the future, llamafile
+        # may support multiple models, so leaving this method as-is for now.
+        self._logger.debug(f"Retrieved models: {_models}")
+
+        # Clean up model names:
+        # 1. Remove file extension
+        # 2. Remove quantization info
+        # e.g. 'mistral-7b-instruct-v0.2.Q5_K_M.gguf'
+        #   -> 'mistral-7b-instruct-v0.2'
+        # e.g. '/Users/kate/models/mistral-7b-instruct-v0.2.Q5_K_M.gguf'
+        #   ->                    'mistral-7b-instruct-v0.2'
+        # e.g. 'llava-v1.5-7b-q4.gguf'
+        #   -> 'llava-v1.5-7b'
+        def clean_model_name(model_file: str) -> str:
+            name_without_ext = Path(model_file).name.rsplit(".", 1)[0]
+            name_without_Q = re.match(
+                r"^[a-zA-Z0-9]+([.\-](?!([qQ]|B?F)\d{1,2})[a-zA-Z0-9]+)*",
+                name_without_ext,
+            )
+            return name_without_Q.group() if name_without_Q else name_without_ext
+
+        clean_model_ids = [clean_model_name(m.id) for m in _models]
+        self._logger.debug(f"Cleaned model IDs: {clean_model_ids}")
+
+        return [
+            LLAMAFILE_CHAT_MODELS[id]
+            for id in clean_model_ids
+            if id in LLAMAFILE_CHAT_MODELS
+        ]
+
+    def get_tokenizer(self, model_name: LlamafileModelName) -> LlamafileTokenizer:
+        return LlamafileTokenizer(self._credentials)
+
+    def count_message_tokens(
+        self,
+        messages: ChatMessage | list[ChatMessage],
+        model_name: LlamafileModelName,
+    ) -> int:
+        if isinstance(messages, ChatMessage):
+            messages = [messages]
+
+        if model_name == LlamafileModelName.MISTRAL_7B_INSTRUCT:
+            # For mistral-instruct, num added tokens depends on if the message
+            # is a prompt/instruction or an assistant-generated message.
+            # - prompt gets 
, 
 added and the first instruction
+            # begins with '
' ('beginning-of-sentence' token).
+            # - assistant-generated messages get '
' added
+            # see: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
+            #
+            prompt_added = 1  # one for '
' token
+            assistant_num_added = 0
+            ntokens = 0
+            for message in messages:
+                if (
+                    message.role == ChatMessage.Role.USER
+                    # note that 'system' messages will get converted
+                    # to 'user' messages before being sent to the model
+                    or message.role == ChatMessage.Role.SYSTEM
+                ):
+                    # 5 tokens for 
, 
, which actually get
+                    # tokenized into "[, INST, ]" and "[, /, INST, ]"
+                    # by the mistral tokenizer
+                    prompt_added += 5
+                elif message.role == ChatMessage.Role.ASSISTANT:
+                    assistant_num_added += 1  # for 

+                else:
+                    raise ValueError(
+                        f"{model_name} does not support role: {message.role}"
+                    )
+
+                ntokens += self.count_tokens(message.content, model_name)
+
+            total_token_count = prompt_added + assistant_num_added + ntokens
+            return total_token_count
+
+        else:
+            raise NotImplementedError(
+                f"count_message_tokens not implemented for model {model_name}"
+            )
+
+    def _get_chat_completion_args(
+        self,
+        prompt_messages: list[ChatMessage],
+        model: LlamafileModelName,
+        functions: list[CompletionModelFunction] | None = None,
+        max_output_tokens: int | None = None,
+        **kwargs,
+    ) -> tuple[
+        list[ChatCompletionMessageParam], CompletionCreateParams, dict[str, Any]
+    ]:
+        messages, completion_kwargs, parse_kwargs = super()._get_chat_completion_args(
+            prompt_messages, model, functions, max_output_tokens, **kwargs
+        )
+
+        if model == LlamafileModelName.MISTRAL_7B_INSTRUCT:
+            messages = self._adapt_chat_messages_for_mistral_instruct(messages)
+
+        if "seed" not in kwargs:
+            # FIXME: temporarily hard-coded for reproducibility, instead the
+            #  seed should be set from config
+            completion_kwargs["seed"] = 0
+
+        return messages, completion_kwargs, parse_kwargs
+
+    def _adapt_chat_messages_for_mistral_instruct(
+        self, messages: list[ChatCompletionMessageParam]
+    ) -> list[ChatCompletionMessageParam]:
+        """
+        Munge the messages to be compatible with the mistral-7b-instruct chat
+        template, which:
+        - only supports 'user' and 'assistant' roles.
+        - expects messages to alternate between user/assistant roles.
+
+        See details here:
+        https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2#instruction-format
+        """
+        adapted_messages: list[ChatCompletionMessageParam] = []
+        for message in messages:
+            # convert 'system' role to 'user' role as mistral-7b-instruct does
+            # not support 'system'
+            if message["role"] == ChatMessage.Role.SYSTEM:
+                message["role"] = ChatMessage.Role.USER
+
+            if (
+                len(adapted_messages) == 0
+                or message["role"] != (last_message := adapted_messages[-1])["role"]
+            ):
+                adapted_messages.append(message)
+            else:
+                if not message.get("content"):
+                    continue
+
+                # if the curr message has the same role as the previous one,
+                # concat the current message content to the prev message
+                if message["role"] == "user" and last_message["role"] == "user":
+                    # user messages can contain other types of content blocks
+                    if not isinstance(last_message["content"], list):
+                        last_message["content"] = [
+                            {"type": "text", "text": last_message["content"]}
+                        ]
+
+                    last_message["content"].extend(
+                        message["content"]
+                        if isinstance(message["content"], list)
+                        else [{"type": "text", "text": message["content"]}]
+                    )
+                elif message["role"] != "user" and last_message["role"] != "user":
+                    last_message["content"] = (
+                        (last_message.get("content") or "")
+                        + "\n\n"
+                        + (message.get("content") or "")
+                    ).strip()
+
+        return adapted_messages
+
+    def _parse_assistant_tool_calls(
+        self,
+        assistant_message: ChatCompletionMessage,
+        compat_mode: bool = False,
+        **kwargs,
+    ):
+        tool_calls: list[AssistantToolCall] = []
+        parse_errors: list[Exception] = []
+
+        if compat_mode and assistant_message.content:
+            try:
+                tool_calls = list(
+                    _tool_calls_compat_extract_calls(assistant_message.content)
+                )
+            except Exception as e:
+                parse_errors.append(e)
+
+        return tool_calls, parse_errors
+
+
+def _tool_calls_compat_extract_calls(response: str) -> Iterator[AssistantToolCall]:
+    import re
+    import uuid
+
+    logging.debug(f"Trying to extract tool calls from response:\n{response}")
+
+    response = response.strip()  # strip off any leading/trailing whitespace
+    if response.startswith("```"):
+        # attempt to remove any extraneous markdown artifacts like "```json"
+        response = response.strip("```")
+        if response.startswith("json"):
+            response = response.strip("json")
+        response = response.strip()  # any remaining whitespace
+
+    if response[0] == "[":
+        tool_calls: list[AssistantToolCallDict] = json_loads(response)
+    else:
+        block = re.search(r"```(?:tool_calls)?\n(.*)\n```\s*$", response, re.DOTALL)
+        if not block:
+            raise ValueError("Could not find tool_calls block in response")
+        tool_calls: list[AssistantToolCallDict] = json_loads(block.group(1))
+
+    for t in tool_calls:
+        t["id"] = str(uuid.uuid4())
+        # t["function"]["arguments"] = str(t["function"]["arguments"])  # HACK
+
+        yield AssistantToolCall.parse_obj(t)
diff --git a/forge/forge/llm/providers/multi.py b/forge/forge/llm/providers/multi.py
index 93784cd2eae5..ee42c2a8446b 100644
--- a/forge/forge/llm/providers/multi.py
+++ b/forge/forge/llm/providers/multi.py
@@ -7,6 +7,7 @@
 
 from .anthropic import ANTHROPIC_CHAT_MODELS, AnthropicModelName, AnthropicProvider
 from .groq import GROQ_CHAT_MODELS, GroqModelName, GroqProvider
+from .llamafile import LLAMAFILE_CHAT_MODELS, LlamafileModelName, LlamafileProvider
 from .openai import OPEN_AI_CHAT_MODELS, OpenAIModelName, OpenAIProvider
 from .schema import (
     AssistantChatMessage,
@@ -24,10 +25,15 @@
 
 _T = TypeVar("_T")
 
-ModelName = AnthropicModelName | GroqModelName | OpenAIModelName
+ModelName = AnthropicModelName | GroqModelName | LlamafileModelName | OpenAIModelName
 EmbeddingModelProvider = OpenAIProvider
 
-CHAT_MODELS = {**ANTHROPIC_CHAT_MODELS, **GROQ_CHAT_MODELS, **OPEN_AI_CHAT_MODELS}
+CHAT_MODELS = {
+    **ANTHROPIC_CHAT_MODELS,
+    **GROQ_CHAT_MODELS,
+    **LLAMAFILE_CHAT_MODELS,
+    **OPEN_AI_CHAT_MODELS,
+}
 
 
 class MultiProvider(BaseChatModelProvider[ModelName, ModelProviderSettings]):
@@ -153,6 +159,7 @@ def _get_provider_class(
             return {
                 ModelProviderName.ANTHROPIC: AnthropicProvider,
                 ModelProviderName.GROQ: GroqProvider,
+                ModelProviderName.LLAMAFILE: LlamafileProvider,
                 ModelProviderName.OPENAI: OpenAIProvider,
             }[provider_name]
         except KeyError:
@@ -162,4 +169,10 @@ def __repr__(self):
         return f"{self.__class__.__name__}()"
 
 
-ChatModelProvider = AnthropicProvider | GroqProvider | OpenAIProvider | MultiProvider
+ChatModelProvider = (
+    AnthropicProvider
+    | GroqProvider
+    | LlamafileProvider
+    | OpenAIProvider
+    | MultiProvider
+)
diff --git a/forge/forge/llm/providers/schema.py b/forge/forge/llm/providers/schema.py
index 11b522e77960..297db5a5ce9b 100644
--- a/forge/forge/llm/providers/schema.py
+++ b/forge/forge/llm/providers/schema.py
@@ -55,6 +55,7 @@ class ModelProviderName(str, enum.Enum):
     OPENAI = "openai"
     ANTHROPIC = "anthropic"
     GROQ = "groq"
+    LLAMAFILE = "llamafile"
 
 
 class ChatMessage(BaseModel):


Code review comments:
path: autogpt/scripts/llamafile/serve.py
text: @user1:
If this is just as example then fine but otherwise `LLAMAFILE` could be provided as an argument.

@user2:
Implemented

diff_hunk: @@ -0,0 +1,317 @@
+import enum
+import logging
+import re
+from pathlib import Path
+from typing import Any, Iterator, Optional, Sequence
+
+import requests
+from openai.types.chat import (
+    ChatCompletionMessage,
+    ChatCompletionMessageParam,
+    CompletionCreateParams,
+)
+from pydantic import SecretStr
+
+from forge.json.parsing import json_loads
+from forge.models.config import UserConfigurable
+
+from .._openai_base import BaseOpenAIChatProvider
+from ..schema import (
+    AssistantToolCall,
+    AssistantToolCallDict,
+    ChatMessage,
+    ChatModelInfo,
+    CompletionModelFunction,
+    ModelProviderConfiguration,
+    ModelProviderCredentials,
+    ModelProviderName,
+    ModelProviderSettings,
+    ModelTokenizer,
+)
+
+
+class LlamafileModelName(str, enum.Enum):
+    MISTRAL_7B_INSTRUCT = "mistral-7b-instruct-v0.2"
+
+
+LLAMAFILE_CHAT_MODELS = {
+    info.name: info
+    for info in [
+        ChatModelInfo(
+            name=LlamafileModelName.MISTRAL_7B_INSTRUCT,
+            provider_name=ModelProviderName.LLAMAFILE,
+            prompt_token_cost=0.0,
+            completion_token_cost=0.0,
+            max_tokens=32768,
+            has_function_call_api=False,
+        ),
+    ]
+}
+
+LLAMAFILE_EMBEDDING_MODELS = {}
+
+
+class LlamafileCredentials(ModelProviderCredentials):
+    api_key = SecretStr("sk-no-key-required")
+    api_base: SecretStr = UserConfigurable(  # type: ignore
+        default=SecretStr("http://localhost:8080/v1"), from_env="LLAMAFILE_API_BASE"
+    )
+
+
+class LlamafileTokenizer(ModelTokenizer[int]):
+    def __init__(self, credentials: LlamafileCredentials):
+        self._credentials = credentials
+
+    @property
+    def _tokenizer_base_url(self):
+        # The OpenAI-chat-compatible base url should look something like
+        # 'http://localhost:8080/v1' but the tokenizer endpoint is
+        # 'http://localhost:8080/tokenize'. So here we just strip off the '/v1'.
+        api_base = self._credentials.api_base.get_secret_value()
+        return api_base.strip("/v1")
+
+    def encode(self, text: str) -> list[int]:
+        response = requests.post(
+            url=f"{self._tokenizer_base_url}/tokenize", json={"content": text}
+        )
+        response.raise_for_status()
+        return response.json()["tokens"]
+
+    def decode(self, tokens: list[int]) -> str:
+        response = requests.post(
+            url=f"{self._tokenizer_base_url}/detokenize", json={"tokens": tokens}
+        )
+        response.raise_for_status()
+        return response.json()["content"]
+
+
+class LlamafileSettings(ModelProviderSettings):
+    credentials: Optional[LlamafileCredentials]  # type: ignore
+
+
+class LlamafileProvider(
+    BaseOpenAIChatProvider[LlamafileModelName, LlamafileSettings],
+    # TODO: add and test support for embedding models
+    # BaseOpenAIEmbeddingProvider[LlamafileModelName, LlamafileSettings],
+):
+    EMBEDDING_MODELS = LLAMAFILE_EMBEDDING_MODELS
+    CHAT_MODELS = LLAMAFILE_CHAT_MODELS
+    MODELS = {**CHAT_MODELS, **EMBEDDING_MODELS}
+
+    default_settings = LlamafileSettings(
+        name="llamafile_provider",
+        description=(
+            "Provides chat completion and embedding services "
+            "through a llamafile instance"
+        ),
+        configuration=ModelProviderConfiguration(),
+    )
+
+    _settings: LlamafileSettings  # type: ignore
+    _credentials: LlamafileCredentials  # type: ignore
+
+    async def get_available_models(self) -> Sequence[ChatModelInfo[LlamafileModelName]]:
+        _models = (await self._client.models.list()).data
+        # note: at the moment, llamafile only serves one model at a time (so this
+        # list will only ever have one value). however, in the future, llamafile
+        # may support multiple models, so leaving this method as-is for now.
+        self._logger.debug(f"Retrieved models: {_models}")
+
+        # Clean up model names:
+        # 1. Remove file extension
+        # 2. Remove quantization info
+        # e.g. 'mistral-7b-instruct-v0.2.Q5_K_M.gguf'
+        #   -> 'mistral-7b-instruct-v0.2'
+        # e.g. '/Users/kate/models/mistral-7b-instruct-v0.2.Q5_K_M.gguf'
+        #   ->                    'mistral-7b-instruct-v0.2'
+        # e.g. 'llava-v1.5-7b-q4.gguf'
+        #   -> 'llava-v1.5-7b'
+        def clean_model_name(model_file: str) -> str:
path: forge/forge/llm/providers/llamafile/llamafile.py
text: @user1:
I like that the explanation is added here!
nit: we use `"""comment"""` just below the member:

```suggestion
        def clean_model_name(model_file: str) -> str:
        """Clean up model names:
            1. Remove file extension
            2. Remove quantization info
            e.g. 'mistral-7b-instruct-v0.2.Q5_K_M.gguf'
              -> 'mistral-7b-instruct-v0.2'
            e.g. '/Users/kate/models/mistral-7b-instruct-v0.2.Q5_K_M.gguf'
              ->                    'mistral-7b-instruct-v0.2'
            e.g. 'llava-v1.5-7b-q4.gguf'
              -> 'llava-v1.5-7b'"""
```