Repository: Significant-Gravitas/AutoGPT
Pull request: feat(forge/llm): Add `GroqProvider`
### Background

This has issues with the profile generator, so maybe merge #7121 first

### Changes 🏗️

* Add `GroqProvider` in `forge.llm.providers.groq`
  * Add to `llm.providers.multi`
  * Add `groq` dependency (v0.8.0)

* Update AutoGPT docs & config template
  * Update .env.template
  * Update docs

### PR Quality Scorecard ✨

<!--
Check out our contribution guide:
https://github.com/Significant-Gravitas/Nexus/wiki/Contributing

1. Avoid duplicate work, issues, PRs etc.
2. Also consider contributing something other than code; see the [contribution guide]
   for options.
3. Clearly explain your changes.
4. Avoid making unnecessary changes, especially if they're purely based on personal
   preferences. Doing so is the maintainers' job. ;-)
-->

- [x] Have you used the PR description template? &ensp; `+2 pts`
- [x] Is your pull request atomic, focusing on a single change? &ensp; `+5 pts`
- [ ] Have you linked the GitHub issue(s) that this PR addresses? &ensp; `+5 pts`
- [x] Have you documented your changes clearly and comprehensively? &ensp; `+5 pts`
- [x] Have you changed or added a feature? &ensp; `-4 pts`
  - [x] Have you added/updated corresponding documentation? &ensp; `+4 pts`
  - [ ] Have you added/updated corresponding integration tests? &ensp; `+5 pts`
- [ ] Have you changed the behavior of AutoGPT? &ensp; `-5 pts`
  - [ ] Have you also run `agbenchmark` to verify that these changes do not regress performance? &ensp; `+10 pts`


___

### **PR Type**
Enhancement


___

### **Description**
- Introduced `GroqProvider` to manage interactions with the Groq API, including methods for chat completions and API retries.
- Updated `multi.py` to integrate `GroqProvider` alongside other model providers.
- Extended `ModelProviderName` enum to include `GROQ`.
- Added the `groq` package to project dependencies to ensure API interaction capabilities.



___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement
</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>groq.py</strong><dd><code>Implement GroqProvider for Groq API Interaction</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>
      
autogpts/autogpt/autogpt/core/resource/model_providers/groq.py

<li>Introduced a new <code>GroqProvider</code> class for handling interactions with the <br>Groq API.<br> <li> Defined model configurations, credentials, and settings for Groq.<br> <li> Implemented methods for creating chat completions and handling API <br>retries.<br>


</details>
    

  </td>
  <td><a href="https://github.com/Significant-Gravitas/AutoGPT/pull/7130/files#diff-b484baabe66ef89640cabd5812fe67213f22935ae2755cea3958ae26cbe93311">+417/-0</a>&nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
      <summary><strong>multi.py</strong><dd><code>Integrate GroqProvider into MultiProvider</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>
      
autogpts/autogpt/autogpt/core/resource/model_providers/multi.py

<li>Added <code>GroqProvider</code> to the list of model providers.<br> <li> Updated the <code>ModelName</code> type to include <code>GroqModelName</code>.<br> <li> Included <code>GROQ_CHAT_MODELS</code> in the combined chat models dictionary.<br>


</details>
    

  </td>
  <td><a href="https://github.com/Significant-Gravitas/AutoGPT/pull/7130/files#diff-a7cd8d4101179a4457674f2c507ddac2732498e5d7629a71d9ed7a0c4c402027">+6/-4</a>&nbsp; &nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
      <summary><strong>schema.py</strong><dd><code>Update ModelProviderName Enum with GROQ</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>
      
autogpts/autogpt/autogpt/core/resource/model_providers/schema.py

- Added `GROQ` to the `ModelProviderName` enum.



</details>
    

  </td>
  <td><a href="https://github.com/Significant-Gravitas/AutoGPT/pull/7130/files#diff-a1eb47c2850b98017895095802dc1f608732dd7b104b5e0ad7d2abe4e264aec5">+1/-0</a>&nbsp; &nbsp; &nbsp; </td>

</tr>                    
</table></td></tr><tr><td><strong>Dependencies
</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>pyproject.toml</strong><dd><code>Add Groq Dependency in pyproject.toml</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>
      
autogpts/autogpt/pyproject.toml

- Added `groq` package as a dependency.



</details>
    

  </td>
  <td><a href="https://github.com/Significant-Gravitas/AutoGPT/pull/7130/files#diff-9693a37f5a2c961a6bc8acf875e6d3b573adb5467d30d95872f3c006f1c8fc7b">+1/-0</a>&nbsp; &nbsp; &nbsp; </td>

</tr>                    
</table></td></tr></tr></tbody></table>

___

> 💡 **PR-Agent usage**:
>Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions



Problem statement:
Support using other/local LLMs
You can modify the code to accept a config file as input, and read the Chosen_Model flag to select the appropriate AI model. Here's an example of how to achieve this:

Create a sample config file named config.ini:

[AI]
Chosen_Model = gpt-4

Offload the call_ai_fuction from the ai_functions.py to a separate library. Modify the call_ai_function function to read the model from the config file:

```
import configparser

def call_ai_function(function, args, description, config_path="config.ini"):
    # Load the configuration file
    config = configparser.ConfigParser()
    config.read(config_path)

    # Get the chosen model from the config file
    model = config.get("AI", "Chosen_Model", fallback="gpt-4")

    # Parse args to comma separated string
    args = ", ".join(args)
    messages = [
        {
            "role": "system",
            "content": f"You are now the following python function: ```# {description}\n{function}```\n\nOnly respond with your `return` value.",
        },
        {"role": "user", "content": args},
    ]

  # Use different AI APIs based on the chosen model
    if model == "gpt-4":
        response = openai.ChatCompletion.create(
            model=model, messages=messages, temperature=0
        )
    elif model == "some_other_api":
        # Add code to call another AI API with the appropriate parameters
        response = some_other_api_call(parameters)
    else:
        raise ValueError(f"Unsupported model: {model}")

    return response.choices[0].message["content"]

```
In this modified version, the call_ai_function takes an additional parameter config_path which defaults to "config.ini". The function reads the config file, retrieves the Chosen_Model value, and uses it as the model for the OpenAI API call. If the Chosen_Model flag is not found in the config file, it defaults to "gpt-4".

 the if/elif structure is used to call different AI APIs based on the chosen model from the configuration file. Replace some_other_api with the name of the API you'd like to use, and replace parameters with the appropriate parameters required by that API. You can extend the if/elif structure to include more AI APIs as needed.

Hints from issue discussion:
Excellent work. Lots of people are asking for this submit a pull request!

In order to fully support gpt3.5 (and other models) we need to harden the prompt.

@Koobah had some success by adding this line to the end of prompt.txt:
```
Before submitting the response, simulate parsing the response with Python json.loads. Don't submit unless it can be parsed.
```

This would also help out #21 

I see you looking over my shoulder. Thoughts?

I've got a friend who is going to clone the branch and test for me. (hopefully) I don't have a working environment atm.
I also changed the user prompt from NEXT COMMAND to GENERATE NEXT COMMAND JSON. Basically reminding it to use JSON whenever possible
It's still not 100% working, though. 
https://github.com/DataBassGit/Auto-GPT/blob/master/scripts/ai_function_lib.py

@Koobah This is basically what I'm working with atm. I think we can probably add a verify_json function to a gpt-3.5 segment of that function.
Pull request on this is submitted. I'm going to start looking into more models and platforms that can be incorporated.
Would be huge if it can run llama.cpp locally.
I might be able to swing that. Let's see if this merge gets approved. I'm also looking at implementing GPT4all.
@DataBassGit I see that PR got closed. What's the status of your fork?
They moved the API call to GPT-4 to an external library in main.py, but there are still some scripts that call openai directly, like chat.py, browse.py etc. 

GPT4all doesn't support x64 architecture. I also tried some APIs on hugging face, but it seems that it truncates responses on the free API endpoints.

I just converted BabyAGI to Oobabooga, but it's untested. Should be getting to that tonight. If it works, I will start working on porting AutoGPT to Oobabooga as well. The nice thing about this method is that it allows for local or remote hosting, and can handle many different language models without much issue.

Hugging face should work, though. I need to review Microsoft/Jarvis. They make heavy use of HF APIs.
GPT4all supports x64 and every architecture llama.cpp supports, which is every architecture (even non-POSIX, and webassemly). Their moto is "Can it run ~Doom~ LLaMA" for a reason.

Ooga supports GPT4all (and all llama.cpp ggml models), since it packages llama.cpp and the llamacpp python bindings library. So porting it to ooba would effectively resolve this.

Commit message: Merge branch 'master' into groq

Patch to review:
diff --git a/autogpts/autogpt/.env.template b/autogpts/autogpt/.env.template
index 14c7bcaa5d51..6810b8de35bf 100644
--- a/autogpts/autogpt/.env.template
+++ b/autogpts/autogpt/.env.template
@@ -8,6 +8,9 @@
 ## ANTHROPIC_API_KEY - Anthropic API Key (Example: sk-ant-api03-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
 # ANTHROPIC_API_KEY=
 
+## GROQ_API_KEY - Groq API Key (Example: gsk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
+# GROQ_API_KEY=
+
 ## TELEMETRY_OPT_IN - Share telemetry on errors and other issues with the AutoGPT team, e.g. through Sentry.
 ##   This helps us to spot and solve problems earlier & faster. (Default: DISABLED)
 # TELEMETRY_OPT_IN=true
diff --git a/autogpts/autogpt/autogpt/core/resource/model_providers/groq.py b/autogpts/autogpt/autogpt/core/resource/model_providers/groq.py
new file mode 100644
index 000000000000..1e0c74e5ffd8
--- /dev/null
+++ b/autogpts/autogpt/autogpt/core/resource/model_providers/groq.py
@@ -0,0 +1,420 @@
+from __future__ import annotations
+
+import enum
+import logging
+from typing import TYPE_CHECKING, Callable, Optional, ParamSpec, TypeVar
+
+import sentry_sdk
+import tenacity
+import tiktoken
+from groq import APIConnectionError, APIStatusError
+from pydantic import SecretStr
+
+from autogpt.core.configuration import Configurable, UserConfigurable
+from autogpt.core.resource.model_providers.schema import (
+    AssistantChatMessage,
+    AssistantFunctionCall,
+    AssistantToolCall,
+    ChatMessage,
+    ChatModelInfo,
+    ChatModelProvider,
+    ChatModelResponse,
+    CompletionModelFunction,
+    ModelProviderBudget,
+    ModelProviderConfiguration,
+    ModelProviderCredentials,
+    ModelProviderName,
+    ModelProviderSettings,
+    ModelTokenizer,
+)
+from autogpt.core.utils.json_utils import json_loads
+
+from .utils import validate_tool_calls
+
+if TYPE_CHECKING:
+    from groq.types.chat import ChatCompletion, CompletionCreateParams
+    from groq.types.chat.chat_completion import ChoiceMessage as ChatCompletionMessage
+    from groq.types.chat.completion_create_params import Message as MessageParam
+
+_T = TypeVar("_T")
+_P = ParamSpec("_P")
+
+
+class GroqModelName(str, enum.Enum):
+    LLAMA3_8B = "llama3-8b-8192"
+    LLAMA3_70B = "llama3-70b-8192"
+    MIXTRAL_8X7B = "mixtral-8x7b-32768"
+    GEMMA_7B = "gemma-7b-it"
+
+
+GROQ_CHAT_MODELS = {
+    info.name: info
+    for info in [
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_8B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.05 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_70B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.59 / 1e6,
+            completion_token_cost=0.79 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.MIXTRAL_8X7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.27 / 1e6,
+            completion_token_cost=0.27 / 1e6,
+            max_tokens=32768,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.GEMMA_7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.10 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+    ]
+}
+
+
+class GroqConfiguration(ModelProviderConfiguration):
+    fix_failed_parse_tries: int = UserConfigurable(3)
+
+
+class GroqCredentials(ModelProviderCredentials):
+    """Credentials for Groq."""
+
+    api_key: SecretStr = UserConfigurable(from_env="GROQ_API_KEY")
+    api_base: Optional[SecretStr] = UserConfigurable(
+        default=None, from_env="GROQ_API_BASE_URL"
+    )
+
+    def get_api_access_kwargs(self) -> dict[str, str]:
+        return {
+            k: (v.get_secret_value() if type(v) is SecretStr else v)
+            for k, v in {
+                "api_key": self.api_key,
+                "base_url": self.api_base,
+            }.items()
+            if v is not None
+        }
+
+
+class GroqSettings(ModelProviderSettings):
+    configuration: GroqConfiguration
+    credentials: Optional[GroqCredentials]
+    budget: ModelProviderBudget
+
+
+class GroqProvider(Configurable[GroqSettings], ChatModelProvider):
+    default_settings = GroqSettings(
+        name="groq_provider",
+        description="Provides access to Groq's API.",
+        configuration=GroqConfiguration(
+            retries_per_request=7,
+        ),
+        credentials=None,
+        budget=ModelProviderBudget(),
+    )
+
+    _settings: GroqSettings
+    _configuration: GroqConfiguration
+    _credentials: GroqCredentials
+    _budget: ModelProviderBudget
+
+    def __init__(
+        self,
+        settings: Optional[GroqSettings] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        if not settings:
+            settings = self.default_settings.copy(deep=True)
+        if not settings.credentials:
+            settings.credentials = GroqCredentials.from_env()
+
+        super(GroqProvider, self).__init__(settings=settings, logger=logger)
+
+        from groq import AsyncGroq
+
+        self._client = AsyncGroq(**self._credentials.get_api_access_kwargs())
+
+    async def get_available_models(self) -> list[ChatModelInfo]:
+        _models = (await self._client.models.list()).data
+        return [GROQ_CHAT_MODELS[m.id] for m in _models if m.id in GROQ_CHAT_MODELS]
+
+    def get_token_limit(self, model_name: str) -> int:
+        """Get the token limit for a given model."""
+        return GROQ_CHAT_MODELS[model_name].max_tokens
+
+    @classmethod
+    def get_tokenizer(cls, model_name: GroqModelName) -> ModelTokenizer:
+        # HACK: No official tokenizer is available for Claude 3
+        return tiktoken.encoding_for_model(model_name)
+
+    @classmethod
+    def count_tokens(cls, text: str, model_name: GroqModelName) -> int:
+        return 0  # HACK: No official tokenizer is available for Claude 3
+
+    @classmethod
+    def count_message_tokens(
+        cls,
+        messages: ChatMessage | list[ChatMessage],
+        model_name: GroqModelName,
+    ) -> int:
+        return 0  # HACK: No official tokenizer is available for Claude 3
+
+    async def create_chat_completion(
+        self,
+        model_prompt: list[ChatMessage],
+        model_name: GroqModelName,
+        completion_parser: Callable[[AssistantChatMessage], _T] = lambda _: None,
+        functions: Optional[list[CompletionModelFunction]] = None,
+        max_output_tokens: Optional[int] = None,
+        prefill_response: str = "",
+        **kwargs,
+    ) -> ChatModelResponse[_T]:
+        """Create a completion using the Groq API."""
+        groq_messages, completion_kwargs = self._get_chat_completion_args(
+            prompt_messages=model_prompt,
+            model=model_name,
+            functions=functions,
+            max_output_tokens=max_output_tokens,
+            **kwargs,
+        )
+
+        total_cost = 0.0
+        attempts = 0
+        while True:
+            completion_kwargs["messages"] = groq_messages.copy()
+            _response, _cost, t_input, t_output = await self._create_chat_completion(
+                completion_kwargs
+            )
+            total_cost += _cost
+
+            # If parsing the response fails, append the error to the prompt, and let the
+            # LLM fix its mistake(s).
+            attempts += 1
+            parse_errors: list[Exception] = []
+
+            _assistant_msg = _response.choices[0].message
+
+            tool_calls, _errors = self._parse_assistant_tool_calls(_assistant_msg)
+            parse_errors += _errors
+
+            # Validate tool calls
+            if not parse_errors and tool_calls and functions:
+                parse_errors += validate_tool_calls(tool_calls, functions)
+
+            assistant_msg = AssistantChatMessage(
+                content=_assistant_msg.content,
+                tool_calls=tool_calls or None,
+            )
+
+            parsed_result: _T = None  # type: ignore
+            if not parse_errors:
+                try:
+                    parsed_result = completion_parser(assistant_msg)
+                except Exception as e:
+                    parse_errors.append(e)
+
+            if not parse_errors:
+                if attempts > 1:
+                    self._logger.debug(
+                        f"Total cost for {attempts} attempts: ${round(total_cost, 5)}"
+                    )
+
+                return ChatModelResponse(
+                    response=AssistantChatMessage(
+                        content=_assistant_msg.content,
+                        tool_calls=tool_calls or None,
+                    ),
+                    parsed_result=parsed_result,
+                    model_info=GROQ_CHAT_MODELS[model_name],
+                    prompt_tokens_used=t_input,
+                    completion_tokens_used=t_output,
+                )
+
+            else:
+                self._logger.debug(
+                    f"Parsing failed on response: '''{_assistant_msg}'''"
+                )
+                parse_errors_fmt = "\n\n".join(
+                    f"{e.__class__.__name__}: {e}" for e in parse_errors
+                )
+                self._logger.warning(
+                    f"Parsing attempt #{attempts} failed: {parse_errors_fmt}"
+                )
+                for e in parse_errors:
+                    sentry_sdk.capture_exception(
+                        error=e,
+                        extras={"assistant_msg": _assistant_msg, "i_attempt": attempts},
+                    )
+
+                if attempts < self._configuration.fix_failed_parse_tries:
+                    groq_messages.append(_assistant_msg.dict(exclude_none=True))
+                    groq_messages.append(
+                        {
+                            "role": "system",
+                            "content": (
+                                f"ERROR PARSING YOUR RESPONSE:\n\n{parse_errors_fmt}"
+                            ),
+                        }
+                    )
+                    continue
+                else:
+                    raise parse_errors[0]
+
+    def _get_chat_completion_args(
+        self,
+        prompt_messages: list[ChatMessage],
+        model: GroqModelName,
+        functions: Optional[list[CompletionModelFunction]] = None,
+        max_output_tokens: Optional[int] = None,
+        **kwargs,  # type: ignore
+    ) -> tuple[list[MessageParam], CompletionCreateParams]:
+        """Prepare chat completion arguments and keyword arguments for API call.
+
+        Args:
+            model_prompt: List of ChatMessages.
+            model_name: The model to use.
+            functions: Optional list of functions available to the LLM.
+            kwargs: Additional keyword arguments.
+
+        Returns:
+            list[ChatCompletionMessageParam]: Prompt messages for the OpenAI call
+            dict[str, Any]: Any other kwargs for the OpenAI call
+        """
+        kwargs: CompletionCreateParams = kwargs  # type: ignore
+        kwargs["model"] = model
+        if max_output_tokens:
+            kwargs["max_tokens"] = max_output_tokens
+
+        if functions:
+            kwargs["tools"] = [
+                {"type": "function", "function": f.schema} for f in functions
+            ]
+            if len(functions) == 1:
+                # force the model to call the only specified function
+                kwargs["tool_choice"] = {
+                    "type": "function",
+                    "function": {"name": functions[0].name},
+                }
+
+        if extra_headers := self._configuration.extra_request_headers:
+            # 'extra_headers' is not on CompletionCreateParams, but is on chat.create()
+            kwargs["extra_headers"] = kwargs.get("extra_headers", {})  # type: ignore
+            kwargs["extra_headers"].update(extra_headers.copy())  # type: ignore
+
+        groq_messages: list[MessageParam] = [
+            message.dict(
+                include={"role", "content", "tool_calls", "tool_call_id", "name"},
+                exclude_none=True,
+            )
+            for message in prompt_messages
+        ]
+
+        if "messages" in kwargs:
+            groq_messages += kwargs["messages"]
+            del kwargs["messages"]  # type: ignore - messages are added back later
+
+        return groq_messages, kwargs
+
+    async def _create_chat_completion(
+        self, completion_kwargs: CompletionCreateParams
+    ) -> tuple[ChatCompletion, float, int, int]:
+        """
+        Create a chat completion using the Groq API with retry handling.
+
+        Params:
+            completion_kwargs: Keyword arguments for an Groq Messages API call
+
+        Returns:
+            Message: The message completion object
+            float: The cost ($) of this completion
+            int: Number of input tokens used
+            int: Number of output tokens used
+        """
+
+        @self._retry_api_request
+        async def _create_chat_completion_with_retry(
+            completion_kwargs: CompletionCreateParams,
+        ) -> ChatCompletion:
+            return await self._client.chat.completions.create(**completion_kwargs)
+
+        response = await _create_chat_completion_with_retry(completion_kwargs)
+
+        cost = self._budget.update_usage_and_cost(
+            model_info=GROQ_CHAT_MODELS[completion_kwargs["model"]],
+            input_tokens_used=response.usage.prompt_tokens,
+            output_tokens_used=response.usage.completion_tokens,
+        )
+        return (
+            response,
+            cost,
+            response.usage.prompt_tokens,
+            response.usage.completion_tokens,
+        )
+
+    def _parse_assistant_tool_calls(
+        self, assistant_message: ChatCompletionMessage, compat_mode: bool = False
+    ):
+        tool_calls: list[AssistantToolCall] = []
+        parse_errors: list[Exception] = []
+
+        if assistant_message.tool_calls:
+            for _tc in assistant_message.tool_calls:
+                try:
+                    parsed_arguments = json_loads(_tc.function.arguments)
+                except Exception as e:
+                    err_message = (
+                        f"Decoding arguments for {_tc.function.name} failed: "
+                        + str(e.args[0])
+                    )
+                    parse_errors.append(
+                        type(e)(err_message, *e.args[1:]).with_traceback(
+                            e.__traceback__
+                        )
+                    )
+                    continue
+
+                tool_calls.append(
+                    AssistantToolCall(
+                        id=_tc.id,
+                        type=_tc.type,
+                        function=AssistantFunctionCall(
+                            name=_tc.function.name,
+                            arguments=parsed_arguments,
+                        ),
+                    )
+                )
+
+            # If parsing of all tool calls succeeds in the end, we ignore any issues
+            if len(tool_calls) == len(assistant_message.tool_calls):
+                parse_errors = []
+
+        return tool_calls, parse_errors
+
+    def _retry_api_request(self, func: Callable[_P, _T]) -> Callable[_P, _T]:
+        return tenacity.retry(
+            retry=(
+                tenacity.retry_if_exception_type(APIConnectionError)
+                | tenacity.retry_if_exception(
+                    lambda e: isinstance(e, APIStatusError) and e.status_code >= 500
+                )
+            ),
+            wait=tenacity.wait_exponential(),
+            stop=tenacity.stop_after_attempt(self._configuration.retries_per_request),
+            after=tenacity.after_log(self._logger, logging.DEBUG),
+        )(func)
+
+    def __repr__(self):
+        return "GroqProvider()"
diff --git a/autogpts/autogpt/autogpt/core/resource/model_providers/multi.py b/autogpts/autogpt/autogpt/core/resource/model_providers/multi.py
index f194e02564d6..81e30ef8acee 100644
--- a/autogpts/autogpt/autogpt/core/resource/model_providers/multi.py
+++ b/autogpts/autogpt/autogpt/core/resource/model_providers/multi.py
@@ -8,6 +8,7 @@
 from autogpt.core.configuration import Configurable
 
 from .anthropic import ANTHROPIC_CHAT_MODELS, AnthropicModelName, AnthropicProvider
+from .groq import GROQ_CHAT_MODELS, GroqModelName, GroqProvider
 from .openai import OPEN_AI_CHAT_MODELS, OpenAIModelName, OpenAIProvider
 from .schema import (
     AssistantChatMessage,
@@ -25,9 +26,9 @@
 
 _T = TypeVar("_T")
 
-ModelName = AnthropicModelName | OpenAIModelName
+ModelName = AnthropicModelName | GroqModelName | OpenAIModelName
 
-CHAT_MODELS = {**ANTHROPIC_CHAT_MODELS, **OPEN_AI_CHAT_MODELS}
+CHAT_MODELS = {**ANTHROPIC_CHAT_MODELS, **GROQ_CHAT_MODELS, **OPEN_AI_CHAT_MODELS}
 
 
 class MultiProvider(Configurable[ModelProviderSettings], ChatModelProvider):
@@ -143,16 +144,17 @@ def _get_provider(self, provider_name: ModelProviderName) -> ChatModelProvider:
     @classmethod
     def _get_model_provider_class(
         cls, model_name: ModelName
-    ) -> type[AnthropicProvider | OpenAIProvider]:
+    ) -> type[AnthropicProvider | GroqProvider | OpenAIProvider]:
         return cls._get_provider_class(CHAT_MODELS[model_name].provider_name)
 
     @classmethod
     def _get_provider_class(
         cls, provider_name: ModelProviderName
-    ) -> type[AnthropicProvider | OpenAIProvider]:
+    ) -> type[AnthropicProvider | GroqProvider | OpenAIProvider]:
         try:
             return {
                 ModelProviderName.ANTHROPIC: AnthropicProvider,
+                ModelProviderName.GROQ: GroqProvider,
                 ModelProviderName.OPENAI: OpenAIProvider,
             }[provider_name]
         except KeyError:
diff --git a/autogpts/autogpt/autogpt/core/resource/model_providers/schema.py b/autogpts/autogpt/autogpt/core/resource/model_providers/schema.py
index bb2e29490ea4..0de34e4a9108 100644
--- a/autogpts/autogpt/autogpt/core/resource/model_providers/schema.py
+++ b/autogpts/autogpt/autogpt/core/resource/model_providers/schema.py
@@ -45,6 +45,7 @@ class ModelProviderService(str, enum.Enum):
 class ModelProviderName(str, enum.Enum):
     OPENAI = "openai"
     ANTHROPIC = "anthropic"
+    GROQ = "groq"
 
 
 class ChatMessage(BaseModel):
diff --git a/autogpts/autogpt/poetry.lock b/autogpts/autogpt/poetry.lock
index 251cbb3bd794..2a3b7ea2feb1 100644
--- a/autogpts/autogpt/poetry.lock
+++ b/autogpts/autogpt/poetry.lock
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.7.1 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.
 
 [[package]]
 name = "agbenchmark"
@@ -2441,6 +2441,25 @@ files = [
 docs = ["Sphinx", "furo"]
 test = ["objgraph", "psutil"]
 
+[[package]]
+name = "groq"
+version = "0.5.0"
+description = "The official Python library for the groq API"
+optional = false
+python-versions = ">=3.7"
+files = [
+    {file = "groq-0.5.0-py3-none-any.whl", hash = "sha256:a7e6be1118bcdfea3ed071ec00f505a34d4e6ec28c435adb5a5afd33545683a1"},
+    {file = "groq-0.5.0.tar.gz", hash = "sha256:d476cdc3383b45d2a4dc1876142a9542e663ea1029f9e07a05de24f895cae48c"},
+]
+
+[package.dependencies]
+anyio = ">=3.5.0,<5"
+distro = ">=1.7.0,<2"
+httpx = ">=0.23.0,<1"
+pydantic = ">=1.9.0,<3"
+sniffio = "*"
+typing-extensions = ">=4.7,<5"
+
 [[package]]
 name = "grpc-google-iam-v1"
 version = "0.13.0"
@@ -7258,4 +7277,4 @@ benchmark = ["agbenchmark"]
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.10"
-content-hash = "ad1e3c4706465733d04ddab975af630975bd528efce152c1da01eded53069eca"
+content-hash = "78c90344e75e92b9a4f191dbe0abf83d08b28ec68b8f7c59cf583bfea19a4d02"
diff --git a/autogpts/autogpt/pyproject.toml b/autogpts/autogpt/pyproject.toml
index 99f58774b624..5cfb7805bb6e 100644
--- a/autogpts/autogpt/pyproject.toml
+++ b/autogpts/autogpt/pyproject.toml
@@ -74,6 +74,7 @@ agbenchmark = { path = "../../benchmark", optional = true }
 google-cloud-logging = "^3.8.0"
 google-cloud-storage = "^2.13.0"
 psycopg2-binary = "^2.9.9"
+groq = "^0.5.0"
 
 [tool.poetry.extras]
 benchmark = ["agbenchmark"]
diff --git a/docs/content/AutoGPT/configuration/options.md b/docs/content/AutoGPT/configuration/options.md
index 17602102ba65..d5a666e2ad20 100644
--- a/docs/content/AutoGPT/configuration/options.md
+++ b/docs/content/AutoGPT/configuration/options.md
@@ -23,6 +23,7 @@ Configuration is controlled through the `Config` object. You can set configurati
 - `GITHUB_USERNAME`: GitHub Username. Optional.
 - `GOOGLE_API_KEY`: Google API key. Optional.
 - `GOOGLE_CUSTOM_SEARCH_ENGINE_ID`: [Google custom search engine ID](https://programmablesearchengine.google.com/controlpanel/all). Optional.
+- `GROQ_API_KEY`: Set this if you want to use Groq models with AutoGPT
 - `HEADLESS_BROWSER`: Use a headless browser while AutoGPT uses a web browser. Setting to `False` will allow you to see AutoGPT operate the browser. Default: True
 - `HUGGINGFACE_API_TOKEN`: HuggingFace API, to be used for both image generation and audio to text. Optional.
 - `HUGGINGFACE_AUDIO_TO_TEXT_MODEL`: HuggingFace audio to text model. Default: CompVis/stable-diffusion-v1-4


Code review comments:
diff_hunk: @@ -0,0 +1,420 @@
+from __future__ import annotations
+
+import enum
+import logging
+from typing import TYPE_CHECKING, Callable, Optional, ParamSpec, TypeVar
+
+import sentry_sdk
+import tenacity
+import tiktoken
+from groq import APIConnectionError, APIStatusError
+from pydantic import SecretStr
+
+from autogpt.core.configuration import Configurable, UserConfigurable
+from autogpt.core.resource.model_providers.schema import (
+    AssistantChatMessage,
+    AssistantFunctionCall,
+    AssistantToolCall,
+    ChatMessage,
+    ChatModelInfo,
+    ChatModelProvider,
+    ChatModelResponse,
+    CompletionModelFunction,
+    ModelProviderBudget,
+    ModelProviderConfiguration,
+    ModelProviderCredentials,
+    ModelProviderName,
+    ModelProviderSettings,
+    ModelTokenizer,
+)
+from autogpt.core.utils.json_utils import json_loads
+
+from .utils import validate_tool_calls
+
+if TYPE_CHECKING:
+    from groq.types.chat import ChatCompletion, CompletionCreateParams
+    from groq.types.chat.chat_completion import ChoiceMessage as ChatCompletionMessage
+    from groq.types.chat.completion_create_params import Message as MessageParam
+
+_T = TypeVar("_T")
+_P = ParamSpec("_P")
+
+
+class GroqModelName(str, enum.Enum):
+    LLAMA3_8B = "llama3-8b-8192"
+    LLAMA3_70B = "llama3-70b-8192"
+    MIXTRAL_8X7B = "mixtral-8x7b-32768"
+    GEMMA_7B = "gemma-7b-it"
+
+
+GROQ_CHAT_MODELS = {
+    info.name: info
+    for info in [
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_8B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.05 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_70B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.59 / 1e6,
+            completion_token_cost=0.79 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.MIXTRAL_8X7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.27 / 1e6,
+            completion_token_cost=0.27 / 1e6,
+            max_tokens=32768,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.GEMMA_7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.10 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+    ]
+}
+
+
+class GroqConfiguration(ModelProviderConfiguration):
+    fix_failed_parse_tries: int = UserConfigurable(3)
+
+
+class GroqCredentials(ModelProviderCredentials):
+    """Credentials for Groq."""
+
+    api_key: SecretStr = UserConfigurable(from_env="GROQ_API_KEY")
+    api_base: Optional[SecretStr] = UserConfigurable(
+        default=None, from_env="GROQ_API_BASE_URL"
+    )
+
+    def get_api_access_kwargs(self) -> dict[str, str]:
+        return {
+            k: (v.get_secret_value() if type(v) is SecretStr else v)
+            for k, v in {
+                "api_key": self.api_key,
+                "base_url": self.api_base,
+            }.items()
+            if v is not None
+        }
+
+
+class GroqSettings(ModelProviderSettings):
+    configuration: GroqConfiguration
+    credentials: Optional[GroqCredentials]
+    budget: ModelProviderBudget
+
+
+class GroqProvider(Configurable[GroqSettings], ChatModelProvider):
+    default_settings = GroqSettings(
+        name="groq_provider",
+        description="Provides access to Groq's API.",
+        configuration=GroqConfiguration(
+            retries_per_request=7,
+        ),
+        credentials=None,
+        budget=ModelProviderBudget(),
+    )
+
+    _settings: GroqSettings
+    _configuration: GroqConfiguration
+    _credentials: GroqCredentials
+    _budget: ModelProviderBudget
+
+    def __init__(
+        self,
+        settings: Optional[GroqSettings] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        if not settings:
+            settings = self.default_settings.copy(deep=True)
+        if not settings.credentials:
+            settings.credentials = GroqCredentials.from_env()
+
+        super(GroqProvider, self).__init__(settings=settings, logger=logger)
+
+        from groq import AsyncGroq
+
+        self._client = AsyncGroq(**self._credentials.get_api_access_kwargs())
+
+    async def get_available_models(self) -> list[ChatModelInfo]:
+        _models = (await self._client.models.list()).data
+        return [GROQ_CHAT_MODELS[m.id] for m in _models if m.id in GROQ_CHAT_MODELS]
+
+    def get_token_limit(self, model_name: str) -> int:
+        """Get the token limit for a given model."""
+        return GROQ_CHAT_MODELS[model_name].max_tokens
+
+    @classmethod
+    def get_tokenizer(cls, model_name: GroqModelName) -> ModelTokenizer:
+        # HACK: No official tokenizer is available for Claude 3
path: autogpts/autogpt/autogpt/core/resource/model_providers/groq.py
text: @user1:
Remove comment here

diff_hunk: @@ -0,0 +1,420 @@
+from __future__ import annotations
+
+import enum
+import logging
+from typing import TYPE_CHECKING, Callable, Optional, ParamSpec, TypeVar
+
+import sentry_sdk
+import tenacity
+import tiktoken
+from groq import APIConnectionError, APIStatusError
+from pydantic import SecretStr
+
+from autogpt.core.configuration import Configurable, UserConfigurable
+from autogpt.core.resource.model_providers.schema import (
+    AssistantChatMessage,
+    AssistantFunctionCall,
+    AssistantToolCall,
+    ChatMessage,
+    ChatModelInfo,
+    ChatModelProvider,
+    ChatModelResponse,
+    CompletionModelFunction,
+    ModelProviderBudget,
+    ModelProviderConfiguration,
+    ModelProviderCredentials,
+    ModelProviderName,
+    ModelProviderSettings,
+    ModelTokenizer,
+)
+from autogpt.core.utils.json_utils import json_loads
+
+from .utils import validate_tool_calls
+
+if TYPE_CHECKING:
+    from groq.types.chat import ChatCompletion, CompletionCreateParams
+    from groq.types.chat.chat_completion import ChoiceMessage as ChatCompletionMessage
+    from groq.types.chat.completion_create_params import Message as MessageParam
+
+_T = TypeVar("_T")
+_P = ParamSpec("_P")
+
+
+class GroqModelName(str, enum.Enum):
+    LLAMA3_8B = "llama3-8b-8192"
+    LLAMA3_70B = "llama3-70b-8192"
+    MIXTRAL_8X7B = "mixtral-8x7b-32768"
+    GEMMA_7B = "gemma-7b-it"
+
+
+GROQ_CHAT_MODELS = {
+    info.name: info
+    for info in [
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_8B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.05 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_70B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.59 / 1e6,
+            completion_token_cost=0.79 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.MIXTRAL_8X7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.27 / 1e6,
+            completion_token_cost=0.27 / 1e6,
+            max_tokens=32768,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.GEMMA_7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.10 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+    ]
+}
+
+
+class GroqConfiguration(ModelProviderConfiguration):
+    fix_failed_parse_tries: int = UserConfigurable(3)
+
+
+class GroqCredentials(ModelProviderCredentials):
+    """Credentials for Groq."""
+
+    api_key: SecretStr = UserConfigurable(from_env="GROQ_API_KEY")
+    api_base: Optional[SecretStr] = UserConfigurable(
+        default=None, from_env="GROQ_API_BASE_URL"
+    )
+
+    def get_api_access_kwargs(self) -> dict[str, str]:
+        return {
+            k: (v.get_secret_value() if type(v) is SecretStr else v)
+            for k, v in {
+                "api_key": self.api_key,
+                "base_url": self.api_base,
+            }.items()
+            if v is not None
+        }
+
+
+class GroqSettings(ModelProviderSettings):
+    configuration: GroqConfiguration
+    credentials: Optional[GroqCredentials]
+    budget: ModelProviderBudget
+
+
+class GroqProvider(Configurable[GroqSettings], ChatModelProvider):
+    default_settings = GroqSettings(
+        name="groq_provider",
+        description="Provides access to Groq's API.",
+        configuration=GroqConfiguration(
+            retries_per_request=7,
+        ),
+        credentials=None,
+        budget=ModelProviderBudget(),
+    )
+
+    _settings: GroqSettings
+    _configuration: GroqConfiguration
+    _credentials: GroqCredentials
+    _budget: ModelProviderBudget
+
+    def __init__(
+        self,
+        settings: Optional[GroqSettings] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        if not settings:
+            settings = self.default_settings.copy(deep=True)
+        if not settings.credentials:
+            settings.credentials = GroqCredentials.from_env()
+
+        super(GroqProvider, self).__init__(settings=settings, logger=logger)
+
+        from groq import AsyncGroq
+
+        self._client = AsyncGroq(**self._credentials.get_api_access_kwargs())
+
+    async def get_available_models(self) -> list[ChatModelInfo]:
+        _models = (await self._client.models.list()).data
+        return [GROQ_CHAT_MODELS[m.id] for m in _models if m.id in GROQ_CHAT_MODELS]
+
+    def get_token_limit(self, model_name: str) -> int:
+        """Get the token limit for a given model."""
+        return GROQ_CHAT_MODELS[model_name].max_tokens
+
+    @classmethod
+    def get_tokenizer(cls, model_name: GroqModelName) -> ModelTokenizer:
+        # HACK: No official tokenizer is available for Claude 3
+        return tiktoken.encoding_for_model(model_name)
+
+    @classmethod
+    def count_tokens(cls, text: str, model_name: GroqModelName) -> int:
+        return 0  # HACK: No official tokenizer is available for Claude 3
+
+    @classmethod
+    def count_message_tokens(
+        cls,
+        messages: ChatMessage | list[ChatMessage],
+        model_name: GroqModelName,
+    ) -> int:
+        return 0  # HACK: No official tokenizer is available for Claude 3
path: autogpts/autogpt/autogpt/core/resource/model_providers/groq.py
text: @user1:
Remove comment here, and give an attempt at counting tokens

@user1:
for now, I'd take tiktoken and hope for the best

diff_hunk: @@ -0,0 +1,420 @@
+from __future__ import annotations
+
+import enum
+import logging
+from typing import TYPE_CHECKING, Callable, Optional, ParamSpec, TypeVar
+
+import sentry_sdk
+import tenacity
+import tiktoken
+from groq import APIConnectionError, APIStatusError
+from pydantic import SecretStr
+
+from autogpt.core.configuration import Configurable, UserConfigurable
+from autogpt.core.resource.model_providers.schema import (
+    AssistantChatMessage,
+    AssistantFunctionCall,
+    AssistantToolCall,
+    ChatMessage,
+    ChatModelInfo,
+    ChatModelProvider,
+    ChatModelResponse,
+    CompletionModelFunction,
+    ModelProviderBudget,
+    ModelProviderConfiguration,
+    ModelProviderCredentials,
+    ModelProviderName,
+    ModelProviderSettings,
+    ModelTokenizer,
+)
+from autogpt.core.utils.json_utils import json_loads
+
+from .utils import validate_tool_calls
+
+if TYPE_CHECKING:
+    from groq.types.chat import ChatCompletion, CompletionCreateParams
+    from groq.types.chat.chat_completion import ChoiceMessage as ChatCompletionMessage
+    from groq.types.chat.completion_create_params import Message as MessageParam
+
+_T = TypeVar("_T")
+_P = ParamSpec("_P")
+
+
+class GroqModelName(str, enum.Enum):
+    LLAMA3_8B = "llama3-8b-8192"
+    LLAMA3_70B = "llama3-70b-8192"
+    MIXTRAL_8X7B = "mixtral-8x7b-32768"
+    GEMMA_7B = "gemma-7b-it"
+
+
+GROQ_CHAT_MODELS = {
+    info.name: info
+    for info in [
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_8B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.05 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.LLAMA3_70B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.59 / 1e6,
+            completion_token_cost=0.79 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.MIXTRAL_8X7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.27 / 1e6,
+            completion_token_cost=0.27 / 1e6,
+            max_tokens=32768,
+            has_function_call_api=True,
+        ),
+        ChatModelInfo(
+            name=GroqModelName.GEMMA_7B,
+            provider_name=ModelProviderName.GROQ,
+            prompt_token_cost=0.10 / 1e6,
+            completion_token_cost=0.10 / 1e6,
+            max_tokens=8192,
+            has_function_call_api=True,
+        ),
+    ]
+}
+
+
+class GroqConfiguration(ModelProviderConfiguration):
+    fix_failed_parse_tries: int = UserConfigurable(3)
+
+
+class GroqCredentials(ModelProviderCredentials):
+    """Credentials for Groq."""
+
+    api_key: SecretStr = UserConfigurable(from_env="GROQ_API_KEY")
+    api_base: Optional[SecretStr] = UserConfigurable(
+        default=None, from_env="GROQ_API_BASE_URL"
+    )
+
+    def get_api_access_kwargs(self) -> dict[str, str]:
+        return {
+            k: (v.get_secret_value() if type(v) is SecretStr else v)
+            for k, v in {
+                "api_key": self.api_key,
+                "base_url": self.api_base,
+            }.items()
+            if v is not None
+        }
+
+
+class GroqSettings(ModelProviderSettings):
+    configuration: GroqConfiguration
+    credentials: Optional[GroqCredentials]
+    budget: ModelProviderBudget
+
+
+class GroqProvider(Configurable[GroqSettings], ChatModelProvider):
+    default_settings = GroqSettings(
+        name="groq_provider",
+        description="Provides access to Groq's API.",
+        configuration=GroqConfiguration(
+            retries_per_request=7,
+        ),
+        credentials=None,
+        budget=ModelProviderBudget(),
+    )
+
+    _settings: GroqSettings
+    _configuration: GroqConfiguration
+    _credentials: GroqCredentials
+    _budget: ModelProviderBudget
+
+    def __init__(
+        self,
+        settings: Optional[GroqSettings] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        if not settings:
+            settings = self.default_settings.copy(deep=True)
+        if not settings.credentials:
+            settings.credentials = GroqCredentials.from_env()
+
+        super(GroqProvider, self).__init__(settings=settings, logger=logger)
+
+        from groq import AsyncGroq
+
+        self._client = AsyncGroq(**self._credentials.get_api_access_kwargs())
+
+    async def get_available_models(self) -> list[ChatModelInfo]:
+        _models = (await self._client.models.list()).data
+        return [GROQ_CHAT_MODELS[m.id] for m in _models if m.id in GROQ_CHAT_MODELS]
+
+    def get_token_limit(self, model_name: str) -> int:
+        """Get the token limit for a given model."""
+        return GROQ_CHAT_MODELS[model_name].max_tokens
+
+    @classmethod
+    def get_tokenizer(cls, model_name: GroqModelName) -> ModelTokenizer:
+        # HACK: No official tokenizer is available for Claude 3
+        return tiktoken.encoding_for_model(model_name)
+
+    @classmethod
+    def count_tokens(cls, text: str, model_name: GroqModelName) -> int:
+        return 0  # HACK: No official tokenizer is available for Claude 3
path: autogpts/autogpt/autogpt/core/resource/model_providers/groq.py
text: @user1:
Remove comment here, and give an attempt at counting tokens lol