Repository: Significant-Gravitas/AutoGPT
Pull request: Fixed #4229
Resolves: https://github.com/Significant-Gravitas/Auto-GPT/issues/4229

### Background
The defaults need to be adapted for people without access to GPT4 - or the corresponding functions won't work. The default model for people without GPT4 access should be cfg.fast_llm_model and not cfg.smart_llm_model.

### Changes
Added check_model fn in configurator that sends a request to the OpenAI API using create_chat_completion with the model provided. If an InvalidRequestError is raised, it returns a hard-coded value of "gpt-3.5-turbo".

### Documentation
Comments added to the function and test.

### Test Plan
Checks if fn returns original model argument if no error.
Checks if fn returns "gpt-3.5-turbo" when InvalidRequestError is raised.

### PR Quality Checklist
- [x] My pull request is atomic and focuses on a single change.
- [x] I have thoroughly tested my changes with multiple different prompts.
- [x] I have considered potential risks and mitigations for my changes.
- [x] I have documented my changes clearly and comprehensively.
- [x] I have not snuck in any "extra" small tweaks changes. <!-- Submit these as separate Pull Requests, they are the easiest to merge! -->
- [x] I have run `black .` and `isort .` against my code to ensure it passes our linter.


Problem statement:
The model: `gpt-4` does not exist / fixing defaults in llm_utils.py
### Duplicates

- [X] I have searched the existing issues

### Summary 💡

the defaults need to be adapted for people without access to GPT4 - or the corresponding functions won't work.
The default model for people without GPT4 access should be cfg.fast_llm_model and not cfg.smart_llm_model

It would probably make sense to check once during startup what model is available and then set up the default accordingly.
In general, GPT4 should only be a default setting once it is verified to be available.


### Examples 🌈

_No response_

### Motivation 🔦

_No response_

Hints from issue discussion:
Pull request: https://github.com/Significant-Gravitas/Auto-GPT/pull/4278

Commit message: Merge branch 'master' into gpt-4-exists

Patch to review:
diff --git a/autogpt/llm/llm_utils.py b/autogpt/llm/llm_utils.py
index a77bccbc06ac..41942b990952 100644
--- a/autogpt/llm/llm_utils.py
+++ b/autogpt/llm/llm_utils.py
@@ -9,7 +9,7 @@
 import openai
 import tiktoken
 from colorama import Fore, Style
-from openai.error import APIError, RateLimitError, Timeout
+from openai.error import APIError, InvalidRequestError, RateLimitError, Timeout
 
 from autogpt.config import Config
 from autogpt.llm.api_manager import ApiManager
@@ -293,3 +293,24 @@ def create_embedding(
     )  # normalize the length to one
     chunk_embeddings = chunk_embeddings.tolist()
     return chunk_embeddings
+
+
+def check_smart_llm_model_access() -> None:
+    """Check if smart model is available for use. If not, set to fast model."""
+    cfg = Config()
+
+    messages = [
+        {"role": "user", "content": ""},
+    ]
+    try:
+        create_chat_completion(
+            model=cfg.smart_llm_model, messages=messages, temperature=0
+        )
+    except InvalidRequestError:
+        logger.typewriter_log(
+            "WARNING: ",
+            Fore.YELLOW,
+            f"You do not have access to {cfg.smart_llm_model}. Setting default "
+            f"to {cfg.fast_llm_model}.",
+        )
+        cfg.smart_llm_model = cfg.fast_llm_model
diff --git a/autogpt/main.py b/autogpt/main.py
index 792496095af2..9d06fb7540a6 100644
--- a/autogpt/main.py
+++ b/autogpt/main.py
@@ -9,6 +9,7 @@
 from autogpt.commands.command import CommandRegistry
 from autogpt.config import Config, check_openai_api_key
 from autogpt.configurator import create_config
+from autogpt.llm.llm_utils import check_smart_llm_model_access
 from autogpt.logs import logger
 from autogpt.memory import get_memory
 from autogpt.plugins import scan_plugins
@@ -47,6 +48,7 @@ def run_auto_gpt(
     cfg = Config()
     # TODO: fill in llm values here
     check_openai_api_key()
+
     create_config(
         continuous,
         continuous_limit,
@@ -63,6 +65,8 @@ def run_auto_gpt(
         skip_news,
     )
 
+    check_smart_llm_model_access()
+
     if cfg.continuous_mode:
         for line in get_legal_warning().split("\n"):
             logger.warn(markdown_to_ansi_style(line), "LEGAL:", Fore.RED)
diff --git a/tests/test_config.py b/tests/test_config.py
index e1fc67454542..0f4b173946ac 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -2,10 +2,11 @@
 Test cases for the Config class, which handles the configuration settings
 for the AI and ensures it behaves as a singleton.
 """
+from unittest.mock import patch
 
-import pytest
+from openai import InvalidRequestError
 
-from autogpt.config import Config
+from autogpt.llm.llm_utils import check_smart_llm_model_access
 
 
 def test_initial_values(config):
@@ -117,3 +118,21 @@ def test_set_debug_mode(config):
 
     # Reset debug mode
     config.set_debug_mode(debug_mode)
+
+
+def test_check_smart_llm_model_access(config):
+    with patch("openai.ChatCompletion.create") as mock_create_chat_completion:
+        # Test when no InvalidRequestError is raised
+        smart_model = config.smart_llm_model
+        check_smart_llm_model_access()  # should not change smart_llm_model
+        assert config.smart_llm_model == smart_model
+
+        # Test when InvalidRequestError is raised
+        mock_create_chat_completion.side_effect = InvalidRequestError(
+            "error message", "error param"
+        )
+        check_smart_llm_model_access()  # should set smart_llm_model to fast_llm_model
+        assert config.smart_llm_model == config.fast_llm_model
+
+        # Reset smart_llm_model
+        config.set_smart_llm_model(smart_model)


Code review comments:
diff_hunk: @@ -293,3 +293,24 @@ def create_embedding(
     )  # normalize the length to one
     chunk_embeddings = chunk_embeddings.tolist()
     return chunk_embeddings
+
+
+def check_smart_llm_model_access() -> None:
+    """Check if smart model is available for use. If not, set to fast model."""
+    cfg = Config()
+
+    messages = [
+        {"role": "user", "content": ""},
+    ]
+    try:
+        create_chat_completion(
+            model=cfg.smart_llm_model, messages=messages, temperature=0
+        )
+    except InvalidRequestError:
+        logger.typewriter_log(
+            "WARNING: ",
+            Fore.YELLOW,
+            f"You do not have access to {cfg.smart_llm_model}. Setting default "
+            f"to {cfg.fast_llm_model}.",
+        )
+        cfg.smart_llm_model = cfg.fast_llm_model
path: autogpt/llm/llm_utils.py
text: @user1:
What if they have set for both fast and smart GPT4?

@author:
Hey @user2-boikov, changes made:

- check_model() fn added which takes model name and model type, returns original model name if valid, or hard-coded "gpt-3.5-turbo" if not valid
- Models are retrieved from get_models() in APIManager, which uses openai.Model.list()
- Checks for both fast_llm_model and smart_llm_model are done at top of create_config()

Let me know if you'd like any other updates or another approach.