Repository: lastmile-ai/aiconfig (language: python)

Issue:
Have a consistent way to distinguish model parsers from models
A single ModelParser can be used to run inference for multiple models. For example, the `AnyscaleEndpoint` model parser (https://github.com/lastmile-ai/aiconfig/blob/main/python/src/aiconfig/default_parsers/anyscale_endpoint.py) supports inference for LLaMA, Mistral, CodeLLaMA, LLaMAGuard, and other models.

Similarly, Hugging Face task model parsers can each support inference for thousands of models.

In aiconfig today, it's hard to distinguish between models and model parsers. A "model parser" is really a "model inference provider" for all models that can be run in the same way.

For example, the following prompt is valid:
```
  {
      "name": "gen_itinerary",
      "input": "Generate an itinerary ordered by {{order_by}} for these activities: {{get_activities.output}}.",
      "metadata": {
        "model": {
          "name": "AnyscaleEndpoint",
          "settings": {
            "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
            "max_tokens": 3000,
            "system_prompt": "You are an expert travel coordinator with exquisite taste."
          }
        },
        "parameters": {
          "order_by": "geographic location"
        }
      }
    }
```

But, if we add a `model_parsers` mapping to the global metadata
```
    "model_parsers": {
      "meta-llama/Llama-2-7b-chat-hf": "AnyscaleEndpoint",
   }
```

then the same prompt can be expressed as:
```
    {
      "name": "gen_itinerary",
      "input": "Generate an itinerary ordered by {{order_by}} for these activities: {{get_activities.output}}.",
      "metadata": {
        "model": {
          "name": "mistralai/Mixtral-8x7B-Instruct-v0.1",
          "settings": {
            "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
            "max_tokens": 3000,
            "system_prompt": "You are an expert travel coordinator with exquisite taste."
          }
        },
        "parameters": {
          "order_by": "geographic location"
        }
      }
    }
```

Some things we need to clean up and simplify:
1. confusion between `model.name` and `model.settings.model` -- they are often duplicated for no good reason.
2. having cleaner distinguishability between the concept of "model parser" and the "model id"



Discussion / hints:
A couple more thoughts here:
- Ideally we could also just express as
```
{
      "name": "gen_itinerary",
      "input": "Generate an itinerary ordered by {{order_by}} for these activities: {{get_activities.output}}.",
      "metadata": {
        "model": {
          "name": "mistralai/Mixtral-8x7B-Instruct-v0.1",
          "settings": {
            "max_tokens": 3000,
            "system_prompt": "You are an expert travel coordinator with exquisite taste."
          }
        },
        "parameters": {
          "order_by": "geographic location"
        }
      }
    }
```
if the model parser registry has a registered parser for `mistralai/Mixtral-8x7B-Instruct-v0.1`.


We currently also support registering model parsers for different models in 2 ways:
```
gpt_models = [
    "gpt-4",
    "gpt-4-0314",
    "gpt-4-0613",
    "gpt-4-32k",
    "gpt-4-32k-0314",
    "gpt-4-32k-0613",
    "gpt-3.5-turbo",
    "gpt-3.5-turbo-16k",
    "gpt-3.5-turbo-0301",
    "gpt-3.5-turbo-0613",
    "gpt-3.5-turbo-16k-0613",
]
for model in gpt_models:
    ModelParserRegistry.register_model_parser(DefaultOpenAIParser(model))
```
and 
```
ModelParserRegistry.register_model_parser(DefaultOpenAIParser(), gpt_models)
```
with the former instantiating a new DefaultOpenAIParser class for each model ID (with the parser id being set to model id), and the latter reusing the same class instance (with default parser id) and associating it to multiple model ids.

We probably don't want to support both cases

Fix patch:
diff --git a/cookbooks/HuggingFace/hf.py b/cookbooks/HuggingFace/hf.py
index 3958e5c4e..20405a6e6 100644
--- a/cookbooks/HuggingFace/hf.py
+++ b/cookbooks/HuggingFace/hf.py
@@ -209,7 +209,7 @@ async def deserialize(
         Returns:
             dict: Model-specific completion parameters.
         """
-        resolved_prompt = resolve_prompt(prompt, params, aiconfig)
+        resolved_prompt = resolve_prompt(prompt, params if params is not None else {}, aiconfig)
 
         # Build Completion data
         model_settings = self.get_model_settings(prompt, aiconfig)
diff --git a/python/src/aiconfig/default_parsers/dalle.py b/python/src/aiconfig/default_parsers/dalle.py
index 1c1958982..feb59fa2c 100644
--- a/python/src/aiconfig/default_parsers/dalle.py
+++ b/python/src/aiconfig/default_parsers/dalle.py
@@ -25,7 +25,7 @@
 
 
 # TODO: Centralize this into utils function with the HuggingFaceTextParser class
-def refine_image_completion_params(model_settings):
+def refine_image_completion_params(model_settings, aiconfig, prompt):
     """
     Refines the completion params for the Dall-E request API. Removes any unsupported params.
     The supported keys were found by looking at the OpenAI Dall-E API: https://platform.openai.com/docs/api-reference/images/create?lang=python`
@@ -37,6 +37,11 @@ def refine_image_completion_params(model_settings):
         if key.lower() in supported_keys:
             completion_data[key.lower()] = model_settings[key]
 
+    # Explicitly set the model to use if not already specified
+    if completion_data.get("model") is None:
+        model_name = aiconfig.get_model_name(prompt)
+        completion_data["model"] = model_name
+
     return completion_data
 
 
@@ -141,11 +146,11 @@ async def deserialize(self, prompt: Prompt, aiconfig: "AIConfigRuntime", params:
             dict: Model-specific completion parameters.
         """
         # Get inputs from aiconfig
-        resolved_prompt = resolve_prompt(prompt, params, aiconfig)
+        resolved_prompt = resolve_prompt(prompt, params if params is not None else {}, aiconfig)
         model_settings = self.get_model_settings(prompt, aiconfig)
 
         # Build Completion data
-        completion_data = refine_image_completion_params(model_settings)
+        completion_data = refine_image_completion_params(model_settings, aiconfig, prompt)
         completion_data["prompt"] = resolved_prompt
         return completion_data
 
diff --git a/python/src/aiconfig/default_parsers/openai.py b/python/src/aiconfig/default_parsers/openai.py
index a7e6b03a1..c6a59ab14 100644
--- a/python/src/aiconfig/default_parsers/openai.py
+++ b/python/src/aiconfig/default_parsers/openai.py
@@ -160,7 +160,7 @@ async def deserialize(self, prompt: Prompt, aiconfig: "AIConfigRuntime", params:
         # Build Completion params
         model_settings = self.get_model_settings(prompt, aiconfig)
 
-        completion_params = refine_chat_completion_params(model_settings)
+        completion_params = refine_chat_completion_params(model_settings, aiconfig, prompt)
 
         # In the case thhat the messages array weren't saves as part of the model settings, build it here. Messages array is used for conversation history.
         if not completion_params.get("messages"):
@@ -407,10 +407,10 @@ def multi_choice_message_reducer(messages: Union[Dict[int, dict], None], chunk:
     return messages
 
 
-def refine_chat_completion_params(model_settings):
+def refine_chat_completion_params(model_settings, aiconfig, prompt):
     # completion parameters to be used for openai's chat completion api
     # system prompt handled separately
-    # streaming handled seperatly.
+    # streaming handled separately.
     # Messages array built dynamically and handled separately
     supported_keys = {
         "frequency_penalty",
@@ -436,6 +436,11 @@ def refine_chat_completion_params(model_settings):
         if key in model_settings:
             completion_data[key] = model_settings[key]
 
+    # Explicitly set the model to use if not already specified
+    if completion_data.get("model") is None:
+        model_name = aiconfig.get_model_name(prompt)
+        completion_data["model"] = model_name
+
     return completion_data
 
 
diff --git a/python/src/aiconfig/default_parsers/palm.py b/python/src/aiconfig/default_parsers/palm.py
index edf5fb323..749c2e540 100644
--- a/python/src/aiconfig/default_parsers/palm.py
+++ b/python/src/aiconfig/default_parsers/palm.py
@@ -90,7 +90,7 @@ async def deserialize(self, prompt: Prompt, aiconfig: "AIConfigRuntime", params:
         # Build Completion data
         model_settings = self.get_model_settings(prompt, aiconfig)
 
-        completion_data = refine_chat_completion_params(model_settings)
+        completion_data = refine_chat_completion_params(model_settings, aiconfig, prompt)
 
         prompt_str = resolve_prompt(prompt, params, aiconfig)
 
@@ -246,7 +246,7 @@ async def deserialize(self, prompt: Prompt, aiconfig: "AIConfigRuntime", params:
         # Build Completion data
         model_settings = self.get_model_settings(prompt, aiconfig)
 
-        completion_data = refine_chat_completion_params(model_settings)
+        completion_data = refine_chat_completion_params(model_settings, aiconfig, prompt)
 
         # TODO: handle if user specifies previous messages in settings
         completion_data["messages"] = []
@@ -366,7 +366,7 @@ def get_output_text(
         return ""
 
 
-def refine_chat_completion_params(model_settings):
+def refine_chat_completion_params(model_settings, aiconfig, prompt):
     # completion parameters to be used for Palm's chat completion api
     # messages handled seperately
     supported_keys = {
@@ -384,6 +384,11 @@ def refine_chat_completion_params(model_settings):
         if key.lower() in supported_keys:
             completion_data[key.lower()] = model_settings[key]
 
+    # Explicitly set the model to use if not already specified
+    if completion_data.get("model") is None:
+        model_name = aiconfig.get_model_name(prompt)
+        completion_data["model"] = model_name
+
     return completion_data
 
 
diff --git a/python/src/aiconfig/model_parser.py b/python/src/aiconfig/model_parser.py
index 13e559020..6ec455a75 100644
--- a/python/src/aiconfig/model_parser.py
+++ b/python/src/aiconfig/model_parser.py
@@ -175,10 +175,10 @@ def get_model_settings(self, prompt: Prompt, aiconfig: "AIConfigRuntime") -> Dic
             # Merge config and prompt settings with prompt settings taking precedent
             model_settings = {}
             global_settings = aiconfig.get_global_settings(model_metadata.name)
-            prompt_setings = prompt.metadata.model.settings if prompt.metadata.model.settings is not None else {}
+            prompt_settings = prompt.metadata.model.settings if prompt.metadata.model.settings is not None else {}
 
             model_settings.update(global_settings)
-            model_settings.update(prompt_setings)
+            model_settings.update(prompt_settings)
 
             return model_settings
 


Test patch:
diff --git a/python/tests/parsers/test_openai_util.py b/python/tests/parsers/test_openai_util.py
index e8ae540db..81a92f0c6 100644
--- a/python/tests/parsers/test_openai_util.py
+++ b/python/tests/parsers/test_openai_util.py
@@ -4,20 +4,32 @@
 from aiconfig.default_parsers.openai import refine_chat_completion_params
 from mock import patch
 
-from aiconfig.schema import ExecuteResult, Prompt, PromptInput, PromptMetadata
+from aiconfig.schema import ExecuteResult, Prompt, PromptInput, PromptMetadata, ModelMetadata
 
 from ..conftest import mock_openai_chat_completion
 from ..util.file_path_utils import get_absolute_file_path_from_relative
 
 
 def test_refine_chat_completion_params():
-    model_settings_with_stream_and_system_prompt = {
-        "n": "3",
-        "stream": True,
-        "system_prompt": "system_prompt",
-        "random_attribute": "value_doesn't_matter",
-    }
-    refined_params = refine_chat_completion_params(model_settings_with_stream_and_system_prompt)
+    prompt = Prompt(
+        name="test",
+        input="test",
+        metadata=PromptMetadata(
+            model=ModelMetadata(
+                name="gpt-4",
+                settings={
+                    "n": "3",
+                    "stream": True,
+                    "system_prompt": "system_prompt",
+                    "random_attribute": "value_doesn't_matter",
+                },
+            )
+        ),
+    )
+
+    aiconfig = AIConfigRuntime.create(name="test_refine_chat_completion_params", prompts=[prompt])
+
+    refined_params = refine_chat_completion_params(prompt.metadata.model.settings, aiconfig, prompt)
 
     assert "system_prompt" not in refined_params
     assert "stream" in refined_params