Repository: SeldonIO/MLServer

Issue:
[huggingface] Merge predictive unit parameters from env with model settings
# Background
## What are PREDICTIVE_UNIT_PARAMETERS
This is a collection of parameters passed though the environment to a HuggingFace model. This is opposed to declaring them in a `model-settings.json` file as it is the case in Seldon Core v2. As of this moment, those parameters are injected via Seldon Core v1 only, where as Seldon Core v2 uses `model-settings.json` to provide the metadata for the model.

PREDICTIVE_UNIT_PARAMETERS are used only  by the HuggingFace runtime and injected only by SCv1.

You can find the code for creating `HuggingFaceSettings` in `./runtimes/huggingface/ml-server-huggingface/settings.py` fle along with functions for parsing those params from env vars or from `model-settings.json`  

# What is the problem
Currently, `HuggingFaceSettings` are created either from parsing the PREDICTIVE_UNIT_PARAMETERS from the environment OR from the `model-settings.json`. Meaning that if there is at least one parameter set in the env var, the `model-settings.json` extra parameters will be ignored. This makes it cumbersome when a deployment is created from the UI because additional params such as `task`, `pretrained_model`, `pretrained_tokenizer`, `framework`, etc. will have to be added one by one in the Wizard. Why they have to be added from the wizard and not just specified in `model-settings.json` - because currently SCv1 always injects `model_uri` param so the PREDICTIVE_UNIT_PARAMETERS env var so it's not empty. Because this var is not empty, the HF settings are initialised from it and the `model-settings.json` is ignored.

# What needs to be done
When creating HuggingFace settings, env vars needs to be merged with params from `model-settings.json`, giving priority to env vars. For example:
If such env var exists:
```
PREDICTIVE_UNIT_PARAMETERS = [{"name":"model_uri","value":"/mnt/models","type":"STRING"}, {"name":"task_suffix","value":"else","type":"STRING"}]
```
and such `model-settings.json` file exists:
```
{
    "name": "transformer",
    "implementation": "mlserver_huggingface.HuggingFaceRuntime",
    "parameters": {
        "extra": {
            "task": "text-generation",
            "task_suffix": "something",
            "framework": "pt"
        }
    }
}
```
The outcome should be that the `task` parameter doesn't need to be specified in the wizard and
The HuggingFace settings should contain values: task = text-generation, task_suffix = else, framework = pt

# Scope
This only relating to `HuggingFace` runtime and when it's used from SCv1 and only valid as long as SCv1 is still operational and related code is present in MLServer.


Discussion:

Resolving patch:
diff --git a/runtimes/huggingface/mlserver_huggingface/settings.py b/runtimes/huggingface/mlserver_huggingface/settings.py
--- a/runtimes/huggingface/mlserver_huggingface/settings.py
+++ b/runtimes/huggingface/mlserver_huggingface/settings.py
@@ -1,7 +1,7 @@
 import os
 import orjson
 
-from typing import Optional, Dict
+from typing import Optional, Dict, Union, NewType
 from pydantic import BaseSettings
 from distutils.util import strtobool
 from transformers.pipelines import SUPPORTED_TASKS
@@ -110,23 +110,33 @@ def task_name(self):
         return self.task
 
 
-def parse_parameters_from_env() -> Dict:
+EXTRA_TYPE_DICT = {
+    "INT": int,
+    "FLOAT": float,
+    "DOUBLE": float,
+    "STRING": str,
+    "BOOL": bool,
+}
+
+ExtraDict = NewType("ExtraDict", Dict[str, Union[str, bool, float, int]])
+
+
+def parse_parameters_from_env() -> ExtraDict:
     """
     This method parses the environment variables injected via SCv1.
+
+    At least an empty dict is always returned.
     """
     # TODO: Once support for SCv1 is deprecated, we should remove this method and rely
     # purely on settings coming via the `model-settings.json` file.
     parameters = orjson.loads(os.environ.get(PARAMETERS_ENV_NAME, "[]"))
 
-    type_dict = {
-        "INT": int,
-        "FLOAT": float,
-        "DOUBLE": float,
-        "STRING": str,
-        "BOOL": bool,
-    }
+    parsed_parameters: ExtraDict = ExtraDict({})
+
+    # Guard: Exit early if there's no parameters
+    if len(parameters) == 0:
+        return parsed_parameters
 
-    parsed_parameters = {}
     for param in parameters:
         name = param.get("name")
         value = param.get("value")
@@ -135,22 +145,20 @@ def parse_parameters_from_env() -> Dict:
             parsed_parameters[name] = bool(strtobool(value))
         else:
             try:
-                parsed_parameters[name] = type_dict[type_](value)
+                parsed_parameters[name] = EXTRA_TYPE_DICT[type_](value)
             except ValueError:
                 raise InvalidModelParameter(name, value, type_)
             except KeyError:
                 raise InvalidModelParameterType(type_)
+
     return parsed_parameters
 
 
 def get_huggingface_settings(model_settings: ModelSettings) -> HuggingFaceSettings:
-    env_params = parse_parameters_from_env()
-    if not env_params and (
-        not model_settings.parameters or not model_settings.parameters.extra
-    ):
-        raise MissingHuggingFaceSettings()
+    """Get the HuggingFace settings provided to the runtime"""
 
-    extra = env_params or model_settings.parameters.extra  # type: ignore
+    env_params = parse_parameters_from_env()
+    extra = merge_huggingface_settings_extra(model_settings, env_params)
     hf_settings = HuggingFaceSettings(**extra)  # type: ignore
 
     if hf_settings.task not in SUPPORTED_TASKS:
@@ -161,3 +169,35 @@ def get_huggingface_settings(model_settings: ModelSettings) -> HuggingFaceSettin
             raise InvalidOptimumTask(hf_settings.task, SUPPORTED_OPTIMUM_TASKS.keys())
 
     return hf_settings
+
+
+def merge_huggingface_settings_extra(
+    model_settings: ModelSettings, env_params: ExtraDict
+) -> ExtraDict:
+    """
+    This function returns the Extra field of the Settings.
+
+    It merges them, iff they're both present, from the
+    environment AND model settings file. Precedence is
+    giving to the environment.
+    """
+
+    # Both `parameters` and `extra` are Optional, so we
+    # need to get the value, or nothing.
+    settings_params = (
+        model_settings.parameters.extra
+        if model_settings.parameters is not None
+        else None
+    )
+
+    if settings_params is None and env_params == {}:
+        # There must be settings provided by at least the environment OR model settings
+        raise MissingHuggingFaceSettings()
+
+    # Set the default value
+    settings_params = settings_params or {}
+
+    # Overwrite any conflicting keys, giving precedence to the environment
+    settings_params.update(env_params)
+
+    return ExtraDict(settings_params)


Test patch:
diff --git a/runtimes/huggingface/tests/test_settings.py b/runtimes/huggingface/tests/test_settings.py
new file mode 100644
--- /dev/null
+++ b/runtimes/huggingface/tests/test_settings.py
@@ -0,0 +1,169 @@
+import pytest
+
+from typing import Dict
+
+from mlserver.settings import ModelSettings, ModelParameters
+
+from mlserver_huggingface.runtime import HuggingFaceRuntime
+from mlserver_huggingface.settings import (
+    HuggingFaceSettings,
+    ExtraDict,
+    PARAMETERS_ENV_NAME,
+    get_huggingface_settings,
+    merge_huggingface_settings_extra,
+)
+from mlserver_huggingface.errors import MissingHuggingFaceSettings
+
+
+@pytest.fixture()
+def model_settings_extra_task():
+    return ModelSettings(
+        name="foo",
+        implementation=HuggingFaceRuntime,
+        parameters=ModelParameters(
+            extra={"task": "text-generation", "pretrained_model": "distilgpt2"}
+        ),
+    )
+
+
+@pytest.fixture()
+def model_settings_extra_none():
+    return ModelSettings(
+        name="foo",
+        implementation=HuggingFaceRuntime,
+        parameters=ModelParameters(extra=None),
+    )
+
+
+@pytest.fixture()
+def model_settings_extra_empty():
+    return ModelSettings(
+        name="foo",
+        implementation=HuggingFaceRuntime,
+        parameters=ModelParameters(extra={}),
+    )
+
+
+@pytest.mark.parametrize(
+    "model_settings,env_params,expected",
+    [
+        (
+            "model_settings_extra_task",
+            {"task": "question-answering"},
+            {"task": "question-answering", "pretrained_model": "distilgpt2"},
+        ),
+        (
+            "model_settings_extra_task",
+            {},
+            {"task": "text-generation", "pretrained_model": "distilgpt2"},
+        ),
+        (
+            "model_settings_extra_none",
+            {"task": "question-answering"},
+            {"task": "question-answering"},
+        ),
+        (
+            "model_settings_extra_empty",
+            {"task": "question-answering"},
+            {"task": "question-answering"},
+        ),
+    ],
+)
+def test_merge_huggingface_settings_extra(
+    model_settings: str,
+    env_params: ExtraDict,
+    expected: Dict,
+    request: pytest.FixtureRequest,
+):
+    assert expected == merge_huggingface_settings_extra(
+        request.getfixturevalue(model_settings), env_params
+    )
+
+
+def test_merge_huggingface_settings_extra_raises(model_settings_extra_none):
+    with pytest.raises(MissingHuggingFaceSettings):
+        merge_huggingface_settings_extra(model_settings_extra_none, {})
+
+
+@pytest.mark.parametrize(
+    "model_settings,env_params,expected",
+    [
+        (
+            "model_settings_extra_task",
+            '[{"name": "task", "value": "question-answering", "type": "STRING"}]',
+            HuggingFaceSettings(
+                task="question-answering",
+                task_suffix="",
+                pretrained_model="distilgpt2",
+                pretrained_tokenizer=None,
+                framework=None,
+                optimum_model=False,
+                device=-1,
+                inter_op_threads=None,
+                intra_op_threads=None,
+            ),
+        ),
+        (
+            "model_settings_extra_task",
+            "[]",
+            HuggingFaceSettings(
+                task="text-generation",
+                task_suffix="",
+                pretrained_model="distilgpt2",
+                pretrained_tokenizer=None,
+                framework=None,
+                optimum_model=False,
+                device=-1,
+                inter_op_threads=None,
+                intra_op_threads=None,
+            ),
+        ),
+        (
+            "model_settings_extra_none",
+            '[{"name": "task", "value": "question-answering", "type": "STRING"}]',
+            HuggingFaceSettings(
+                task="question-answering",
+                task_suffix="",
+                pretrained_model=None,
+                pretrained_tokenizer=None,
+                framework=None,
+                optimum_model=False,
+                device=-1,
+                inter_op_threads=None,
+                intra_op_threads=None,
+            ),
+        ),
+        (
+            "model_settings_extra_empty",
+            '[{"name": "task", "value": "question-answering", "type": "STRING"}]',
+            HuggingFaceSettings(
+                task="question-answering",
+                task_suffix="",
+                pretrained_model=None,
+                pretrained_tokenizer=None,
+                framework=None,
+                optimum_model=False,
+                device=-1,
+                inter_op_threads=None,
+                intra_op_threads=None,
+            ),
+        ),
+    ],
+)
+def test_get_huggingface_settings(
+    model_settings: str,
+    env_params: str,
+    expected: HuggingFaceSettings,
+    request: pytest.FixtureRequest,
+    monkeypatch: pytest.MonkeyPatch,
+):
+    monkeypatch.setenv(PARAMETERS_ENV_NAME, env_params)
+
+    assert expected == get_huggingface_settings(request.getfixturevalue(model_settings))
+
+    monkeypatch.delenv(PARAMETERS_ENV_NAME)
+
+
+def test_get_huggingface_settings_raises(model_settings_extra_none):
+    with pytest.raises(MissingHuggingFaceSettings):
+        get_huggingface_settings(model_settings_extra_none)