Repository: SeldonIO/MLServer

Issue:
"Maximum batch size" and "Maximum batch time" are not overwritten through env variables
if the two variable `max_batch_time` and `max_batch_size` are defined in the `model-settings.json`:
```json
{
    "name": "node-1",
    "implementation": "models.NodeOne",
    "max_batch_size": 5,
    "max_batch_time": 1,
    "parameters": {
        "uri": "./fakeuri"
    }
}
```
Then they couldn't be overwritten by environment variables `MLSERVER_MODEL_MAX_BATCH_SIZE` and `MLSERVER_MODEL_MAX_BATCH_TIME` and mlserver will always revert back to the values set in the `model-settings.json`. However, it is expected to use the values in the json as defaults and env variable as user arguments for overwriting them.



Discussion:
Hey @saeid93 ,

Yeah, it's a known issue that, when using `mlserver build`, local setting files override environment variables. This is due to how they get injected during the container's startup (which is essentially by injecting them as env vars as well).

Having said that, there wasn't an issue for that yet, so thanks for opening one! 
Hey @adriangonz,

Sure, no problem. Since the setting.json is also injected by env vars why the same env var is not used for the initialization as well? so it will be overwritten by the user-defined var?


Hey @saeid93 ,

When you build a custom image and it has a `settings.json` present, the same issue should occur. As in, the values set up on that local `settings.json` will get injected through env vars, overriding whatever else comes from upstream.

Resolving patch:
diff --git a/hack/generate_dotenv.py b/hack/generate_dotenv.py
--- a/hack/generate_dotenv.py
+++ b/hack/generate_dotenv.py
@@ -82,9 +82,10 @@ def _parse_dict_values(name: str, value: str) -> str:
     try:
         value = value.replace("'", '"')
         json.loads(value)
-        return f"export {name}='{value}'\n"
+        return f"{name}='{value}'\n"
     except JSONDecodeError:
-        return f'export {name}="{value}"\n'
+        # If not JSON, then assume it's a plain string
+        return f'{name}="{value}"\n'
 
 
 @click.command()
diff --git a/mlserver/cli/constants.py b/mlserver/cli/constants.py
--- a/mlserver/cli/constants.py
+++ b/mlserver/cli/constants.py
@@ -48,8 +48,9 @@
 
 USER root
 # Install dependencies system-wide, to ensure that they are available for every
-# user
-RUN ./hack/build-env.sh . ./envs/base && \
+# user and give permissions to (future) environment folder.
+RUN ./hack/build-env.sh . && \\
+    mkdir -p ./envs/base && \\
     chown -R 1000:0 ./envs/base && \\
     chmod -R 776 ./envs/base
 USER 1000
@@ -59,7 +60,7 @@
 
 # Override MLServer's own `CMD` to activate the embedded environment
 # (optionally activating the hot-loaded one as well).
-CMD source ./hack/activate-env.sh ./envs/base.tar.gz ./envs/base && \\
+CMD source ./hack/activate-env.sh ./envs/base.tar.gz && \\
     mlserver start $MLSERVER_MODELS_DIR
 """
 


Test patch:
diff --git a/tests/testdata/env_models.py b/tests/testdata/env_models.py
--- a/tests/testdata/env_models.py
+++ b/tests/testdata/env_models.py
@@ -11,7 +11,7 @@
 
 from mlserver import MLModel
 from mlserver.types import InferenceRequest, InferenceResponse
-from mlserver.codecs import NumpyCodec
+from mlserver.codecs import NumpyRequestCodec
 
 
 class DummySKLearnModel(MLModel):
@@ -27,8 +27,9 @@ async def load(self) -> bool:
         return self.ready
 
     async def predict(self, payload: InferenceRequest) -> InferenceResponse:
-        decoded = self.decode(payload.inputs[0])
+        decoded = self.decode_request(payload, default_codec=NumpyRequestCodec)
         prediction = self._model.predict(decoded)
 
-        output = NumpyCodec().encode(name="prediction", payload=prediction)
-        return InferenceResponse(id=payload.id, model_name=self.name, outputs=[output])
+        return NumpyRequestCodec.encode_response(
+            model_name=self.name, payload=prediction, model_version=self.version
+        )
diff --git a/tests/testdata/environment.yml b/tests/testdata/environment.yml
--- a/tests/testdata/environment.yml
+++ b/tests/testdata/environment.yml
@@ -3,4 +3,4 @@ dependencies:
   - python == 3.7
   - scikit-learn
   - pip:
-      - mlserver == 1.1.0.dev6
+      - mlserver == 1.2.0.dev6