Repository: bentoml/BentoML

Issue:
Add python version mismatch warning
**Is your feature request related to a problem? Please describe.**

Currently, when the user saves a BentoML bundle, the `bentoml.yml` file under the bundle directory contains the Python version in which the bundle was created.

When deploying with Docker, bentoml ensures that the python version matches the saved bundle python version. Although when the user is using a custom docker image, or when the user is loading directly with the `bentoml.load` API, there's no guarantee the python version matches.

This can cause issues in some cases, especially where PickleArtifact or PyTorchArtifact is being used.

We should add a warning when loading a BentoML saved bundle if the current Python version does not match the saved bundle's python version.


**Describe the solution you'd like**
Add a warning to tell the user about Python version mismatch

**Describe alternatives you've considered**
Instead of warning, raise an exception.  

We can still allow user to specifically set a config to allow loading BentoML bundle of a different python version, e.g.  'BENTOML__CORE__PYPASS_BUNDLE_PY_VERSION_CHECK=true' 

**Additional context**
n/a


Discussion:
Hi @parano. I've started to look into this a little. :)

> We should add a warning when loading a BentoML saved bundle if the current Python version does not match the saved bundle's python version.

For figuring out the saved bundle's python version, I see there is a `python_version` in the `env` category in a saved bundle's `bentoml.yml` config file. This value comes from the user's installed python version I believe, see the following links:
* [[**code that saves env config to .yml**]](https://github.com/bentoml/BentoML/blob/master/bentoml/saved_bundle/config.py#L89)
* [[**code where version is assigned to `BentoServiceEnv`**]](https://github.com/bentoml/BentoML/blob/master/bentoml/service_env.py#L116)
*  [[**code where version is determined from `sys.version_info`**]](https://github.com/bentoml/BentoML/blob/master/bentoml/service_env.py#L37)

But, if we're always taking the python version from `sys.version_info`, it seems like it wouldn't necessarily be helpful for determining the version of a user-supplied custom base image... 

I did some reading (e.g. [this page](https://docs.bentoml.org/en/latest/concepts.html?highlight=base%20image#using-other-docker-base-images) in the documentation), but I am a bit new to Docker, so it is not yet clear to me how to detect the version of python that's inside an arbitrary Docker image.

@joshuacwnewton great questions! The problem we are trying to solve is when the training environment and serving environment has a different Python version.

For example, I run the quickstart guide on my laptop which has python 3.6 installed:
```bash
cd bentoml
python guides/quick-start/main.py
```

Now since BentoML takes the python version from current environment, in this saved bundle, the python_version in bentoml.yml file will be 3.6 matching the version of my laptop.  The default docker image that BentoML provide, ensures that the docker image will have the identical python version, it essentially installs the target python version with conda: https://github.com/bentoml/BentoML/blob/v0.8.3/bentoml/saved_bundle/bentoml-init.sh

Recently we introduced the custom docker base image API allowing users to use their own docker image for deployment. And if their base image does not have `conda` installed, BentoML will just use the system python. And it might not be the expected python version.

Let's say I build a custom docker image that has python 3.7 installed and it does not have conda installed, and I specify it to be the base image when creating a BentoML saved bundle on my laptop with Python 3.6.  Then when I run the API server docker,  `sys.version_info` in the python session within the docker container will return `3.7`, but the `bentoml.yml` file contains `3.6`.  And that's when we should show a warning to the user, telling them about the version mismatch.

This warning is also meaningful when the user tries to manually deploy to a server without docker, and they might accidentally install a different python version.
Apologies for the delay -- I was finishing up with scikit-learn so I could focus on BentoML full-time for the rest of the MLH program. :)

Thank you so much for your patience, and for taking the time to explain, @parano. I realize now that I didn't understand Docker very well when I asked my question. I have learned more since, and your explanation makes much more sense now, so thank you.

Just to confirm, we are warning just on minor version mismatches (e.g. `3.6` vs. `3.7` as you mention) and not micro version mismatches (e.g. `3.7.6` vs. `3.7.5`)?

Resolving patch:
diff --git a/bentoml/saved_bundle/config.py b/bentoml/saved_bundle/config.py
--- a/bentoml/saved_bundle/config.py
+++ b/bentoml/saved_bundle/config.py
@@ -16,6 +16,7 @@
 import logging
 from datetime import datetime
 from pathlib import Path
+from sys import version_info
 
 from ruamel.yaml import YAML
 
@@ -35,6 +36,7 @@
 logger = logging.getLogger(__name__)
 DEFAULT_MAX_LATENCY = config("marshal_server").getint("default_max_latency")
 DEFAULT_MAX_BATCH_SIZE = config("marshal_server").getint("default_max_batch_size")
+PYTHON_VERSION = f"{version_info.major}.{version_info.minor}.{version_info.micro}"
 
 
 def _get_apis_list(bento_service):
@@ -99,6 +101,7 @@ def load(cls, filepath):
             yml_content = config_file.read()
         conf.config = conf._yaml.load(yml_content)
         ver = str(conf["version"])
+        py_ver = conf.config["env"]["python_version"]
 
         if ver != BENTOML_VERSION:
             msg = (
@@ -117,6 +120,13 @@ def load(cls, filepath):
             else:  # Otherwise just show a warning.
                 logger.warning(msg)
 
+        if py_ver != PYTHON_VERSION:
+            logger.warning(
+                f"Saved BentoService Python version mismatch: loading "
+                f"BentoService bundle created with Python version {py_ver}, "
+                f"but current environment version is {PYTHON_VERSION}."
+            )
+
         return conf
 
     def get_bento_service_metadata_pb(self):


Test patch:
diff --git a/tests/test_save_and_load.py b/tests/test_save_and_load.py
--- a/tests/test_save_and_load.py
+++ b/tests/test_save_and_load.py
@@ -1,3 +1,4 @@
+import logging
 import uuid
 import mock
 import pytest
@@ -125,3 +126,32 @@ def test_save_duplicated_bento_exception_raised(example_bento_service_class):
 
     delete_saved_bento_service(svc_metadata.name, svc_metadata.version)
     delete_saved_bento_service(svc_metadata_new.name, svc_metadata_new.version)
+
+
+def test_pyversion_warning_on_load(
+    tmp_path_factory, capsys, example_bento_service_class
+):
+    # Set logging level so version mismatch warnings are outputted
+    bentoml.configure_logging(logging_level=logging.WARNING)
+    # (Note that logger.warning() is captured by pytest in stdout, NOT stdlog.
+    #  So the warning is in capsys.readouterr().out, NOT caplog.text.)
+
+    test_model = TestModel()
+    svc = example_bento_service_class()
+    svc.pack('model', test_model)
+
+    # Should not warn for default `_python_version` value
+    match_dir = tmp_path_factory.mktemp("match")
+    svc.save_to_dir(match_dir)
+    _ = bentoml.load(str(match_dir))
+    assert "Python version mismatch" not in capsys.readouterr().out
+
+    # Should warn for any version mismatch (major, minor, or micro)
+    svc.env._python_version = "X.Y.Z"
+    mismatch_dir = tmp_path_factory.mktemp("mismatch")
+    svc.save_to_dir(mismatch_dir)
+    _ = bentoml.load(str(mismatch_dir))
+    assert "Python version mismatch" in capsys.readouterr().out
+
+    # Reset logging level to default
+    bentoml.configure_logging()