Repository: mlflow/mlflow

Issue:
[BUG] `load_model()` adding code directories to system path that shouldn't be added to path
### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### MLflow version

1.26.1

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.02
- **Python version**: 3.8
- **yarn version, if running the dev UI**: N/A


### Describe the problem

## Overview

The main problem is that MLflow is adding _all directories_ under the `/code` directory to the system path, which is not what one would always expect to be the case. This is causing imports to break / collide in a way that doesn't otherwise happen when running the same code outside of MLflow. It seems to me that one normally expects the top-level modules to be on the system path i.e. that only `<model_directory>/code` should be on the path, and not everything underneath it.

## Minimum Reproducible Example

See attached file for MLflow that this can be reproduced with:
[mlflow-issue-test.zip](https://github.com/mlflow/mlflow/files/8836294/mlflow-issue-test.zip)

#### Model Structure
```
mlflow-issue-test
├── code
│   ├── dir_one
│   │   ├── entrypoint.py
│   │   └── __init__.py
│   └── dir_two
│       ├── redis.py
│       └── __init__.py
├── conda.yml
├── data
└── MLmodel

```

#### Problem Setup

I have two directories in this model, `dir_one` and `dir_two`. `dir_one/entrypoint.py` is the entrypoint to the model and is pointed to by the `MLmodel` entrypoint field. The problem is that this entrypoint is going to try to import a 3rd party dependency (in this case, `redis`) that has a name that collides with the name of a file inside of another directory, `code/dir_two`. 

Under normal operation this doesn't matter since `dir_two/redis.py` isn't on the system path - it can only be imported by doing `from dir_two import redis`. However, since MLflow adds *all* directories under `/code` to the system path, this import is now conflicting with the system import and is breaking inference for us. Specifically, [this code](https://github.com/mlflow/mlflow/blob/master/mlflow/utils/model_utils.py#L120) is the offending logic.

#### Example Code

`code/dir_one/entrypoint.py`: the entrypoint
```python
import mlflow
import pandas as pd

# This should import the system redis-py "redis"
# but it actually grabs `/code/dir_two/redis.py` because
# MLflow messes with the import paths
import redis

class _ModelWrapper(mlflow.pyfunc.PythonModel):
    def predict(self, input_df: pd.DataFrame) -> pd.DataFrame:
        # Toy predict example
        return input_df

def _load_pyfunc(data_path: str) -> _ModelWrapper:
    return _ModelWrapper()
```

`code/dir_two/redis.py`: the colliding file
```python
# This file should not be imported!

raise RuntimeError(f"code/dir_two/redis.py has been imported!")
```

`MLmodel`
```yaml
flavors:
  python_function:
    code: code
    data: data
    env: conda.yml
    loader_module: dir_one.entrypoint
model_uuid: 66a6a42db8444965b7dae6ce6163549d
utc_time_created: '2022-06-03 20:49:12.836282'
```

#### Running the Example

```
>>> mlflow.pyfunc.load_model("mlflow-issue-test")
2022/06/04 00:38:36 WARNING mlflow.pyfunc: The specified model does not have a specified Python version. It may be incompatible with the version of Python that is currently running: Python 3.8.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.env/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py", line 735, in load_model
    model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/mlflow-issue-test/code/dir_one/entrypoint.py", line 7, in <module>
    import redis
  File "/tmp/mlflow-issue-test/code/dir_two/redis.py", line 4, in <module>
    raise RuntimeError(f"code/dir_two/redis.py has been imported!")
RuntimeError: code/dir_two/redis.py has been imported!
```

#### Example Breakdown

Notice that when the entrypoint code calls `import redis`, it ends up importing `code/dir_two/redis.py` instead of the system `redis` module that would be expected. It's not expected that all files under parallel directories are magically on the system path.



### Tracking information

_No response_

### Code to reproduce issue

See attached file for MLflow that this can be reproduced with:
[mlflow-issue-test.zip](https://github.com/mlflow/mlflow/files/8836294/mlflow-issue-test.zip)

Simple unzip the file as an MLflow model and then try to load the model

```bash
unzip mlflow-issue-test.zip -d mlflow-issue-test
python3
```

```python3
>>> import mlflow
>>> mlflow.pyfunc.load_model("mlflow-issue-test")
````

Produces the outputs showed in the next section

### Other info / logs

```python
>>> mlflow.pyfunc.load_model("mlflow-issue-test")
2022/06/04 00:38:36 WARNING mlflow.pyfunc: The specified model does not have a specified Python version. It may be incompatible with the version of Python that is currently running: Python 3.8.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.env/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py", line 735, in load_model
    model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/mlflow-issue-test/code/dir_one/entrypoint.py", line 7, in <module>
    import redis
  File "/tmp/mlflow-issue-test/code/dir_two/redis.py", line 4, in <module>
    raise RuntimeError(f"code/dir_two/redis.py has been imported!")
RuntimeError: code/dir_two/redis.py has been imported!
```

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [X] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [X] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
I guess one possible solution is for me to just move all of my source code down by one directory? i.e. make a directory `<model>/code/package` and place both `dir_one` and `dir_two` under `/code/package`.. that would fix this error
Isn't a top-level definition considered an external namespace declaration? I've always been in the habit of: 
* Placing internal-facing namespace declarations behind internal paths (as you referenced in your self-answer). That's definitely a recommended action.
* Never introduce namespace collisions by 'shadowing' public names of modules (do you need your internal module to be called 'redis'? That might not be the best course of action.)
@BenWilson2 I'm actually not convinced that this is resolved or that MLflow is doing the right thing here. Take this traceback I'm seeing 

```
 File "/opt/ml/model/code/application_package/__init__.py", line 9, in <module>
    from .core import (  # noqa: F401
  File "/opt/ml/model/code/application_package/core.py", line 10, in <module>
    from operators.workflows import WorkflowDAG
  File "/opt/ml/model/code/operators/workflows.py", line 6, in <module>
    import networkx as nx
  File "/miniconda/envs/custom_env/lib/python3.8/site-packages/networkx/__init__.py", line 98, in <module>
    import networkx.utils
  File "/miniconda/envs/custom_env/lib/python3.8/site-packages/networkx/utils/__init__.py", line 2, in <module>
    from networkx.utils.decorators import *
  File "/miniconda/envs/custom_env/lib/python3.8/site-packages/networkx/utils/decorators.py", line 14, in <module>
    from decorator import decorator
  File "/opt/ml/model/code/api_utils/decorator.py", line 3, in <module>
    from fastapi import APIRouter, FastAPI, Request
```

My code is structured as follows

```
mlflow-issue-test
├── code
│   ├── api_utils
│   │   ├── decorator.py
│   │   └── __init__.py
│   └── operators
│       ├── workflows.py
│       └── __init__.py
├── conda.yml
├── data
└── MLmodel
```

Now `workflows.py` does `import networkx`, and `networkx` has a call `from decorator import decorator` (hoping to import the built-in `decorator` utility that ships with Python), this is error'ing out because it's finding *my* `decorator.py` that's under `code/api_utils` directory in my code.

This issue doesn't normally arise because the only way for anyone to access my `decorator.py` is to do `from api_utils import decorator`, but because MLflow adds every directory under `/code` to the PATH, I'm now seeing a conflict. 

Can you explain the logic for why MLflow is doing this in the first place? Your suggestion for putting all of my code under one global namespace wouldn't make a difference here.. imagine that I had this scenario

```
mlflow-issue-test
├── code
│   ├── my_module
│   │   ├── decorator
│   │    |    |── decorator.py
│   │    |    |── __init__.py
├── conda.yml
├── data
└── MLmodel
```

(I moved my code under `my_module` but now have a directory called `decorator` under `my_module`. I've namespaced all my code, but if an external package was doing `from decorator import decorator`, it would conflict with my code because MLflow has placed `/code/my_module/decorator` on the PYTHONPATH.
There was actually totally a bug in 1.24.0 on [this line](https://github.com/mlflow/mlflow/blob/v1.24.0/mlflow/pyfunc/utils.py#L23) that caused models to work because MLflow wasn't actually detecting directories under `/code`. Once upgrading to 1.26.1, now our models break because MLflow is mucking with the system path so much
@jhallard Thank you for raising this issue! I completely agree that the conflict with `from decorator import decorator` is a pretty egregious bug. Ideally, I think it would be sufficient to add the model's `code` directory (instead of every directory recursively) to the PYTHONPATH, then `from api_utils import decorator` would work properly. Let me know what you think here. Would you be interested in contributing this fix?
Yup! And we can confirm that it's the right fix because that's what we were inadvertently doing in version==1.24.0 because of a bug where weren't detecting any of the directories as directories, which left the code such that none of the subdirectories were added on the PYTHONPATH. It's only after that bug was fixed, and the code started working as intended, that these bugs were introduced.

I'm definitely happy to contribute here, but I think it's going to be as simple as just deleting the logic of adding subdirectories?
Perfect! Yep, I agree that this is the right fix. Simply deleting the subdirectory logic should do the trick. Thank you so much - very excited about this contribution!
It seems like this might break some existing models out there that expect those subdirectories to already be on the path, for example, this is the issue that caused the bug to be fixed in the first place: https://github.com/mlflow/mlflow/issues/5786

If I undo that logic, this person's model will break with a `module not found` error. How should that person structure their `sklearn` model with external dependencies such that there's won't break?
Posted a comment here https://github.com/mlflow/mlflow/issues/5786 to try and figure out how to fix this for everyone. We might need to change the way that `sklearn` and others use the `code_paths` argument when logging a model
I wouldn't worry too much about things breaking if someone is relying on recursive path additions to all scoped namespaces. That's definitely not correct. Thanks for continuing to dig in to this! If they have an issue, we can respond with referencing this ticket to elucidate why it's definitely not correct to recursively add all subpaths (I imagine anything named 'utils' or 'tools' or 'config' would blow up!). 
Please let us know when your PR is filed so we can get it in for the 2.0 release. 
Thanks again!
@BenWilson2 sounds good, but just one point of clarity - the code today isn't recursively adding paths, it's just going down level which is enough to cause things to blow up in most cases. This feels like some sort of directory structure off-by-one error

Resolving patch:
diff --git a/mlflow/utils/model_utils.py b/mlflow/utils/model_utils.py
--- a/mlflow/utils/model_utils.py
+++ b/mlflow/utils/model_utils.py
@@ -78,23 +78,6 @@ def _get_flavor_configuration_from_ml_model_file(ml_model_file, flavor_name):
     return model_conf.flavors[flavor_name]
 
 
-def _get_code_dirs(src_code_path, dst_code_path=None):
-    """
-    Obtains the names of the subdirectories contained under the specified source code
-    path and joins them with the specified destination code path.
-    :param src_code_path: The path of the source code directory for which to list subdirectories.
-    :param dst_code_path: The destination directory path to which subdirectory names should be
-                          joined.
-    """
-    if not dst_code_path:
-        dst_code_path = src_code_path
-    return [
-        (os.path.join(dst_code_path, x))
-        for x in os.listdir(src_code_path)
-        if os.path.isdir(os.path.join(src_code_path, x)) and not x == "__pycache__"
-    ]
-
-
 def _validate_code_paths(code_paths):
     if code_paths is not None:
         if not isinstance(code_paths, list):
@@ -122,7 +105,7 @@ def _validate_and_copy_code_paths(code_paths, path, default_subpath="code"):
 
 
 def _add_code_to_system_path(code_path):
-    sys.path = [code_path] + _get_code_dirs(code_path) + sys.path
+    sys.path = [code_path] + sys.path
     # Delete cached modules so they will get reloaded anew from the correct code path
     # Otherwise python will use the cached modules
     modules = [


Test patch:
diff --git a/tests/utils/test_model_utils.py b/tests/utils/test_model_utils.py
--- a/tests/utils/test_model_utils.py
+++ b/tests/utils/test_model_utils.py
@@ -75,8 +75,12 @@ def test_add_code_to_system_path(sklearn_knn_model, model_path):
     mlflow.sklearn.save_model(
         sk_model=sklearn_knn_model,
         path=model_path,
-        code_paths=["tests/utils/test_resources/dummy_module.py"],
+        code_paths=[
+            "tests/utils/test_resources/dummy_module.py",
+            "tests/utils/test_resources/dummy_package",
+        ],
     )
+
     sklearn_flavor_config = mlflow_model_utils._get_flavor_configuration(
         model_path=model_path, flavor_name=mlflow.sklearn.FLAVOR_NAME
     )
@@ -85,5 +89,11 @@ def test_add_code_to_system_path(sklearn_knn_model, model_path):
         # verify that source code paths and their subdirectories are resolved correctly
         with pytest.raises(ModuleNotFoundError, match="No module named 'dummy_module'"):
             import dummy_module
+
         mlflow_model_utils._add_code_from_conf_to_system_path(model_path, sklearn_flavor_config)
         import dummy_module
+
+    # If this raises an exception it's because dummy_package.test imported
+    # dummy_package.operator and not the built-in operator module. This only
+    # happens if MLflow is misconfiguring the system path.
+    from dummy_package import base  # pylint: disable=W0611
diff --git a/tests/utils/test_resources/dummy_package/__init__.py b/tests/utils/test_resources/dummy_package/__init__.py
new file mode 100644
diff --git a/tests/utils/test_resources/dummy_package/base.py b/tests/utils/test_resources/dummy_package/base.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_resources/dummy_package/base.py
@@ -0,0 +1,3 @@
+# This call should import the **system** operator, not the dummy_package.operator
+# module that is adjacent to this file
+import operator  # pylint: disable=W0611
diff --git a/tests/utils/test_resources/dummy_package/operator.py b/tests/utils/test_resources/dummy_package/operator.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_resources/dummy_package/operator.py
@@ -0,0 +1,5 @@
+# This module is meant to test shadowing of the built-in operator module
+raise Exception(
+    "This package should not have been imported!"
+    "This means that the sys.path was not configured correctly"
+)