Repository: mlflow/mlflow

Issue:
[BUG] Double runs created using databricks projects backend, `_mlflow_local_backend_run_id` seems ignored.
### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [x] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.04
- **MLflow installed from (source or binary)**: source
- **MLflow version (run ``mlflow --version``)**: 1.12
- **Python version**: 3.7
- **npm version, if running the dev UI**: not applicable
- **Exact command to reproduce**: `mlflow run . -b databricks --backend-config cluster-config.json`

### Describe the problem
When running mlflow projects using the Databricks projects backend two distinct run_id's seem to be created. The first run id is created by the client submitting the shell command job to databricks. This creates a run on the tracking server and connects the job status to that run. The MLFlow library then creates a bash command job as a databricks job by running the following command on the databricks job

```
mlflow run PATH_ON_DATABRICKS --entry-point MY_ENTRY_POINT
 -c '{"_mlflow_local_backend_run_id": "THE RUN ID OF THE RUN CREATED BY THE CLIENT"}'
```

However, when this command is executed in the databricks job the config for the local backend passed by `_mlflow_local_backend_run_id` seems to be ignored. This results in a new run being created on the tracking server. This is the run where the databricks job will log all its parameters to.

It seems easy to fix this issue by, instead of passing the client run id to the databricks job via that (seemingly private) `_mlflow_local_backend_run_id` parameter as the backend config, using the `--run-id` cli option. I've created a small MR to show how this could be resolved (#3788 ).

### Code to reproduce issue
I am not able to reproduce this issue locally, it seems only in the databricks job the local backend config `_mlflow_local_backend_run_id` is ignored.

To reproduce, create any simple mlflow project and try to run that with the databricks backend.

### Other info / logs
[This](https://ibb.co/9w2Fjny) is a picture of how runs show up in the UI. The bottom run is created by the client running `mlflow run . -b databricks` the top run is the run created by the bash job executed on databricks that actually performs the work. The bottom run will always have state UNFINISHED since it is never completed by any running job.



### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [x] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [ ] `area/server-infra`: MLflow server, JavaScript dev server
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [x] `integrations/databricks`: Databricks integrations



Discussion:
Me and my colleague @SebGr did a bit more investigation on this issue and it seems that this was issue was introduced in `1.9`. `mlflow==1.8` does not exhibit this behaviour.  `1.9` is also the version where `mlflow` started using the local backend config to pass the `run_id` to the databricks job instead of relying on the `--run-id` command line argument (as I'm proposing to revert to as a fix in #3788).
@FlorisHoogenboom thank you for filing this issue. I am having trouble reproducing when using a Databricks backend. Would you mind providing the following:

a) Version of mlflow specified in your project's `conda.yaml`.
b) [Databricks runtime version](https://docs.databricks.com/release-notes/runtime/releases.html) (eg "Databricks Runtime 7.5 for ML")
@AveshCSingh . Thank you for looking into this. We we're running still on 6.6-ml (needing to upgrade to 7.3-ml-lts). As it turns out the issue only exists when running on 6.6-ml and not on 7.3-ml. I've created a small repo for you to reproduce. 

See [here](https://github.com/FlorisHoogenboom/mlflow-double-runs/tree/master) to reproduce the issue on 6.6-ml. See here to run the same project on [7.3-ml-lts ](https://github.com/FlorisHoogenboom/mlflow-double-runs/tree/dbr7.3)(which does not create double runs). Both use `mlflow=1.12.1` in their `environment.yaml` file ([here](https://github.com/FlorisHoogenboom/mlflow-double-runs/blob/4ed0cae5a04579441d1a037d7e44343cafe16260/mlflow-environment.yml#L8)).

It is interesting to see it does not appear on `7.3-ml`. To me that seems to point how that local backend config is interpreted on the backend.
Ah I see. The problem is that these earlier versions of the ML runtime run mlflow <= 1.8.0. The backend_config which specifies `_MLFLOW_LOCAL_BACKEND_RUN_ID_CONFIG` is not backwards compatible to these earlier versions of MLflow. One possible fix is to both pass the backend config as well as `--run-id`.

If you would like to implement this, I would be happy to review. Otherwise, I can see if the engineering team at Databricks has the bandwidth.
Ahh, that explains a lot. Thank you so much for helping out here. We've been banging our head against this one for quite some time! I do feel that this is actually quite a tricky thing: the MLFlow version to start the run using the local backend is not the version specified in your projects `conda.yaml` or the version specified as the cluster dependency (as part of the submitted job). This now means that it is very easy to end up with a project where you have three different versions of MLFlow:

- The version of your client
- The version you specify in your projects `conda.yaml` and hence use to log
- The version that comes preinstalled on the databricks runtime (and basically serves as the "outer" mlflow version that is used to kick of a local run on the cluster).

I do think that this might be something that would be good to document a bit more (happy to submit an MR for that if you agree here), but also it might be better to just pin the version of MLFlow that is installed on the cluster level. Now if there is by default an older version of MLFlow on the runtime that is allowed, but I think it might be better to just pin the version to the same version as the client here: https://github.com/mlflow/mlflow/blob/c7b5c5dd3ab7281f6fd9b0d44f78589353989b46/mlflow/projects/databricks.py#L199

That solves the issue of the "outer" MLFlow on databricks running a different version than the local client you are running locally and hence would prevent similar issues like this one in the future! I'm also happy to adjust the linked MR to implement the solution you proposed, but maybe this is a more generic solution. Let me know what you think.
Thanks for summarizing this so clearly! I think using the same version for `(2) The version you specify in your projects conda.yaml and hence use to log` and `(3) The version that comes preinstalled on the databricks runtime.` would reduce the cognitive overhead here.

Unfortunately, we would need this change to be backwards compatible. Meaning that MLProjects running against existing Databricks runtimes should continue to operate as-is, while new runtimes can fix the version of mlflow used in (3).

Another worry is that we may be breaking some use cases. @dbczumar @juntai-zheng @smurching is it common for a user to run an MLflow project against an MLflow backend, but want to use a different version of MLflow than that backend comes with?
I spoke with @smurching. Pinning the version of mlflow here
https://github.com/mlflow/mlflow/blob/c71bae2365177e57d7b50004c5b9964bd455bacc/mlflow/projects/databricks.py#L199
as you suggested sounds good. Though we will need to add some special logic to work around the issue described in that comment. Something like this pseudo-code:
```
if VERSION.ends_with('.dev0'):
  libraries = [{"pypi": {"package": "'mlflow<=%s'" % VERSION}}] 
else:
  libraries = [{"pypi": {"package": "'mlflow=%s'" % VERSION}}] 
```

Resolving patch:
diff --git a/mlflow/projects/databricks.py b/mlflow/projects/databricks.py
--- a/mlflow/projects/databricks.py
+++ b/mlflow/projects/databricks.py
@@ -25,7 +25,7 @@
     MLFLOW_DATABRICKS_WEBAPP_URL,
 )
 from mlflow.utils.uri import is_databricks_uri, is_http_uri
-from mlflow.version import VERSION
+from mlflow.version import is_release_version, VERSION
 
 # Base directory within driver container for storing files related to MLflow
 DB_CONTAINER_BASE = "/databricks/mlflow"
@@ -193,10 +193,20 @@ def _run_shell_command_job(self, project_uri, command, env_vars, cluster_spec):
                  Databricks
                  `Runs Get <https://docs.databricks.com/api/latest/jobs.html#runs-get>`_ API.
         """
-        # NB: We use <= on the version specifier to allow running projects on pre-release
-        # versions, where we will select the most up-to-date mlflow version available.
-        # Also note, that we escape this so '<' is not treated as a shell pipe.
-        libraries = [{"pypi": {"package": "'mlflow<=%s'" % VERSION}}]
+        if is_release_version():
+            libraries = [{"pypi": {"package": "mlflow==%s" % VERSION}}]
+        else:
+            # When running a non-release version as the client the same version will not be
+            # available within Databricks.
+            _logger.warning(
+                (
+                    "Your client is running a non-release version of MLFlow. "
+                    "This version is not avaialable on the databricks runtime. "
+                    "MLFlow will fallback the MLFlow version provided by the runtime. "
+                    "This might lead to unforeseen issues. "
+                )
+            )
+            libraries = [{"pypi": {"package": "'mlflow<=%s'" % VERSION}}]
 
         # Check syntax of JSON - if it contains libraries and new_cluster, pull those out
         if "new_cluster" in cluster_spec:
diff --git a/mlflow/version.py b/mlflow/version.py
--- a/mlflow/version.py
+++ b/mlflow/version.py
@@ -1,4 +1,9 @@
 # Copyright 2018 Databricks, Inc.
+import re
 
 
 VERSION = "1.12.2.dev0"
+
+
+def is_release_version():
+    return bool(re.match(r"^\d+\.\d+\.\d+$", VERSION))


Test patch:
diff --git a/tests/test_version.py b/tests/test_version.py
new file mode 100644
--- /dev/null
+++ b/tests/test_version.py
@@ -0,0 +1,9 @@
+from mlflow import version
+
+
+def test_is_release_version(monkeypatch):
+    monkeypatch.setattr(version, "VERSION", "1.19.0")
+    assert version.is_release_version()
+
+    monkeypatch.setattr(version, "VERSION", "1.19.0.dev0")
+    assert not version.is_release_version()