Repository: mlflow/mlflow
Pull request: Fix `validate_docker_installation` to throw when docker daemon is not running
Signed-off-by: harupy <hkawamura0130@gmail.com>

## What changes are proposed in this pull request?

Fixes https://github.com/mlflow/mlflow/issues/5851

## How is this patch tested?

Existing tests

## Does this PR change the documentation?

- [x] No. You can skip the rest of this section.
- [ ] Yes. Make sure the changed pages / sections render correctly by following the steps below.

1. Check the status of the `ci/circleci: build_doc` check. If it's successful, proceed to the
   next step, otherwise fix it.
2. Click `Details` on the right to open the job page of CircleCI.
3. Click the `Artifacts` tab.
4. Click `docs/build/html/index.html`.
5. Find the changed pages / sections and make sure they render correctly.

## Release Notes

### Is this a user-facing change?

- [x] No. You can skip the rest of this section.
- [ ] Yes. Give a description of this change to be included in the release notes for MLflow users.

(Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change.)

### What component(s), interfaces, languages, and integrations does this PR affect?
Components
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [x] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [x] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
Insert an empty named anchor here to allow jumping to this section with a fragment URL
(e.g. https://github.com/mlflow/mlflow/pull/123#user-content-release-note-category).
Note that GitHub prefixes anchor names in markdown with "user-content-".
-->
<a name="release-note-category"></a>
### How should the PR be classified in the release notes? Choose one:

- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [x] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [ ] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes


Problem statement:
[BUG] Obscure error when Docker not started
### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### System information

- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: Nope - pretty standard
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: OSX
- **MLflow installed from (source or binary)**: pip
- **MLflow version (run ``mlflow --version``)**: mlflow, version 1.24.0
- **Python version**: Python 3.8.2
- **npm version, if running the dev UI**:


### Describe the problem

When docker daemon is not running on OSX, `mlflow run .` with docker env returns a cryptic "file not found error".

```
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
```

### Tracking information

_No response_

### Code to reproduce issue

```yaml
name: foo

docker_env:
  image: gcr.io/foo/bar-mlflow-base

entry_points:
  main:
    parameters:
      timesteps: { type: int, default: 100 }
      record_timesteps: { type: int, default: 1000 }
    command: "python pipeline.py --timesteps {timesteps} --record-timesteps {record_timesteps}"
```

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [X] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [X] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

Hints from issue discussion:
@stobias123 Can you share the full stack trace?
```
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/user/bin/mlflow", line 8, in <module>
    sys.exit(cli())
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/mlflow/cli.py", line 168, in run
    projects.run(
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/mlflow/projects/__init__.py", line 293, in run
    submitted_run_obj = _run(
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/mlflow/projects/__init__.py", line 145, in _run
    image = build_docker_image(
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/mlflow/projects/docker.py", line 69, in build_docker_image
    client = docker.from_env()
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/Users/user/.virtualenvs/user/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
```

Are you sure docker is running on your system? Can you run `docker version`?
https://stackoverflow.com/questions/64952238/docker-errors-dockerexception-error-while-fetching-server-api-version seems related
No that's what I said is the bug.
> When docker daemon is not running on OSX, mlflow run . with docker env returns a cryptic "file not found error".


I would expect this to be a simple pre-execution check. Instead I wait for quite some time, then get a cryptic error message as shown in stacktrace above. Took me a while to realize what was going on.
Sorry, I misunderstood. Filed #5853

Commit message: join command

Signed-off-by: harupy <hkawamura0130@gmail.com>

Patch to review:
diff --git a/mlflow/projects/docker.py b/mlflow/projects/docker.py
index 0eb2d389a34c8..7d01407684b66 100644
--- a/mlflow/projects/docker.py
+++ b/mlflow/projects/docker.py
@@ -26,18 +26,23 @@
 
 def validate_docker_installation():
     """
-    Verify if Docker is installed on host machine.
+    Verify if Docker is installed and running on host machine.
     """
-    try:
-        docker_path = "docker"
-        process._exec_cmd([docker_path, "--help"], throw_on_error=False)
-    except EnvironmentError:
+    if shutil.which("docker") is None:
         raise ExecutionException(
             "Could not find Docker executable. "
             "Ensure Docker is installed as per the instructions "
             "at https://docs.docker.com/install/overview/."
         )
 
+    cmd = ["docker", "ps"]
+    prc = process._exec_cmd(cmd, throw_on_error=False)
+    if prc.returncode != 0:
+        joined_cmd = " ".join(cmd)
+        raise ExecutionException(
+            f"Ran `{joined_cmd}` to ensure docker daemon is running but it failed: {prc.stderr}"
+        )
+
 
 def validate_docker_env(project):
     if not project.name:


Code review comments:
diff_hunk: @@ -26,18 +26,23 @@
 
 def validate_docker_installation():
     """
-    Verify if Docker is installed on host machine.
+    Verify if Docker is installed and running on host machine.
     """
-    try:
-        docker_path = "docker"
-        process._exec_cmd([docker_path, "--help"], throw_on_error=False)
-    except EnvironmentError:
+    if shutil.which("docker") is None:
         raise ExecutionException(
             "Could not find Docker executable. "
             "Ensure Docker is installed as per the instructions "
             "at https://docs.docker.com/install/overview/."
         )
 
+    cmd = ["docker", "ps"]
+    prc = process._exec_cmd(cmd, throw_on_error=False)
+    if prc.returncode != 0:
+        joined_cmd = " ".join(cmd)
+        raise ExecutionException(
+            f"Ran `{joined_cmd}` to ensure docker daemon is running but it failed: {prc.stderr}"
path: mlflow/projects/docker.py
text: @user1:
Nit: I suggest to also log the "prc.stdout" which might be useful for showing what's wrong.

@author:
@user1 Done!