Repository: mlflow/mlflow

Issue:
[BUG] Kubernetes Projects cannot push to private Docker repositories
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)
for information on what types of issues we address.

For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).
  
Please do not delete this template unless you are sure your issue is outside its scope.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: OSX 10.14
- **MLflow installed from (source or binary)**: PyPI
- **MLflow version (run ``mlflow --version``)**: 1.1.0
- **Python version**: 2.7
- **npm version, if running the dev UI**:
- **Exact command to reproduce**: See below

### Describe the problem
When executing an MLflow Project on Kubernetes, MLflow builds a docker image containing the project's contents and attempts to push it to a Docker repository specified by the backend configuration file (see https://mlflow.org/docs/latest/projects.html#execution-guide). When the Docker repository is a private repository, MLflow fails to push the Docker image. This failure occurs even if the user has authenticated with the Docker repository in their shell via `docker login` or provided access credentials in the `~/.docker/config.json` file.

### Code to reproduce issue
The following steps reproduce the issue using the [mlflow/examples/docker example](https://github.com/mlflow/mlflow/tree/master/examples/docker).

1. Clone the MLflow repository and check out `master`.

2. Specify a private Docker repository in the [mlflow/examples/docker/kubernetes_config.json file](https://github.com/mlflow/mlflow/blob/master/examples/docker/kubernetes_config.json). For example, I used the repository `dbczumar/mlflow-k8s-test`:
```
{
    "kube-context": "docker-for-desktop",
    "kube-job-template-path": "examples/docker/kubernetes_job_template.yaml",
    "repository-uri": "dbczumar/mlflow-k8s-test"
}
```

3. Authenticate with the private Docker repository in your shell (either via `docker login` or by providing credentials in your `~/.docker/config.json` file). Confirm that you can push images to this repository.

4. In the same shell, navigate to the root directory of your MLflow repository and run the following command:
```sh
$ mlflow run examples/docker --backend kubernetes --backend-config examples/docker/kubernetes_config.json -P alpha=0.5
```

5. Observe that the Docker image for the Project builds successfully, but the push process to the private repository fails with a 500-level error:
```
2019/07/26 16:38:23 INFO mlflow.projects: === Building docker image dbczumar/mlflow-k8s-test:96eb9e5 ===
2019/07/26 16:38:30 INFO mlflow.projects.kubernetes: === Pushing docker image dbczumar/mlflow-k8s-test:96eb9e5 ===
Traceback (most recent call last):
  File "/Users/czumar/anaconda2/bin/mlflow", line 11, in <module>
    load_entry_point('mlflow', 'console_scripts', 'mlflow')()
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/czumar/mlflow/mlflow/cli.py", line 137, in run
    run_id=run_id
  File "/Users/czumar/mlflow/mlflow/projects/__init__.py", line 265, in run
    use_conda=use_conda, storage_dir=storage_dir, synchronous=synchronous, run_id=run_id)
  File "/Users/czumar/mlflow/mlflow/projects/__init__.py", line 171, in _run
    image_digest = kb.push_image_to_registry(image.tags[0])
  File "/Users/czumar/mlflow/mlflow/projects/kubernetes.py", line 23, in push_image_to_registry
    return client.images.get_registry_data(image_tag).id
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/docker/models/images.py", line 333, in get_registry_data
    attrs=self.client.api.inspect_distribution(name),
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/docker/utils/decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/docker/api/image.py", line 266, in inspect_distribution
    self._get(self._url("/distribution/{0}/json", image)), True
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/docker/api/client.py", line 262, in _result
    self._raise_for_status(response)
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/docker/api/client.py", line 258, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/Users/czumar/anaconda2/lib/python2.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("errors:
denied: requested access to the resource is denied
unauthorized: authentication required
")
```


Discussion:
@marcusrehm Can you take a look at this?
Sure @dbczumar! I'll try to have a look at it this week ok?

For what I read in the issue you are trying to use it with docker hub registry, right?
@marcusrehm Thank you! Yes, I've tested project execution with private Docker repositories and ECR repositories - this issue occurs in both cases.
Hi @dbczumar , sorry for the late reply.

I've tested it downloading mlflow==1.1.0 from pypi and using the tag 1.1.0 from master branch. Both ways worked for me. I tried it with master branch and it worked too.

Can you try to clean up your environment and try again?


  
The only thing I see different in my tests is the OS config. I'm using Ubuntu 16.04 while you are using OSX 10.14.
@marcusrehm I'll try to reproduce this on a fresh Ubuntu installation and will respond back.
Hi @dbczumar , did you have time to try it again? Need some help?
@marcusrehm Sorry for the delay. I reproduced the issue on Ubuntu 16.04; it turns out to be caused by an outdated version of the Python `docker` library (version `3.7.2`). It turns out that the image upload process was successful with version `3.7.2`, but authorization failures were encountered when attempting to read metadata from the uploaded image (see this line: https://github.com/mlflow/mlflow/blob/master/mlflow/projects/kubernetes.py#L23). This seems to have been fixed when running with `docker>=4.0.0`. 

Accordingly, I'm going to submit a small PR requiring a minimum `docker` version of `4.0.0`; I will assign you as a reviewer. In the mean time, can you try reproducing the issue with `docker<=3.7.2` as a sanity check?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -45,11 +45,10 @@ def package_files(directory):
         'pyyaml',
         'querystring_parser',
         'simplejson',
-        'docker>=3.6.0',
+        'docker>=4.0.0',
         'entrypoints',
         'sqlparse',
         'sqlalchemy',
-        'docker>=3.6.0',
         'gorilla',
     ],
     extras_require={


Test patch: