Repository: mlflow/mlflow

Issue:
[BUG] change in 1.12.0 causes S3 SSLError
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ x] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**:
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:  macOS 10.15.6
- **MLflow installed from (source or binary)**: binary
- **MLflow version (run ``mlflow --version``)**: 1.12.0
- **Python version**: 3.8.5
- **npm version, if running the dev UI**:
- **Exact command to reproduce**:
S3ArtifactRepository(my_uri).list_artifacts(my_path)

### Describe the problem
Describe the problem clearly here. Include descriptions of the expected behavior and the actual behavior.

As of this version I'm getting an error:
"certificate verify failed: self signed certificate in certificate chain." 

I'm working with private corporate endpoints. This worked fine before 1.12.0. I've isolated it to changes in mlflow.store.artifact.s3_artifact_repo, line 54. Simply adding verify=True explicitly seems to have caused this, even though this should be the default in boto3. 

### Code to reproduce issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Tricky 

### Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [x ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [ ] `area/server-infra`: MLflow server, JavaScript dev server
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
Hey @einarbmag, we have experienced the same. I don't know the exact cause (might be also related directly to boto3 or certifi) however, this on-liner fixes the issue:

```
cat $REQUESTS_CA_BUNDLE >> $(echo -e "import certifi\nprint(certifi.where())" | python)
```
I basically adds a copy of the certificates into your python environment.

Just tried it, works for me too. So that's maybe a bug in boto3 that it
uses certifi.where() instead of the config/env var specified certificates
when verify=True is supplied explicitly?

Its also possible to do verify=path/to/certificates, I believe. Could be a fix

On Fri, 15 Jan 2021, 10:28 sklingel, <notifications@github.com> wrote:

> Hey @einarbmag <https://github.com/einarbmag>, we have experienced the
> same. I don't know the exact cause (might be also related directly to boto3
> or certifi) however, this on-liner fixes the issue:
>
> cat $REQUESTS_CA_BUNDLE >> $(echo -e "import certifi\nprint(certifi.where())" | python)
>
> I basically adds a copy of the certificates into your python environment.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlflow/mlflow/issues/3986#issuecomment-760809976>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABG2GELLXKDJDVHIF4VR5N3S2AKFXANCNFSM4WDZKC3A>
> .
>

Same issue here. The problem is indeed that `verify=True` is not supported by boto3 and should be either `verify=None` to use default certificates, or `verify=False` to disable TLS verification. No need to supply `verify=path/to/certificates` since setting it to `None` would be the same (documentation here: https://github.com/boto/boto3/blob/develop/boto3/session.py#L212)
I could maybe do a PR for this! This would be the main change, at https://github.com/mlflow/mlflow/blob/master/mlflow/store/artifact/s3_artifact_repo.py#L45
```
        verify = None
        if ignore_tls and ignore_tls.lower() in ["true", "yes", "1"]:
            verify = False
```

Resolving patch:
diff --git a/mlflow/store/artifact/s3_artifact_repo.py b/mlflow/store/artifact/s3_artifact_repo.py
--- a/mlflow/store/artifact/s3_artifact_repo.py
+++ b/mlflow/store/artifact/s3_artifact_repo.py
@@ -42,9 +42,14 @@ def _get_s3_client(self):
         s3_endpoint_url = os.environ.get("MLFLOW_S3_ENDPOINT_URL")
         ignore_tls = os.environ.get("MLFLOW_S3_IGNORE_TLS")
 
-        verify = True
+        do_verify = True
         if ignore_tls:
-            verify = ignore_tls.lower() not in ["true", "yes", "1"]
+            do_verify = ignore_tls.lower() not in ["true", "yes", "1"]
+
+        # The valid verify argument value is None/False/path to cert bundle file, See
+        # https://github.com/boto/boto3/blob/73865126cad3938ca80a2f567a1c79cb248169a7/
+        # boto3/session.py#L212
+        verify = None if do_verify else False
 
         # NOTE: If you need to specify this env variable, please file an issue at
         # https://github.com/mlflow/mlflow/issues so we know your use-case!


Test patch:
diff --git a/tests/store/artifact/test_s3_artifact_repo.py b/tests/store/artifact/test_s3_artifact_repo.py
--- a/tests/store/artifact/test_s3_artifact_repo.py
+++ b/tests/store/artifact/test_s3_artifact_repo.py
@@ -10,6 +10,8 @@
 from tests.helper_functions import set_boto_credentials  # pylint: disable=unused-import
 from tests.helper_functions import mock_s3_bucket  # pylint: disable=unused-import
 
+from unittest import mock
+
 
 @pytest.fixture
 def s3_artifact_root(mock_s3_bucket):
@@ -53,6 +55,17 @@ def test_file_artifact_is_logged_with_content_metadata(s3_artifact_root, tmpdir)
     assert response.get("ContentEncoding") is None
 
 
+@pytest.mark.parametrize("ignore_tls_env, verify", [("", None), ("true", False), ("false", None)])
+def test_get_s3_client_verify_param_set_correctly(s3_artifact_root, ignore_tls_env, verify):
+    from unittest.mock import ANY
+
+    with mock.patch.dict("os.environ", {"MLFLOW_S3_IGNORE_TLS": ignore_tls_env}, clear=True):
+        with mock.patch("boto3.client") as mock_get_s3_client:
+            repo = get_artifact_repository(posixpath.join(s3_artifact_root, "some/path"))
+            repo._get_s3_client()
+            mock_get_s3_client.assert_called_with("s3", config=ANY, endpoint_url=ANY, verify=verify)
+
+
 def test_file_artifacts_are_logged_with_content_metadata_in_batch(s3_artifact_root, tmpdir):
     subdir_path = str(tmpdir.mkdir("subdir"))
     nested_path = os.path.join(subdir_path, "nested")