Repository: iterative/dvc

Issue:
ssl_verify key in remote config does not accept custom CA bundle path and aws config is ignored
The `ssl_verify` key in the remote config gets passed through to the `S3FileSystem` `client_kwargs`:
https://github.com/iterative/dvc/blob/89b40afee740146af42efeb8563c08053f984a88/dvc/fs/s3.py#L105
https://github.com/iterative/dvc/blob/89b40afee740146af42efeb8563c08053f984a88/dvc/fs/fsspec_wrapper.py#L17
https://github.com/iterative/dvc/blob/89b40afee740146af42efeb8563c08053f984a88/dvc/fs/s3.py#L154

These are in turn passed to the `aiobotocore.AioSession` :
https://github.com/dask/s3fs/blob/a3d7a946f85b6dbef62ab75c61fe1319a482c8ba/s3fs/core.py#L366

In the AioSession it checks if the `verify` key is set and if it isn't then it looks in the aws config:
https://github.com/aio-libs/aiobotocore/blob/2a7c7f5a8c7a61daebe484bc5a6f2232607af82c/aiobotocore/session.py#L70-L71
`verify` can either be a boolean _or_ a string, with the latter being a path to a custom CA bundle:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html (see the `verify` argument of the `client` method.)

However, the config schema for DVC only allows boolean for `ssl_verify` and defaults true:
https://github.com/iterative/dvc/blob/89b40afee740146af42efeb8563c08053f984a88/dvc/config_schema.py#L148

The result is that the aws config is never checked and a custom CA bundle cannot be used. If such a CA bundle is needed when trying to communicate to remote (e.g. using push or pull) the result is
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
```

I ran into this problem because my company uses a self-hosted S3 clone with a bundle of internally signed certificates. Setting the `AWS_CA_BUNDLE` environment variable did not resolve the issue. But modifying the config schema to accept a string:
```
Optional("ssl_verify", default=True): Any(Bool, str),
```
and running
```
dvc remote modify object-store ssl_verify "$HOME/.aws/cabundle.pem"
```
resolved the issue for me.

I'm happy to open a pull request to make the change to the config schema if that solution is acceptable, but it would be my first contribution (for any OSS project!), so it'll take extra time for me to setup my environment, etc.


Discussion:
Looks like we have 2 (related) action points here

- `client["verify"] = config.get("ssl_verify", True) ` should default to passing None into boto instead of True if `ssl_verify` is not explicitly set for a DVC remote
- DVC's `ssl_verify` should also accept a ca_bundle string

(and finally the docs for `ssl_verify` will need to be updated)

@rgvanwesep if you'd like to work on this just let us know in this ticket. I'd say the first change is higher priority since it will make `aiobotocore` fall back to reading `ca_bundle` from the user's AWS config by default, but both can be done together.

pinging @isidentical to double check this
@pmrowla Good point on the first change. It would be good to be able to let boto fall back to the config.

I would like to work on this. I can make both changes at the same time.
related https://github.com/iterative/dvc/issues/5972

Resolving patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -145,7 +145,7 @@ class RelPath(str):
                     "session_token": str,
                     Optional("listobjects", default=False): Bool,  # obsoleted
                     Optional("use_ssl", default=True): Bool,
-                    Optional("ssl_verify", default=True): Bool,
+                    "ssl_verify": Any(Bool, str),
                     "sse": str,
                     "sse_kms_key_id": str,
                     "acl": str,
diff --git a/dvc/fs/s3.py b/dvc/fs/s3.py
--- a/dvc/fs/s3.py
+++ b/dvc/fs/s3.py
@@ -102,7 +102,7 @@ def _prepare_credentials(self, **config):
         client = login_info["client_kwargs"]
         client["region_name"] = config.get("region")
         client["endpoint_url"] = config.get("endpointurl")
-        client["verify"] = config.get("ssl_verify", True)
+        client["verify"] = config.get("ssl_verify")
 
         # encryptions
         additional = login_info["s3_additional_kwargs"]


Test patch:
diff --git a/tests/unit/fs/test_s3.py b/tests/unit/fs/test_s3.py
--- a/tests/unit/fs/test_s3.py
+++ b/tests/unit/fs/test_s3.py
@@ -39,7 +39,15 @@ def test_verify_ssl_default_param(dvc):
     }
     fs = S3FileSystem(**config)
 
-    assert fs.fs_args["client_kwargs"]["verify"]
+    assert "client_kwargs" not in fs.fs_args
+
+    config = {
+        "url": url,
+        "endpointurl": "https://my.custom.s3:1234",
+    }
+    fs = S3FileSystem(**config)
+
+    assert "verify" not in fs.fs_args["client_kwargs"]
 
 
 def test_s3_config_credentialpath(dvc, monkeypatch):
@@ -74,6 +82,29 @@ def test_ssl_verify_bool_param(dvc):
     assert fs.fs_args["client_kwargs"]["verify"] == config["ssl_verify"]
 
 
+def test_ssl_verify_path_param(dvc):
+    config = {"url": url, "ssl_verify": "/path/to/custom/cabundle.pem"}
+    fs = S3FileSystem(**config)
+
+    assert fs.fs_args["client_kwargs"]["verify"] == config["ssl_verify"]
+
+
+def test_ssl_verify_none_param(dvc):
+    config = {"url": url, "ssl_verify": None}
+    fs = S3FileSystem(**config)
+
+    assert "client_kwargs" not in fs.fs_args
+
+    config = {
+        "url": url,
+        "endpointurl": "https://my.custom.s3:1234",
+        "ssl_verify": None,
+    }
+    fs = S3FileSystem(**config)
+
+    assert "verify" not in fs.fs_args["client_kwargs"]
+
+
 def test_grants(dvc):
     config = {
         "url": url,
diff --git a/tests/unit/test_config.py b/tests/unit/test_config.py
--- a/tests/unit/test_config.py
+++ b/tests/unit/test_config.py
@@ -1,4 +1,5 @@
 import os
+import textwrap
 
 import pytest
 
@@ -33,3 +34,39 @@ def test_get_fs(tmp_dir, scm):
     assert config._get_fs("local") == config.wfs
     assert config._get_fs("global") == config.wfs
     assert config._get_fs("system") == config.wfs
+
+
+def test_s3_ssl_verify(tmp_dir, dvc):
+    config = Config(validate=False)
+    with config.edit() as conf:
+        conf["remote"]["remote-name"] = {"url": "s3://bucket/dvc"}
+
+    assert "ssl_verify" not in config["remote"]["remote-name"]
+
+    with config.edit() as conf:
+        section = conf["remote"]["remote-name"]
+        section["ssl_verify"] = False
+
+    assert (tmp_dir / ".dvc" / "config").read_text() == textwrap.dedent(
+        """\
+        [core]
+            no_scm = True
+        ['remote "remote-name"']
+            url = s3://bucket/dvc
+            ssl_verify = False
+        """
+    )
+
+    with config.edit() as conf:
+        section = conf["remote"]["remote-name"]
+        section["ssl_verify"] = "/path/to/custom/cabundle.pem"
+
+    assert (tmp_dir / ".dvc" / "config").read_text() == textwrap.dedent(
+        """\
+        [core]
+            no_scm = True
+        ['remote "remote-name"']
+            url = s3://bucket/dvc
+            ssl_verify = /path/to/custom/cabundle.pem
+        """
+    )