Repository: iterative/dvc

Issue:
 dvc push: unable to validate self-signed certificate from S3 service
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug. 

Example: `repro: doesn't detect input changes`
-->

## Description
Attempting dvc push to s3 remote which uses SSL termination. Getting certificate validation error since we are using self-signed certificates
<!--
A clear and concise description of what the bug is.
-->

### Reproduce

1. dvc init
2. modify remote to s3 endpoint with https scheme
3. copy certificate bundle from remote se to local directory
4. update AWS_CA_BUNDLE environment variable to point to local certificate bundle 
3. dvc add
4. dvc push


<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

### Expected
data sent to s3 remote

<!--
A clear and concise description of what you expect to happen.
-->

### Environment information
MacOs
Homebrew DVC installation

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**
DVC version: 2.0.18 (brew)
---------------------------------
Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git
```console
$ dvc doctor
```

**Additional Information (if any):**
I placed the output of dvc push -v below along with relevant environment variables
```
johnathan@Johnathans-MacBook-Pro repo % dvc push -v
2021-05-05 13:56:31,064 DEBUG: Check for update is enabled.
+------------------------------------------+
|                                          |
|     Update available 2.0.18 -> 2.1.0     |
|          Run `brew upgrade dvc`          |
|                                          |
+------------------------------------------+

2021-05-05 13:56:31,329 DEBUG: Preparing to upload data to 's3://internal-test-files'
2021-05-05 13:56:31,330 DEBUG: Preparing to collect status from s3://internal-test-files
2021-05-05 13:56:31,330 DEBUG: Collecting information from local cache...
2021-05-05 13:56:31,332 DEBUG: Collecting information from remote cache...                                                                                                            
2021-05-05 13:56:31,332 DEBUG: Querying 1 hashes via object_exists
2021-05-05 13:56:43,907 ERROR: unexpected error - SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/main.py", line 55, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/command/data_sync.py", line 56, in run
    processed_files_count = self.repo.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/push.py", line 41, in push
    return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/data_cloud.py", line 66, in push
    return remote.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 56, in wrapper
    return f(obj, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 466, in push
    ret = self._process(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 323, in _process
    dir_status, file_status, dir_contents = self._status(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 169, in _status
    remote_exists.update(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 242, in _indexed_dir_hashes
    dir_exists.update(self.odb.list_hashes_exists(dir_md5s - dir_exists))
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 338, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 329, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 233, in exists
    return self.isfile(path_info) or self.isdir(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 263, in isfile
    return path_info.path in self._list_paths(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 284, in _list_paths
    for obj_summary in obj_summaries:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 166, in pages
    for page in pages:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 341, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
2021-05-05 13:56:44,478 DEBUG: Version info for developers:
DVC version: 2.0.18 (brew)
---------------------------------
Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-05-05 13:56:44,479 DEBUG: Analytics is enabled.
2021-05-05 13:56:44,717 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'
2021-05-05 13:56:44,722 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'
```
```
johnathan@Johnathans-MacBook-Pro repo % printenv
USER=johnathan
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
...
TERM=xterm-256color
AWS_CA_BUNDLE=</path/to/bundle.crt>
REQUESTS_CA_BUNDLE=REQUESTS_CA_BUNDLE=</path/to/bundle.crt>
_=/usr/bin/printenv
johnathan@Johnathans-MacBook-Pro repo % cat cat /Users/johnathan/.bash_profile             
cat: cat: No such file or directory
export PATH="/usr/local/bin:$PATH"
export PATH="/usr/local/opt/python@3.7/bin:$PATH"
export GODEBUG="x509ignoreCN=0"
export AWS_CA_BUNDLE="</path/to/bundle.crt>"
export REQUESTS_CA_BUNDLE="</path/to/bundle.crt>"
export SSL_CERT_FILE="</path/to/bundle.crt>"
```

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->



Discussion:
@JoaoDeDynam can you try to install dvc from the master, `pip install "dvc[s3] @ git+https://github.com/iterative/dvc` and try again? We recently rewrote all our s3 code, so it would be nice if you could check that version out and let us know whether you still experience an odd behavior. 
after setting up a virtualenv and installing with pip and pip3, I run into
the same issues I was having on this issue
<https://github.com/iterative/dvc/issues/5900#issuecomment-829634366>

On Thu, May 6, 2021 at 4:14 AM Batuhan Taskaya ***@***.***>
wrote:

> @JoaoDeDynam <https://github.com/JoaoDeDynam> can you try to install dvc
> from the master, pip install "dvc[s3] @ git+
> https://github.com/iterative/dvc and try again? We recently rewrote all
> our s3 code, so it would be nice if you could check that version out and
> let us know whether you still experience an odd behavior.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/iterative/dvc/issues/5972#issuecomment-833441591>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AT4QSBM7RXQW53CDUP46PSDTMJ2ZFANCNFSM44F2BCPQ>
> .
>

-- 
This e-mail message, including any attachments, is for the sole use of the 
intended recipient(s) and may contain confidential and privileged 
information. Unauthorized review, use, disclosure or distribution is 
prohibited. If you are not the intended recipient, please contact the 
sender by reply e-mail and destroy all copies of the original message. 
Thank you.

There is an investigation of the issue:
https://github.com/boto/botocore/issues/2339#issuecomment-816068725

boto3 library reads AWS_CA_BUNDLE/REQUESTS_CA_BUNDLE environment variables to get a path of certificate bundle which is then used to verify SSL handshake.

It is also possible to pass a `verify` option explicitly into the boto3 client class.
If it is passed, `False` means disabling SSL check and some str means to use this value as a path to certificates bundle.
But `True` means not only to enable SSL check but also to completely ignore value of both AWS_CA_BUNDLE and REQUESTS_CA_BUNDLE environment variables.
They are being parsed only if `version` option value is `None` or if this option has not been passed into a client at all.

But dvc uses the same config value `ssl_verify` for both HTTP and S3 storage types, and the default value for this option is `True`.  

This unpexpected behavior also caused an error in MLflow project: https://github.com/mlflow/mlflow/pull/4047
I have a similar problem when i push my data to s3. 
Following message is the error message in my case. 

```sh
$ dvc push
ERROR: unexpected error - Cannot connect to host xxx.s3.amazonaws.com:443 ssl:default [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)]: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
dvc push  4.14s user 0.32s system 24% cpu 18.118 total
```

my dvc version was 2.3.0

```
$ dvc --version
2.3.0
```


```
[mike@mike-pc /run/media/mike/70ecef3b-5ec4-4400-8028-7955b30f3a46/bush-dataset][10:58:40] !+(master)[O][I][%][λ]> dvc doctor
DVC version: 2.3.0 (snap)
---------------------------------
Platform: Python 3.6.9 on Linux-5.10.41-1-MANJARO-x86_64-with-ManjaroLinux-21.0.6-Ornara
Supports: azure, gdrive, gs, http, https, s3, ssh
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git

```
This issue should be resolved in 2.3.0

In 2.3.0 `ssl_verify` for S3 remotes can be used to point directly to a custom CA bundle, the same way you would use `verify` with boto. If `ssl_verify` is unset in your remote config, DVC will fall back (via boto) to the CA bundle specified in your `~/.aws` config (previously leaving `ssl_verify` unset would force the default value to `True`)
@pmrowla thanks! After setup `use_ssl = false`  in dvc config. SSL error was not displayed !

```
[core]
    remote = main
['remote "main"']
    url = s3://xxx
    use_ssl = false

```
@mgoldchild just so you are aware, that will disable SSL (HTTPS) entirely. If you were looking to use SSL with a self-signed certificate, you should be using the `ssl_verify` config option, and not disabling `use_ssl`.
@pmrowla I experience the same issue with dvc 2.5.0. I setup a custom s3 storage using [minio](https://min.io) with a self-signed certificate. with setting `ssl_verify` to `false` I get past the first SSL-Error (probably for authentication?) but for each file I try to pull I still get this error. Disabling ssl at all with `use_ssl` does not help either.

The output from `dvc doctor` is:

```bash
➜ dvc doctor
DVC version: 2.5.0 (brew)
---------------------------------
Platform: Python 3.9.6 on macOS-11.4-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc, git
```
@justusschock, did you set the `endpoint_url`? Could you please try posting the error that you get ? Please try adding the `-v` flag to the command that you are using.
@skshetry Yes, I set the `endpoint_url` to my internal domain (replaced with MYURL below).

I get the following error:

```
2021-07-08 14:08:45,104 DEBUG: Uploading '.dvc/cache/d1/a1f686dc2f807b33a81bc6c02e5c57' to 's3://dvc/Dens/d1/a1f686dc2f807b33a81bc6c02e5c57'
2021-07-08 14:08:47,309 ERROR: failed to upload '.dvc/cache/ac/13f137d80741105dc543f08858c4ae' to 's3://dvc/Dens/ac/13f137d80741105dc543f08858c4ae' - SSL validation failed for https://MYURL:9000/dvc/Dens/ac/13f137d80741105dc543f08858c4ae [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 36, in wrapper
    func(from_info, to_info, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/base.py", line 233, in upload
    self._upload(  # noqa, pylint: disable=no-member
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 160, in wrapper
    raise translate_boto_error(exc)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 156, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 210, in _upload
    obj.upload_file(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/boto3/s3/inject.py", line 278, in object_upload_file
    return self.meta.client.upload_file(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/boto3/s3/inject.py", line 129, in upload_file
    return transfer.upload_file(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/usr/local/Cellar/dvc/2.5.0/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 341, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://MYURL:9000/dvc/Dens/ac/13f137d80741105dc543f08858c4ae [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)
```


My remote config looks like this:
```
['remote "truenas"']
    url = s3://dvc/Dens
    jobs = 8
    endpointurl = https://MYURL:9000/
    ssl_verify = false
    use_ssl = false
```
Looking at the code, we are using `s3` session during upload/download, which does not have information about `ssl_verify`, etc.

https://github.com/iterative/dvc/blob/87b49fd9f9829caab351a81560d2b90d791feae2/dvc/fs/s3.py#L209

cc @isidentical 

Resolving patch:
diff --git a/dvc/fs/s3.py b/dvc/fs/s3.py
--- a/dvc/fs/s3.py
+++ b/dvc/fs/s3.py
@@ -188,6 +188,7 @@ def s3(self):
             "s3",
             endpoint_url=client_kwargs.get("endpoint_url"),
             use_ssl=login_info["use_ssl"],
+            verify=client_kwargs.get("verify"),
         )
 
     def _get_obj(self, path_info):


Test patch: