Repository: iterative/dvc

Issue:
push: userless ssh url doesn't work
# Bug Report

## Description

When pushing to a ssh remote which url doesn't have a user (the user is set in `~/.ssh/config`) I get the following error:
```
ERROR: unexpected error - No existing session
```

When checking with `dvc push --verbose` I get the following debug message:

```
DEBUG: Establishing ssh connection with 'myhost.example' through port '22' as user 'None'
```

### Reproduce

1. `dvc init`
2. Copy `dataset.zip` to the directory
3. `dvc add dataset.zip`
4. `dvc remote add --default ssh-remote ssh://myhost.example/path/to/folder`
5. Configure `~/.ssh/config` with User and IdentityFile for Host `myhost`

```
Host myhost.example
    HostName myhost.example
    User myuser
    IdentityFile /path/to/my/private/key
```

6. `dvc push`

### Expected

My data is pushed to the ssh remote

### Environment information

* My computer username is different from the one used with ssh remote

**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 2.4.1 (rpm)
---------------------------------
Platform: Python 3.8.10 on Linux-5.12.12-arch1-1-x86_64-with-glibc2.7
Supports: azure, gdrive, gs, webhdfs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Repo: dvc, git
```

**Additional Information (if any):**

```console
$ dvc push --verbose
2021-06-25 15:16:19,871 DEBUG: Check for update is enabled.
2021-06-25 15:16:20,289 DEBUG: Preparing to upload data to 'ssh://myhost.example/path/to/folder'
2021-06-25 15:16:20,289 DEBUG: Preparing to collect status from ssh://myhost.example/path/to/folder
2021-06-25 15:16:20,289 DEBUG: Collecting information from local cache...
2021-06-25 15:16:20,290 DEBUG: Collecting information from remote cache...                          
2021-06-25 15:16:20,291 DEBUG: Matched '0' indexed hashes
2021-06-25 15:16:20,293 DEBUG: Establishing ssh connection with 'myhost.example' through port '22' as user 'None'
2021-06-25 15:16:20,937 ERROR: unexpected error - No existing session                               
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/fs/pool.py", line 51, in get_connection
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc/main.py", line 55, in main
  File "dvc/command/base.py", line 50, in do_run
  File "dvc/command/data_sync.py", line 57, in run
  File "dvc/repo/__init__.py", line 51, in wrapper
  File "dvc/repo/push.py", line 44, in push
  File "dvc/data_cloud.py", line 79, in push
  File "dvc/remote/base.py", line 57, in wrapper
  File "dvc/remote/base.py", line 494, in push
  File "dvc/remote/base.py", line 351, in _process
  File "dvc/remote/base.py", line 195, in _status
  File "dvc/remote/base.py", line 145, in hashes_exist
  File "dvc/objects/db/ssh.py", line 73, in hashes_exist
  File "concurrent/futures/_base.py", line 619, in result_iterator
  File "concurrent/futures/_base.py", line 444, in result
  File "concurrent/futures/_base.py", line 389, in __get_result
  File "concurrent/futures/thread.py", line 57, in run
  File "dvc/objects/db/ssh.py", line 64, in exists_with_progress
  File "dvc/objects/db/ssh.py", line 31, in batch_exists
  File "contextlib.py", line 113, in __enter__
  File "dvc/fs/pool.py", line 11, in get_connection
  File "dvc/fs/pool.py", line 53, in get_connection
  File "dvc/fs/ssh/connection.py", line 59, in __init__
  File "paramiko/client.py", line 435, in connect
  File "paramiko/client.py", line 764, in _auth
  File "paramiko/client.py", line 740, in _auth
  File "paramiko/transport.py", line 1570, in auth_publickey
paramiko.ssh_exception.SSHException: No existing session
------------------------------------------------------------
2021-06-25 15:16:23,055 DEBUG: Version info for developers:
DVC version: 2.4.1 (rpm)
---------------------------------
Platform: Python 3.8.10 on Linux-5.12.12-arch1-1-x86_64-with-glibc2.7
Supports: azure, gdrive, gs, webhdfs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/mapper/VolGroup00-lvolhome
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-06-25 15:16:23,058 DEBUG: Analytics is enabled.
2021-06-25 15:16:23,297 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpr0kchyzn']'
2021-06-25 15:16:23,300 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpr0kchyzn']'

```


<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->

**Workaround:**

Overriding `url` in local config by adding the user (e.g. `ssh://myuser@myhost.example/path/to/folder`)

**Possible lead:**

One possible fix to the issue is:

```diff
diff --git a/dvc/fs/ssh/__init__.py b/dvc/fs/ssh/__init__.py
index 192e3ef3..622954f6 100644
--- a/dvc/fs/ssh/__init__.py
+++ b/dvc/fs/ssh/__init__.py
@@ -126,9 +126,9 @@ class SSHFileSystem(BaseFileSystem):  # pylint:disable=abstract-method
 
         return get_connection(
             SSHConnection,
-            path_info.host,
-            username=path_info.user,
-            port=path_info.port,
+            self.host,
+            username=self.user,
+            port=self.port,
             key_filename=self.keyfile,
             timeout=self.timeout,
             password=self.password,
```

However I don't know if it breaks other actions.



Discussion:
The SSH backend used in DVC (paramiko) won't read per-host User and IdentityFile from your OpenSSH (`~/.ssh`) config.

As you noted, the solution here is to just set the username when you configure your remote. You can either set it in the URL, or using in the `user` config field for the DVC remote (see the `Click for SSH` section in: https://dvc.org/doc/command-reference/remote/modify#available-parameters-per-storage-type)

We are in the process of switching SSH backends to asyncssh (https://github.com/iterative/dvc/issues/6064), which afaik is better about reading OpenSSH configs than paramiko, so it's possible that this use case will also be handled without needing to explicitly set your username after that switch is made.

But either way, for DVC SSH remotes you should just set the appropriate `user` in the DVC remote config.
@isidentical Btw, does asyncssh handle that? Just wondering.
@pmrowla I tried setting `user` with:
```console
$ dvc remote modify --local ssh-remote user myuser
```
But it has the same issue.
> The SSH backend used in DVC (paramiko) won't read per-host User and IdentityFile from your OpenSSH (`~/.ssh`) config.

DVC reads the OpenSSH config [here](https://github.com/iterative/dvc/blob/4e792ae61c5927ab2e5f6a6914d985d43aa705b4/dvc/fs/ssh/__init__.py#L50). And it works. The issue is that you don't use the username that you retrieved when creating the ssh connection [here](https://github.com/iterative/dvc/blob/4e792ae61c5927ab2e5f6a6914d985d43aa705b4/dvc/fs/ssh/__init__.py#L129-L131)
Looks like it's related to the changes in https://github.com/iterative/dvc/pull/6059

Previously, `fs.path_info` was populated using `self.user`, but that was lost when `path_info` was removed from fs/remote into the ODB classes
I'm also running into this issue. 

I verified in dvc/remote/base.py in `Remote.__init__` the `**config` does indeed contain the "user" field, but in dvc/fs/ssh/connection.py in `SSHConnection.__Init__`, the **kwargs does not contain "username". 

I'm not sure if the difference between "user" and "username" is important or not.

In `dvc/fs/ssh/__init__.py` would a solution being changing: 

```python
    def ssh(self, path_info):
        self.ensure_credentials()

        from .connection import SSHConnection

        return get_connection(
            SSHConnection,
            path_info.host,
            username=path_info.user,
            port=path_info.port,
            key_filename=self.keyfile,
            timeout=self.timeout,
            password=self.password,
            gss_auth=self.gss_auth,
            sock=self.sock,
            allow_agent=self.allow_agent,
        )
```

to 

```python
    def ssh(self, path_info):
        self.ensure_credentials()

        from .connection import SSHConnection

        return get_connection(
            SSHConnection,
            path_info.host,
            username=self.user,
            port=path_info.port,
            key_filename=self.keyfile,
            timeout=self.timeout,
            password=self.password,
            gss_auth=self.gss_auth,
            sock=self.sock,
            allow_agent=self.allow_agent,
        )
```

Basically    `username=path_info.user,` ->  `username=self.user,`. The self.user variable seems to have the correct information in it. 

@pmrowla @Erotemic Thanks for the research, guys! Indeed, it is a regression from #6059 :slightly_frowning_face: I'll send a fix shortly while we are waiting for https://github.com/iterative/sshfs , which will remove this code anyway in the future.

Resolving patch:
diff --git a/dvc/fs/ssh/__init__.py b/dvc/fs/ssh/__init__.py
--- a/dvc/fs/ssh/__init__.py
+++ b/dvc/fs/ssh/__init__.py
@@ -119,16 +119,16 @@ def ensure_credentials(self):
         if self.ask_password and self.password is None:
             self.password = ask_password(self.host, self.user, self.port)
 
-    def ssh(self, path_info):
+    def ssh(self):
         self.ensure_credentials()
 
         from .connection import SSHConnection
 
         return get_connection(
             SSHConnection,
-            path_info.host,
-            username=path_info.user,
-            port=path_info.port,
+            self.host,
+            username=self.user,
+            port=self.port,
             key_filename=self.keyfile,
             timeout=self.timeout,
             password=self.password,
@@ -141,7 +141,7 @@ def ssh(self, path_info):
     def open(self, path_info, mode="r", encoding=None, **kwargs):
         assert mode in {"r", "rt", "rb", "wb"}
 
-        with self.ssh(path_info) as ssh, closing(
+        with self.ssh() as ssh, closing(
             ssh.sftp.open(path_info.path, mode)
         ) as fd:
             if "b" in mode:
@@ -150,19 +150,19 @@ def open(self, path_info, mode="r", encoding=None, **kwargs):
                 yield io.TextIOWrapper(fd, encoding=encoding)
 
     def exists(self, path_info) -> bool:
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             return ssh.exists(path_info.path)
 
     def isdir(self, path_info):
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             return ssh.isdir(path_info.path)
 
     def isfile(self, path_info):
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             return ssh.isfile(path_info.path)
 
     def walk_files(self, path_info, **kwargs):
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             for fname in ssh.walk_files(path_info.path):
                 yield path_info.replace(path=fname)
 
@@ -170,32 +170,32 @@ def remove(self, path_info):
         if path_info.scheme != self.scheme:
             raise NotImplementedError
 
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             ssh.remove(path_info.path)
 
     def makedirs(self, path_info):
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             ssh.makedirs(path_info.path)
 
     def move(self, from_info, to_info):
         if from_info.scheme != self.scheme or to_info.scheme != self.scheme:
             raise NotImplementedError
 
-        with self.ssh(from_info) as ssh:
+        with self.ssh() as ssh:
             ssh.move(from_info.path, to_info.path)
 
     def copy(self, from_info, to_info):
         if not from_info.scheme == to_info.scheme == self.scheme:
             raise NotImplementedError
 
-        with self.ssh(from_info) as ssh:
+        with self.ssh() as ssh:
             ssh.atomic_copy(from_info.path, to_info.path)
 
     def symlink(self, from_info, to_info):
         if not from_info.scheme == to_info.scheme == self.scheme:
             raise NotImplementedError
 
-        with self.ssh(from_info) as ssh:
+        with self.ssh() as ssh:
             ssh.symlink(from_info.path, to_info.path)
 
     def hardlink(self, from_info, to_info):
@@ -205,7 +205,7 @@ def hardlink(self, from_info, to_info):
         # See dvc/remote/local/__init__.py - hardlink()
         if self.getsize(from_info) == 0:
 
-            with self.ssh(to_info) as ssh:
+            with self.ssh() as ssh:
                 ssh.sftp.open(to_info.path, "w").close()
 
             logger.debug(
@@ -215,18 +215,18 @@ def hardlink(self, from_info, to_info):
             )
             return
 
-        with self.ssh(from_info) as ssh:
+        with self.ssh() as ssh:
             ssh.hardlink(from_info.path, to_info.path)
 
     def reflink(self, from_info, to_info):
         if from_info.scheme != self.scheme or to_info.scheme != self.scheme:
             raise NotImplementedError
 
-        with self.ssh(from_info) as ssh:
+        with self.ssh() as ssh:
             ssh.reflink(from_info.path, to_info.path)
 
     def md5(self, path_info):
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             return HashInfo(
                 "md5",
                 ssh.md5(path_info.path),
@@ -234,7 +234,7 @@ def md5(self, path_info):
             )
 
     def info(self, path_info):
-        with self.ssh(path_info) as ssh:
+        with self.ssh() as ssh:
             return ssh.info(path_info.path)
 
     def _upload_fobj(self, fobj, to_info, **kwargs):
@@ -243,7 +243,7 @@ def _upload_fobj(self, fobj, to_info, **kwargs):
             shutil.copyfileobj(fobj, fdest)
 
     def _download(self, from_info, to_file, name=None, no_progress_bar=False):
-        with self.ssh(from_info) as ssh:
+        with self.ssh() as ssh:
             ssh.download(
                 from_info.path,
                 to_file,
@@ -254,7 +254,7 @@ def _download(self, from_info, to_file, name=None, no_progress_bar=False):
     def _upload(
         self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
     ):
-        with self.ssh(to_info) as ssh:
+        with self.ssh() as ssh:
             ssh.upload(
                 from_file,
                 to_info.path,
diff --git a/dvc/objects/db/ssh.py b/dvc/objects/db/ssh.py
--- a/dvc/objects/db/ssh.py
+++ b/dvc/objects/db/ssh.py
@@ -28,7 +28,7 @@ def _exists(chunk_and_channel):
                 callback(path)
             return ret
 
-        with self.fs.ssh(path_infos[0]) as ssh:
+        with self.fs.ssh() as ssh:
             channels = ssh.open_max_sftp_channels()
             max_workers = len(channels)
 
@@ -78,7 +78,7 @@ def _list_paths(self, prefix=None, progress_callback=None):
             root = posixpath.join(self.path_info.path, prefix[:2])
         else:
             root = self.path_info.path
-        with self.fs.ssh(self.path_info) as ssh:
+        with self.fs.ssh() as ssh:
             if prefix and not ssh.exists(root):
                 return
             # If we simply return an iterator then with above closes instantly


Test patch: