Repository: fsspec/universal_pathlib (language: python)

Issue:
SSH / SFTP Implementations
The following script works differently between `0.2.2` and `0.0.24` and I'm trying to figure the cleanest way to account for it:

```py
import upath

ssh_path = upath.UPath("ssh://juftin@192.168.1.123:22/").resolve()
```

(yeah, I know SFTP/SSH isn't a current implementation - I'd be happy to help to implement it, it looks to work pretty seamlessly without any custom functions)

### `0.0.24`

```py
>>> ssh_path
UPath('ssh://juftin@192.168.1.123:22/)
>>> ssh_path.path
'/'
>>> str(ssh_path)
'ssh://juftin@192.168.1.123:22/'
>>> ssh_path.fs.ftp.normalize(ssh_path.path)
'/'
>>> upath.__version__
'0.0.24'
```

### `0.2.2`

```py
>>> ssh_path
UPath('ssh://.')
>>> ssh_path.path
'.'
>>> str(ssh_path)
'ssh://.'
>>> ssh_path.fs.ftp.normalize(ssh_path.path)
'/home/juftin'
>>> upath.__version__
'0.2.2'
```

The difference here is that `0.2.2` assumes you are starting from the user's home directory and every path is relative from there, this means paths like `ssh://juftin@192.168.1.123:22/var/local` are no longer working.

In my case the previous way the SFTP/SSH filesystem went to the root of the directory tree was more intuitive. I've been working on getting this to work for [browsr](https://github.com/juftin/browsr) / [textual-universal-directorytree](https://github.com/juftin/textual-universal-directorytree) , the below code works for me:

```py
class SFTPTextualPath(UPath):
    """
    SFTPTextualPath
    """

    @property
    def path(self) -> str:
        """
        Always return the path relative to the root
        """
        pth = super().path
        if pth.startswith("."):
            return f"/{pth[1:]}"
        elif pth.startswith("/"):
            return pth
        else:
            return "/" + pth

    def __str__(self) -> str:
        """
        Add the protocol prefix + extras to the string representation
        """
        string_representation = f"{self.protocol}://"
        if "username" in self.storage_options:
            string_representation += f"{self.storage_options['username']}@"
        string_representation += f"{self.storage_options['host']}"
        if "port" in self.storage_options:
            string_representation += f":{self.storage_options['port']}"
        string_representation += self.path
        return string_representation
```


Discussion / hints:
Hi @juftin,

thank you for reporting! This is a "bug" in the sense, that even though ssh filesystems are untested, it should be possible to avoid this for all untested filesystems where the netloc would be stripped or end up in the path anchor.

Correct behavior can be restored by setting a special internal setting in UPath's flavour implementation. Note that the following code fixes behavior for ssh:

```python
import upath

# ------- v0.2.2 specific
# Please don't use these two lines in your code:

from upath._flavour import WrappedFileSystemFlavour
WrappedFileSystemFlavour.protocol_config["netloc_is_anchor"].add("ssh")

# There's going to be a correct fix implemented in `universal_pathlib`
# -------

ssh_path = upath.UPath("ssh://juftin@192.168.1.123:22/var/local")

print(f"{ssh_path.parts=}")  # ssh_path.parts=('/', 'var', 'local')
print(f"{ssh_path.root=}")   # ssh_path.root='/'

```

For a general fix, I need to check if it's possible to infer the `netloc_is_anchor` setting from the return value of a filesystem's `_strip_protocol` method, if provided with a generic uri with non-empty netloc.

But anyways, it would of course be great to add an SSHPath implementation and thoroughly test all path functionality.

Cheers,
Andreas
Oh very neat. I saw the [SFTPFileSystemFlavour](https://github.com/fsspec/universal_pathlib/blob/380144c18f291f0f0a15fe8a02bc265233dd594b/upath/_flavour_sources.py#L732) - but wasn't sure what might be needed there. I'd be happy to get a SSHPath implementation going - that would need this upstream `netloc_is_anchor` inference feature first though, right?
> that would need this upstream netloc_is_anchor inference feature first though, right?

No, not really. You could add `"ssh"` and `"sftp"` here:
https://github.com/fsspec/universal_pathlib/blob/380144c18f291f0f0a15fe8a02bc265233dd594b/upath/_flavour.py#L106-L118

and make a SFTPPath / SSHPath implementation like the others. Most of the work will be to make a test fixture that sets up a local ssh server to test against. https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/tests/test_sftp.py this should be a good source for how to set this up.
Hi, just to say that we bumped into this problem when going for 0.2.0 to 0.2.2: `UPath("ssh://myserver.domain.org/data/mydata.nc").path` was producing `/data/mydata.nc` until 0.2.1, where it became `data/mydata.nc`.

I can also confirm that the workaround works for me:
```python
from upath._flavour import WrappedFileSystemFlavour
WrappedFileSystemFlavour.protocol_config["netloc_is_anchor"].add("ssh")
```
Ran into this issue also, the workaround works but would be nice to get SSH working more seamlessly.

Fix patch:
diff --git a/pyproject.toml b/pyproject.toml
index a9189e9..d6ee777 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -51,6 +51,7 @@ dev = [
     "s3fs",
     "moto[s3,server]",
     "webdav4[fsspec]",
+    "paramiko",
     "wsgidav",
     "cheroot",
     # "hadoop-test-cluster",
diff --git a/upath/_flavour.py b/upath/_flavour.py
index 08c2fff..128b828 100644
--- a/upath/_flavour.py
+++ b/upath/_flavour.py
@@ -109,6 +109,8 @@ class WrappedFileSystemFlavour:  # (pathlib_abc.FlavourBase)
             "https",
             "s3",
             "s3a",
+            "sftp",
+            "ssh",
             "smb",
             "gs",
             "gcs",
diff --git a/upath/implementations/sftp.py b/upath/implementations/sftp.py
new file mode 100644
index 0000000..ed9b447
--- /dev/null
+++ b/upath/implementations/sftp.py
@@ -0,0 +1,24 @@
+from __future__ import annotations
+
+import sys
+from typing import Any
+from typing import Generator
+
+if sys.version_info >= (3, 11):
+    from typing import Self
+else:
+    from typing_extensions import Self
+
+from upath import UPath
+
+_unset: Any = object()
+
+
+class SFTPPath(UPath):
+    __slots__ = ()
+
+    def iterdir(self) -> Generator[Self, None, None]:
+        if not self.is_dir():
+            raise NotADirectoryError(str(self))
+        else:
+            return super().iterdir()
diff --git a/upath/registry.py b/upath/registry.py
index c886e39..27e5d01 100644
--- a/upath/registry.py
+++ b/upath/registry.py
@@ -76,6 +76,8 @@ class _Registry(MutableMapping[str, "type[upath.UPath]"]):
         "memory": "upath.implementations.memory.MemoryPath",
         "s3": "upath.implementations.cloud.S3Path",
         "s3a": "upath.implementations.cloud.S3Path",
+        "sftp": "upath.implementations.sftp.SFTPPath",
+        "ssh": "upath.implementations.sftp.SFTPPath",
         "webdav": "upath.implementations.webdav.WebdavPath",
         "webdav+http": "upath.implementations.webdav.WebdavPath",
         "webdav+https": "upath.implementations.webdav.WebdavPath",


Test patch:
diff --git a/upath/tests/conftest.py b/upath/tests/conftest.py
index 8a57299..b0907d9 100644
--- a/upath/tests/conftest.py
+++ b/upath/tests/conftest.py
@@ -16,6 +16,7 @@
 from fsspec.registry import _registry
 from fsspec.registry import register_implementation
 from fsspec.utils import stringify_path
+from packaging.version import Version
 
 from .utils import posixify
 
@@ -464,3 +465,61 @@ def smb_fixture(local_testdir, smb_url, smb_container):
     smb.put(local_testdir, "/home/testdir", recursive=True)
     yield url
     smb.delete("/home/testdir", recursive=True)
+
+
+@pytest.fixture(scope="module")
+def ssh_container():
+    if shutil.which("docker") is None:
+        pytest.skip("docker not installed")
+
+    name = "fsspec_test_ssh"
+    stop_docker(name)
+    cmd = (
+        "docker run"
+        " -d"
+        f" --name {name}"
+        " -e USER_NAME=user"
+        " -e PASSWORD_ACCESS=true"
+        " -e USER_PASSWORD=pass"
+        " -p 2222:2222"
+        " linuxserver/openssh-server:latest"
+    )
+    try:
+        subprocess.run(shlex.split(cmd))
+        time.sleep(1)
+        yield {
+            "host": "localhost",
+            "port": 2222,
+            "username": "user",
+            "password": "pass",
+        }
+    finally:
+        stop_docker(name)
+
+
+@pytest.fixture
+def ssh_fixture(ssh_container, local_testdir, monkeypatch):
+    pytest.importorskip("paramiko", reason="sftp tests require paramiko")
+
+    cls = fsspec.get_filesystem_class("ssh")
+    if cls.put != fsspec.AbstractFileSystem.put:
+        monkeypatch.setattr(cls, "put", fsspec.AbstractFileSystem.put)
+    if Version(fsspec.__version__) < Version("2022.10.0"):
+        from fsspec.callbacks import _DEFAULT_CALLBACK
+
+        monkeypatch.setattr(_DEFAULT_CALLBACK, "relative_update", lambda *args: None)
+
+    fs = fsspec.filesystem(
+        "ssh",
+        host=ssh_container["host"],
+        port=ssh_container["port"],
+        username=ssh_container["username"],
+        password=ssh_container["password"],
+    )
+    fs.put(local_testdir, "/app/testdir", recursive=True)
+    try:
+        yield "ssh://{username}:{password}@{host}:{port}/app/testdir/".format(
+            **ssh_container
+        )
+    finally:
+        fs.delete("/app/testdir", recursive=True)
diff --git a/upath/tests/implementations/test_sftp.py b/upath/tests/implementations/test_sftp.py
new file mode 100644
index 0000000..7ef299f
--- /dev/null
+++ b/upath/tests/implementations/test_sftp.py
@@ -0,0 +1,40 @@
+import pytest
+
+from upath import UPath
+from upath.tests.cases import BaseTests
+from upath.tests.utils import skip_on_windows
+from upath.tests.utils import xfail_if_version
+
+_xfail_old_fsspec = xfail_if_version(
+    "fsspec",
+    lt="2022.7.0",
+    reason="fsspec<2022.7.0 sftp does not support create_parents",
+)
+
+
+@skip_on_windows
+class TestUPathSFTP(BaseTests):
+
+    @pytest.fixture(autouse=True)
+    def path(self, ssh_fixture):
+        self.path = UPath(ssh_fixture)
+
+    @_xfail_old_fsspec
+    def test_mkdir(self):
+        super().test_mkdir()
+
+    @_xfail_old_fsspec
+    def test_mkdir_exists_ok_true(self):
+        super().test_mkdir_exists_ok_true()
+
+    @_xfail_old_fsspec
+    def test_mkdir_exists_ok_false(self):
+        super().test_mkdir_exists_ok_false()
+
+    @_xfail_old_fsspec
+    def test_mkdir_parents_true_exists_ok_false(self):
+        super().test_mkdir_parents_true_exists_ok_false()
+
+    @_xfail_old_fsspec
+    def test_mkdir_parents_true_exists_ok_true(self):
+        super().test_mkdir_parents_true_exists_ok_true()
diff --git a/upath/tests/test_registry.py b/upath/tests/test_registry.py
index e7fa162..f0562de 100644
--- a/upath/tests/test_registry.py
+++ b/upath/tests/test_registry.py
@@ -22,7 +22,9 @@
     "memory",
     "s3",
     "s3a",
+    "sftp",
     "smb",
+    "ssh",
     "webdav",
     "webdav+http",
     "webdav+https",