Repository: fsspec/universal_pathlib
Pull request #248 (2024-08-05 11:38:37)

Update moto[s3,server] remove upper version bound
Updates the requirements on [moto[s3,server]](https://github.com/getmoto/moto) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/getmoto/moto/blob/master/CHANGELOG.md">moto[s3,server]'s changelog</a>.</em></p>
<blockquote>
<h2>5.0.12</h2>
<p>Docker Digest for 5.0.12: <em>sha256:e1bde8f908f2fdf0878c8e4398561badd016b51b4d9fd8dcb9f4daef936a4427</em></p>
<pre><code>General:
    * The MotoProxy can now be run on Windows
<p>New Services:
* DirectConnect:
* create_connection()
* delete_connection()
* describe_connections()
* update_connection()</p>
<pre><code>* DynamoDB:
    * describe_export()
    * export_table_to_point_in_time()
    * list_export()

* NetworkManager:
    * create_device()
    * create_link()
    * create_link()
    * delete_device()
    * delete_link()
    * delete_site()
    * get_devices()
    * get_links()
    * get_sites()
    * list_tags_for_resource()

* SageMaker:
    * list_endpoints()
    * list_endpoint_configs()
    * create_auto_ml_job_v2()
    * describe_auto_ml_job_v2()
    * list_auto_ml_jobs()
    * stop_auto_ml_job()
    * create_compilation_job()
    * describe_compilation_job()
    * list_compilation_jobs()
    * delete_compilation_job()
    * create_domain()
    * describe_domain()
    * list_domains()
    * delete_domain()
    * create_model_explainability_job_definition()
    * describe_model_explainability_job_definition()
    * list_model_explainability_job_definitions()
    * delete_model_explainability_job_definition()
</code></pre>
<p></code></pre></p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/getmoto/moto/commit/707b272d293bd0c3fdae4ae979de46187904aa0f"><code>707b272</code></a> Pre-Release: Up Version Number</li>
<li><a href="https://github.com/getmoto/moto/commit/434c7a1aab50ac4b0854bae9388a56b9ccaca15a"><code>434c7a1</code></a> Prep release 5.0.12 (<a href="https://redirect.github.com/getmoto/moto/issues/7929">#7929</a>)</li>
<li><a href="https://github.com/getmoto/moto/commit/0f25f2f6823e28c6aec8f089ee765b195a0ee394"><code>0f25f2f</code></a> CloudFormation: update_stack() - Improve UsePreviousValue behaviour (<a href="https://redirect.github.com/getmoto/moto/issues/7928">#7928</a>)</li>
<li><a href="https://github.com/getmoto/moto/commit/2ca9df6ab4688a8ca4fedd52a1c7a3e63d7a4be2"><code>2ca9df6</code></a> CognitoIDP: ID-token is different when username=email (<a href="https://redirect.github.com/getmoto/moto/issues/7927">#7927</a>)</li>
<li><a href="https://github.com/getmoto/moto/commit/fbec8b607b58097b9ddd3c62d78df361a7584b56"><code>fbec8b6</code></a> CognitoIDP: initiate_auth with USERNAME_PASSWORD_AUTH and SMS/Software Token ...</li>
<li><a href="https://github.com/getmoto/moto/commit/743044f9af5eae43f0b46324fb523cb4062d2954"><code>743044f</code></a> ACM: DomainValidationOptions should have SUCCESS-status (<a href="https://redirect.github.com/getmoto/moto/issues/7926">#7926</a>)</li>
<li><a href="https://github.com/getmoto/moto/commit/7cd27632c28aaa7fb32beac144931a4e51da1ac1"><code>7cd2763</code></a> Consolidate <code>Neptune</code> and <code>RDS</code> backends (<a href="https://redirect.github.com/getmoto/moto/issues/7925">#7925</a>)</li>
<li><a href="https://github.com/getmoto/moto/commit/210a2a3a6ea1d1b151cdaa49c67706b95c32a164"><code>210a2a3</code></a> Feature: Sagemaker Model Quality Job Definition (<a href="https://redirect.github.com/getmoto/moto/issues/7916">#7916</a>)</li>
<li><a href="https://github.com/getmoto/moto/commit/ec769529b89c41b784ef2947a4d2e37180a1bb13"><code>ec76952</code></a> chore: update SSM default AMI's (<a href="https://redirect.github.com/getmoto/moto/issues/7912">#7912</a>)</li>
<li><a href="https://github.com/getmoto/moto/commit/ac092748193c9f0f44185d1fc135ebe613e2ffb9"><code>ac09274</code></a> Feature: Sagemaker Hyper Parameter Tuning Job (<a href="https://redirect.github.com/getmoto/moto/issues/7917">#7917</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/getmoto/moto/compare/0.0.7...5.0.12">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

dependabot[bot]: The following labels could not be found: `maintenance`.

Patch:
diff --git a/.gitignore b/.gitignore
index e331257..059c618 100644
--- a/.gitignore
+++ b/.gitignore
@@ -139,3 +139,6 @@ cython_debug/
 
 # setuptools_scm
 upath/_version.py
+
+# vscode workspace settings
+.vscode/
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a438469..020fe14 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 ...
 
+## [0.2.3] - 2024-08-23
+### Added
+- upath: add st_birthtime as standard field (#254)
+- upath: added SMBPath and tests (#219)
+- ci: added typesafety checks (#212)
+
+### Fixed
+- upath: fix UPath.is_absolute on <3.12 (#256)
+- upath: fix UPath.rename for absolute paths (#225)
+- upath._flavour: fix path parsing due to change in urllib.parse.SplitResult behavior (#236)
+- upath: fixed typing regressions (#212)
+- upath: update flavour sources (#224, #237, #252)
+- docs: fix link to filesystem spec optional dependencies (#232)
+
 ## [0.2.2] - 2024-03-04
 ### Fixed
 - upath: fixed comparison with pathlib.Path on py<3.12 (#203)
@@ -126,7 +140,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 - started a changelog to keep track of significant changes
 
-[Unreleased]: https://github.com/fsspec/universal_pathlib/compare/v0.2.2...HEAD
+[Unreleased]: https://github.com/fsspec/universal_pathlib/compare/v0.2.3...HEAD
+[0.2.3]: https://github.com/fsspec/universal_pathlib/compare/v0.2.2...v0.2.3
 [0.2.2]: https://github.com/fsspec/universal_pathlib/compare/v0.2.1...v0.2.2
 [0.2.1]: https://github.com/fsspec/universal_pathlib/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/fsspec/universal_pathlib/compare/v0.1.4...v0.2.0
diff --git a/dev/requirements.txt b/dev/requirements.txt
index b854418..800c137 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -1,7 +1,7 @@
 fsspec[git,hdfs,dask,http,sftp,smb]==2024.6.1
 
 # these dependencies define their own filesystems
-adlfs==2024.4.1
+adlfs==2024.7.0
 boxfs==0.3.0
 dropboxdrivefs==1.4.1
 gcsfs==2024.6.1
@@ -10,9 +10,9 @@ ocifs==1.3.1
 webdav4[fsspec]==0.10.0
 # gfrivefs @ git+https://github.com/fsspec/gdrivefs@master broken ...
 morefs[asynclocalfs]==0.2.2
-dvc==3.51.2
-huggingface_hub==0.23.4
-lakefs-spec==0.9.0
+dvc==3.53.1
+huggingface_hub==0.23.5
+lakefs-spec==0.10.0
 ossfs==2023.12.0
 fsspec-xrootd==0.3.0
 wandbfs==0.0.2
diff --git a/noxfile.py b/noxfile.py
index 88b1cea..8f654a0 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -107,7 +107,8 @@ def typesafety(session):
     )
 
 
-@nox.session()
-def smoke(session):
-    print("please tun `nox -s tests` instead")
-    raise SystemExit(1)
+@nox.session(python="3.12")
+def generate_flavours(session):
+    session.install("-r", "dev/requirements.txt")
+    with open("upath/_flavour_sources.py", "w") as target:
+        session.run("python", "dev/generate_flavours.py", stdout=target)
diff --git a/setup.cfg b/setup.cfg
index 6d67430..0f8b335 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -43,7 +43,7 @@ dev =
     requests
     gcsfs
     s3fs
-    moto[s3,server]<5
+    moto[s3,server]
     webdav4[fsspec]
     wsgidav
     cheroot
diff --git a/upath/_flavour_sources.py b/upath/_flavour_sources.py
index d3ccf58..7523f1b 100644
--- a/upath/_flavour_sources.py
+++ b/upath/_flavour_sources.py
@@ -199,7 +199,7 @@ class AsyncLocalFileSystemFlavour(AbstractFileSystemFlavour):
 
 class AzureBlobFileSystemFlavour(AbstractFileSystemFlavour):
     __orig_class__ = 'adlfs.spec.AzureBlobFileSystem'
-    __orig_version__ = '2024.4.1'
+    __orig_version__ = '2024.7.0'
     protocol = ('abfs', 'az', 'abfss')
     root_marker = ''
     sep = '/'
@@ -272,7 +272,7 @@ class AzureBlobFileSystemFlavour(AbstractFileSystemFlavour):
 
 class AzureDatalakeFileSystemFlavour(AbstractFileSystemFlavour):
     __orig_class__ = 'adlfs.gen1.AzureDatalakeFileSystem'
-    __orig_version__ = '2024.4.1'
+    __orig_version__ = '2024.7.0'
     protocol = ('adl',)
     root_marker = ''
     sep = '/'
@@ -571,7 +571,7 @@ class HadoopFileSystemFlavour(AbstractFileSystemFlavour):
 
 class HfFileSystemFlavour(AbstractFileSystemFlavour):
     __orig_class__ = 'huggingface_hub.hf_file_system.HfFileSystem'
-    __orig_version__ = '0.23.4'
+    __orig_version__ = '0.23.5'
     protocol = ('hf',)
     root_marker = ''
     sep = '/'
@@ -587,7 +587,7 @@ class JupyterFileSystemFlavour(AbstractFileSystemFlavour):
 
 class LakeFSFileSystemFlavour(AbstractFileSystemFlavour):
     __orig_class__ = 'lakefs_spec.spec.LakeFSFileSystem'
-    __orig_version__ = '0.9.0'
+    __orig_version__ = '0.10.0'
     protocol = ('lakefs',)
     root_marker = ''
     sep = '/'
@@ -958,7 +958,7 @@ class ZipFileSystemFlavour(AbstractFileSystemFlavour):
 
 class _DVCFileSystemFlavour(AbstractFileSystemFlavour):
     __orig_class__ = 'dvc.fs.dvc._DVCFileSystem'
-    __orig_version__ = '3.51.2'
+    __orig_version__ = '3.53.1'
     protocol = ('dvc',)
     root_marker = '/'
     sep = '/'
diff --git a/upath/_stat.py b/upath/_stat.py
index f2cbece..5a4f0b1 100644
--- a/upath/_stat.py
+++ b/upath/_stat.py
@@ -281,6 +281,26 @@ class UPathStatResult:
                 pass
         return self._seq[9]
 
+    @property
+    def st_birthtime(self) -> int | float:
+        """time of creation"""
+        for key in [
+            "birthtime",
+            "created",
+            "creation_time",
+            "timeCreated",
+            "created_at",
+        ]:
+            try:
+                raw_value = self._info[key]
+            except KeyError:
+                continue
+            try:
+                return _convert_value_to_timestamp(raw_value)
+            except (TypeError, ValueError):
+                pass
+        raise AttributeError("birthtime")
+
     # --- extra fields ------------------------------------------------
 
     def __getattr__(self, item):
@@ -288,22 +308,6 @@ class UPathStatResult:
             return 0  # fallback default value
         raise AttributeError(item)
 
-    if "st_birthtime" in _fields_extra:
-
-        @property
-        def st_birthtime(self) -> int | float:
-            """time of creation"""
-            for key in ["created", "creation_time", "timeCreated", "created_at"]:
-                try:
-                    raw_value = self._info[key]
-                except KeyError:
-                    continue
-                try:
-                    return _convert_value_to_timestamp(raw_value)
-                except (TypeError, ValueError):
-                    pass
-            return 0
-
     # --- os.stat_result tuple interface ------------------------------
 
     def __len__(self) -> int:
diff --git a/upath/core.py b/upath/core.py
index b160eee..3e11dfa 100644
--- a/upath/core.py
+++ b/upath/core.py
@@ -91,6 +91,9 @@ def _make_instance(cls, args, kwargs):
     return cls(*args, **kwargs)
 
 
+_unset: Any = object()
+
+
 # accessors are deprecated
 _FSSpecAccessor = FSSpecAccessorShim
 
@@ -937,6 +940,9 @@ class UPath(PathlibPathShim, Path):
     def absolute(self) -> Self:
         return self
 
+    def is_absolute(self) -> bool:
+        return self._flavour.isabs(str(self))
+
     def resolve(self, strict: bool = False) -> Self:
         _parts = self.parts
 
@@ -1005,21 +1011,40 @@ class UPath(PathlibPathShim, Path):
     def rename(
         self,
         target: str | os.PathLike[str] | UPath,
-        *,
-        recursive: bool = False,
-        maxdepth: int | None = None,
+        *,  # note: non-standard compared to pathlib
+        recursive: bool = _unset,
+        maxdepth: int | None = _unset,
         **kwargs: Any,
-    ) -> UPath:  # fixme: non-standard
-        target_: UPath
-        if not isinstance(target, UPath):
-            target_ = self.parent.joinpath(target).resolve()
+    ) -> Self:
+        if isinstance(target, str) and self.storage_options:
+            target = UPath(target, **self.storage_options)
+        target_protocol = get_upath_protocol(target)
+        if target_protocol:
+            if target_protocol != self.protocol:
+                raise ValueError(
+                    f"expected protocol {self.protocol!r}, got: {target_protocol!r}"
+                )
+            if not isinstance(target, UPath):
+                target_ = UPath(target, **self.storage_options)
+            else:
+                target_ = target
+            # avoid calling .resolve for subclasses of UPath
+            if ".." in target_.parts or "." in target_.parts:
+                target_ = target_.resolve()
         else:
-            target_ = target
+            parent = self.parent
+            # avoid calling .resolve for subclasses of UPath
+            if ".." in parent.parts or "." in parent.parts:
+                parent = parent.resolve()
+            target_ = parent.joinpath(os.path.normpath(target))
+        assert isinstance(target_, type(self)), "identical protocols enforced above"
+        if recursive is not _unset:
+            kwargs["recursive"] = recursive
+        if maxdepth is not _unset:
+            kwargs["maxdepth"] = maxdepth
         self.fs.mv(
             self.path,
             target_.path,
-            recursive=recursive,
-            maxdepth=maxdepth,
             **kwargs,
         )
         return target_
diff --git a/upath/implementations/smb.py b/upath/implementations/smb.py
index c072165..f8505a2 100644
--- a/upath/implementations/smb.py
+++ b/upath/implementations/smb.py
@@ -1,9 +1,21 @@
+from __future__ import annotations
+
+import os
+import sys
 import warnings
+from typing import Any
+
+if sys.version_info >= (3, 11):
+    from typing import Self
+else:
+    from typing_extensions import Self
 
 import smbprotocol.exceptions
 
 from upath import UPath
 
+_unset: Any = object()
+
 
 class SMBPath(UPath):
     __slots__ = ()
@@ -29,24 +41,24 @@ class SMBPath(UPath):
         else:
             return super().iterdir()
 
-    def rename(self, target, **kwargs):
-        if "recursive" in kwargs:
+    def rename(
+        self,
+        target: str | os.PathLike[str] | UPath,
+        *,
+        recursive: bool = _unset,
+        maxdepth: int | None = _unset,
+        **kwargs: Any,
+    ) -> Self:
+        if recursive is not _unset:
             warnings.warn(
                 "SMBPath.rename(): recursive is currently ignored.",
                 UserWarning,
                 stacklevel=2,
             )
-        if "maxdepth" in kwargs:
+        if maxdepth is not _unset:
             warnings.warn(
                 "SMBPath.rename(): maxdepth is currently ignored.",
                 UserWarning,
                 stacklevel=2,
             )
-        if not isinstance(target, UPath):
-            target = self.parent.joinpath(target).resolve()
-        self.fs.mv(
-            self.path,
-            target.path,
-            **kwargs,
-        )
-        return target
+        return super().rename(target, **kwargs)


Test patch:
diff --git a/typesafety/test_upath_interface.yml b/typesafety/test_upath_interface.yml
index 219b3a4..116411f 100644
--- a/typesafety/test_upath_interface.yml
+++ b/typesafety/test_upath_interface.yml
@@ -565,3 +565,13 @@
       from upath import UPath
 
       reveal_type(UPath("abc").walk())  # N: Revealed type is "typing.Iterator[tuple[upath.core.UPath, builtins.list[builtins.str], builtins.list[builtins.str]]]"
+
+- case: upath_rename_extra_kwargs
+  disable_cache: false
+  main: |
+    from upath import UPath
+
+    UPath("abc").rename("efg")
+    UPath("recursive bool").rename("efg", recursive=True)
+    UPath("maxdepth int").rename("efg", maxdepth=1)
+    UPath("untyped extras").rename("efg", overwrite=True, something="else")
diff --git a/upath/tests/cases.py b/upath/tests/cases.py
index ae04ee6..d865d53 100644
--- a/upath/tests/cases.py
+++ b/upath/tests/cases.py
@@ -115,6 +115,9 @@ class BaseTests:
 
         assert not (self.path / "not-existing-file.txt").is_file()
 
+    def test_is_absolute(self):
+        assert self.path.is_absolute() is True
+
     def test_is_mount(self):
         assert self.path.is_mount() is False
 
@@ -282,6 +285,11 @@ class BaseTests:
         assert target == moved
         assert not upath.exists()
         assert moved.exists()
+        # reverse with an absolute path as str
+        back = moved.rename(str(upath))
+        assert back == upath
+        assert not moved.exists()
+        assert back.exists()
 
     def test_rename2(self):
         upath = self.path.joinpath("folder1/file2.txt")
@@ -291,6 +299,11 @@ class BaseTests:
         assert target_path == moved
         assert not upath.exists()
         assert moved.exists()
+        # reverse with a relative path as UPath
+        back = moved.rename(UPath("file2.txt"))
+        assert back == upath
+        assert not moved.exists()
+        assert back.exists()
 
     def test_replace(self):
         pass
diff --git a/upath/tests/conftest.py b/upath/tests/conftest.py
index 976623e..8a57299 100644
--- a/upath/tests/conftest.py
+++ b/upath/tests/conftest.py
@@ -140,7 +140,7 @@ def s3_server():
     port = 5555
     endpoint_uri = f"http://127.0.0.1:{port}/"
     proc = subprocess.Popen(
-        shlex.split(f"moto_server s3 -p {port}"),
+        shlex.split(f"moto_server -p {port}"),
         stderr=subprocess.DEVNULL,
         stdout=subprocess.DEVNULL,
     )
diff --git a/upath/tests/test_stat.py b/upath/tests/test_stat.py
index 66d9668..4922ae1 100644
--- a/upath/tests/test_stat.py
+++ b/upath/tests/test_stat.py
@@ -25,15 +25,45 @@ def test_stat_as_info(pth_file):
 
 
 def test_stat_atime(pth_file):
-    assert isinstance(pth_file.stat().st_atime, (float, int))
+    atime = pth_file.stat().st_atime
+    assert isinstance(atime, (float, int))
+
+
+@pytest.mark.xfail(reason="fsspec does not return 'atime'")
+def test_stat_atime_value(pth_file):
+    atime = pth_file.stat().st_atime
+    assert atime > 0
 
 
 def test_stat_mtime(pth_file):
-    assert isinstance(pth_file.stat().st_mtime, (float, int))
+    mtime = pth_file.stat().st_mtime
+    assert isinstance(mtime, (float, int))
+
+
+def test_stat_mtime_value(pth_file):
+    mtime = pth_file.stat().st_mtime
+    assert mtime > 0
 
 
 def test_stat_ctime(pth_file):
-    assert isinstance(pth_file.stat().st_ctime, (float, int))
+    ctime = pth_file.stat().st_ctime
+    assert isinstance(ctime, (float, int))
+
+
+@pytest.mark.xfail(reason="fsspec returns 'created' but not 'ctime'")
+def test_stat_ctime_value(pth_file):
+    ctime = pth_file.stat().st_ctime
+    assert ctime > 0
+
+
+def test_stat_birthtime(pth_file):
+    birthtime = pth_file.stat().st_birthtime
+    assert isinstance(birthtime, (float, int))
+
+
+def test_stat_birthtime_value(pth_file):
+    birthtime = pth_file.stat().st_birthtime
+    assert birthtime > 0
 
 
 def test_stat_seq_interface(pth_file):
diff --git a/upath/tests/utils.py b/upath/tests/utils.py
index 463ed0a..4158738 100644
--- a/upath/tests/utils.py
+++ b/upath/tests/utils.py
@@ -39,9 +39,11 @@ def xfail_if_version(module, *, reason, **conditions):
 def xfail_if_no_ssl_connection(func):
     try:
         import requests
-
+    except ImportError:
+        return pytest.mark.skip(reason="requests not installed")(func)
+    try:
         requests.get("https://example.com")
-    except (ImportError, requests.exceptions.SSLError):
+    except (requests.exceptions.ConnectionError, requests.exceptions.SSLError):
         return pytest.mark.xfail(reason="No SSL connection")(func)
     else:
         return func