Package: universal-pathlib
Universal PathlibUniversal Pathlib is a Python library that extends thepathlib.PathAPI to support a variety of backend filesystems viafilesystem_spec.InstallationInstall the latest version ofuniversal_pathlibwith pip or conda. Please note
that while this will installfsspecas a dependency, for some filesystems, you
have to install additional packages. For example, to use S3, you need to installs3fs, or better depend onfsspec[s3]:PyPIpython-mpipinstalluniversal_pathlibcondacondainstall-cconda-forgeuniversal_pathlibAdding universal_pathlib to your projectBelow is apyproject.tomlbased example for addinguniversal_pathlibto your
project as a dependency if you want to use it withs3andhttpfilesystems:[project]name="myproject"requires-python=">=3.8"dependencies=["universal_pathlib>=0.2.0","fsspec[s3,http]",]Seefilesystem_spec/setup.pyfor an overview of the available
fsspec extras.Basic Usage# pip install universal_pathlib fsspec[s3]>>>fromupathimportUPath>>>>>>s3path=UPath("s3://test_bucket")/"example.txt">>>s3path.nameexample.txt>>>s3path.stemexample>>>s3path.suffix.txt>>>s3path.exists()True>>>s3path.read_text()'Hello World'For more examples, see theexample notebook here.Currently supported filesystems (and protocols)file:Local filesystemmemory:Ephemeral filesystem in RAMaz:,adl:,abfs:andabfss:Azure Storage(requiresadlfs)data:RFC 2397 style data URLs(requiresfsspec>=2023.12.2)github:GitHub repository filesystemhttp:andhttps:HTTP(S)-based filesystemhdfs:Hadoop distributed filesystemgs:andgcs:Google Cloud Storage(requiresgcsfs)s3:ands3a:AWS S3(requiress3fsto be installed)webdav,webdav+http:andwebdav+https:WebDAV-based filesystem on top of
HTTP(S)(requireswebdav4[fsspec])It is likely, that other fsspec-compatible filesystems are supported through the
default implementation. But because they are not tested in the universal_pathlib
test-suite, correct behavior is not guaranteed. If you encounter any issues with
a specific filesystem using the default implementation, please open an issue. We
are happy to add support for other filesystems via custom UPath implementations.
And of course, contributions for new filesystems are welcome!Class hierarchyThe class hierarchy forUPathimplementations and their relation to the stdlibpathlibclasses are visualized in the following diagram:flowchart TB
  subgraph s0[pathlib]
    A---> B
    A--> AP
    A--> AW

    B--> BP
    AP---> BP
    B--> BW
    AW---> BW
  end
  subgraph s1[upath]
    B ---> U
    U --> UP
    U --> UW
    BP --> UP
    BW --> UW
    U --> UL
    U --> US3
    U --> UH
    U -.-> UO
  end

  A(PurePath)
  AP(PurePosixPath)
  AW(PureWindowsPath)
  B(Path)
  BP(PosixPath)
  BW(WindowsPath)

  U(UPath)
  UP(PosixUPath)
  UW(WindowsUPath)
  UL(FilePath)
  US3(S3Path)
  UH(HttpPath)
  UO(...Path)

  classDef np fill:#f7f7f7,stroke:#2166ac,stroke-width:2px,color:#333
  classDef nu fill:#f7f7f7,stroke:#b2182b,stroke-width:2px,color:#333

  class A,AP,AW,B,BP,BW,UP,UW np
  class U,UL,US3,UH,UO nu

  style UO stroke-dasharray: 3 3

  style s0 fill:none,stroke:#07b,stroke-width:3px,stroke-dasharray:3,color:#07b
  style s1 fill:none,stroke:#d02,stroke-width:3px,stroke-dasharray:3,color:#d02When instantiatingUPaththe returned instance type is determined by the path,
or better said, the "protocol" that was provided to the constructor. TheUPathclass will return a registered implementation for the protocol, if available. If
no specialized implementation can be found but the protocol is available throughfsspec, it will return aUPathinstance and provide filesystem access with a
default implementation. Please note the default implementation can not guarantee
correct behavior for filesystems that are not tested in the test-suite.Local paths and url pathsIf a local path is providedUPathwill return aPosixUPathorWindowsUPathinstance. These two implementations are 100% compatible with thePosixPathandWindowsPathclasses of their specific Python version. They're tested against a
large subset of the CPython pathlib test-suite to ensure compatibility.If a local urlpath is provided, i.e. a "file://" or "local://" URI, the returned
instance type will be aFilePathinstance. This class is a subclass ofUPaththat provides file access viaLocalFileSystemfromfsspec. You can use it to
ensure that all your local file access is done throughfsspecas well.UPath public class APIThe public class interface ofUPathextendspathlib.Pathvia attributes that
simplify interaction withfilesystem_spec. Think of theUPathclass in terms
of the following code:frompathlibimportPathfromtypingimportAny,MappingfromfsspecimportAbstractFileSystemclassUPath(Path):# the real implementation is more complex, but this is the general idea@propertydefprotocol(self)->str:"""The fsspec protocol for the path."""@propertydefstorage_options(self)->Mapping[str,Any]:"""The fsspec storage options for the path."""@propertydefpath(self)->str:"""The path that a fsspec filesystem can use."""@propertydeffs(self)->AbstractFileSystem:"""The cached fsspec filesystem instance for the path."""These attributes are used to provide a public interface to move from theUPathinstance to more fsspec specific code:fromupathimportUPathfromfsspecimportfilesystemp=UPath("s3://bucket/file.txt",anon=True)fs=filesystem(p.protocol,**p.storage_options)# equivalent to p.fswithfs.open(p.path)asf:data=f.read()Advanced UsageIf you want to create your own UPath implementations, there are multiple ways to
customize your subclass behavior. Here are a few things to keep in mind when you
create your own UPath implementation:UPath's constructor,upath.registry, and subclassingWhen instantiatingUPath(...)theUPath.__new__()method determines the path
protocol and returns a registered implementation for the protocol, if available.
The registered implementations are mapped in theupath.registrymodule. When a
protocol is not registered,universal_pathlibchecks if the protocol is mapped
to anfsspecfilesystem. If so, it returns an instance ofUPathand provides
filesystem access through the default implementation. The protocol is determined
by either looking at the URI scheme of the first argument to the constructor, or
by using theprotocolkeyword argument:fromupathimportUPathfromupath.implementations.cloudimportS3Pathfromupath.implementations.memoryimportMemoryPathp0=UPath("s3://bucket/file.txt")assertp0.protocol=="s3"asserttype(p0)isS3Pathassertisinstance(p0,UPath)p1=UPath("/some/path/file.txt",protocol="memory")assertp1.protocol=="memory"asserttype(p1)isMemoryPathassertisinstance(p1,UPath)# the ftp filesystem current has no custom UPath implementation and is not# tested in the universal_pathlib test-suite. Therefore, the default UPath# implementation is returned, and a warning is emitted on instantiation.p2=UPath("ftp://ftp.ncbi.nih.gov/snp/archive")assertp2.protocol=="ftp"asserttype(p2)isUPathThis has some implications for custom UPath subclasses. We'll go through the two
main cases where you might want to create a custom UPath implementation:Case 1: Custom filesystem works with default UPath implementationLet's say you would like to add a new implementation of your "myproto" protocol.
You already built a custom AbstractFileSystem implementation for "myproto" which
you have registered throughfsspec.registry. In some cases it is possible that
the custom filesystem class already works withUPath's default implementation,
and you don't need to necessarily create a custom UPath implementation:importfsspec.registryfromfsspec.specimportAbstractFileSystemclassMyProtoFileSystem(AbstractFileSystem):protocol=("myproto",)...# your custom implementationfsspec.registry.register_implementation("myproto",MyProtoFileSystem)fromupathimportUPathp=UPath("myproto:///my/proto/path")asserttype(p)isUPathassertp.protocol=="myproto"assertisinstance(p.fs,MyProtoFileSystem)Case 2: Custom filesystem requires a custom UPath implementationSometimes the default implementation isn't sufficient and some method(s) have to
be overridden to provide correct behavior. In this case, create a customUPathimplementation:fromupathimportUPathclassMyProtoPath(UPath):defmkdir(self,mode=0o777,parents=False,exist_ok=False):something={...:...}# fixes to make MyProtoFileSystem.mkdir workself.fs.mkdir(self.path,**something)defpath(self):path=super().pathifpath.startswith("/"):returnpath[1:]# MyProtoFileSystem needs the path without "/"returnpathIf you use your implementation directly viaMyProtoPath("myproto:///a/b"), you
can use this implementation already as is. If you want a call toUPath(...)to
return your custom implementation when the detected protocol is"myproto", you
need to register your implementation. The next section explains your options.Also note: In case you develop a customUPathimplementation, please feel free
to open an issue to discuss integrating it inuniversal_pathlib.Implementation registration dynamically from PythonYou can register your custom UPath implementation dynamically from Python:# for example: mymodule/submodule.pyfromupathimportUPathfromupath.registryimportregister_implementationclassMyProtoPath(UPath):...# your custom implementationregister_implementation("myproto",MyProtoPath)Implementation registration on installation via entry pointsIf you distribute your implementation in your own Python package, you can informuniversal_pathlibabout your implementation via theentry_pointsmechanism:# pyproject.toml
[project.entry-points."unversal_pathlib.implementations"]
myproto = "my_module.submodule:MyPath"# setup.cfg
[options.entry_points]
universal_pathlib.implementations =
    myproto = my_module.submodule:MyPathChose the method that fits your use-case best. If you have questions, open a new
issue in theuniversal_pathlibrepository. We are happy to help you!Customization options for UPath subclassesFilesystem access methodsOnce you thoroughly test your custom UPath implementation, it's likely that some
methods need to be overridden to provide correct behavior compared tostdlib'spathlib.Pathclass. The most common issue is that for certain edge cases, your
implementation is not raising the same exceptions compared to thepathlib.Pathclass. Or that theUPath.pathproperty needs some prefix removed or added.classMyProtoPath(UPath):@propertydefpath(self)->str:ifp:=self.path.startswith("/"):p=p[1:]returnpdefmkdir(self,mode=0o777,parents=False,exist_ok=False):ifsome_edge_case:raiseFileExistsError(str(self))super().mkdir(mode=mode,parents=parents,exist_ok=exist_ok)defis_file(self):returnself.fs.isfile(self.path,myproto_option=123)Storage option parsingIt's possible that you might want to extract additional storage options from the
user provided arguments to you constructor. You can provide a custom classmethod
for_parse_storage_options:importosclassMyProtoPath(UPath):@classmethoddef_parse_storage_options(cls,urlpath:str,protocol:str,storage_options:Mapping[str,Any])->dict[str,Any]:if"SOME_VAR"inos.environ:storage_options["some_var"]=os.environ["SOME_VAR"]storage_options["my_proto_caching"]=Truestorage_options["extra"]=get_setting_from_path(urlpath)returnstorage_optionsFsspec filesystem instantiationTo have more control over fsspec filesystem instantiation you can write a custom_fs_factoryclassmethod:classMyProtoPath(UPath):@classmethoddef_fs_factory(cls,urlpath:str,protocol:str,storage_options:Mapping[str,Any])->AbstractFileSystem:myfs=...# custom code that creates a AbstractFileSystem instancereturnmyfsInit argument parsingIn special cases you need to take more control over how the init args are parsed
for your custom subclass. You can override__init__or theUPathclassmethod_transform_init_args. The latter handles pickling of your custom subclass in a
better way in case you modify storage options or the protocol.classMyProtoPath(UPath):@classmethoddef_transform_init_args(cls,args:tuple[str|os.PathLike,...],protocol:str,storage_options:dict[str,Any],)->tuple[tuple[str|os.PathLike,...],str,dict[str,Any]]:# check the cloud, http or webdav implementations for examples...returnargs,protocol,storage_optionsStopping UPath's subclass dispatch mechanismThere are cases for which you want to disable the protocol dispatch mechanism of
theUPath.__new__constructor. For example if you want to extend the class API
of yourUPathimplementation, and use it as the base class for other, directly
instantiated subclasses. Together with other customization options this can be a
useful feature. Please be aware that in this case all protocols are handled with
the default implementation in UPath. Please always feel free to open an issue in
the issue tracker to discuss your use case. We're happy to help with finding the
most maintainable solution.classExtraUPath(UPath):_protocol_dispatch=False# disable the registry return an ExtraUPathdefsome_extra_method(self)->str:return"hello world"assertExtraUPath("s3://bucket/file.txt").some_extra_method()=="hello world"Migration GuideUPath's internal implementation is likely going to change with larger changes in
CPython's stdlibpathliblanding in the next Python versions (3.13,3.14).
To reduce the problems for user code, when these changes are landing inUPath,
there have been some significant changes inv0.2.0. This migration guide tries
to help migrating code that extensively relies on private implementation details
of theUPathclass of versionsv0.1.xto the new and better supported public
interface ofv0.2.0migrating tov0.2.0_FSSpecAccessor subclasses with custom filesystem access methodsIf you implemented a custom accessor subclass, it is now recommended to override
the correspondingUPathmethods in your subclass directly:# OLD: v0.1.xfromupath.coreimportUPath,_FSSpecAccessorclassMyAccessor(_FSSpecAccessor):defexists(self,path,**kwargs):# custom codereturnpath.fs.exists(self._format_path(path),**kwargs)deftouch(self,path,**kwargs):# customreturnpath.fs.touch(self._format_path(path),**kwargs)classMyPath(UPath):_default_accessor=MyAccessor# NEW: v0.2.0+fromupathimportUPathclassMyPath(UPath):defexists(self,*,follow_symlinks=True):kwargs={}# custom codereturnself.fs.exists(self.path,**kwargs)deftouch(self,mode=0o666,exist_ok=True):kwargs={}# custom codeself.fs.touch(self.path,**kwargs)_FSSpecAccessor subclasses with custom__init__methodIf you implemented a custom__init__method for your accessor subclass usually
the intention is to customize how the fsspec filesystem instance is created. You
have two options to recreate this with the new implementation. Chose one or both
dependent on the level of control you need.# OLD: v0.1.ximportfsspecfromupath.coreimportUPath,_FSSpecAccessorclassMyAccessor(_FSSpecAccessor):def__init__(self,parsed_url:SplitResult|None,**kwargs:Any)->None:# custom codeprotocol=...storage_options=...self._fs=fsspec.filesystem(protocol,storage_options)classMyPath(UPath):_default_accessor=MyAccessor# NEW: v0.2.0+fromupathimportUPathclassMyPath(UPath):@classmethoddef_parse_storage_options(cls,urlpath:str,protocol:str,storage_options:Mapping[str,Any])->dict[str,Any]:# custom code to change storage_optionsstorage_options=...returnstorage_options@classmethoddef_fs_factory(cls,urlpath:str,protocol:str,storage_options:Mapping[str,Any])->AbstractFileSystem:# custom code to instantiate fsspec filesystemprotocol=...storage_options=...# note changes to storage_options here won't# show up in MyPath().storage_optionsreturnfsspec.filesystem(protocol,**storage_options)Access to._accessorThe_accessorattribute and the_FSSpecAccessorclass is deprecated. In case
you need direct access to the underlying filesystem, just accessUPath().fs.# OLD: v0.1.xfromupath.coreimportUPathclassMyPath(UPath):defmkdir(self,mode=0o777,parents=False,exist_ok=False):self._accessor.mkdir(...)# custom access to the underlying fs...# NEW: v0.2.0+fromupathimportUPathclassMyPath(UPath):defmkdir(self,mode=0o777,parents=False,exist_ok=False):self.fs.mkdir(...)Access to._path,._kwargs,._drv,._root,._partsIf you access one of the listed private attributes directly, move your code over
to the following public versions:deprecatedv0.2.0+UPath()._pathUPath().pathUPath()._kwargsUPath().storage_optionsUPath()._drvUPath().driveUPath()._rootUPath().rootUPath()._partsUPath().partsAccess to._urlBe aware that the._urlattribute will likely be deprecated onceUPath()has
support for uri fragments and uri query parameters through a public api. In case
you are interested in contributing this functionality, please open an issue!Calling_from_parts,_parse_args,_format_parsed_partsIf your code is currently calling any of the three above listed classmethods, it
relies on functionality based on the implementation ofpathlibin Python up to3.11. Inuniversal_pathlibwe vendor code that allows theUPath()class to
be based on the3.12implementation ofpathlib.Pathalone. Usually, usage of
those classmethods occurs when copying some code of the internal implementations
of methods of theUPath0.1.4classes.To reproduce custom_format_parsed_partsmethods inv0.2.0, try overridingUPath().pathand/orUPath().with_segments().Custom_from_partsand_parse_argsclassmethods can now be implemented via
the_transform_init_argsmethod or via more functionality in the new flavour
class. Please open an issue for discussion in case you have this use case.Custom_URIFlavourclassesThe_URIFlavourclass was removed fromuniversal_pathliband the new flavour
class for fsspec filesystem path operations now lives inupath._flavour. As of
now the internal FSSpecFlavour is experimental. In a future Python version, it's
likely that a flavour or flavour-like base class will become public, that allows
us to base our internal implementation on. Until then, if you find yourself in a
situation where a custom path flavour would solve your problem, please feel free
to open an issue for discussion. We're happy to find a maintainable solution.Using.parse_parts(),.casefold(),.join_parsed_parts()of._flavourThese methods of the._flavourattribute ofpathlib.Path()andUPath()are
specific topathlibof Python versions up to3.11.UPath()is now based on
the3.12implementation ofpathlib.Path. Please refer to the implementations
of theupath._flavoursubmodule to see how you could avoid using them.Known issues solvable by installing newer upstream dependenciesSome issues inUPath's behavior with specific fsspec filesystems are fixed via
installation of a newer version of its upstream dependencies. Below you can find
a list of known issues and their solutions. We attempt to keep this list updated
whenever we encounter more:UPath().glob():fsspecfixed glob behavior when handling**patterns infsspec>=2023.9.0GCSPath().mkdir():
a few mkdir quirks are solved by installinggcsfs>=2022.7.1fsspec.filesystem(WebdavPath().protocol)the webdav protocol was added to fsspec in versionfsspec>=2022.5.0stat.S_ISDIR(HTTPPath().stat().st_mode)requiresfsspec>=2024.2.0to correctly returnTruefor directoriesContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theMIT license,universal_pathlibis free and open source software.IssuesIf you encounter any problems, or if you create your own implementations and run
into limitations, pleasefile an issuewith a detailed description. We
are always happy to help with any problems you might encounter.