Package: pathlibfs
pathlibfspathlib 🤝 fsspecLike pathlib, Python standard library module, manipulate remote filesystem paths.Installationpip install pathlibfsGetting StartedIt only providePathclass:frompathlibfsimportPathp=Path('your/path.txt')p_s3=Path('s3://bucket/key.txt')p_gcs=Path('gs://bucket/key.txt')pathlibfs usesfsspecas a backend filesystem.
So if you want to use some specific remote filesystem, you need to install extra dependencies such ass3fsorgcsfs.Seeknown implementationsto check out supported protocols.Special Environment VariablesnamedescriptionPATHLIBFS_S3_SESSION_CACHEIf defined, store S3 session cache locally likeboto3-session-cacheAPIsPathclass has many methods, and it's almost same aspathlibandfsspec.AbstractFileSystem.For example, like pathlib, you can join path with/:p = Path('some/dir') / 'subdir'  # -> Path('some/dir/subdir')propertiesGotcha.namedescriptionfsbackend filesystem of fsspecpathpath without protocol. e.g. Path('s3://bucket/key') == 'bucket/key'fullpathpath with protocol. e.g. Path('a.txt') == 'file://a.txt'urlpathpath with full chain. e.g. Path('simplecache::s3://bucket/key') == 'simplecache::s3://bucket/key'drivesame as pathlibrootsame as pathlibpartssame as pathlibanchorsame as pathlibparentssame as pathlib, return List[Path]parentsame as pathlib, return Pathhas_parentsame aspath.parent != pathnamesame as pathlibsuffixsame as pathlibsuffixessame as pathlibstemsame as pathlibsepseparator of backend filesystem, such as '/'protocolbackend protocol. e.g. 's3', 'gcs'pathlib based operationsnamedescriptionas_posix()only for local filesystem, otherwise raise anPathlibfsExceptionchmod(mode: int)only for local filesystem, otherwise raise anPathlibfsExceptiongroup()only for local filesystem, otherwise raise anPathlibfsExceptionis_mount()only for local filesystem, otherwise raise anPathlibfsExceptionis_symlink()only for local filesystem, otherwise raise anPathlibfsExceptionis_socket()only for local filesystem, otherwise raise anPathlibfsExceptionis_fifo()only for local filesystem, otherwise raise anPathlibfsExceptionis_block_device()only for local filesystem, otherwise raise anPathlibfsExceptionis_char_device()only for local filesystem, otherwise raise anPathlibfsExceptionowner()only for local filesystem, otherwise raise anPathlibfsExceptionsymlink_to(target: PathLike)only for local filesystem, otherwise raise anPathlibfsExceptionresolve()only for local filesystem, otherwise returnselfis_absolute()only for local filesystem, otherwise return Trueis_reserved()only for local filesystem, otherwise return Falsejoinpath(*p)same as pathlibmatch(pattern: str)same as pathlibwith_name(name: str)same as pathlibwith_suffix(suffix: str)same as pathlibread_bytes()same as pathlibread_text()same as pathlibwrite_text(data)same as pathlibwrite_bytes(data)same as pathlibis_dir()same as pathlibis_file()same as pathlibunlink()same as pathlibrelative_to(path: PathLike)same as pathlib, butreturn str.path == other / path.relative_to(other)iterdir(**kwargs)same as pathlib. It's just an wrapper of ls(),it's not efficient.Usels().stat()alias of info.It's not same as pathlib, but fsspec'sinfo()rglob(pattern, **kwargs)same meaning as pathlib, and accept fsspec'sglob()fsspec based mathodsnamedescriptionls(**kwargs)same as fsspeclistdir()alias of lsfind(maxdepth: Optional[int] = None, withdirs: bool = False, **kwargs)same as fsspecglob(pattern, **kwargs)same as fsspecexpand_path(recursive: bool = False, maxdepth: Optional[int] = None, **kwargs)same as fsspecwalk(maxdepth: Optional[int] = None, **kwargs)same asos.walkand fsspec'swalk, yield(dirpath, dirnames, filenames). AnddirpathisPathinstance.exists(**kwargs)same as fsspecisdir()same as fsspecisfile()same as fsspeclexists()same as fsspecukey()same as fsspecchecksum()same as fsspecsign(expiration: int = 100, **kwargs)same as fsspecsize()same as fsspeccreated()same as fsspecmodified()same as fsspecdu(total: bool = True, maxdepth: Optional[int] = None, **kwargs)same as fsspecdisk_usage()alias of duinfo(**kwargs)same as fsspecopen()same as fsspeccat(recursive: bool = False, on_error: str = "raise", **kwargs)same as fsspecread_block(offset: int, length: int, delimiter: Optional[bytes] = None)same as fsspechead(size: int = 1024)same as fsspectail(size: int = 1024)same as fsspecmkdir(parents: bool = False, exist_ok: bool = False, **kwargs)same as fsspecmakedir()alias of mkdirmakedirs(exist_ok: bool = False, **kwargs)same asmkdir(parents=True)mkdirs()alias of makedirs`touch(mode: int = 0o666, exist_ok: bool = True, truncate: bool = False, **kwargs)same as fsspecpipe_file(data)same as fsspecrm_file()same as fsspecrm(recursive: bool = False, maxdepth: Optional[int] = None)same as fsspecdelete()alias of rminvalidate_cache()same as fsspecclear_instance_cache()same as fsspeccopy(dst: PathLike, recursive: bool = False, on_error: Optional[str] = None, **kwargs)copy the path to dst.copy()can handle any protocol combinations so you don't need to callput()orget()for almost all cases.cp()alias of copymove(dst: PathLike, recursive=False, maxdepth=None, **kwargs)similar to copy, but delete source path after copy.mv(),rename(), `replace()alias of moveput(target: PathLike, recursive: bool = False, callback=fsspec.callbacks._DEFAULT_CALLBACK, **kwargs)Uploadlocaltarget to the path.upload()alias of putget(arget: PathLike, recursive: bool = False, callback=fsspec.callbacks._DEFAULT_CALLBACK, **kwargs)Downalod the path intolocaltarget.download()alias of getothersnamedescriptionislocal()True if protocol is local filesystem.clone(path: Optional[str] = None)copy self instance with different path (optional).samefile(target: PathLike)same asself == targetHow to testStart mock server for testing.docker-compose up -dRun test:pytest