Package: lakefs-spec
lakeFS-spec: An fsspec backend for lakeFSWelcome to lakeFS-spec, afilesystem-specbackend implementation for thelakeFSdata lake.
Our primary goal is to streamline versioned data operations in lakeFS, enabling seamless integration with popular data science tools such as Pandas, Polars, and DuckDB directly from Python.Highlights:Simple repository operations in lakeFSEasy access to underlying storage and versioning operationsSeamless integration with the fsspec ecosystemDirectly access lakeFS objects from popular data science libraries (including Pandas, Polars, DuckDB, PyArrow) with minimal codeTransaction support for reliable data version controlSmart data transfers through client-side caching (up-/download)Auto-discovery configuration[!NOTE]
We are seeking early adopters who would like to actively participate in our feedback process and shape the future of the library.
If you are interested in using the library and want to get in touch with us, please reach out viaGithub Discussions.InstallationlakeFS-spec is published on PyPI, you can simply install it using your favorite package manager:$pipinstalllakefs-spec# or$poetryaddlakefs-specUsageThe following usage examples showcase two major ways of using lakeFS-spec: as a low-level filesystem abstraction, and through third-party (data science) libraries.For a more thorough overview of the features and use cases for lakeFS-spec, see theuser guideandtutorialssections in the documentation.Low-level: As a fsspec filesystemThe following example shows how to upload a file, create a commit, and read back the committed data using the bare lakeFS filesystem implementation.
It assumes you have already created a repository namedrepoand havelakectlcredentials set up on your machine in~/.lakectl.yaml(see thelakeFS quickstart guideif you are new to lakeFS and need guidance).frompathlibimportPathfromlakefs_specimportLakeFSFileSystemREPO,BRANCH="repo","main"# Prepare example local datalocal_path=Path("demo.txt")local_path.write_text("Hello, lakeFS!")# Upload to lakeFS and create a commitfs=LakeFSFileSystem()# will auto-discover config from ~/.lakectl.yaml# Upload a file on a temporary transaction branchwithfs.transaction(repository=REPO,base_branch=BRANCH)astx:fs.put(local_path,f"{REPO}/{tx.branch.id}/{local_path.name}")tx.commit(message="Add demo data")# Read back committed filef=fs.open(f"{REPO}/{BRANCH}/demo.txt","rt")print(f.readline())# "Hello, lakeFS!"High-level: Via third-party librariesA variety of widely-used data science tools are building on fsspec to access remote storage resources and can thus work with lakeFS data lakes directly through lakeFS-spec (see thefsspec docsfor details).
The examples assume you have a lakeFS instance with thequickstartrepositorycontaining sample data available.# Pandas -- see https://pandas.pydata.org/docs/user_guide/io.html#reading-writing-remote-filesimportpandasaspddata=pd.read_parquet("lakefs://quickstart/main/lakes.parquet")print(data.head())# Polars -- see https://pola-rs.github.io/polars/user-guide/io/cloud-storage/importpolarsaspldata=pl.read_parquet("lakefs://quickstart/main/lakes.parquet",use_pyarrow=True)print(data.head())# DuckDB -- see https://duckdb.org/docs/guides/python/filesystems.htmlimportduckdbimportfsspecduckdb.register_filesystem(fsspec.filesystem("lakefs"))res=duckdb.read_parquet("lakefs://quickstart/main/lakes.parquet")res.show()ContributingWe encourage and welcome contributions from the community to enhance the project.
Please checkdiscussionsor raise anissueon GitHub for any problems you encounter with the library.For information on the general development workflow, see thecontribution guide.LicenseThe lakeFS-spec library is distributed under theApache-2 license.