Repository: aai-institute/lakefs-spec

Issue:
fs.exists() on the root of a branch throws an exception
### What happened?

I have a LakeFS repository 'demo1', and a 'main' branch. I expect `lakefs://demo1/main` to represent the root of a filesystem corresponding to that branch. However an `exists("lakefs://demo1/main")`, with or without a trailing "/" throws an exception.

Example test:
```
from lakefs_spec import LakeFSFileSystem

def test_lakefs_exists():
    fs = LakeFSFileSystem( )
    assert fs.exists("lakefs://demo1/main/")  # throws OSError
```
../.venv/lib/python3.12/site-packages/lakefs_spec/spec.py:233: OSError

System details:
Python 3.12
pip freeze | grep -i 'lakefs\|fsspec'
fsspec==2024.6.1
lakefs==0.7.1
lakefs-sdk==1.37.0
lakefs-spec==0.10.0

### What did you expect to happen?

If the repo and ref exist, then the root of the filesystem at that ref should always exist. As a reference, the [S3FS implementation](https://github.com/fsspec/s3fs/blob/main/s3fs/core.py#L1056-L1057) always returns true for "" and "/" provided as the path being checked.

The current lakefs-spec implementation seems to defer the check to the backend as an object check. I'm not sure if that is actually required, instead of checking the repo + ref exist and always returning `true` as in the S3FS implementation.

Note that this behavior is also inconsistent with the behavior of `fs.ls()`, `fs.walk()`, etc.:
```
from lakefs_spec import LakeFSFileSystem

def test_lakefs_ls():
    fs = LakeFSFileSystem()

    path = "lakefs://demo1/main/"
    files = fs.ls(path)  # No exception
    assert len(files) > 0
```

### lakeFS-spec version

Version: 0.10.0

### lakeFS version

Version: 0.7.1

Discussion:
@AdrianoKF: Hi @ssbusc1,

Thanks for reporting the issue and providing a thorough description!

We have implemented a fix in #292, I will shortly release a new version of the package on PyPI (as [`0.11.0`](https://pypi.org/project/lakefs-spec/0.11.0/)).

`fs.exists("lakefs://repo/<ref>/")` will be supported and basically check if the specific ref (branch/commit) exists in the repository. Note that the trailing slash is required.

Feel free to ping me if you notice any other problems with the latest code!@ssbusc1: Thanks @AdrianoKF for the quick fix and support! I'm now able to move ahead with version 0.11.0 and will report back if I run into any other issues.

Solution patch:
diff --git a/.github/workflows/python.yaml b/.github/workflows/python.yaml
index 2377836a..24d5328e 100644
--- a/.github/workflows/python.yaml
+++ b/.github/workflows/python.yaml
@@ -17,8 +17,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
       - name: Install uv
         uses: astral-sh/setup-uv@v3
       - name: Install Python 3.10 interpreter
@@ -28,8 +26,13 @@ jobs:
       - name: Run pre-commit checks
         run: uvx pre-commit run --all-files --verbose --show-diff-on-failure
   test:
-    name: Test lakefs-spec on ubuntu-latest
-    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        python-version: ["3.10", "3.11", "3.12"]
+    name: Run tests on Python ${{ matrix.python-version }} on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
     services:
       lakefs:
         image: treeverse/lakefs:latest
@@ -44,12 +47,10 @@ jobs:
           LAKEFS_BLOCKSTORE_TYPE: "local"
     steps:
       - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
       - name: Install uv
         uses: astral-sh/setup-uv@v3
-      - name: Install Python 3.10 interpreter (oldest supported)
-        run: uv python install 3.10
+      - name: Install Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
       - name: Install the project
         run: uv sync --extra dev
       - name: Execute python tests
@@ -58,6 +59,8 @@ jobs:
         uses: codecov/codecov-action@v4
         env:
           CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+        with:
+          flags: ${{ matrix.python-version }}
   docs:
     name: Build documentation for lakefs-spec
     runs-on: ubuntu-latest
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index cf4a5a58..7cd353fb 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -42,7 +42,7 @@ To get started with development, you can follow these steps (requires an install
     To spin up a local lakeFS instance quickly for testing, you can use the Docker Compose file bundled with this repository:
 
     ```shell
-    docker-compose -f hack/docker-compose.yml up
+    docker-compose -f hack/compose.yml up
     ```
 
 ## Updating dependencies
diff --git a/codecov.yml b/codecov.yml
index 220ef821..f0398edb 100644
--- a/codecov.yml
+++ b/codecov.yml
@@ -4,3 +4,6 @@ coverage:
       default:
         # Allow slight decreases in coverage for PRs
         threshold: "1%"
+flag_management:
+  default_rules:
+    carryforward: true
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 1110bb05..52184daa 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -68,16 +68,16 @@ Or, if you want to try the latest pre-release version directly from GitHub:
 
 If you don't already have access to a lakeFS server, you can quickly start a local instance using Docker Compose. Before continuing, please make sure Docker is installed and running on your machine.
 
-The lakeFS quickstart deployment can be launched directly with a [configuration file](https://github.com/aai-institute/lakefs-spec/blob/main/hack/docker-compose.yml) provided in the lakeFS-spec repository:
+The lakeFS quickstart deployment can be launched directly with a [configuration file](https://github.com/aai-institute/lakefs-spec/blob/main/hack/compose.yml) provided in the lakeFS-spec repository:
 
 ```shell
-$ curl https://raw.githubusercontent.com/aai-institute/lakefs-spec/main/hack/docker-compose.yml | docker-compose -f - up
+$ curl https://raw.githubusercontent.com/aai-institute/lakefs-spec/main/hack/compose.yml | docker-compose -f - up
 ```
 
-If you do not have `curl` installed on your machine or would like to examine and/or customize the container configuration, you can also create a `docker-compose.yml` file locally and use it with `docker-compose up`:
+If you do not have `curl` installed on your machine or would like to examine and/or customize the container configuration, you can also create a `compose.yml` file locally and use it with `docker-compose up`:
 
-```yaml title="docker-compose.yml"
---8<-- "https://raw.githubusercontent.com/aai-institute/lakefs-spec/main/hack/docker-compose.yml:3:"
+```yaml title="compose.yml"
+--8<-- "https://raw.githubusercontent.com/aai-institute/lakefs-spec/main/hack/compose.yml:3:"
 ```
 
 In order to allow lakeFS-spec to automatically discover credentials to access this lakeFS instance, create a `.lakectl.yaml` in your home directory containing the credentials for the quickstart environment (you can also use `lakectl config` to create this file interactively if you have the `lakectl` tool installed on your machine):
diff --git a/hack/README.md b/hack/README.md
index d2345ba7..fba184f6 100644
--- a/hack/README.md
+++ b/hack/README.md
@@ -2,7 +2,7 @@
 
 This document contains information on the resources in this directory, and how they can be used in development and testing.
 
-## `docker-compose.yml` - local lakeFS quickstart instance
+## `compose.yml` - local lakeFS quickstart instance
 
 This Docker Compose file bootstraps a local [lakeFS quickstart instance](https://docs.lakefs.io/quickstart/launch.html).
 It does not come with an associated volume, so you can spin it up and down without creating dangling resources.
@@ -13,7 +13,7 @@ Requirements:
 To bootstrap, run the following command:
 
 ```shell
-docker compose -f hack/docker-compose.yml up
+docker compose -f hack/compose.yml up
 ```
 
 To stop the container again, exit with `Ctrl-c`.
diff --git a/hack/docker-compose.yml b/hack/compose.yml
similarity index 91%
rename from hack/docker-compose.yml
rename to hack/compose.yml
index a9e8ed4b..5fde4569 100644
--- a/hack/docker-compose.yml
+++ b/hack/compose.yml
@@ -1,10 +1,8 @@
 # This docker-compose file provides an ephemeral lakeFS quickstart instance for local development.
 
-version: "3"
-
 services:
   lakefs:
-    image: treeverse/lakefs:1.7.0
+    image: treeverse/lakefs:1.37.0
     ports:
       - 8000:8000
     environment:
diff --git a/hack/lakefs-s3-local.yml b/hack/lakefs-s3-local.yml
index 6ace58e6..f85cfebf 100644
--- a/hack/lakefs-s3-local.yml
+++ b/hack/lakefs-s3-local.yml
@@ -1,7 +1,4 @@
 # This docker-compose file provides an ephemeral lakeFS quickstart instance for local development.
-
-version: "3"
-
 services:
   seaweedfs:
     container_name: sandbox-s3
@@ -13,7 +10,7 @@ services:
       - ./config/s3.json:/etc/seaweedfs/s3.json
 
   lakefs:
-    image: treeverse/lakefs:1.7.0
+    image: treeverse/lakefs:1.37.0
     network_mode: host
     depends_on:
       - seaweedfs
diff --git a/src/lakefs_spec/spec.py b/src/lakefs_spec/spec.py
index d20b703e..8da521f6 100644
--- a/src/lakefs_spec/spec.py
+++ b/src/lakefs_spec/spec.py
@@ -198,6 +198,9 @@ def exists(self, path: str | os.PathLike[str], **kwargs: Any) -> bool:
 
         Input paths can either be files or directories.
 
+        If the path refers to the root of the repository, this method will return
+        ``True`` if the reference or branch exists.
+
         Parameters
         ----------
         path: str | os.PathLike[str]
@@ -219,6 +222,11 @@ def exists(self, path: str | os.PathLike[str], **kwargs: Any) -> bool:
         repository, ref, resource = parse(path)
         try:
             reference = lakefs.Reference(repository, ref, client=self.client)
+
+            # Repo root (i.e., empty resource) boils down to checking if the ref exists
+            if resource == "":
+                return reference.get_commit() is not None
+
             if reference.object(resource).exists():
                 return True
             # if it isn't an object, it might be a common prefix (i.e. "directory").
@@ -226,6 +234,8 @@ def exists(self, path: str | os.PathLike[str], **kwargs: Any) -> bool:
                 max_amount=1, prefix=resource.rstrip("/") + "/", delimiter="/"
             )
             return len(list(children)) > 0
+        except NotFoundException:
+            return False
         except ServerException as e:
             # in case of an error other than "not found", existence cannot be
             # decided, so raise the translated error.
diff --git a/src/lakefs_spec/util.py b/src/lakefs_spec/util.py
index 7480fba7..7e1cff3b 100644
--- a/src/lakefs_spec/util.py
+++ b/src/lakefs_spec/util.py
@@ -7,7 +7,7 @@
 import os
 import re
 import sys
-from collections.abc import Callable, Generator, Iterable
+from collections.abc import Callable, Generator, Iterable, Iterator
 from typing import Any, Protocol
 
 from lakefs_sdk import Pagination
@@ -50,20 +50,23 @@ def depaginate(
         kwargs["after"] = resp.pagination.next_offset
 
 
-def batched(iterable: Iterable, n: int) -> Iterable[tuple]:
-    pyversion = tuple(sys.version_info[:3])
+def _batched(iterable: Iterable, n: int) -> Iterator[tuple]:
+    # "roughly equivalent" block from
+    # https://docs.python.org/3/library/itertools.html#itertools.batched
+    if n < 1:
+        raise ValueError("n must be at least one")
+    iterator = iter(iterable)
+    while batch := tuple(itertools.islice(iterator, n)):
+        yield batch
+
+
+def batched(iterable: Iterable, n: int) -> Iterator[tuple]:
     # itertools.batched was added in Python 3.12.
-    if pyversion >= (3, 12):
+    if sys.version_info >= (3, 12):
         # TODO(nicholasjng): Remove once target Python version is 3.12
-        return itertools.batched(iterable, n)  # type: ignore
+        yield from itertools.batched(iterable, n)
     else:
-        # "roughly equivalent" block from
-        # https://docs.python.org/3/library/itertools.html#itertools.batched
-        if n < 1:
-            raise ValueError("n must be at least one")
-        iterator = iter(iterable)
-        while batch := tuple(itertools.islice(iterator, n)):
-            yield batch
+        yield from _batched(iterable, n)
 
 
 def md5_checksum(lpath: str | os.PathLike[str], blocksize: int = 2**22) -> str:


Test patch:
diff --git a/tests/test_exists.py b/tests/test_exists.py
index 75c67474..07456ff0 100644
--- a/tests/test_exists.py
+++ b/tests/test_exists.py
@@ -38,3 +38,23 @@ def test_exists_on_staged_file(
     # upload, verify existence.
     fs.put(lpath, rpath)
     assert fs.exists(rpath)
+
+
+def test_exists_repo_root(
+    fs: LakeFSFileSystem,
+    repository: Repository,
+) -> None:
+    """Test `fs.exists` on the repository root."""
+
+    # Existing repo and branch should return true
+    assert fs.exists(f"lakefs://{repository.id}/main/")
+
+    # Existing repo and commit should return true
+    head_ref = lakefs.Branch(repository.id, "main", client=fs.client).head
+    assert fs.exists(f"lakefs://{repository.id}/{head_ref.id}/")
+
+    # Nonexistent branch should return false
+    assert not fs.exists(f"lakefs://{repository.id}/nonexistent/")
+
+    # Nonexistent repo should return false
+    assert not fs.exists("lakefs://nonexistent/main/")
diff --git a/tests/test_util.py b/tests/test_util.py
new file mode 100644
index 00000000..979d3ec8
--- /dev/null
+++ b/tests/test_util.py
@@ -0,0 +1,28 @@
+import pytest
+
+from lakefs_spec.util import _batched
+
+
+def test_batched_empty_iterable():
+    result = list(_batched([], 3))
+    assert result == []
+
+
+def test_batched_single_batch():
+    result = list(_batched([1, 2, 3], 3))
+    assert result == [(1, 2, 3)]
+
+
+def test_batched_multiple_batches():
+    result = list(_batched([1, 2, 3, 4, 5, 6, 7], 3))
+    assert result == [(1, 2, 3), (4, 5, 6), (7,)]
+
+
+def test_batched_batch_size_greater_than_iterable():
+    result = list(_batched([1, 2], 5))
+    assert result == [(1, 2)]
+
+
+def test_batched_invalid_batch_size():
+    with pytest.raises(ValueError, match="n must be at least one"):
+        list(_batched([1, 2, 3], 0))