Repository: aai-institute/lakefs-spec

Issue:
Feature Request: Enable `@` and `~N` Syntax in `lakefs-spec` Paths
### What is the motivation and/or use case?

The [performance best practices documentation](https://docs.lakefs.io/understand/performance-best-practices.html#read-data-using-the-commit-id) highlights the advantages of accessing committed data using syntax like `lakefs://repo/main@/path`. Specifically:

> In cases where you are only interested in reading committed data:  
> Add `@` before the path `lakefs://repo/main@/path`.  
> When accessing data using the branch name (e.g., `lakefs://repo/main/path`), lakeFS will also try to fetch uncommitted data...

This feature works seamlessly with `lakectl`. However, `lakefs-spec` currently does not support the `@` or `~N` syntax in paths, resulting in the following error:

```python
ValueError: not a valid lakeFS URI: ...
```

### Desired Functionality

Extend `lakefs-spec` to support the `@` and `~N` syntax in paths, enabling users to directly specify:
1. **Exact commit reference:** Use `@` to specify a path as of a specific commit on a branch.
2. **Relative commit reference:** Use `~N` to access data from `N` commits prior to the latest commit on a branch.

For example:
```python
# Desired path syntax with `@` for a specific commit
file_path = "lakefs://example-repo/main@/data.parquet"

# Desired path syntax with `~N` for relative commits
file_path = "lakefs://example-repo/main~1/data.parquet"
```

### Benefits

1. **Improved Syntax Guarantee:** 
   The current approach involves extracting the commit ID programmatically:
   ```python
   prev_commit = repo.ref(f"{main.id}@").get_commit().id
   file_path = f"lakefs://example-repo/{prev_commit}/data.parquet"
   ```
   While this works, it introduces unnecessary complexity. Adding support for `@` or `~N` would streamline workflows and reduce boilerplate code.

2. **Consistency Across Tools:**
   Aligning `lakefs-spec` behavior with `lakectl` would create a unified and predictable user experience across lakeFS tools.

3. **Improved Usability:** 
   The syntactic sugar would significantly simplify code and make it easier for users to adopt and use lakeFS effectively.



### How can we implement this feature?

### Proposed Implementation

- Update the URI parsing logic in `lakefs-spec` to recognize and correctly handle:
  - The `@` syntax for exact commit references.
  - The `~N` syntax for relative commit references.

- Ensure backward compatibility with existing paths.

### Example Usage After Implementation

With the updated functionality, users could directly specify:
```python
file_path = "lakefs://example-repo/main@/data.parquet"  # Use specific commit
df = pd.read_parquet(file_path)

file_path = "lakefs://example-repo/main~1/data.parquet"  # Use N commits back
df = pd.read_parquet(file_path)
```

This would eliminate the need for the manual `repo.ref` and `prev_commit` extraction process.

Discussion:
@AdrianoKF: Thanks for your detailed report, @grepinsight! 🎉 

This indeed looks like a shortcoming in our URI validation, which should be addressed. I'll have a more detailed look today and get back to you soon.@AdrianoKF: I've pushed a fix in PR #316.

@grepinsight, can you please test if this indeed solves your issue? (`pip install git+https://github.com/aai-institute/lakefs-spec.git@314-commit-ref-syntax` should work).@gh-allee: @AdrianoKF This is awesome, and your fix works as expected

```python
import lakefs_spec
lakefs_spec.__version__
```
```
'0.11.2.dev9+g25ee73f'
```
```python
whats_on_stage = pd.read_parquet("lakefs://repo-name/main/test.parquet")
new_syntax_df = pd.read_parquet("lakefs://repo-name/main@/test.parquet")

print(f"whats_on_stage: {whats_on_stage.shape}")
print(f"new_syntax_df: {new_syntax_df.shape}")
```
```
whats_on_stage: (5, 14)
new_syntax_df: (4170933, 14)
```
@AdrianoKF: Awesome - thanks for your feedback, @grepinsight! I'll merge this PR and release v0.11.2 right away, so you can update your dependencies.

Solution patch:
diff --git a/src/lakefs_spec/util.py b/src/lakefs_spec/util.py
index 78288b1e..93396183 100644
--- a/src/lakefs_spec/util.py
+++ b/src/lakefs_spec/util.py
@@ -97,6 +97,14 @@ def md5_checksum(lpath: str | os.PathLike[str], blocksize: int = 2**22) -> str:
     return file_hash.hexdigest()
 
 
+_uri_parts = {
+    "protocol": r"^(?:lakefs://)?",  # leading lakefs:// protocol (optional)
+    "repository": r"(?P<repository>[a-z0-9][a-z0-9\-]{2,62})/",
+    "ref expression": r"(?P<ref>\w[\w\-.]*(([~\^]\d*)*|@)?)/",  # ref name with optional @, ~N, ^N suffixes
+    "resource": r"(?P<resource>.*)",
+}
+
+
 def parse(path: str) -> tuple[str, str, str]:
     """
     Parses a lakeFS URI in the form ``lakefs://<repo>/<ref>/<resource>``.
@@ -118,16 +126,9 @@ def parse(path: str) -> tuple[str, str, str]:
         If the path does not conform to the lakeFS URI format.
     """
 
-    uri_parts = {
-        "protocol": r"^(?:lakefs://)?",  # leading lakefs:// protocol (optional)
-        "repository": r"(?P<repository>[a-z0-9][a-z0-9\-]{2,62})/",
-        "ref expression": r"(?P<ref>\w[\w\-.^~]*)/",
-        "resource": r"(?P<resource>.*)",
-    }
-
     groups: dict[str, str] = {}
     start = 0
-    for group, regex in uri_parts.items():
+    for group, regex in _uri_parts.items():
         # we parse iteratively to improve the error message for the user if an invalid URI is given.
         # by going front to back and parsing each part successively, we obtain the current path segment,
         # and print it out to the user if it does not conform to our assumption of the lakeFS URI spec.


Test patch:
diff --git a/tests/regression/test_gh_314.py b/tests/regression/test_gh_314.py
new file mode 100644
index 00000000..9c035771
--- /dev/null
+++ b/tests/regression/test_gh_314.py
@@ -0,0 +1,31 @@
+from lakefs import Branch, Repository
+
+from lakefs_spec.spec import LakeFSFileSystem
+
+
+def test_gh_314(
+    fs: LakeFSFileSystem,
+    repository: Repository,
+    temp_branch: Branch,
+) -> None:
+    """
+    Regression test for GitHub issue 314: Enable `@` and `~N` syntax
+    https://github.com/aai-institute/lakefs-spec/issues/314
+    """
+
+    prefix = f"lakefs://{repository.id}/{temp_branch.id}"
+    datapath = f"{prefix}/data.txt"
+
+    # add new file, and immediately commit.
+    fs.pipe(datapath, b"data1")
+    temp_branch.commit(message="Add data.txt")
+
+    fs.pipe(datapath, b"data2")
+    # Reading the committed version of the file should yield the correct data.
+    committed_head_path = f"{prefix}@/data.txt"
+    assert fs.read_text(committed_head_path) == "data1"
+
+    # Reading a relative commit should yield the correct data.
+    temp_branch.commit(message="Update data.txt")
+    relative_commit_path = f"{prefix}~1/data.txt"
+    assert fs.read_text(relative_commit_path) == "data1"
diff --git a/tests/test_util.py b/tests/test_util.py
index 979d3ec8..61e292ba 100644
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -1,6 +1,8 @@
+import re
+
 import pytest
 
-from lakefs_spec.util import _batched
+from lakefs_spec.util import _batched, _uri_parts
 
 
 def test_batched_empty_iterable():
@@ -26,3 +28,44 @@ def test_batched_batch_size_greater_than_iterable():
 def test_batched_invalid_batch_size():
     with pytest.raises(ValueError, match="n must be at least one"):
         list(_batched([1, 2, 3], 0))
+
+
+class TestLakeFSUriPartRegexes:
+    @pytest.mark.parametrize(
+        "repo_name, valid",
+        [
+            ("my-repo", True),
+            ("@@repo", False),
+            ("", False),
+            ("a", False),
+            ("a" * 63, True),
+            ("a" * 64, False),
+        ],
+    )
+    def test_repository(self, repo_name: str, valid: bool) -> None:
+        result = re.match(_uri_parts["repository"], repo_name + "/")
+        if valid:
+            assert result is not None
+        else:
+            assert result is None
+
+    @pytest.mark.parametrize(
+        "refexp, valid",
+        [
+            ("", False),
+            ("main", True),
+            ("main@", True),
+            ("main~", True),
+            ("main^", True),
+            ("main^2", True),
+            ("main^^^", True),
+            ("main^1^1", True),
+            ("main^1~1", True),
+        ],
+    )
+    def test_ref_expression(self, refexp: str, valid: bool) -> None:
+        result = re.match(_uri_parts["ref expression"], refexp + "/")
+        if valid:
+            assert result is not None
+        else:
+            assert result is None