Repository: LLNL/repligit
Pull request #13: Add async client
Author: login: cmelone
Labels: name: dependencies
description: Modifications to a Dependency File
Created: 2025-03-26T23:39:03Z | Merged: 2025-03-31T18:34:20Z

closes https://github.com/LLNL/repligit/issues/2

I added an `asyncio` module under `src/repligit`. It has the same API as
the sync client.

import example:

```
from repligit.asyncio import fetch_pack, ls_remote, send_pack
```

Full example located at `examples/git_mirror_async.py`.

I tried to share as much logic as possible between the sync and async
clients, but I had to write some handlers to deal with streaming and
decoding.

The sync client is mostly unchanged. I've confirmed the funcionality
works, but I'm open to feedback on how to improve the organization of
the code and if any optimizations can be made to performance.

Also:
- enforces a minimum of python 3.10
- sets aiohttp as an optional dependency (repligit can be installed with
  `pip install repligit[aiohttp]`)


Diff:
diff --git a/examples/git_mirror.py b/examples/git_mirror.py
index f710733..0092423 100644
--- a/examples/git_mirror.py
+++ b/examples/git_mirror.py
@@ -9,8 +9,18 @@ def main():
 
     target_ref = f"refs/heads/{branch_name}"
 
+    # Authentication credentials
+    # Note: Only provide credentials when authentication is required
+    # src_username = "<username>"  # Uncomment if source repo requires auth
+    # src_password = "<token>"     # Uncomment if source repo requires auth
+    dest_username = "<username>"  # For destination repo write access
+    dest_password = "<token>"  # For destination repo write access
+
+    # List references from source repository (without authentication)
     gh_refs = ls_remote(src_remote_url)
-    gl_refs = ls_remote(dest_remote_url)
+
+    # List references from destination repository (with authentication)
+    gl_refs = ls_remote(dest_remote_url, username=dest_username, password=dest_password)
 
     want_sha = gh_refs[target_ref]
     have_shas = gl_refs.values()
@@ -21,11 +31,16 @@ def main():
         print("Everything is up to date")
         return
 
-    packfile = fetch_pack(src_remote_url, want_sha, have_shas)
-
-    dest_username = "<username>"
-    dest_password = "<token>"
+    # Fetch the packfile from source repository
+    packfile = fetch_pack(
+        src_remote_url,
+        want_sha,
+        have_shas,
+        # username=src_username,  # Uncomment if source repo requires auth
+        # password=src_password,  # Uncomment if source repo requires auth
+    )
 
+    # Upload packfile to destination repository
     send_pack(
         dest_remote_url,
         target_ref,
diff --git a/examples/git_mirror_async.py b/examples/git_mirror_async.py
new file mode 100644
index 0000000..a714b76
--- /dev/null
+++ b/examples/git_mirror_async.py
@@ -0,0 +1,60 @@
+import asyncio
+
+from repligit.asyncio import fetch_pack, ls_remote, send_pack
+
+
+async def main():
+    src_remote_url = "https://github.com/spack/spack.git"
+    dest_remote_url = "https://gitlab.com/test-org/test-repo.git"
+
+    branch_name = "main"
+
+    target_ref = f"refs/heads/{branch_name}"
+
+    # Authentication credentials
+    # Note: Only provide credentials when authentication is required
+    # src_username = "<username>"  # Uncomment if source repo requires auth
+    # src_password = "<token>"     # Uncomment if source repo requires auth
+    dest_username = "<username>"  # For destination repo write access
+    dest_password = "<token>"  # For destination repo write access
+
+    # List references from source repository (without authentication)
+    gh_refs = await ls_remote(src_remote_url)
+
+    # List references from destination repository (with authentication)
+    gl_refs = await ls_remote(
+        dest_remote_url, username=dest_username, password=dest_password
+    )
+
+    want_sha = gh_refs[target_ref]
+    have_shas = gl_refs.values()
+
+    from_sha = gl_refs.get(target_ref) or ("0" * 40)
+
+    if want_sha in have_shas:
+        print("Everything is up to date")
+        return
+
+    # Fetch the packfile from source repository
+    packfile = await fetch_pack(
+        src_remote_url,
+        want_sha,
+        have_shas,
+        # username=src_username,  # Uncomment if source repo requires auth
+        # password=src_password,  # Uncomment if source repo requires auth
+    )
+
+    # Upload packfile to destination repository
+    await send_pack(
+        dest_remote_url,
+        target_ref,
+        from_sha,
+        want_sha,
+        packfile,
+        username=dest_username,
+        password=dest_password,
+    )
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
diff --git a/pyproject.toml b/pyproject.toml
index 75de627..65e87c1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -13,7 +13,7 @@ authors = [
 ]
 description = "A python library implementing the git http transfer protocol."
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 classifiers = [
     "Programming Language :: Python :: 3",
     "Operating System :: OS Independent",
@@ -24,6 +24,9 @@ license = "Apache-2.0 WITH LLVM-exception"
 Homepage = "https://github.com/llnl/repligit"
 Issues = "https://github.com/llnl/repligit/issues"
 
+[project.optional-dependencies]
+aiohttp = ["aiohttp"]
+
 [tool.pytest.ini_options]
 pythonpath = [
   "src"
diff --git a/spack.yaml b/spack.yaml
index 80d77ef..bae31c7 100644
--- a/spack.yaml
+++ b/spack.yaml
@@ -11,6 +11,7 @@ spack:
   - py-pytest
   - py-repligit
   - py-ruff
+  - py-aiohttp
   - python
   view: true
   concretizer:
diff --git a/src/repligit/__init__.py b/src/repligit/__init__.py
index f7bc199..3c771bd 100644
--- a/src/repligit/__init__.py
+++ b/src/repligit/__init__.py
@@ -1 +1,3 @@
-__all__ = ["repligit"]
+from repligit.client import fetch_pack, ls_remote, send_pack
+
+__all__ = ["ls_remote", "fetch_pack", "send_pack"]
diff --git a/src/repligit/asyncio/__init__.py b/src/repligit/asyncio/__init__.py
new file mode 100644
index 0000000..7697934
--- /dev/null
+++ b/src/repligit/asyncio/__init__.py
@@ -0,0 +1,8 @@
+try:
+    import aiohttp  # noqa
+except ModuleNotFoundError:
+    raise ModuleNotFoundError("aiohttp is required to use the async client") from None
+
+from repligit.asyncio.client import fetch_pack, ls_remote, send_pack
+
+__all__ = ["ls_remote", "fetch_pack", "send_pack"]
diff --git a/src/repligit/asyncio/client.py b/src/repligit/asyncio/client.py
new file mode 100644
index 0000000..0adbf3b
--- /dev/null
+++ b/src/repligit/asyncio/client.py
@@ -0,0 +1,91 @@
+from typing import List
+
+import aiohttp
+
+from repligit.asyncio.parse import decode_lines, iter_lines
+from repligit.parse import generate_fetch_pack_request, generate_send_pack_header
+
+
+async def ls_remote(url: str, username: str = None, password: str = None):
+    """Get commit hash of remote master branch, return SHA-1 hex string or
+    None if no remote commits.
+    """
+
+    url = f"{url}/info/refs?service=git-upload-pack"
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+    async with aiohttp.ClientSession(auth=auth) as session:
+        async with session.get(url, raise_for_status=True) as resp:
+            lines = decode_lines(iter_lines(resp, encoding="utf-8"))
+            service_line = await anext(lines)
+            assert service_line == "# service=git-upload-pack"
+
+            # `async for` inside `dict()` not supported so no dict comprehension
+            result = {}
+            async for line in lines:
+                if not line:
+                    continue
+                sha, ref = line.split()
+                result[ref] = sha
+            return result
+
+
+async def fetch_pack(
+    url: str, want_sha: str, have_shas: List[str], username=None, password=None
+):
+    """Download a packfile from a remote server."""
+    url = f"{url}/git-upload-pack"
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+
+    request = generate_fetch_pack_request(want_sha, have_shas)
+
+    async with aiohttp.ClientSession(auth=auth) as session:
+        async with session.post(
+            url,
+            headers={
+                "Content-type": "application/x-git-upload-pack-request",
+            },
+            data=request,
+            raise_for_status=True,
+            timeout=None,
+        ) as resp:
+            length_bytes = await resp.content.readexactly(4)
+            line_length = int(length_bytes, 16)
+
+            line = await resp.content.readexactly(line_length - 4)
+            if line[:3] == b"NAK" or line[:3] == b"ACK":
+                # this is a difference in API between sync and async
+                # has to be read within this context to be used in the caller
+                return await resp.content.read()
+            else:
+                return None
+
+
+async def send_pack(
+    url: str,
+    ref: str,
+    from_sha: str,
+    to_sha: str,
+    packfile,
+    username: str = None,
+    password: str = None,
+):
+    """Send a packfile to a remote server."""
+    url = f"{url}/git-receive-pack"
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+
+    header = generate_send_pack_header(ref, from_sha, to_sha)
+    # unlike in the sync version the packfile is already read into memory
+    receive_pack_request = header + packfile
+
+    async with aiohttp.ClientSession(auth=auth) as session:
+        async with session.post(
+            url,
+            headers={
+                "Content-type": "application/x-git-receive-pack-request",
+            },
+            data=receive_pack_request,
+            raise_for_status=True,
+        ) as resp:
+            lines = decode_lines(iter_lines(resp, encoding="utf-8"))
+            assert await anext(lines) == "unpack ok"
+            assert await anext(lines) == f"ok {ref}"
diff --git a/src/repligit/asyncio/parse.py b/src/repligit/asyncio/parse.py
new file mode 100644
index 0000000..4bd2f8c
--- /dev/null
+++ b/src/repligit/asyncio/parse.py
@@ -0,0 +1,38 @@
+# --------- asynchronous client helpers ---------
+from typing import AsyncIterable, AsyncIterator
+
+import aiohttp
+
+
+async def decode_lines(line_stream: AsyncIterable) -> AsyncIterator:
+    """(async) Decode server response iterator into usable lines."""
+    async for line in line_stream:
+        line_length = int(line[:4], 16)
+        yield line[4:line_length]
+
+
+async def iter_lines(
+    resp: aiohttp.ClientResponse, encoding: str, chunk_size: int = 512
+):
+    """
+    Stream lines from an HTTP response. Yields raw lines as bytes by default.
+
+    If `encoding` is set, yields decoded strings using the given encoding.
+
+    This is a reimplementation of requests.models.iter_lines for async,
+    the default `chunk_size` value is inherited from that definition.
+    """
+    buffer = bytearray()
+
+    async for chunk in resp.content.iter_chunked(chunk_size):
+        buffer.extend(chunk)
+        while b"\n" in buffer:
+            idx = buffer.index(b"\n")
+            line = buffer[:idx].rstrip(b"\r")
+            del buffer[: idx + 1]
+            yield line.decode(encoding) if encoding else bytes(line)
+
+    # if there's anything left in the response yield it
+    if buffer:
+        line = buffer.rstrip(b"\r")
+        yield line.decode(encoding) if encoding else bytes(line)
diff --git a/src/repligit/repligit.py b/src/repligit/client.py
similarity index 57%
rename from src/repligit/repligit.py
rename to src/repligit/client.py
index ac8fbbf..9ac6972 100644
--- a/src/repligit/repligit.py
+++ b/src/repligit/client.py
@@ -1,30 +1,15 @@
-from typing import Generator, List, Union
+from typing import List
 
 import requests
 
+from repligit.parse import (
+    decode_lines,
+    generate_fetch_pack_request,
+    generate_send_pack_header,
+)
 
-def decode_lines(lines: Generator[str]) -> Generator[str]:
-    """Decode server response iterator into usable lines."""
-    for line in lines:
-        line_length = int(line[:4], 16)
-        yield line[4:line_length]
 
-
-def encode_lines(lines: List[Union[bytes, str]]) -> bytes:
-    """Build byte string from given lines to send to server."""
-    result = []
-    for line in lines:
-        if type(line) is str:
-            line = line.encode("utf-8")
-
-        result.append(f"{len(line) + 5:04x}".encode())
-        result.append(line)
-        result.append(b"\n")
-
-    return b"".join(result)
-
-
-def ls_remote(url, username=None, password=None):
+def ls_remote(url: str, username: str = None, password: str = None):
     """Get commit hash of remote master branch, return SHA-1 hex string or
     None if no remote commits.
     """
@@ -38,23 +23,12 @@ def ls_remote(url, username=None, password=None):
         resp.encoding = "utf-8"
 
     lines = decode_lines(resp.iter_lines(decode_unicode=True))
-
     service_line = next(lines)
     assert service_line == "# service=git-upload-pack"
 
     return dict(reversed(line.split()) for line in lines if line)
 
 
-def generate_fetch_pack_request(want: str, haves: List[str]) -> bytes:
-    """Generate a git-upload packfile request to retrieve a packfile from a
-    remote server based on a list of have and want commits.
-    """
-    want_cmds = encode_lines([f"want {want}".encode()])
-    have_cmds = encode_lines([f"have {sha}".encode() for sha in haves])
-
-    return want_cmds + b"0000" + have_cmds + encode_lines([b"done"])
-
-
 def fetch_pack(
     url: str, want_sha: str, have_shas: List[str], username=None, password=None
 ):
@@ -74,7 +48,6 @@ def fetch_pack(
         stream=True,
         timeout=None,
     )
-
     resp.raise_for_status()
 
     line_length = int(resp.raw.read(4), 16)
@@ -86,10 +59,6 @@ def fetch_pack(
         return None
 
 
-def generate_send_pack_header(ref: str, from_sha: str, to_sha: str) -> bytes:
-    return encode_lines([f"{from_sha} {to_sha} {ref}\x00 report-status"]) + b"0000"
-
-
 def send_pack(
     url: str,
     ref: str,
@@ -99,11 +68,12 @@ def send_pack(
     username: str = None,
     password: str = None,
 ):
+    """Send a packfile to a remote server."""
     url = f"{url}/git-receive-pack"
     auth = (username, password) if username or password else None
 
     header = generate_send_pack_header(ref, from_sha, to_sha)
-    recieve_pack_request = header + packfile.read()
+    receive_pack_request = header + packfile.read()
 
     resp = requests.post(
         url,
@@ -112,11 +82,13 @@ def send_pack(
         },
         stream=True,
         auth=auth,
-        data=recieve_pack_request,
+        data=receive_pack_request,
     )
-
     resp.raise_for_status()
 
+    if resp.encoding is None:
+        resp.encoding = "utf-8"
+
     lines = decode_lines(resp.iter_lines(decode_unicode=True))
-    assert next(lines) == b"unpack ok"
-    assert next(lines) == f"ok {ref}".encode()
+    assert next(lines) == "unpack ok"
+    assert next(lines) == f"ok {ref}"
diff --git a/src/repligit/parse.py b/src/repligit/parse.py
new file mode 100644
index 0000000..46cac18
--- /dev/null
+++ b/src/repligit/parse.py
@@ -0,0 +1,37 @@
+# --------- shared client helpers ---------
+from typing import Generator, List, Union
+
+
+def decode_lines(lines: Generator[str]) -> Generator[str]:
+    """Decode server response iterator into usable lines."""
+    for line in lines:
+        line_length = int(line[:4], 16)
+        yield line[4:line_length]
+
+
+def encode_lines(lines: List[Union[bytes, str]]) -> bytes:
+    """Build byte string from given lines to send to server."""
+    result = []
+    for line in lines:
+        if type(line) is str:
+            line = line.encode("utf-8")
+
+        result.append(f"{len(line) + 5:04x}".encode())
+        result.append(line)
+        result.append(b"\n")
+
+    return b"".join(result)
+
+
+def generate_send_pack_header(ref: str, from_sha: str, to_sha: str) -> bytes:
+    return encode_lines([f"{from_sha} {to_sha} {ref}\x00 report-status"]) + b"0000"
+
+
+def generate_fetch_pack_request(want: str, haves: List[str]) -> bytes:
+    """Generate a git-upload packfile request to retrieve a packfile from a
+    remote server based on a list of have and want commits.
+    """
+    want_cmds = encode_lines([f"want {want}".encode()])
+    have_cmds = encode_lines([f"have {sha}".encode() for sha in haves])
+
+    return want_cmds + b"0000" + have_cmds + encode_lines([b"done"])
diff --git a/tests/test_repligit.py b/tests/test_client.py
similarity index 85%
rename from tests/test_repligit.py
rename to tests/test_client.py
index 73d0606..5a0a7a0 100644
--- a/tests/test_repligit.py
+++ b/tests/test_client.py
@@ -1,4 +1,4 @@
-import repligit.repligit as repligit
+from repligit.parse import decode_lines, encode_lines, generate_send_pack_header
 
 
 def test_decode_lines():
@@ -14,7 +14,7 @@ def test_decode_lines():
         b"",
     ]
 
-    lines = list(repligit.decode_lines(raw_lines))
+    lines = list(decode_lines(raw_lines))
     assert decoded_lines == lines
 
 
@@ -29,7 +29,7 @@ def test_encode_lines_from_bytes():
         b"003f358aa046cd57dbca306e80d4c3fbb86edc5b36af refs/pull/96/head\n"
     )
 
-    output_lines = repligit.encode_lines(input_lines)
+    output_lines = encode_lines(input_lines)
     assert encoded_lines == output_lines
 
 
@@ -44,7 +44,7 @@ def test_encode_lines_from_str():
         b"003f358aa046cd57dbca306e80d4c3fbb86edc5b36af refs/pull/96/head\n"
     )
 
-    output_lines = repligit.encode_lines(input_lines)
+    output_lines = encode_lines(input_lines)
     assert encoded_lines == output_lines
 
 
@@ -59,5 +59,5 @@ def test_generate_send_pack_header():
     to_sha = "b03ab96b18ed6633c877221318db41d36b15e3d7"
     ref = "refs/heads/main"
 
-    output_header = repligit.generate_send_pack_header(ref, from_sha, to_sha)
+    output_header = generate_send_pack_header(ref, from_sha, to_sha)
     assert expected_header == output_header


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR adds an asynchronous client for the git HTTP transfer protocol while refactoring the client API and updating related dependencies and examples.  
- Introduces new asyncio modules (client, parse, and util) that mirror the synchronous API.  
- Refactors public client API by replacing the legacy repligit.py with the new client.py.  
- Updates dependencies (py-aiohttp) and bumps the minimum Python version to 3.10.

### Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

<details>
<summary>Show a summary per file</summary>

| File                                 | Description                                                         |
| ------------------------------------ | ------------------------------------------------------------------- |
| src/repligit/util.py                 | Adds shared utility functions for both sync and async clients.      |
| src/repligit/repligit.py             | Removes the legacy synchronous client implementation.               |
| src/repligit/parse.py                | Introduces helpers for encoding/decoding and processing server lines. |
| src/repligit/client.py               | Implements the updated synchronous client using the shared utilities. |
| src/repligit/asyncio/util.py         | Creates an async equivalent for streaming HTTP response lines.        |
| src/repligit/asyncio/parse.py        | Provides async helpers to decode and process ls-remote responses.       |
| src/repligit/asyncio/client.py       | Implements the asynchronous client with similar API as the sync version.|
| src/repligit/asyncio/__init__.py     | Aggregates the public API for the async client.                       |
| src/repligit/__init__.py             | Updates the public API exports to point to the new client implementation.|
| spack.yaml                           | Adds py-aiohttp dependency for async functionality.                   |
| pyproject.toml                       | Bumps the minimum required Python version to 3.10.                      |
| examples/git_mirror_async.py         | Provides an asynchronous example of mirroring a git repository.        |
| examples/git_mirror.py               | Updates the synchronous example to reflect the new import API.          |
</details>





state: COMMENTED
author_association: NONE
submitted_at: 2025-03-27T16:42:28Z
user: login: cmelone
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T16:44:47Z
user: login: alecbcs
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T16:18:08Z
user: login: alecbcs
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T16:19:59Z
user: login: alecbcs
body: Still looking through the implementation for the asynchronous library, but thought I'd post a few comments I've written so far. This is looking great!
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T22:15:16Z
user: login: cmelone
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T22:46:30Z
user: login: alecbcs
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T12:12:26Z
user: login: alecbcs
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T12:22:35Z
user: login: alecbcs
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T17:03:13Z
user: login: alecbcs
body: Just a few last nitpicks. Otherwise looks good to me! Thanks @cmelone!
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-31T17:37:33Z
user: login: alecbcs
body: Looks good to me. Thanks @cmelone!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-31T18:34:13Z

Review comments:
diff_hunk: @@ -0,0 +1,105 @@
+"""
+Asynchronous implementation of repligit operations.
+"""
+
+import ssl
+from typing import List
+
+import aiohttp
+import certifi
+
+from repligit.asyncio.parse import decode_lines, process_ls_remote
+from repligit.asyncio.util import iter_lines
+from repligit.parse import generate_fetch_pack_request, generate_send_pack_header
+from repligit.util import (
+    check_fetch_pack_resp,
+    fmt_fetch_pack_url,
+    fmt_ls_remote_url,
+    fmt_send_pack_url,
+    get_receive_pack_headers,
+    get_upload_pack_headers,
+    validate_send_pack_resp,
+    validate_service_line,
+)
+
+# TODO debug why this doesn't work
+# ssl_context = ssl.create_default_context(cafile=certifi.where())
+ssl_context = ssl._create_unverified_context()
+
path: src/repligit/asyncio/client.py
user: login: Copilot
body: Using an unverified SSL context can expose the application to security risks. Consider resolving why the verified context creation (using ssl.create_default_context) is not working to ensure secure connections.
```suggestion
ssl_context = ssl.create_default_context(cafile=certifi.where())
# ssl_context = ssl._create_unverified_context()
```
created_at: 2025-03-27T16:42:27Z
author_association: NONE
diff_hunk: @@ -0,0 +1,105 @@
+"""
+Asynchronous implementation of repligit operations.
+"""
+
+import ssl
+from typing import List
+
+import aiohttp
+import certifi
+
+from repligit.asyncio.parse import decode_lines, process_ls_remote
+from repligit.asyncio.util import iter_lines
+from repligit.parse import generate_fetch_pack_request, generate_send_pack_header
+from repligit.util import (
+    check_fetch_pack_resp,
+    fmt_fetch_pack_url,
+    fmt_ls_remote_url,
+    fmt_send_pack_url,
+    get_receive_pack_headers,
+    get_upload_pack_headers,
+    validate_send_pack_resp,
+    validate_service_line,
+)
+
+# TODO debug why this doesn't work
+# ssl_context = ssl.create_default_context(cafile=certifi.where())
+ssl_context = ssl._create_unverified_context()
+
+
+async def ls_remote(url: str, username: str = None, password: str = None):
+    """Get commit hash of remote master branch, return SHA-1 hex string or
+    None if no remote commits.
+    """
+
+    url = fmt_ls_remote_url(url)
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+    async with aiohttp.ClientSession(
+        auth=auth, connector=aiohttp.TCPConnector(ssl=ssl_context)
+    ) as session:
+        async with session.get(url, raise_for_status=True) as resp:
+            lines = decode_lines(iter_lines(resp, encoding="utf-8"))
+            return await process_ls_remote(lines, validate_service_line)
+
+
+async def fetch_pack(url: str, want_sha: str, have_shas: List[str], username=None, password=None):
+    """Download a packfile from a remote server."""
+    url = fmt_fetch_pack_url(url)
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+
+    request = generate_fetch_pack_request(want_sha, have_shas)
+
+    async with aiohttp.ClientSession(
+        auth=auth, connector=aiohttp.TCPConnector(ssl=ssl_context)
+    ) as session:
+        async with session.post(
+            url,
+            headers=get_upload_pack_headers(),
+            data=request,
+            raise_for_status=True,
+            timeout=None,
+        ) as resp:
+            length_bytes = await resp.content.readexactly(4)
+            line_length = int(length_bytes, 16)
+            line = await resp.content.readexactly(line_length - 4)
+
+            if check_fetch_pack_resp(line):
+                # TODO address: this is a difference in API between sync and async
+                # has to be read within this context to be used in the caller
+                return await resp.content.read()
path: src/repligit/asyncio/client.py
user: login: Copilot
body: The async fetch_pack function returns the entire packfile content via 'await resp.content.read()', differing from the synchronous API that returns a streaming response. This inconsistency may affect handling of large packfiles; consider documenting this behavior or providing a streaming alternative.
```suggestion
                content = b""
                async for chunk in resp.content.iter_chunked(1024):
                    content += chunk
                return content
```
created_at: 2025-03-27T16:42:28Z
author_association: NONE
diff_hunk: @@ -0,0 +1,105 @@
+"""
+Asynchronous implementation of repligit operations.
+"""
+
+import ssl
+from typing import List
+
+import aiohttp
+import certifi
+
+from repligit.asyncio.parse import decode_lines, process_ls_remote
+from repligit.asyncio.util import iter_lines
+from repligit.parse import generate_fetch_pack_request, generate_send_pack_header
+from repligit.util import (
+    check_fetch_pack_resp,
+    fmt_fetch_pack_url,
+    fmt_ls_remote_url,
+    fmt_send_pack_url,
+    get_receive_pack_headers,
+    get_upload_pack_headers,
+    validate_send_pack_resp,
+    validate_service_line,
+)
+
+# TODO debug why this doesn't work
+# ssl_context = ssl.create_default_context(cafile=certifi.where())
+ssl_context = ssl._create_unverified_context()
+
+
+async def ls_remote(url: str, username: str = None, password: str = None):
+    """Get commit hash of remote master branch, return SHA-1 hex string or
+    None if no remote commits.
+    """
+
+    url = fmt_ls_remote_url(url)
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+    async with aiohttp.ClientSession(
+        auth=auth, connector=aiohttp.TCPConnector(ssl=ssl_context)
+    ) as session:
+        async with session.get(url, raise_for_status=True) as resp:
+            lines = decode_lines(iter_lines(resp, encoding="utf-8"))
+            return await process_ls_remote(lines, validate_service_line)
+
+
+async def fetch_pack(url: str, want_sha: str, have_shas: List[str], username=None, password=None):
+    """Download a packfile from a remote server."""
+    url = fmt_fetch_pack_url(url)
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+
+    request = generate_fetch_pack_request(want_sha, have_shas)
+
+    async with aiohttp.ClientSession(
+        auth=auth, connector=aiohttp.TCPConnector(ssl=ssl_context)
+    ) as session:
+        async with session.post(
+            url,
+            headers=get_upload_pack_headers(),
+            data=request,
+            raise_for_status=True,
+            timeout=None,
+        ) as resp:
+            length_bytes = await resp.content.readexactly(4)
+            line_length = int(length_bytes, 16)
+            line = await resp.content.readexactly(line_length - 4)
+
+            if check_fetch_pack_resp(line):
+                # TODO address: this is a difference in API between sync and async
+                # has to be read within this context to be used in the caller
+                return await resp.content.read()
path: src/repligit/asyncio/client.py
user: login: cmelone
body: isn't this effectively the same thing as loading it all into memory?              
created_at: 2025-03-27T16:44:47Z
author_association: MEMBER
diff_hunk: @@ -9,6 +9,7 @@ def main():
 
     target_ref = f"refs/heads/{branch_name}"
 
+    # if either src or dest require auth for read access, add the username and password args
path: examples/git_mirror.py
user: login: alecbcs
body: I'd add a commented out `src_{username,password}=<username,token>` and a commented out fetch_pack that uses them to make their comment easier to understand.
created_at: 2025-03-28T16:08:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,105 @@
+"""
+Asynchronous implementation of repligit operations.
+"""
+
+import ssl
+from typing import List
+
+import aiohttp
+import certifi
path: src/repligit/asyncio/client.py
user: login: alecbcs
body: ```suggestion
```

We can drop the `certifi` import here since it's unused. 
created_at: 2025-03-28T16:13:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,105 @@
+"""
+Asynchronous implementation of repligit operations.
+"""
+
+import ssl
+from typing import List
+
+import aiohttp
+import certifi
+
+from repligit.asyncio.parse import decode_lines, process_ls_remote
+from repligit.asyncio.util import iter_lines
+from repligit.parse import generate_fetch_pack_request, generate_send_pack_header
+from repligit.util import (
+    check_fetch_pack_resp,
+    fmt_fetch_pack_url,
+    fmt_ls_remote_url,
+    fmt_send_pack_url,
+    get_receive_pack_headers,
+    get_upload_pack_headers,
+    validate_send_pack_resp,
+    validate_service_line,
+)
+
+# TODO debug why this doesn't work
+# ssl_context = ssl.create_default_context(cafile=certifi.where())
+ssl_context = ssl._create_unverified_context()
+
+
+async def ls_remote(url: str, username: str = None, password: str = None):
+    """Get commit hash of remote master branch, return SHA-1 hex string or
+    None if no remote commits.
+    """
+
+    url = fmt_ls_remote_url(url)
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+    async with aiohttp.ClientSession(
+        auth=auth, connector=aiohttp.TCPConnector(ssl=ssl_context)
+    ) as session:
+        async with session.get(url, raise_for_status=True) as resp:
+            lines = decode_lines(iter_lines(resp, encoding="utf-8"))
+            return await process_ls_remote(lines, validate_service_line)
+
+
+async def fetch_pack(url: str, want_sha: str, have_shas: List[str], username=None, password=None):
+    """Download a packfile from a remote server."""
+    url = fmt_fetch_pack_url(url)
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+
+    request = generate_fetch_pack_request(want_sha, have_shas)
+
+    async with aiohttp.ClientSession(
+        auth=auth, connector=aiohttp.TCPConnector(ssl=ssl_context)
+    ) as session:
+        async with session.post(
+            url,
+            headers=get_upload_pack_headers(),
+            data=request,
+            raise_for_status=True,
+            timeout=None,
+        ) as resp:
+            length_bytes = await resp.content.readexactly(4)
+            line_length = int(length_bytes, 16)
+            line = await resp.content.readexactly(line_length - 4)
+
+            if check_fetch_pack_resp(line):
+                # TODO address: this is a difference in API between sync and async
+                # has to be read within this context to be used in the caller
+                return await resp.content.read()
path: src/repligit/asyncio/client.py
user: login: alecbcs
body: Yeah that looks like it to me.
created_at: 2025-03-28T16:19:59Z
author_association: MEMBER
diff_hunk: @@ -9,8 +9,19 @@ def main():
 
     target_ref = f"refs/heads/{branch_name}"
 
-    gh_refs = ls_remote(src_remote_url)
-    gl_refs = ls_remote(dest_remote_url)
+    # note: provide credentials in the following situations:
+    # ls_remote: source repo requires auth to read
+    # ls_remote: destination repo requires auth to read
+    # fetch_pack: source repo requires auth to read
+    # send_pack: destination repo requires auth to write
+    # username and password args are not needed otherwise
+    src_username = "<username>"
+    src_password = "<token>"
+    dest_username = "<username>"
+    dest_password = "<token>"
+
+    gh_refs = ls_remote(src_remote_url, username=src_username, password=src_password)
+    gl_refs = ls_remote(dest_remote_url, username=dest_username, password=dest_password)
path: examples/git_mirror.py
user: login: alecbcs
body: ```suggestion
    # Authentication credentials
    # Note: Only provide credentials when authentication is required
    # src_username = "<username>"  # Uncomment if source repo requires auth
    # src_password = "<token>"     # Uncomment if source repo requires auth
    dest_username = "<username>"   # For destination repo write access
    dest_password = "<token>"      # For destination repo write access

    # List references from source repository (without authentication in this example)
    gh_refs = ls_remote(src_remote_url)

    # List references from destination repository (with authentication)
    gl_refs = ls_remote(dest_remote_url, username=dest_username, password=dest_password)
```

What would you think about rewriting this to be something like this to show both an authenticated and unauthenticated `ls_remote()` call example. (Let's also do the same for the asyncio example.)
created_at: 2025-03-28T22:07:40Z
author_association: MEMBER
diff_hunk: @@ -21,10 +32,13 @@ def main():
         print("Everything is up to date")
         return
 
-    packfile = fetch_pack(src_remote_url, want_sha, have_shas)
-
-    dest_username = "<username>"
-    dest_password = "<token>"
+    packfile = fetch_pack(
+        src_remote_url,
+        want_sha,
+        have_shas,
+        username=src_username,
+        password=src_password,
+    )
path: examples/git_mirror.py
user: login: alecbcs
body: ```suggestion
    # Fetch the packfile from source repository
    packfile = fetch_pack(
        src_remote_url,
        want_sha,
        have_shas,
        # username=src_username,  # Uncomment if source repo requires auth
        # password=src_password,  # Uncomment if source repo requires auth
    )
```

Similar updates to show unauthenticated pull from src.
created_at: 2025-03-28T22:08:59Z
author_association: MEMBER
diff_hunk: @@ -10,6 +10,7 @@ spack:
   - py-pytest
   - py-requests
   - py-ruff
+  - py-aiohttp
path: spack.yaml
user: login: alecbcs
body: Do you want to rebase this PR on top of #15 so that we can drop this?
created_at: 2025-03-28T22:11:24Z
author_association: MEMBER
diff_hunk: @@ -1 +1,3 @@
-__all__ = ["repligit"]
+from repligit.client import fetch_pack, ls_remote, send_pack
+
+__all__ = ["ls_remote", "fetch_pack", "send_pack"]
path: src/repligit/__init__.py
user: login: alecbcs
body: This looks great! Thanks for fixing this!
created_at: 2025-03-28T22:11:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,8 @@
+try:
+    import aiohttp  # noqa
+except ModuleNotFoundError:
+    raise ModuleNotFoundError("aiohttp is required to use the async client") from None
+
+from repligit.asyncio.client import fetch_pack, ls_remote, send_pack
+
+__all__ = ["ls_remote", "fetch_pack", "send_pack"]
path: src/repligit/asyncio/__init__.py
user: login: alecbcs
body: This is super useful! Good thinking!
created_at: 2025-03-28T22:12:39Z
author_association: MEMBER
diff_hunk: @@ -9,8 +9,19 @@ def main():
 
     target_ref = f"refs/heads/{branch_name}"
 
-    gh_refs = ls_remote(src_remote_url)
-    gl_refs = ls_remote(dest_remote_url)
+    # note: provide credentials in the following situations:
+    # ls_remote: source repo requires auth to read
+    # ls_remote: destination repo requires auth to read
+    # fetch_pack: source repo requires auth to read
+    # send_pack: destination repo requires auth to write
+    # username and password args are not needed otherwise
+    src_username = "<username>"
+    src_password = "<token>"
+    dest_username = "<username>"
+    dest_password = "<token>"
+
+    gh_refs = ls_remote(src_remote_url, username=src_username, password=src_password)
+    gl_refs = ls_remote(dest_remote_url, username=dest_username, password=dest_password)
path: examples/git_mirror.py
user: login: cmelone
body: sounds good!
created_at: 2025-03-28T22:46:30Z
author_association: MEMBER
diff_hunk: @@ -9,9 +9,17 @@ def main():
 
     target_ref = f"refs/heads/{branch_name}"
 
+    # Authentication credentials
+    # Note: Only provide credentials when authentication is required
+    # src_username = "<username>"  # Uncomment if source repo requires auth
+    # src_password = "<token>"     # Uncomment if source repo requires auth
+    dest_username = "<username>"  # For destination repo write access
+    dest_password = "<token>"  # For destination repo write access
+
+    # List references from source repository (without authentication in this example)
     gh_refs = ls_remote(src_remote_url)
-    gl_refs = ls_remote(dest_remote_url)
-
+    # List references from destination repository (with authentication)
+    gl_refs = ls_remote(dest_remote_url, username=dest_username, password=dest_password)
     want_sha = gh_refs[target_ref]
path: examples/git_mirror.py
user: login: alecbcs
body: ```suggestion
    # List references from destination repository (with authentication)
    gl_refs = ls_remote(dest_remote_url, username=dest_username, password=dest_password)
    
    want_sha = gh_refs[target_ref]
```
Quick fix to the line breaks to make them a bit easier to understand.
created_at: 2025-03-31T12:12:25Z
author_association: MEMBER
diff_hunk: @@ -21,10 +29,14 @@ def main():
         print("Everything is up to date")
         return
 
-    packfile = fetch_pack(src_remote_url, want_sha, have_shas)
-
-    dest_username = "<username>"
-    dest_password = "<token>"
+    # Fetch the packfile from source repository
+    packfile = fetch_pack(
+        src_remote_url,
+        want_sha,
+        have_shas,
+        # username=src_username,  # Uncomment if source repo requires auth
+        # password=src_password,  # Uncomment if source repo requires auth
+    )
 
     send_pack(
         dest_remote_url,
path: examples/git_mirror.py
user: login: alecbcs
body: ```suggestion
    # Upload packfile to destination repository
    send_pack(
        dest_remote_url,
```
created_at: 2025-03-31T12:22:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,58 @@
+import asyncio
+
+from repligit.asyncio import fetch_pack, ls_remote, send_pack
+
+
+async def main():
+    src_remote_url = "https://github.com/spack/spack.git"
+    dest_remote_url = "https://gitlab.com/test-org/test-repo.git"
+
+    branch_name = "main"
+
+    target_ref = f"refs/heads/{branch_name}"
+
+    # Authentication credentials
+    # Note: Only provide credentials when authentication is required
+    # src_username = "<username>"  # Uncomment if source repo requires auth
+    # src_password = "<token>"     # Uncomment if source repo requires auth
+    dest_username = "<username>"  # For destination repo write access
+    dest_password = "<token>"  # For destination repo write access
+
+    # List references from source repository (without authentication in this example)
+    gh_refs = await ls_remote(src_remote_url)
+    # List references from destination repository (with authentication)
+    gl_refs = await ls_remote(
+        dest_remote_url, username=dest_username, password=dest_password
+    )
path: examples/git_mirror_async.py
user: login: alecbcs
body: ```suggestion
    # List references from source repository (without authentication)
    gh_refs = await ls_remote(src_remote_url)
    
    # List references from destination repository (with authentication)
    gl_refs = await ls_remote(
        dest_remote_url, username=dest_username, password=dest_password
    )
```

Small formatting fix
created_at: 2025-03-31T16:08:11Z
author_association: MEMBER
diff_hunk: @@ -9,9 +9,17 @@ def main():
 
     target_ref = f"refs/heads/{branch_name}"
 
+    # Authentication credentials
+    # Note: Only provide credentials when authentication is required
+    # src_username = "<username>"  # Uncomment if source repo requires auth
+    # src_password = "<token>"     # Uncomment if source repo requires auth
+    dest_username = "<username>"  # For destination repo write access
+    dest_password = "<token>"  # For destination repo write access
+
+    # List references from source repository (without authentication in this example)
     gh_refs = ls_remote(src_remote_url)
path: examples/git_mirror.py
user: login: alecbcs
body: ```suggestion
    # List references from source repository (without authentication)
    gh_refs = ls_remote(src_remote_url)
```

Making the two examples consistent
created_at: 2025-03-31T16:12:54Z
author_association: MEMBER
path: tests/test_repligit.py
user: login: alecbcs
body: We might want to rename this file to `test_client.py` or just `client.py` if pytest will still pick that up.
created_at: 2025-03-31T16:53:48Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,90 @@
+from typing import List
+
+import aiohttp
+
+from repligit.asyncio.parse import decode_lines, iter_lines
+from repligit.parse import generate_fetch_pack_request, generate_send_pack_header
+
+
+async def ls_remote(url: str, username: str = None, password: str = None):
+    """Get commit hash of remote master branch, return SHA-1 hex string or
+    None if no remote commits.
+    """
+
+    url = f"{url}/info/refs?service=git-upload-pack"
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+    async with aiohttp.ClientSession(auth=auth) as session:
+        async with session.get(url, raise_for_status=True) as resp:
+            lines = decode_lines(iter_lines(resp, encoding="utf-8"))
+            service_line = await anext(lines)
+            assert service_line == "# service=git-upload-pack"
+
+            # `async for` inside `dict()` not supported so no dict comprehension
+            result = {}
+            async for line in lines:
+                if line:
+                    sha, ref = line.split()
+                    result[ref] = sha
+            return result
path: src/repligit/asyncio/client.py
user: login: alecbcs
body: ```suggestion
            async for line in lines:
                if not line:
                    continue
                sha, ref = line.split()
                result[ref] = sha
            return result
```

Let's flip this condition to exit early and reduce the indentation level to make it a bit easier to read.
created_at: 2025-03-31T17:31:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,90 @@
+from typing import List
+
+import aiohttp
+
+from repligit.asyncio.parse import decode_lines, iter_lines
+from repligit.parse import generate_fetch_pack_request, generate_send_pack_header
+
+
+async def ls_remote(url: str, username: str = None, password: str = None):
+    """Get commit hash of remote master branch, return SHA-1 hex string or
+    None if no remote commits.
+    """
+
+    url = f"{url}/info/refs?service=git-upload-pack"
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+    async with aiohttp.ClientSession(auth=auth) as session:
+        async with session.get(url, raise_for_status=True) as resp:
+            lines = decode_lines(iter_lines(resp, encoding="utf-8"))
+            service_line = await anext(lines)
+            assert service_line == "# service=git-upload-pack"
+
+            # `async for` inside `dict()` not supported so no dict comprehension
+            result = {}
+            async for line in lines:
+                if line:
+                    sha, ref = line.split()
+                    result[ref] = sha
+            return result
+
+
+async def fetch_pack(
+    url: str, want_sha: str, have_shas: List[str], username=None, password=None
+):
+    """Download a packfile from a remote server."""
+    url = f"{url}/git-upload-pack"
+    auth = aiohttp.BasicAuth(username, password) if username or password else None
+
+    request = generate_fetch_pack_request(want_sha, have_shas)
+
+    async with aiohttp.ClientSession(auth=auth) as session:
+        async with session.post(
+            url,
+            headers={
+                "Content-type": "application/x-git-upload-pack-request",
+            },
+            data=request,
+            raise_for_status=True,
+            timeout=None,
+        ) as resp:
+            length_bytes = await resp.content.readexactly(4)
+            line_length = int(length_bytes, 16)
+            line = await resp.content.readexactly(line_length - 4)
+
+            if line[:3] == b"NAK" or line[:3] == b"ACK":
path: src/repligit/asyncio/client.py
user: login: alecbcs
body: ```suggestion
            length_bytes = await resp.content.readexactly(4)
            line_length = int(length_bytes, 16)
            
            line = await resp.content.readexactly(line_length - 4)
            if line[:3] == b"NAK" or line[:3] == b"ACK":
```

Shifting down for readability.
created_at: 2025-03-31T17:33:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,39 @@
+# --------- asynchronous client helpers ---------
+from typing import AsyncIterable, AsyncIterator
+
+import aiohttp
+
+
+async def decode_lines(line_stream: AsyncIterable) -> AsyncIterator:
+    """(async) Decode server response iterator into usable lines."""
+    async for line in line_stream:
+        line_length = int(line[:4], 16)
+        yield line[4:line_length]
+
+
+async def iter_lines(
+    resp: aiohttp.ClientResponse, encoding: str, chunk_size: int = 512
+):
+    """
+    Stream lines from an HTTP response. Yields raw lines as bytes by default.
+
+    If `encoding` is set, yields decoded strings using the given encoding.
+
+    This is a reimplementation of requests.models.iter_lines for async,
+    the default `chunk_size` value is inherited from that definition.
+    """
+
+    buffer = bytearray()
path: src/repligit/asyncio/parse.py
user: login: alecbcs
body: ```suggestion
    """
    buffer = bytearray()
```
created_at: 2025-03-31T17:35:54Z
author_association: MEMBER

PR comments:
user: login: alecbcs
created_at: 2025-03-28T16:25:00Z
body: I like the changes in `client.py` and `asyncio.client.py`, but I think all of the wrappers in `util.py` make the code harder to read personally. Would it be possible to revert and discuss using util wrappers in a later PR?
author_association: MEMBER
user: login: cmelone
created_at: 2025-03-28T16:29:37Z
body: > Would it be possible to revert and discuss using util wrappers in a later PR?

yep, will do!
author_association: MEMBER
user: login: cmelone
created_at: 2025-03-31T17:04:39Z
body: I accidentally overwrote your last formatting suggestions so will fix that now
author_association: MEMBER
user: login: cmelone
created_at: 2025-03-31T17:13:29Z
body: @alecbcs ready for further review/merging
author_association: MEMBER