Repository: stankudrow/aiofiles
Pull request #1: Implement 'aiofiles.os.walk' via a queue and generators
Author: login: stankudrow
Created: 2025-02-08T17:25:30Z | Merged: 2025-04-25T05:59:42Z

A queue-based solution for the [aiofiles/PR-193](https://github.com/Tinche/aiofiles/pull/193) .


Diff:
diff --git a/.gitignore b/.gitignore
index d5465ad..e3c8fee 100644
--- a/.gitignore
+++ b/.gitignore
@@ -83,6 +83,11 @@ ENV/
 env.bak/
 venv.bak/
 
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
 # PyCharm
 #  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
 #  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
@@ -92,3 +97,6 @@ venv.bak/
 
 # Visual Studio Code
 .vscode/
+
+# ruff stuff:
+.ruff_cache/
diff --git a/Makefile b/Makefile
index 4590f98..d3a7463 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,14 @@
-.PHONY: test lint
+SRC_DIR := src
+TESTS_DIR := tests
+TARGET_DIRS := $(SRC_DIR) $(TESTS_DIR)
+
+.PHONY: format lint test
+
+format:
+	pdm run black $(TARGET_DIRS)
 
 test:
-	pdm run pytest -x --ff tests
+	pdm run pytest -x --ff $(TESTS_DIR)
 
 lint:
-	pdm run flake8 src tests && pdm run black --check src tests
+	pdm run flake8 $(TARGET_DIRS) && pdm run black --check $(TARGET_DIRS)
diff --git a/src/aiofiles/os.py b/src/aiofiles/os.py
index 9f90d61..6ea891e 100644
--- a/src/aiofiles/os.py
+++ b/src/aiofiles/os.py
@@ -2,6 +2,8 @@
 
 import asyncio
 import os
+from functools import partial
+from queue import Queue, Empty
 
 from .base import wrap
 from . import ospath as path
@@ -69,8 +71,30 @@ async def walk(top, topdown=True, onerror=None, followlinks=False):
     Wraps the `os.walk` function.
     """
 
-    for content in os.walk(
-        top, topdown=topdown, onerror=onerror, followlinks=followlinks
-    ):
-        yield content
-        await asyncio.sleep(0)
+    def _walk(q: Queue):
+        nonlocal is_over
+        try:
+            for row in os.walk(
+                top, topdown=topdown, onerror=onerror, followlinks=followlinks
+            ):
+                q.put_nowait(row)
+        finally:
+            is_over = True
+
+    loop = asyncio.get_running_loop()
+    queue = Queue()
+
+    is_over = False
+
+    walker = partial(_walk, q=queue)
+    loop.run_in_executor(None, walker)
+    while True:
+        if is_over and queue.empty():
+            break
+        try:
+            entry = queue.get_nowait()
+            queue.task_done()
+            yield entry
+        except Empty:
+            pass
+    queue.join()
diff --git a/tests/test_os.py b/tests/test_os.py
index 21ab988..8cbe291 100644
--- a/tests/test_os.py
+++ b/tests/test_os.py
@@ -3,9 +3,11 @@
 import asyncio
 import os
 import platform
-from os import stat, walk
+from collections import defaultdict
+from os import stat
 from os.path import dirname, exists, isdir, join
 from pathlib import Path
+from typing import Optional
 
 import pytest
 
@@ -500,34 +502,75 @@ async def test_abspath():
     assert result == abs_filename
 
 
-@pytest.mark.parametrize(("path",), [(".",), ("..",)])
-async def test_walk(path: str):
-    """Test the `walk` call."""
+def _act_on_error(oserror):
+    print(f"aiofiles.os.walk onerror handler: {oserror}")
 
-    os_walk_res = list(walk(top=path))
-    aio_walk_res = [r async for r in aiofiles.os.walk(top=path)]
 
-    assert aio_walk_res == os_walk_res
-
-
-async def test_walk_non_blocking():
-    """Test if the `walk` is non-blocking."""
-
-    async def _set_answer():
-        nonlocal val
-        nonlocal ans
-
-        val = ans
-        await asyncio.sleep(1)
+@pytest.mark.parametrize(
+    ("top", "topdown", "onerror", "followlinks"),
+    [
+        ("non-existent", True, _act_on_error, False),
+        ("README.md", True, None, False),
+        ("README.md", False, _act_on_error, False),
+        ("./src", True, _act_on_error, False),
+        ("./src", False, _act_on_error, False),
+        ("./tests", True, None, True),
+        ("./tests", False, None, True),
+    ],
+)
+async def test_walk(top, topdown, onerror, followlinks):
+    result = [
+        row
+        async for row in aiofiles.os.walk(
+            top=top, topdown=topdown, onerror=onerror, followlinks=followlinks
+        )
+    ]
+    answer = [
+        row
+        for row in os.walk(
+            top=top, topdown=topdown, onerror=onerror, followlinks=followlinks
+        )
+    ]
 
-    async def _walk_away():
-        nonlocal val
-        nonlocal ans
+    assert result == answer
 
-        async for _ in aiofiles.os.walk(top="."):
-            pass
-        assert val == ans
 
-    val, ans = 21, 42
+@pytest.mark.parametrize(
+    "top_path",
+    [
+        "c3Po-ar2D2",  # does not mean to exist
+        "./.github",
+        "./src",
+        "./tests",
+        # ".",  # it is long
+    ],
+)
+async def test_walk_non_blocking(top_path: str):
+    async def _test_walk(
+        top: str, key: str, storage: defaultdict[str, list]
+    ) -> Optional[bool]:
+        non_blocking = None
+        async for datum in aiofiles.os.walk(top=top):
+            storage[key].append(datum)
+            # checking the statistics (common shareable resource) for other tasks
+            other_keys = set(storage) - {key}
+            if not non_blocking:
+                non_blocking = any([storage[other_key] for other_key in other_keys])
+            # without the `asyncio.sleep(0)` the `async for` may block
+            await asyncio.sleep(0)
+
+        return non_blocking
+
+    keys = {f"w_{i}" for i in range(2)}
+    storage = defaultdict(list)
+
+    tasks: list[asyncio.Task] = [
+        asyncio.create_task(_test_walk(top=top_path, key=key, storage=storage))
+        for key in keys
+    ]
+    results = await asyncio.gather(*tasks)
 
-    await asyncio.gather(_walk_away(), _set_answer())
+    # Asserting that each task does not block the event loop.
+    # Empty results are not assumed to be non-blocking by default.
+    if statuses := (r for r in results if r is not None):
+        assert all(statuses)


Reviews:
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-02-08T17:26:27Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-02-08T17:34:09Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-02-08T17:35:43Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-02-08T17:42:31Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-04-12T07:22:16Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-04-19T18:26:04Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-04-19T18:26:48Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-04-19T18:30:52Z
user: login: stankudrow
state: COMMENTED
author_association: OWNER
submitted_at: 2025-04-19T20:02:46Z

Review comments:
diff_hunk: @@ -500,34 +500,34 @@ async def test_abspath():
     assert result == abs_filename
 
 
-@pytest.mark.parametrize(("path",), [(".",), ("..",)])
-async def test_walk(path: str):
-    """Test the `walk` call."""
+def _act_on_error(oserror):
+    print(f"aiofiles.os.walk onerror handler: {oserror}")
 
-    os_walk_res = list(walk(top=path))
-    aio_walk_res = [r async for r in aiofiles.os.walk(top=path)]
 
-    assert aio_walk_res == os_walk_res
-
-
-async def test_walk_non_blocking():
-    """Test if the `walk` is non-blocking."""
-
-    async def _set_answer():
-        nonlocal val
-        nonlocal ans
-
-        val = ans
-        await asyncio.sleep(1)
-
-    async def _walk_away():
-        nonlocal val
-        nonlocal ans
-
-        async for _ in aiofiles.os.walk(top="."):
path: tests/test_os.py
user: login: stankudrow
body: `async for` here is sequential and blocking:

- https://superfastpython.com/asyncio-for-loop/
- https://superfastpython.com/asyncio-async-for/
created_at: 2025-02-08T17:26:27Z
author_association: OWNER
diff_hunk: @@ -500,34 +500,34 @@ async def test_abspath():
     assert result == abs_filename
 
 
-@pytest.mark.parametrize(("path",), [(".",), ("..",)])
-async def test_walk(path: str):
-    """Test the `walk` call."""
+def _act_on_error(oserror):
+    print(f"aiofiles.os.walk onerror handler: {oserror}")
 
-    os_walk_res = list(walk(top=path))
-    aio_walk_res = [r async for r in aiofiles.os.walk(top=path)]
 
-    assert aio_walk_res == os_walk_res
-
-
-async def test_walk_non_blocking():
-    """Test if the `walk` is non-blocking."""
-
-    async def _set_answer():
-        nonlocal val
-        nonlocal ans
-
-        val = ans
-        await asyncio.sleep(1)
-
-    async def _walk_away():
-        nonlocal val
-        nonlocal ans
-
-        async for _ in aiofiles.os.walk(top="."):
path: tests/test_os.py
user: login: stankudrow
body: But if these coros are swapped in the gather tool, i.e., `await asyncio.gather(_set_answer(), _walk_away())` (please refer to https://github.com/stankudrow/aiofiles/pull/1/files#diff-6e32f64be5ec084386460bf639dc465839a6b9bdc9a11b7709c497c51f5f8b36L533 ), this test passes.
created_at: 2025-02-08T17:34:09Z
author_association: OWNER
diff_hunk: @@ -69,8 +70,31 @@ async def walk(top, topdown=True, onerror=None, followlinks=False):
     Wraps the `os.walk` function.
     """
 
-    for content in os.walk(
-        top, topdown=topdown, onerror=onerror, followlinks=followlinks
-    ):
-        yield content
-        await asyncio.sleep(0)
+    def _walk(q: asyncio.Queue):
+        nonlocal is_over
+        try:
+            for row in os.walk(
+                top, topdown=topdown, onerror=onerror, followlinks=followlinks
+            ):
+                q.put_nowait(row)
+        finally:
+            is_over = True
+
+    loop = asyncio.get_running_loop()
+    queue = asyncio.Queue()
+
+    walker = partial(_walk, q=queue)
+    loop.run_in_executor(None, walker)
+
+    is_over = False
+    while True:
+        if is_over and queue.empty():
path: src/aiofiles/os.py
user: login: stankudrow
body: Producer may set `is_over` to True, but the queue may be not empty yet, that is why the loop must keep on.
created_at: 2025-02-08T17:35:43Z
author_association: OWNER
diff_hunk: @@ -69,8 +70,31 @@ async def walk(top, topdown=True, onerror=None, followlinks=False):
     Wraps the `os.walk` function.
     """
 
-    for content in os.walk(
-        top, topdown=topdown, onerror=onerror, followlinks=followlinks
-    ):
-        yield content
-        await asyncio.sleep(0)
+    def _walk(q: asyncio.Queue):
+        nonlocal is_over
+        try:
+            for row in os.walk(
+                top, topdown=topdown, onerror=onerror, followlinks=followlinks
+            ):
+                q.put_nowait(row)
+        finally:
+            is_over = True
+
+    loop = asyncio.get_running_loop()
+    queue = asyncio.Queue()
+
+    walker = partial(_walk, q=queue)
+    loop.run_in_executor(None, walker)
+
+    is_over = False
+    while True:
+        if is_over and queue.empty():
+            break
+        try:
+            entry = queue.get_nowait()
path: src/aiofiles/os.py
user: login: stankudrow
body: If using `await queue.get()`, at this point the loop can get stuck because `is_over` may be False and the emptiness of the queue is not evaluated, then it waits for an item, but the producer may finish at this time and set `is_over` to True and put no item in the queue.
created_at: 2025-02-08T17:42:31Z
author_association: OWNER
diff_hunk: @@ -69,8 +71,30 @@ async def walk(top, topdown=True, onerror=None, followlinks=False):
     Wraps the `os.walk` function.
     """
 
-    for content in os.walk(
-        top, topdown=topdown, onerror=onerror, followlinks=followlinks
-    ):
-        yield content
-        await asyncio.sleep(0)
+    def _walk(q: Queue):
+        nonlocal is_over
+        try:
+            for row in os.walk(
+                top, topdown=topdown, onerror=onerror, followlinks=followlinks
+            ):
+                q.put_nowait(row)
+        finally:
+            is_over = True
+
+    loop = asyncio.get_running_loop()
+    queue = Queue()
path: src/aiofiles/os.py
user: login: stankudrow
body: Now a thread-safe Queue is in use.
created_at: 2025-04-12T07:22:16Z
author_association: OWNER
diff_hunk: @@ -69,8 +71,30 @@ async def walk(top, topdown=True, onerror=None, followlinks=False):
     Wraps the `os.walk` function.
     """
 
-    for content in os.walk(
-        top, topdown=topdown, onerror=onerror, followlinks=followlinks
-    ):
-        yield content
-        await asyncio.sleep(0)
+    def _walk(q: Queue):
+        nonlocal is_over
+        try:
+            for row in os.walk(
+                top, topdown=topdown, onerror=onerror, followlinks=followlinks
+            ):
+                q.put_nowait(row)
+        finally:
+            is_over = True
+
+    loop = asyncio.get_running_loop()
+    queue = Queue()
+
+    is_over = False
+
+    walker = partial(_walk, q=queue)
+    loop.run_in_executor(None, walker)
+    while True:
+        if is_over and queue.empty():
+            break
+        try:
+            entry = queue.get_nowait()
+            queue.task_done()
+            yield entry
+        except Empty:
+            pass
+    queue.join()
path: src/aiofiles/os.py
user: login: stankudrow
body: Finalisation and insuring task processing done right.
created_at: 2025-04-19T18:26:04Z
author_association: OWNER
diff_hunk: @@ -500,34 +503,94 @@ async def test_abspath():
     assert result == abs_filename
 
 
-@pytest.mark.parametrize(("path",), [(".",), ("..",)])
-async def test_walk(path: str):
-    """Test the `walk` call."""
+def _act_on_error(oserror):
+    print(f"aiofiles.os.walk onerror handler: {oserror}")
 
-    os_walk_res = list(walk(top=path))
-    aio_walk_res = [r async for r in aiofiles.os.walk(top=path)]
-
-    assert aio_walk_res == os_walk_res
 
+@pytest.mark.parametrize(
+    ("top", "topdown", "onerror", "followlinks"),
+    [
+        ("non-existent", True, _act_on_error, False),
+        ("README.md", True, None, False),
+        ("README.md", False, _act_on_error, False),
+        ("./src", True, _act_on_error, False),
+        ("./src", False, _act_on_error, False),
+        ("./tests", True, None, True),
+        ("./tests", False, None, True),
+    ],
+)
+async def test_walk(top, topdown, onerror, followlinks):
+    result = [
+        row
+        async for row in aiofiles.os.walk(
+            top=top, topdown=topdown, onerror=onerror, followlinks=followlinks
+        )
+    ]
+    answer = [
+        row
+        for row in os.walk(
+            top=top, topdown=topdown, onerror=onerror, followlinks=followlinks
+        )
+    ]
path: tests/test_os.py
user: login: stankudrow
body: Just comparing that the `async` version works in the same way as the `sync` one does.
created_at: 2025-04-19T18:26:48Z
author_association: OWNER
diff_hunk: @@ -500,34 +503,94 @@ async def test_abspath():
     assert result == abs_filename
 
 
-@pytest.mark.parametrize(("path",), [(".",), ("..",)])
-async def test_walk(path: str):
-    """Test the `walk` call."""
+def _act_on_error(oserror):
+    print(f"aiofiles.os.walk onerror handler: {oserror}")
 
-    os_walk_res = list(walk(top=path))
-    aio_walk_res = [r async for r in aiofiles.os.walk(top=path)]
-
-    assert aio_walk_res == os_walk_res
 
+@pytest.mark.parametrize(
+    ("top", "topdown", "onerror", "followlinks"),
+    [
+        ("non-existent", True, _act_on_error, False),
+        ("README.md", True, None, False),
+        ("README.md", False, _act_on_error, False),
+        ("./src", True, _act_on_error, False),
+        ("./src", False, _act_on_error, False),
+        ("./tests", True, None, True),
+        ("./tests", False, None, True),
+    ],
+)
+async def test_walk(top, topdown, onerror, followlinks):
+    result = [
+        row
+        async for row in aiofiles.os.walk(
+            top=top, topdown=topdown, onerror=onerror, followlinks=followlinks
+        )
+    ]
+    answer = [
+        row
+        for row in os.walk(
+            top=top, topdown=topdown, onerror=onerror, followlinks=followlinks
+        )
+    ]
 
-async def test_walk_non_blocking():
-    """Test if the `walk` is non-blocking."""
+    assert result == answer
 
-    async def _set_answer():
-        nonlocal val
-        nonlocal ans
 
-        val = ans
-        await asyncio.sleep(1)
+@pytest.mark.parametrize(
+    "top_path",
+    [
+        "c3Po-ar2D2",  # does not mean to exist
+        "./src",
+        "./tests",
+        # ".",  # is long
+    ],
+)
+async def test_walk_non_blocking(top_path: str):
path: tests/test_os.py
user: login: stankudrow
body: This test checks that the `async` version is not blocking concerning the wrapper and its associated thread with a thread-safe queue.

When launching a couple of tasks, it is not the case (I hope I did it really right) that tasks execute sequentially.
created_at: 2025-04-19T18:30:52Z
author_association: OWNER
diff_hunk: @@ -1,7 +1,14 @@
-.PHONY: test lint
+SRC_DIR := src
+TESTS_DIR := tests
+TARGET_DIRS := $(SRC_DIR) $(TESTS_DIR)
+
+.PHONY: format lint test
+
+format:
+	pdm run black $(TARGET_DIRS)
path: Makefile
user: login: stankudrow
body: Makes things easier.
created_at: 2025-04-19T20:02:46Z
author_association: OWNER

PR comments:
user: login: stankudrow
created_at: 2025-02-15T10:05:45Z
body: @Tinche, hello, could you review this PR, please?
author_association: OWNER
user: login: Tinche
created_at: 2025-02-15T10:58:37Z
body: @stankudrow `asyncio.Queue` is not threadsafe (see the first line of https://docs.python.org/3/library/asyncio-queue.html), so it can't be used the way you're using it here, unfortunately.
author_association: NONE
user: login: stankudrow
created_at: 2025-02-15T13:27:41Z
body: > @stankudrow `asyncio.Queue` is not threadsafe (see the first line of https://docs.python.org/3/library/asyncio-queue.html), so it can't be used the way you're using it here, unfortunately.

Thank you for the tip, I find it highly valuable. Did you mean that data races are possible because the `asyncio.Queue` is not thread-safe and, regardless a queue.is instantiated within a `walk` coroutine, the nested `_walk` function uses the queue and is run in a separate thread, right?

I need a use case reproducing a problem because of this thread-unsafe nature of the `asyncio.Queue`. Could you help, please?

UPD: there ase thread-safe versions of [queue ADT](https://docs.python.org/3/library/queue.html), but this implementation deals with one consumer created each time the coroutine object gets instantiated. Is this the case?
author_association: OWNER
user: login: stankudrow
created_at: 2025-04-12T07:28:11Z
body: @Tinche , hello, I hope you are doing well.

Now a thread-safe `Queue` is in under the hood, I also add a test on checking the implementation is not blocking and still working as expected.

For now, I don't want to write a standalone `AsyncIOGenerator` as a wrapper for a sync iterative I/O utilities, I think it would be nice to do it later on demand.
author_association: OWNER
user: login: stankudrow
created_at: 2025-04-19T16:34:21Z
body: > @Tinche , hello, I hope you are doing well.
> 
> Now a thread-safe `Queue` is in under the hood, I also add a test on checking the implementation is not blocking and still working as expected.
> 
> For now, I don't want to write a standalone `AsyncIOGenerator` as a wrapper for a sync iterative I/O utilities, I think it would be nice to do it later on demand.

@Tinche , @Martin005, are you happy with this PR? If so, it will be merged into this one ( https://github.com/Tinche/aiofiles/pull/193 ) and will affect the PR into the `aiofiles` project.

I appreciate your time and I hope this work will be beneficial for the project itself. Feel free to share your ideas and propose commits - any coauthoring is welcome. Thank you.
author_association: OWNER
user: login: stankudrow
created_at: 2025-04-19T20:07:55Z
body: Stabilised, ready for reviewing.
author_association: OWNER