Repository: python-trio/trio

Issue:
trio.Path.iterdir wrapping is broken
Given `pathlib.Path.iterdir` returns a generator that does IO access on each iteration, `trio.Path.iterdir` is currently broken given it currently only generates the generator asynchronously (which I suppose is pointless given there is no need for IO at generator creation)

The solution would be to modify  `trio.Path.iterdir` to return an async generator, however this means creating a special case given the current implementation is only an async wrapper on `pathlib.Path.iterdir`.


Discussion:
You're right, good catch.
Also this will require changing the signature – right now it's an async method that returns a synchronous iterator, and it needs to become a synchronous method that returns an async iterator.

I think we can just make the change if we do it soon – you're probably the first person to hit this, and the old behavior is simply wrong.
It feels like there's no real reason that the `listdir` call needs to wait until iteration starts, so it feels like the simplest solution would be to have `trio.Path.iterdir` be an async constructor that calls `listdir` in a worker thread and then returns a regular generator that yields the results.

Doing some fancy wrapping of the stdlib function feels like it'd be a lot trickier.

It would essentially entail something like the following, which was ripped from the stdlib almost verbatim:

```python
async def iterdir(self):
    func = partial(os.listdir, self._wrapped)
    files = await trio.run_sync_in_worker_thread(func)
    def gen():
        for name in files:
            if name in {'.', '..'}:
                continue
            yield self._wrapped._make_child_relpath(name)
            if self._wrapped._closed:
                self._wrapped._raise_closed()
    return gen()
```
Hmm, I assumed that the point of this function was that it didn't pull the whole directory listing into memory at once. Is that wrong?
No, it just calls `os.listdir` and then yields each element.  The following is what's in the stdlib:

```python
def iterdir(self):
    """Iterate over the files in this directory.  Does not yield any
    result for the special paths '.' and '..'.
    """
    if self._closed:
        self._raise_closed()
    for name in self._accessor.listdir(self):
        if name in {'.', '..'}:
            # Yielding a path object for these makes little sense
            continue
        yield self._make_child_relpath(name)
        if self._closed:
            self._raise_closed()
```

where `self._accessor.listdir` is just `os.listdir` unless you pass something else as the `pathlib.Path` template argument, and `os.listdir` just returns a plain list.
Anyway, so doing it this way _would_ change the semantics slightly as we'd be getting the directory listing when creating the generator object rather than when we later start iterating over it, but `pathlib.Path.iterdir` doesn't actually define the semantics of _when_ it actually gets the directory contents, it just happens to be that it currently gets it when you start iterating over it rather than when creating the generator.

(Furthermore, the `template` argument to `pathlib.Path` isn't actually documented, so anyone who would be trying to override the accessors to use a lazy generator rather than `os.list` would be depending on a non-public API.)
Hmm, tricky. The underlying operating systems do expose incremental reads for large directories. And Python even exposes it via `os.scandir`. So ideally in the future `iterdir` might use this. (Also, it'd be nice if pathlib exposed the richer `scandir` interface too: it gives you [`DirEntry`](https://docs.python.org/3/library/os.html#os.DirEntry) objects, which have more information that just the name.) So even if right now `iterdir` just naively calls `listdir` and loads up all the results at once, there's an argument that we should make this an async iterator for future-proofing, so that we have the option to switch to using incremental disk reads in the future.

There are some extra complications though:

* `os.scandir` holds open a file handle that has to be closed eventually. If `iterdir` used it, then users would have to be careful to close `iterdir` to avoid a `ResourceWarning` on CPython / fd leak on PyPI. Not necessarily a big deal, but adding this requirement later would be a pretty subtle change.

* The theoretical benefit of using an incremental OS API for reading the directory is that if you have a tremendously huge directory, we don't have to load the full list into memory all at once. But the naive version, where you keep calling `run_sync_in_worker_thread(scandir.__next__)`, will be *incredibly* slow, because you're adding a thread dispatch overhead for every file, even though the vast majority of them don't have to hit the disk. What you really want to do is amortize this by using a single call to `run_sync_in_worker_thread` to load a whole batch of results into memory all at once. Which... is basically what `iterdir` is already doing, with its naive use of `listdir`. In fact using `listdir` like this is probably optimal for 99.9% of cases, because it's extraordinarily rare that you have a directory listing that's so large that simply holding the list of names causes serious memory problems.

....So I think maybe I've convinced myself that `trio.Path.iterdir` should keep using `listdir`, and then potentially in the future we could imagine adding a `trio.Path.scandir` or something with a cleverer batching algorithm, `DirEntry` support, and an API that nudges you to close the `scandir` fd.

-----

Proposed TODO list for this bug

* [ ] Add a note to the docs for `trio.Path.iterdir`, warning people that despite having `iter` in the name it does load up the whole list into memory (by keeping the call signature the same, we're committing to never changing that, so it makes sense to document)

* [ ] Call `listdir` immediately before returning the iterator, like @kyrias suggested

* [ ] Fix another obvious bug while we're at it: currently `trio.Path.iterdir` yields `pathlib.Path` objects, not `trio.Path` objects :-)


Resolving patch:
diff --git a/trio/_path.py b/trio/_path.py
--- a/trio/_path.py
+++ b/trio/_path.py
@@ -128,6 +128,28 @@ def __init__(self, *args):
 
         self._wrapped = pathlib.Path(*args)
 
+    async def iterdir(self):
+        """
+        Like :meth:`pathlib.Path.iterdir`, but async.
+
+        This is an async method that returns a synchronous iterator, so you
+        use it like::
+        
+           for subpath in await mypath.iterdir():
+               ...
+               
+        Note that it actually loads the whole directory list into memory
+        immediately, during the initial call. (See `issue #501
+        <https://github.com/python-trio/trio/issues/501>`__ for discussion.)
+        
+        """
+
+        def _load_items():
+            return list(self._wrapped.iterdir())
+
+        items = await trio.run_sync_in_worker_thread(_load_items)
+        return (Path(item) for item in items)
+
     def __getattr__(self, name):
         if name in self._forward:
             value = getattr(self._wrapped, name)


Test patch:
diff --git a/trio/tests/test_path.py b/trio/tests/test_path.py
--- a/trio/tests/test_path.py
+++ b/trio/tests/test_path.py
@@ -198,3 +198,17 @@ async def test_path_nonpath():
 async def test_open_file_can_open_path(path):
     async with await trio.open_file(path, 'w') as f:
         assert f.name == fspath(path)
+
+
+async def test_iterdir(path):
+    # Populate a directory
+    await path.mkdir()
+    await (path / 'foo').mkdir()
+    await (path / 'bar.txt').write_bytes(b'')
+
+    entries = set()
+    for entry in await path.iterdir():
+        assert isinstance(entry, trio.Path)
+        entries.add(entry.name)
+
+    assert entries == {'bar.txt', 'foo'}