Repository: pytorch/vision

Issue:
Faster dataset indexing using ImageFolder and DatasetFolder
## 🚀 Feature

We can speed up the indexing of larger datasets by two lines in the `make_dataset` method within https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py.

This is the original code: `4.78 s ± 51.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`
```python
...
for root, _, fnames in sorted(os.walk(target_dir, followlinks=True)):
    for fname in sorted(fnames):
        # here we always join the path and then check whether the file is valid given the full string
        path = os.path.join(root, fname) 
        if is_valid_file(path):
            item = path, class_index
            instances.append(item)

            if target_class not in available_classes:
                available_classes.add(target_class)
```

Suggestion (this is pretty straightforward): `4.7 s ± 39.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`
```python
...
for root, _, fnames in sorted(os.walk(target_dir, followlinks=True)):
    for fname in sorted(fnames):
        if is_valid_file(fname): # we first check whether the fname is valid on the fname only --> then join
            path = os.path.join(root, fname)
            item = path, class_index
            instances.append(item)

            if target_class not in available_classes:
                available_classes.add(target_class)
```

## Motivation & Pitch

The speedup is only ~2% when working with ImageNet. And ~2% is not much. But given the huge number of PyTorch users, **we can make the world a slightly better place by saving unnecessary compute** :)



FYI: If we would switch from `os.walk` to `os.listdir` we could get a more significant speedup. I did a quick test where I dropped support for sub-subfolders (also tested on ImageNet):
`3.99 s ± 27.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`
```python
...
for fname in sorted(os.listdir(target_dir)):
        if is_valid_file(fname): 
            path = os.path.join(target_dir, fname)
            item = path, class_index
            instances.append(item)

            if target_class not in available_classes:
                available_classes.add(target_class)
```

cc @pmeier


Discussion:
Hey @IgorSusmelj, thanks for the analysis. As long as the proposal makes the code faster without making it less maintainable, patches are always appreciated. This is the case here! Would you like to send a PR?
Hi @pmeier, I can add a PR this week. I'll just do the small change. I guess the larger one (replacing `os.walk` with `os.listdir`) might have some unwanted implications when working with sub-sub directories. 
We need the multi-level support for BC so vanilla `os.listdir` is not going to cut it.

Resolving patch:
diff --git a/torchvision/datasets/folder.py b/torchvision/datasets/folder.py
--- a/torchvision/datasets/folder.py
+++ b/torchvision/datasets/folder.py
@@ -86,8 +86,8 @@ def is_valid_file(x: str) -> bool:
             continue
         for root, _, fnames in sorted(os.walk(target_dir, followlinks=True)):
             for fname in sorted(fnames):
-                path = os.path.join(root, fname)
-                if is_valid_file(path):
+                if is_valid_file(fname):
+                    path = os.path.join(root, fname)
                     item = path, class_index
                     instances.append(item)
 


Test patch: