Repository: huggingface/datasets
Pull request: Add imagefolder dataset
A generic imagefolder dataset inspired by `torchvision.datasets.ImageFolder`. 

Resolves #2508 

---

Example Usage:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/954fa8cba4ff806f6147a782fa9efd1a/imagefolder-official-example.ipynb)

Problem statement:
Load Image Classification Dataset from Local 
**Is your feature request related to a problem? Please describe.**
Yes - we would like to load an image classification dataset with datasets without having to write a custom data loader.

**Describe the solution you'd like**

Given a folder structure with images of each class in each folder, the ability to load these folders into a HuggingFace dataset like "cifar10".

**Describe alternatives you've considered**

Implement ViT training outside of the HuggingFace Trainer and without datasets (we did this but prefer to stay on the main path)

Write custom data loader logic

**Additional context**

We're training ViT on custom dataset

Hints from issue discussion:
Hi ! Is this folder structure a standard, a bit like imagenet ?
In this case maybe we can consider having a dataset loader for cifar-like, imagenet-like, squad-like, conll-like etc. datasets ?
```python
from datasets import load_dataset

my_custom_cifar = load_dataset("cifar_like", data_dir="path/to/data/dir")
```

Let me know what you think
Yep that would be sweet - closing for now as we found a workaround. 
@lhoestq I think we'll want a generic `image-folder` dataset (same as 'imagenet-like'). This is like `torchvision.datasets.ImageFolder`, and is something vision folks are used to seeing.
Opening this back up, since I'm planning on tackling this. Already posted a quick version of it on my account on the hub.

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
```

Commit message: :lipstick: style

Patch to review:
diff --git a/datasets/imagefolder/README.md b/datasets/imagefolder/README.md
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/datasets/imagefolder/dummy/0.0.0/dummy_data.zip b/datasets/imagefolder/dummy/0.0.0/dummy_data.zip
new file mode 100644
index 00000000000..8ae06248877
Binary files /dev/null and b/datasets/imagefolder/dummy/0.0.0/dummy_data.zip differ
diff --git a/src/datasets/packaged_modules/__init__.py b/src/datasets/packaged_modules/__init__.py
index 097aa5d3852..71e7e47946d 100644
--- a/src/datasets/packaged_modules/__init__.py
+++ b/src/datasets/packaged_modules/__init__.py
@@ -4,6 +4,7 @@
 from typing import List
 
 from .csv import csv
+from .imagefolder import imagefolder
 from .json import json
 from .pandas import pandas
 from .parquet import parquet
@@ -30,4 +31,5 @@ def hash_python_lines(lines: List[str]) -> str:
     "pandas": (pandas.__name__, hash_python_lines(inspect.getsource(pandas).splitlines())),
     "parquet": (parquet.__name__, hash_python_lines(inspect.getsource(parquet).splitlines())),
     "text": (text.__name__, hash_python_lines(inspect.getsource(text).splitlines())),
+    "imagefolder": (imagefolder.__name__, hash_python_lines(inspect.getsource(imagefolder).splitlines())),
 }
diff --git a/src/datasets/packaged_modules/imagefolder/__init__.py b/src/datasets/packaged_modules/imagefolder/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/src/datasets/packaged_modules/imagefolder/imagefolder.py b/src/datasets/packaged_modules/imagefolder/imagefolder.py
new file mode 100644
index 00000000000..fa9dae5e677
--- /dev/null
+++ b/src/datasets/packaged_modules/imagefolder/imagefolder.py
@@ -0,0 +1,53 @@
+from dataclasses import dataclass
+from pathlib import Path
+from typing import List, Optional
+
+import pyarrow as pa
+
+import datasets
+
+
+logger = datasets.utils.logging.get_logger(__name__)
+
+
+@dataclass
+class ImageFolderConfig(datasets.BuilderConfig):
+    """BuilderConfig for JSON."""
+
+    features: Optional[datasets.Features] = None
+
+    @property
+    def schema(self):
+        return pa.schema(self.features.type) if self.features is not None else None
+
+
+class ImageFolder(datasets.GeneratorBasedBuilder):
+
+    BUILDER_CONFIG_CLASS = ImageFolderConfig
+
+    def _info(self):
+        return datasets.DatasetInfo(features=self.config.features)
+
+    def _split_generators(self, dl_manager):
+        if not self.config.data_files:
+            raise ValueError(f"At least one data file must be specified, but got data_files={self.config.data_files}")
+
+        data_files = dl_manager.download_and_extract(self.config.data_files)
+        if isinstance(data_files, (str, list, tuple)):
+            files = data_files
+            if isinstance(files, str):
+                files = [files]
+            return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"archive_path": data_files})]
+        splits = []
+        for split_name, files in data_files.items():
+            if isinstance(files, str):
+                files = [files]
+            splits.append(datasets.SplitGenerator(name=split_name, gen_kwargs={"archive_path": files}))
+        return splits
+
+    def _generate_examples(self, archive_path):
+        logger.info("generating examples from = %s", archive_path)
+        extensions = {".jpg", ".jpeg", ".png", ".ppm", ".bmp", ".pgm", ".tif", ".tiff", ".webp"}
+        for i, path in enumerate(Path(archive_path).glob("**/*")):
+            if path.suffix in extensions:
+                yield i, {"file": path.as_posix(), "labels": path.parent.name.lower()}
diff --git a/tests/test_dataset_common.py b/tests/test_dataset_common.py
index 1738e766cdb..d5701e94394 100644
--- a/tests/test_dataset_common.py
+++ b/tests/test_dataset_common.py
@@ -75,11 +75,18 @@ def wrapper(self, dataset_name):
 
 
 def get_packaged_dataset_dummy_data_files(dataset_name, path_to_dummy_data):
-    extensions = {"text": "txt", "json": "json", "pandas": "pkl", "csv": "csv", "parquet": "parquet"}
+    extensions = {
+        "text": ".txt",
+        "json": ".json",
+        "pandas": ".pkl",
+        "csv": ".csv",
+        "parquet": ".parquet",
+        "imagefolder": "/",
+    }
     return {
-        "train": os.path.join(path_to_dummy_data, "train." + extensions[dataset_name]),
-        "test": os.path.join(path_to_dummy_data, "test." + extensions[dataset_name]),
-        "dev": os.path.join(path_to_dummy_data, "dev." + extensions[dataset_name]),
+        "train": os.path.join(path_to_dummy_data, "train" + extensions[dataset_name]),
+        "test": os.path.join(path_to_dummy_data, "test" + extensions[dataset_name]),
+        "dev": os.path.join(path_to_dummy_data, "dev" + extensions[dataset_name]),
     }
 
 


Code review comments:
diff_hunk: @@ -0,0 +1,53 @@
+from dataclasses import dataclass
+from pathlib import Path
+from typing import List, Optional
+
+import pyarrow as pa
+
+import datasets
+
+
+logger = datasets.utils.logging.get_logger(__name__)
+
+
+@dataclass
+class ImageFolderConfig(datasets.BuilderConfig):
+    """BuilderConfig for JSON."""
path: src/datasets/packaged_modules/imagefolder/imagefolder.py
text: @author:
typo