Repository: huggingface/datasets
Pull request: Load Dataset from the Hub (NO DATASET SCRIPT)
## Load the data from any Dataset repository on the Hub

This PR adds support for loading datasets from any dataset repository on the hub, without requiring any dataset script.

As a user it's now possible to create a repo and upload some csv/json/text/parquet files, and then be able to load the data in one line. Here is an example with the `allenai/c4` repository that contains a lot of compressed json lines files:
```python
from datasets import load_dataset

data_files = {"train": "en/c4-train.*.json.gz"}
c4 = load_dataset("allenai/c4", data_files=data_files, split="train", streaming=True)

print(c4.n_shards)
# 1024
print(next(iter(c4)))
# {'text': 'Beginners BBQ Class Takin...'}
```

By default it loads all the files, but as shown in the example you can choose the ones you want with unix style patterns.

Of course it's still possible to use dataset scripts since they offer the most flexibility.

## Implementation details

It uses `huggingface_hub` to list the files in a dataset repository.

If you provide a path to a local directory instead of a repository name, it works the same way but it uses `glob`.

Depending on the data files available, or passed in the `data_files` parameter, one of the available builders will be used among the csv, json, text and parquet builders.

Because of this, it's not possible to load both csv and json files at once. In this case you have to load them separately and then concatenate the two datasets for example.

## TODO

- [x] tests
- [x] docs
- [x] when huggingface_hub gets a new release, update the CI and the setup.py

Close https://github.com/huggingface/datasets/issues/2629

Problem statement:
Load datasets from the Hub without requiring a dataset script
As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)

Hints from issue discussion:
This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉

Commit message: style

Patch to review:
diff --git a/docs/source/loading_datasets.rst b/docs/source/loading_datasets.rst
index 02bb5a8c7bc..f5392c57705 100644
--- a/docs/source/loading_datasets.rst
+++ b/docs/source/loading_datasets.rst
@@ -3,16 +3,16 @@ Loading a Dataset
 
 A :class:`datasets.Dataset` can be created from various sources of data:
 
-- from the `HuggingFace Hub <https://huggingface.co/datasets>`__,
-- from local files, e.g. CSV/JSON/text/pandas files, or
+- from the `Hugging Face Hub <https://huggingface.co/datasets>`__,
+- from local or remote files, e.g. CSV/JSON/text/parquet/pandas files, or
 - from in-memory data like python dict or a pandas dataframe.
 
 In this section we study each option.
 
-From the HuggingFace Hub
+From the Hugging Face Hub
 -------------------------------------------------
 
-Over 1,000 datasets for many NLP tasks like text classification, question answering, language modeling, etc, are provided on the `HuggingFace Hub <https://huggingface.co/datasets>`__ and can be viewed and explored online with the `🤗 Datasets viewer <https://huggingface.co/datasets/viewer>`__.
+Over 1,000 datasets for many NLP tasks like text classification, question answering, language modeling, etc, are provided on the `Hugging Face Hub <https://huggingface.co/datasets>`__ and can be viewed and explored online with the `🤗 Datasets viewer <https://huggingface.co/datasets/viewer>`__.
 
 .. note::
 
@@ -25,7 +25,7 @@ All the datasets currently available on the `Hub <https://huggingface.co/dataset
     >>> from datasets import list_datasets
     >>> datasets_list = list_datasets()
     >>> len(datasets_list)
-    1067
+    1103
     >>> print(', '.join(dataset for dataset in datasets_list))
     acronym_identification, ade_corpus_v2, adversarial_qa, aeslc, afrikaans_ner_corpus, ag_news, ai2_arc, air_dialogue, ajgt_twitter_ar,
     allegro_reviews, allocine, alt, amazon_polarity, amazon_reviews_multi, amazon_us_reviews, ambig_qa, amttl, anli, app_reviews, aqua_rat,
@@ -46,7 +46,7 @@ Let's load the **SQuAD dataset for Question Answering**. You can explore this da
 
 This call to :func:`datasets.load_dataset` does the following steps under the hood:
 
-1. Download and import in the library the **SQuAD python processing script** from HuggingFace github repository or AWS bucket if it's not already stored in the library.
+1. Download and import in the library the **SQuAD python processing script** from Hugging Face github repository or AWS bucket if it's not already stored in the library.
 
 .. note::
 
@@ -158,22 +158,52 @@ Apart from :obj:`name` and :obj:`split`, the :func:`datasets.load_dataset` metho
 
 The use of these arguments is discussed in the :ref:`load_dataset_cache_management` section below. You can also find the full details on these arguments on the package reference page for :func:`datasets.load_dataset`.
 
+From a community dataset on the Hugging Face Hub
+-----------------------------------------------------------
+
+The community shares hundreds of datasets on the Hugging Face Hub using **dataset repositories**.
+A dataset repository is a versioned repository of data files.
+Everyone can create a dataset repository on the Hugging Face Hub and upload their data.
+
+For example we have created a demo dataset at https://huggingface.co/datasets/lhoestq/demo1.
+In this dataset repository we uploaded some csv files, and you can load the dataset with:
+
+.. code-block::
+
+    >>> from datasets import load_dataset
+    >>> dataset = load_dataset('lhoestq/demo1')
+
+You can even choose which files to use in a dataset repository.
+For example you can load a subset of the **C4 dataset for language modeling**, hosted by AllenAI on the Hub.
+You can browse the dataset repository at https://huggingface.co/datasets/allenai/c4
+
+In the following example we specify which subset of the files to use with the ``data_files`` parameter:
+
+.. code-block::
+
+    >>> from datasets import load_dataset
+    >>> c4_subset = load_dataset('allenai/c4', data_files='en/c4-train.0000*-of-01024.json.gz')
+
+It will use all the files that match the pattern passed in ``data_files``.
+If you don't specify which data files to use, it will use all the data files (here all C4 is about 13TB of data).
+
 
 .. _loading-from-local-files:
 
-From local files
+From local or remote files
 -----------------------------------------------------------
 
-It's also possible to create a dataset from local files.
+It's also possible to create a dataset from your own local or remote files.
 
 Generic loading scripts are provided for:
 
 - CSV files (with the :obj:`csv` script),
 - JSON files (with the :obj:`json` script),
 - text files (read as a line-by-line dataset with the :obj:`text` script),
+- parquet files (with the :obj:`parquet` script).
 - pandas pickled dataframe (with the :obj:`pandas` script).
 
-If you want to control better how your files are loaded, or if you have a file format exactly reproducing the file format for one of the datasets provided on the `HuggingFace Hub <https://huggingface.co/datasets>`__, it can be more flexible and simpler to create **your own loading script**, from scratch or by adapting one of the provided loading scripts. In this case, please go check the :doc:`add_dataset` chapter.
+If you want to control better how your files are loaded, or if you have a file format exactly reproducing the file format for one of the datasets provided on the `Hugging Face Hub <https://huggingface.co/datasets>`__, it can be more flexible and simpler to create **your own loading script**, from scratch or by adapting one of the provided loading scripts. In this case, please go check the :doc:`add_dataset` chapter.
 
 The :obj:`data_files` argument in :func:`datasets.load_dataset` is used to provide paths to one or several files. This argument currently accepts three types of inputs:
 
@@ -190,6 +220,8 @@ Let's see an example of all the various ways you can provide files to :func:`dat
     >>> dataset = load_dataset('csv', data_files=['my_file_1.csv', 'my_file_2.csv', 'my_file_3.csv'])
     >>> dataset = load_dataset('csv', data_files={'train': ['my_train_file_1.csv', 'my_train_file_2.csv'],
                                                   'test': 'my_test_file.csv'})
+    >>> base_url = 'https://huggingface.co/datasets/lhoestq/demo1/resolve/main/data/'
+    >>> dataset = load_dataset('csv', data_files={'train': base_url + 'train.csv', 'test': base_url + 'test.csv'})
 
 .. note::
 
@@ -218,6 +250,13 @@ Here is an example loading two CSV file to create a ``train`` split (default spl
     >>> from datasets import load_dataset
     >>> dataset = load_dataset('csv', data_files=['my_file_1.csv', 'my_file_2.csv'])
 
+You can also provide the URLs of remote csv files:
+
+.. code-block::
+
+    >>> from datasets import load_dataset
+    >>> dataset = load_dataset('csv', data_files="https://huggingface.co/datasets/lhoestq/demo1/resolve/main/data/train.csv")
+
 The ``csv`` loading script provides a few simple access options to control parsing and reading the CSV files:
 
     - :obj:`skiprows` (int) - Number of first rows in the file to skip (default is 0)
@@ -226,12 +265,6 @@ The ``csv`` loading script provides a few simple access options to control parsi
     - :obj:`quotechar` (1-character string) – The character used optionally for quoting CSV values (default ``"``).
     - :obj:`quoting` (int) – Control quoting behavior (default 0, setting this to 3 disables quoting, refer to `pandas.read_csv documentation <https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html>` for more details).
 
-If you want more control, the ``csv`` script provides full control on reading, parsing and converting through the Apache Arrow `pyarrow.csv.ReadOptions <https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html>`__, `pyarrow.csv.ParseOptions <https://arrow.apache.org/docs/python/generated/pyarrow.csv.ParseOptions.html>`__ and `pyarrow.csv.ConvertOptions <https://arrow.apache.org/docs/python/generated/pyarrow.csv.ConvertOptions.html>`__
-
-    - :obj:`read_options` — Can be provided with a `pyarrow.csv.ReadOptions <https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html>`__ to control all the reading options. If :obj:`skiprows`, :obj:`column_names` or :obj:`autogenerate_column_names` are also provided (see above), they will take priority over the attributes in :obj:`read_options`.
-    - :obj:`parse_options` — Can be provided with a `pyarrow.csv.ParseOptions <https://arrow.apache.org/docs/python/generated/pyarrow.csv.ParseOptions.html>`__ to control all the parsing options. If :obj:`delimiter` or :obj:`quote_char` are also provided (see above), they will take priority over the attributes in :obj:`parse_options`.
-    - :obj:`convert_options` — Can be provided with a `pyarrow.csv.ConvertOptions <https://arrow.apache.org/docs/python/generated/pyarrow.csv.ConvertOptions.html>`__ to control all the conversion options.
-
 
 JSON files
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -258,6 +291,13 @@ You can load such a dataset direcly with:
     >>> from datasets import load_dataset
     >>> dataset = load_dataset('json', data_files='my_file.json')
 
+You can also provide the URLs of remote JSON files:
+
+.. code-block::
+
+    >>> from datasets import load_dataset
+    >>> dataset = load_dataset('json', data_files='https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz')
+
 In real-life though, JSON files can have diverse format and the ``json`` script will accordingly fallback on using python JSON loading methods to handle various JSON file format.
 
 One common occurence is to have a JSON file with a single root dictionary where the dataset is contained in a specific field, as a list of dicts or a dict of lists.
@@ -289,6 +329,13 @@ This is simply done using the ``text`` loading script which will generate a data
     >>> from datasets import load_dataset
     >>> dataset = load_dataset('text', data_files={'train': ['my_text_1.txt', 'my_text_2.txt'], 'test': 'my_test_file.txt'})
 
+You can also provide the URLs of remote text files:
+
+.. code-block::
+
+    >>> from datasets import load_dataset
+    >>> dataset = load_dataset('text', data_files={'train': 'https://huggingface.co/datasets/lhoestq/test/resolve/main/some_text.txt'})
+
 
 Specifying the features of the dataset
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -465,8 +512,8 @@ For example, run the following to skip integrity verifications when loading the
 Loading datasets offline
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Each dataset builder (e.g. "squad") is a python script that is downloaded and cached either from the 🤗 Datasets GitHub repository or from the `HuggingFace Hub <https://huggingface.co/datasets>`__.
-Only the ``text``, ``csv``, ``json`` and ``pandas`` builders are included in ``datasets`` without requiring external downloads.
+Each dataset builder (e.g. "squad") is a python script that is downloaded and cached either from the 🤗 Datasets GitHub repository or from the `Hugging Face Hub <https://huggingface.co/datasets>`__.
+Only the ``text``, ``csv``, ``json``, ``parquet`` and ``pandas`` builders are included in ``datasets`` without requiring external downloads.
 
 Therefore if you don't have an internet connection you can't load a dataset that is not packaged with ``datasets``, unless the dataset is already cached.
 Indeed, if you've already loaded the dataset once before (when you had an internet connection), then the dataset is reloaded from the cache and you can use it offline.
diff --git a/setup.py b/setup.py
index b458ef1b3e0..3051e103e7a 100644
--- a/setup.py
+++ b/setup.py
@@ -85,7 +85,7 @@
     # minimum 2021.05.0 to have the AbstractArchiveFileSystem
     "fsspec>=2021.05.0",
     # To get datasets from the Datasets Hub on huggingface.co
-    "huggingface_hub<0.1.0",
+    "huggingface_hub>=0.0.14,<0.1.0",
     # Utilities from PyPA to e.g., compare versions
     "packaging",
 ]
diff --git a/src/datasets/builder.py b/src/datasets/builder.py
index 4c31b9abe28..b8fb286ae2b 100644
--- a/src/datasets/builder.py
+++ b/src/datasets/builder.py
@@ -25,10 +25,12 @@
 import urllib
 from dataclasses import dataclass
 from functools import partial
+from pathlib import PurePath
 from typing import Dict, List, Optional, Tuple, Union
 
 from datasets.features import Features
 from datasets.utils.mock_download_manager import MockDownloadManager
+from datasets.utils.py_utils import map_nested
 
 from . import config, utils
 from .arrow_dataset import Dataset
@@ -42,7 +44,7 @@
 from .splits import Split, SplitDict, SplitGenerator
 from .utils import logging
 from .utils.download_manager import DownloadManager, GenerateMode
-from .utils.file_utils import DownloadConfig, is_remote_url, request_etags
+from .utils.file_utils import DownloadConfig, is_relative_path, is_remote_url, request_etags, url_or_path_join
 from .utils.filelock import FileLock
 from .utils.info_utils import get_size_checksum_dict, verify_checksums, verify_splits
 
@@ -100,6 +102,7 @@ def create_config_id(
         config_kwargs: dict,
         custom_features: Optional[Features] = None,
         use_auth_token: Optional[Union[bool, str]] = None,
+        base_path: Optional[Union[bool, str]] = None,
     ) -> str:
         """
         The config id is used to build the cache directory.
@@ -154,6 +157,12 @@ def create_config_id(
                 }
             else:
                 raise ValueError("Please provide a valid `data_files` in `DatasetBuilder`")
+
+            def abspath(data_file) -> str:
+                data_file = data_file.as_posix() if isinstance(data_file, PurePath) else str(data_file)
+                return url_or_path_join(base_path, data_file) if is_relative_path(data_file) else data_file
+
+            data_files: Dict[str, List[str]] = map_nested(abspath, data_files)
             remote_urls = [
                 data_file for key in data_files for data_file in data_files[key] if is_remote_url(data_file)
             ]
@@ -376,7 +385,10 @@ def _create_builder_config(self, name=None, custom_features=None, **config_kwarg
 
         # compute the config id that is going to be used for caching
         config_id = builder_config.create_config_id(
-            config_kwargs, custom_features=custom_features, use_auth_token=self.use_auth_token
+            config_kwargs,
+            custom_features=custom_features,
+            use_auth_token=self.use_auth_token,
+            base_path=self.base_path if self.base_path is not None else "",
         )
         is_custom = config_id not in self.builder_configs
         if is_custom:
@@ -575,12 +587,17 @@ def incomplete_dir(dirname):
             # Print is intentional: we want this to always go to stdout so user has
             # information needed to cancel download/preparation if needed.
             # This comes right before the progress bar.
-            print(
-                f"Downloading and preparing dataset {self.info.builder_name}/{self.info.config_name} "
-                f"(download: {utils.size_str(self.info.download_size)}, generated: {utils.size_str(self.info.dataset_size)}, "
-                f"post-processed: {utils.size_str(self.info.post_processing_size)}, "
-                f"total: {utils.size_str(self.info.size_in_bytes)}) to {self._cache_dir}..."
-            )
+            if self.info.size_in_bytes:
+                print(
+                    f"Downloading and preparing dataset {self.info.builder_name}/{self.info.config_name} "
+                    f"(download: {utils.size_str(self.info.download_size)}, generated: {utils.size_str(self.info.dataset_size)}, "
+                    f"post-processed: {utils.size_str(self.info.post_processing_size)}, "
+                    f"total: {utils.size_str(self.info.size_in_bytes)}) to {self._cache_dir}..."
+                )
+            else:
+                print(
+                    f"Downloading and preparing dataset {self.info.builder_name}/{self.info.config_name} to {self._cache_dir}..."
+                )
 
             if self.manual_download_instructions is not None:
                 assert (
@@ -902,8 +919,8 @@ def as_streaming_dataset(
             raise ValueError(f"Builder {self.name} is not streamable.")
         if not config.AIOHTTP_AVAILABLE:
             raise ImportError(
-                f"To be able to use dataset streaming, you need to install dependencies like aiohttp "
-                f'using "pip install \'datasets[streaming]\'" or "pip install aiohttp" for instance'
+                "To be able to use dataset streaming, you need to install dependencies like aiohttp "
+                'using \'datasets[streaming]\'" or "pip install aiohttp" for instance'
             )
 
         from .utils.streaming_download_manager import StreamingDownloadManager
diff --git a/src/datasets/config.py b/src/datasets/config.py
index 5894b2d1832..69d593d6418 100644
--- a/src/datasets/config.py
+++ b/src/datasets/config.py
@@ -21,7 +21,8 @@
 REPO_METRICS_URL = "https://raw.githubusercontent.com/huggingface/datasets/{version}/metrics/{path}/{name}"
 
 # Hub
-HUB_DATASETS_URL = "https://huggingface.co/datasets/{path}/resolve/{version}/{name}"
+HF_ENDPOINT = os.environ.get("HF_ENDPOINT", "https://huggingface.co")
+HUB_DATASETS_URL = HF_ENDPOINT + "/datasets/{path}/resolve/{version}/{name}"
 HUB_DEFAULT_VERSION = "main"
 
 PY_VERSION: str = sys.version.split()[0]
diff --git a/src/datasets/load.py b/src/datasets/load.py
index a0b43f54332..42fc340b84c 100644
--- a/src/datasets/load.py
+++ b/src/datasets/load.py
@@ -16,6 +16,7 @@
 # Lint as: python3
 """Access datasets."""
 import filecmp
+import glob
 import importlib
 import inspect
 import json
@@ -23,11 +24,16 @@
 import re
 import shutil
 import time
-from pathlib import Path
-from typing import Dict, List, Optional, Tuple, Type, Union
+from pathlib import Path, PurePath
+from typing import Counter, Dict, List, Optional, Tuple, Type, Union
 from urllib.parse import urlparse
 
 import fsspec
+import huggingface_hub
+from huggingface_hub import HfApi
+
+from datasets.naming import camelcase_to_snakecase
+from datasets.utils.py_utils import NestedDataStructure
 
 from . import config
 from .arrow_dataset import Dataset
@@ -37,7 +43,7 @@
 from .filesystems import extract_path_from_uri, is_remote_filesystem
 from .iterable_dataset import IterableDataset
 from .metric import Metric
-from .packaged_modules import _PACKAGED_DATASETS_MODULES, hash_python_lines
+from .packaged_modules import _EXTENSION_TO_MODULE, _PACKAGED_DATASETS_MODULES, hash_python_lines
 from .splits import Split
 from .tasks import TaskTemplate
 from .utils.download_manager import GenerateMode
@@ -45,10 +51,11 @@
     DownloadConfig,
     cached_path,
     head_hf_s3,
-    hf_bucket_url,
     hf_github_url,
     hf_hub_url,
     init_hf_modules,
+    is_relative_path,
+    is_remote_url,
     relative_to_absolute_path,
     url_or_path_join,
     url_or_path_parent,
@@ -221,6 +228,120 @@ def get_imports(file_path: str):
     return imports
 
 
+def _resolve_data_files_locally_or_by_urls(
+    base_path: str, patterns: Union[str, List[str], Dict], allowed_extensions: Optional[list] = None
+) -> Union[List[Path], Dict]:
+    """
+    Return the absolute paths to all the files that match the given patterns.
+    It also supports absolute paths in patterns.
+    If an URL is passed, it is returned as is."""
+    data_files_ignore = ["README.md", "config.json"]
+    if isinstance(patterns, str):
+        if is_remote_url(patterns):
+            return [patterns]
+        if is_relative_path(patterns):
+            glob_iter = list(Path(base_path).rglob(patterns))
+        else:
+            glob_iter = [Path(filepath) for filepath in glob.glob(patterns)]
+
+        matched_paths = [
+            filepath.resolve()
+            for filepath in glob_iter
+            if filepath.name not in data_files_ignore and not filepath.name.startswith(".") and filepath.is_file()
+        ]
+        if allowed_extensions is not None:
+            out = [
+                filepath
+                for filepath in matched_paths
+                if any(suffix[1:] in allowed_extensions for suffix in filepath.suffixes)
+            ]
+            if len(out) < len(matched_paths):
+                invalid_matched_files = list(set(matched_paths) - set(out))
+                logger.info(
+                    f"Some files matched the pattern '{patterns}' at {Path(base_path).resolve()} but don't have valid data file extensions: {invalid_matched_files}"
+                )
+        else:
+            out = matched_paths
+        if not out:
+            error_msg = f"Unable to resolve any data file that matches '{patterns}' at {Path(base_path).resolve()}"
+            if allowed_extensions is not None:
+                error_msg += f" with any supported extension {list(allowed_extensions)}"
+            raise FileNotFoundError(error_msg)
+        return out
+    elif isinstance(patterns, dict):
+        return {
+            k: _resolve_data_files_locally_or_by_urls(base_path, v, allowed_extensions=allowed_extensions)
+            for k, v in patterns.items()
+        }
+    else:
+        return sum(
+            [
+                _resolve_data_files_locally_or_by_urls(base_path, pattern, allowed_extensions=allowed_extensions)
+                for pattern in patterns
+            ],
+            [],
+        )
+
+
+def _resolve_data_files_in_dataset_repository(
+    dataset_info: huggingface_hub.hf_api.DatasetInfo,
+    patterns: Union[str, List[str], Dict],
+    allowed_extensions: Optional[list] = None,
+) -> Union[List[PurePath], Dict]:
+    data_files_ignore = ["README.md", "config.json"]
+    if isinstance(patterns, str):
+        all_data_files = [
+            PurePath("/" + dataset_file.rfilename) for dataset_file in dataset_info.siblings
+        ]  # add a / at the beginning to make the pattern **/* match files at the root
+        matched_paths = [
+            filepath.relative_to("/")
+            for filepath in all_data_files
+            if filepath.name not in data_files_ignore
+            and not filepath.name.startswith(".")
+            and filepath.match(patterns)
+        ]
+        if allowed_extensions is not None:
+            out = [
+                filepath
+                for filepath in matched_paths
+                if any(suffix[1:] in allowed_extensions for suffix in filepath.suffixes)
+            ]
+            if len(out) < len(matched_paths):
+                invalid_matched_files = list(set(matched_paths) - set(out))
+                logger.info(
+                    f"Some files matched the pattern {patterns} in dataset repository {dataset_info.id} but don't have valid data file extensions: {invalid_matched_files}"
+                )
+        else:
+            out = matched_paths
+        if not out:
+            error_msg = f"Unable to resolve data_file {patterns} in dataset repository {dataset_info.id}"
+            if allowed_extensions is not None:
+                error_msg += f" with any supported extension {list(allowed_extensions)}"
+            raise FileNotFoundError(error_msg)
+        return out
+    elif isinstance(patterns, dict):
+        return {
+            k: _resolve_data_files_in_dataset_repository(dataset_info, v, allowed_extensions=allowed_extensions)
+            for k, v in patterns.items()
+        }
+    else:
+        return sum(
+            [
+                _resolve_data_files_in_dataset_repository(dataset_info, pattern, allowed_extensions=allowed_extensions)
+                for pattern in patterns
+            ],
+            [],
+        )
+
+
+def _infer_module_for_data_files(data_files: Union[PurePath, List[PurePath], Dict]) -> Optional[str]:
+    extensions_counter = Counter(
+        suffix[1:] for filepath in NestedDataStructure(data_files).flatten() for suffix in filepath.suffixes
+    )
+    if extensions_counter:
+        return _EXTENSION_TO_MODULE[extensions_counter.most_common(1)[0][0]]
+
+
 def prepare_module(
     path: str,
     script_version: Optional[Union[str, Version]] = None,
@@ -230,6 +351,8 @@ def prepare_module(
     force_local_path: Optional[str] = None,
     dynamic_modules_path: Optional[str] = None,
     return_resolved_file_path: bool = False,
+    return_associated_base_path: bool = False,
+    data_files: Optional[Union[Dict, List, str]] = None,
     **download_kwargs,
 ) -> Union[Tuple[str, str], Tuple[str, str, Optional[str]]]:
     r"""
@@ -240,10 +363,30 @@ def prepare_module(
 
     Args:
 
-        path (str):
-            path to the dataset or metric script, can be either:
-                - a path to a local directory containing the dataset processing python script
-                - an url to a github or S3 directory with a dataset processing python script
+        path (str): Path or name of the dataset, or path to a metric script.
+            Depending on ``path``, the module that is returned id either generic moduler (csv, json, text etc.) or a module defined defined a dataset or metric script (a python file).
+
+            For local datasets:
+
+            - if ``path`` is a local directory (but doesn't contain a dataset script)
+              -> load a generic module (csv, json, text etc.) based on the content of the directory
+              e.g. ``'./path/to/directory/with/my/csv/data'``.
+            - if ``path`` is a local dataset or metric script or a directory containing a local dataset script (if the script has the same name as the directory):
+              -> load the module from the dataset or metric script
+              e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``.
+
+            For datasets on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
+
+            - if ``path`` is a canonical dataset or metric on the HF Hub (ex: `glue`, `squad`)
+              -> load the module from the dataset or metric script in the github repository at huggingface/datasets
+              e.g. ``'squad'`` or ``'glue'`` or ``accuracy``.
+            - if ``path`` is a dataset repository on the HF hub (without a dataset script)
+              -> load a generic module (csv, text etc.) based on the content of the repository
+              e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing your data files.
+            - if ``path`` is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
+              -> load the module from the dataset script in the dataset repository
+              e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing a dataset script `'dataset_name.py'`.
+
         script_version (Optional ``Union[str, datasets.Version]``):
             If specified, the module will be loaded from the datasets repository at this version.
             By default:
@@ -260,6 +403,10 @@ def prepare_module(
             By default the datasets and metrics are stored inside the `datasets_modules` module.
         return_resolved_file_path (Optional bool, defaults to False):
             If True, the url or path to the resolved dataset or metric script is returned with the other ouputs
+        return_associated_base_path (Optional bool, defaults to False):
+            If True, the base path associated to the dataset is returned with the other ouputs.
+            It corresponds to the directory or base url where the dataset script/dataset repo is at.
+        data_files (:obj:`Union[Dict, List, str]`, optional): Defining the data_files of the dataset configuration.
         download_kwargs: optional attributes for DownloadConfig() which will override the attributes in download_config if supplied.
 
     Return: Tuple[``str``, ``str``] with
@@ -282,15 +429,20 @@ def prepare_module(
     short_name = name[:-3]
 
     # first check if the module is packaged with the `datasets` package
-    if dataset and path in _PACKAGED_DATASETS_MODULES:
+    def prepare_packaged_module(name):
         try:
-            head_hf_s3(path, filename=name, dataset=dataset, max_retries=download_config.max_retries)
+            head_hf_s3(name, filename=name + ".py", dataset=dataset, max_retries=download_config.max_retries)
         except Exception:
-            logger.debug(f"Couldn't head HF s3 for packaged dataset module '{path}'. Running in offline mode.")
-        module_path, hash = _PACKAGED_DATASETS_MODULES[path]
+            logger.debug(f"Couldn't head HF s3 for packaged dataset module '{name}'. Running in offline mode.")
+        return _PACKAGED_DATASETS_MODULES[name]
+
+    if dataset and path in _PACKAGED_DATASETS_MODULES:
+        output = prepare_packaged_module(path)
         if return_resolved_file_path:
-            return module_path, hash, None
-        return module_path, hash
+            output += (None,)
+        if return_associated_base_path:
+            output += (None,)
+        return output
 
     # otherwise the module is added to the dynamic modules
     dynamic_modules_path = dynamic_modules_path if dynamic_modules_path else init_dynamic_modules()
@@ -305,22 +457,52 @@ def prepare_module(
     else:
         main_folder_path = force_local_path
 
-    # We have three ways to find the processing file:
-    # - if os.path.join(path, name) is a file or a remote url
-    # - if path is a file or a remote url
-    # - otherwise we assume path/name is a path to our S3 bucket
-    combined_path = path if path.endswith(name) else os.path.join(path, name)
-
-    if os.path.isfile(combined_path):
+    # We have several ways to find the processing file:
+    # - if os.path.join(path, name) is a local python file
+    #   -> use the module from the python file
+    # - if path is a local directory (but no python file)
+    #   -> use a packaged module (csv, text etc.) based on content of the directory
+    # - if path has no "/" and is a module on github (in /datasets or in /metrics)
+    #   -> use the module from the python file on github
+    # - if path has one "/" and is dataset repository on the HF hub with a python file
+    #   -> the module from the python file in the dataset repository
+    # - if path has one "/" and is dataset repository on the HF hub without a python file
+    #   -> use a packaged module (csv, text etc.) based on content of the repository
+    resource_type = "dataset" if dataset else "metric"
+    combined_path = os.path.join(path, name)
+    if path.endswith(name):
+        if os.path.isfile(path):
+            file_path = path
+            local_path = path
+            base_path = os.path.dirname(path)
+        else:
+            raise FileNotFoundError(f"Couldn't find a {resource_type} script at {relative_to_absolute_path(path)}")
+    elif os.path.isfile(combined_path):
         file_path = combined_path
         local_path = combined_path
+        base_path = path
     elif os.path.isfile(path):
         file_path = path
         local_path = path
+        base_path = os.path.dirname(path)
+    elif os.path.isdir(path):
+        resolved_data_files = _resolve_data_files_locally_or_by_urls(
+            path, data_files or "*", allowed_extensions=_EXTENSION_TO_MODULE.keys()
+        )
+        infered_module_name = _infer_module_for_data_files(resolved_data_files)
+        if not infered_module_name:
+            raise FileNotFoundError(f"No data files or {resource_type} script found in local directory {path}")
+        output = prepare_packaged_module(infered_module_name)
+        if return_resolved_file_path:
+            output += (None,)
+        if return_associated_base_path:
+            output += (path,)
+        return output
     else:
-        # Try github (canonical datasets/metrics) and then S3 (users datasets/metrics)
+        # Try github (canonical datasets/metrics) and then HF Hub (community datasets)
 
         combined_path_abs = relative_to_absolute_path(combined_path)
+        expected_dir_for_combined_path_abs = os.path.dirname(combined_path_abs)
         try:
             head_hf_s3(path, filename=name, dataset=dataset, max_retries=download_config.max_retries)
             script_version = str(script_version) if script_version is not None else None
@@ -331,9 +513,8 @@ def prepare_module(
                 except FileNotFoundError:
                     if script_version is not None:
                         raise FileNotFoundError(
-                            "Couldn't find remote file with version {} at {}. Please provide a valid version and a valid {} name.".format(
-                                script_version, file_path, "dataset" if dataset else "metric"
-                            )
+                            f"Couldn't find a directory or a {resource_type} named '{path}' using version {script_version}. "
+                            f"It doesn't exist locally at {expected_dir_for_combined_path_abs} or remotely at {file_path}"
                         )
                     else:
                         github_file_path = file_path
@@ -341,36 +522,55 @@ def prepare_module(
                         try:
                             local_path = cached_path(file_path, download_config=download_config)
                             logger.warning(
-                                "Couldn't find file locally at {}, or remotely at {}.\n"
-                                "The file was picked from the master branch on github instead at {}.".format(
-                                    combined_path_abs, github_file_path, file_path
-                                )
+                                f"Couldn't find a directory or a {resource_type} named '{path}'. "
+                                f"It was picked from the master branch on github instead at {file_path}"
                             )
                         except FileNotFoundError:
                             raise FileNotFoundError(
-                                "Couldn't find file locally at {}, or remotely at {}.\n"
-                                "The file is also not present on the master branch on github.".format(
-                                    combined_path_abs, github_file_path
-                                )
+                                f"Couldn't find a directory or a {resource_type} named '{path}'. "
+                                f"It doesn't exist locally at {expected_dir_for_combined_path_abs} or remotely at {github_file_path}"
                             )
             elif path.count("/") == 1:  # users datasets/metrics: s3 path (hub for datasets and s3 for metrics)
-                if dataset:
-                    file_path = hf_hub_url(path=path, name=name, version=script_version)
-                else:
-                    file_path = hf_bucket_url(path, filename=name, dataset=False)
+                file_path = hf_hub_url(path=path, name=name, version=script_version)
+                if not dataset:
+                    # We don't have community metrics on the HF Hub
+                    raise FileNotFoundError(
+                        f"Couldn't find a {resource_type} in a directory at '{path}'. "
+                        f"It doesn't exist locally at {combined_path_abs}"
+                    )
                 try:
                     local_path = cached_path(file_path, download_config=download_config)
                 except FileNotFoundError:
-                    raise FileNotFoundError(
-                        "Couldn't find file locally at {}, or remotely at {}. Please provide a valid {} name.".format(
-                            combined_path_abs, file_path, "dataset" if dataset else "metric"
+                    hf_api = HfApi(config.HF_ENDPOINT)
+                    try:
+                        dataset_info = hf_api.dataset_info(
+                            repo_id=path, revision=script_version, token=download_config.use_auth_token
+                        )
+                    except Exception:
+                        raise FileNotFoundError(
+                            f"Couldn't find a directory or a {resource_type} named '{path}'. "
+                            f"It doesn't exist locally at {expected_dir_for_combined_path_abs} or remotely on {hf_api.endpoint}/datasets"
                         )
+                    resolved_data_files = _resolve_data_files_in_dataset_repository(
+                        dataset_info,
+                        data_files if data_files is not None else "*",
+                        allowed_extensions=_EXTENSION_TO_MODULE.keys(),
                     )
+                    infered_module_name = _infer_module_for_data_files(resolved_data_files)
+                    if not infered_module_name:
+                        raise FileNotFoundError(
+                            f"No data files found in dataset repository '{path}'. Local directory at {expected_dir_for_combined_path_abs} doesn't exist either."
+                        )
+                    output = prepare_packaged_module(infered_module_name)
+                    if return_resolved_file_path:
+                        output += (None,)
+                    if return_associated_base_path:
+                        output += (url_or_path_parent(file_path),)
+                    return output
             else:
                 raise FileNotFoundError(
-                    "Couldn't find file locally at {}. Please provide a valid {} name.".format(
-                        combined_path_abs, "dataset" if dataset else "metric"
-                    )
+                    f"Couldn't find a {resource_type} directory at '{path}'. "
+                    f"It doesn't exist locally at {expected_dir_for_combined_path_abs}"
                 )
         except Exception as e:  # noqa: all the attempts failed, before raising the error we should check if the module already exists.
             if os.path.isdir(main_folder_path):
@@ -389,11 +589,14 @@ def _get_modification_time(module_hash):
                         f"(last modified on {time.ctime(_get_modification_time(hash))}) since it "
                         f"couldn't be found locally at {combined_path_abs}, or remotely ({type(e).__name__})."
                     )
+                    output = (module_path, hash)
                     if return_resolved_file_path:
                         with open(os.path.join(main_folder_path, hash, short_name + ".json")) as cache_metadata:
                             file_path = json.load(cache_metadata)["original file path"]
-                        return module_path, hash, file_path
-                    return module_path, hash
+                        output += (file_path,)
+                    if return_associated_base_path:
+                        output += (url_or_path_parent(file_path),)
+                    return output
             raise
 
     # Load the module in two steps:
@@ -563,9 +766,12 @@ def _get_modification_time(module_hash):
     # make the new module to be noticed by the import system
     importlib.invalidate_caches()
 
+    output = (module_path, hash)
     if return_resolved_file_path:
-        return module_path, hash, file_path
-    return module_path, hash
+        output += (file_path,)
+    if return_associated_base_path:
+        output += (base_path,)
+    return output
 
 
 def load_metric(
@@ -635,7 +841,7 @@ def load_dataset_builder(
     path: str,
     name: Optional[str] = None,
     data_dir: Optional[str] = None,
-    data_files: Union[Dict, List] = None,
+    data_files: Optional[Union[Dict, List, str]] = None,
     cache_dir: Optional[str] = None,
     features: Optional[Features] = None,
     download_config: Optional[DownloadConfig] = None,
@@ -651,15 +857,34 @@ def load_dataset_builder(
 
     Args:
 
-        path (:obj:`str`): Path to the dataset processing script with the dataset builder. Can be either:
+        path (:obj:`str`): Path or name of the dataset.
+            Depending on ``path``, the dataset builder that is returned id either generic dataset builder (csv, json, text etc.) or a dataset builder defined defined a dataset script (a python file).
 
-            - a local path to processing script or the directory containing the script (if the script has the same name as the directory),
+            For local datasets:
+
+            - if ``path`` is a local directory (but doesn't contain a dataset script)
+              -> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
+              e.g. ``'./path/to/directory/with/my/csv/data'``.
+            - if ``path`` is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory):
+              -> load the dataset builder from the dataset script
               e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``.
-            - a dataset identifier in the HuggingFace Datasets Hub (list all available datasets and ids with ``datasets.list_datasets()``)
-              e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'``.
+
+            For datasets on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
+
+            - if ``path`` is a canonical dataset on the HF Hub (ex: `glue`, `squad`)
+              -> load the dataset builder from the dataset script in the github repository at huggingface/datasets
+              e.g. ``'squad'`` or ``'glue'``.
+            - if ``path`` is a dataset repository on the HF hub (without a dataset script)
+              -> load a generic dataset builder (csv, text etc.) based on the content of the repository
+              e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing your data files.
+            - if ``path`` is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
+              -> load the dataset builder from the dataset script in the dataset repository
+              e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing a dataset script `'dataset_name.py'`.
+
+
         name (:obj:`str`, optional): Defining the name of the dataset configuration.
         data_dir (:obj:`str`, optional): Defining the data_dir of the dataset configuration.
-        data_files (:obj:`str`, optional): Defining the data_files of the dataset configuration.
+        data_files (:obj:`Union[Dict, List, str]`, optional): Defining the data_files of the dataset configuration.
         cache_dir (:obj:`str`, optional): Directory to read/write data. Defaults to "~/datasets".
         features (:class:`Features`, optional): Set the features type to use for this dataset.
         download_config (:class:`~utils.DownloadConfig`, optional): Specific download configuration parameters.
@@ -678,24 +903,51 @@ def load_dataset_builder(
 
     """
     # Download/copy dataset processing script
-    module_path, hash, resolved_file_path = prepare_module(
+    module_path, hash, base_path = prepare_module(
         path,
         script_version=script_version,
         download_config=download_config,
         download_mode=download_mode,
         dataset=True,
-        return_resolved_file_path=True,
+        return_associated_base_path=True,
         use_auth_token=use_auth_token,
+        data_files=data_files,
     )
 
     # Get dataset builder class from the processing script
     builder_cls = import_main_class(module_path, dataset=True)
 
-    # Set the base path for downloads as the parent of the script location
-    if resolved_file_path is not None:
-        base_path = url_or_path_parent(resolved_file_path)
-    else:
-        base_path = None
+    # For packaged builder used to load data from a dataset repository or dataset directory (no dataset script)
+    if module_path.startswith("datasets.") and path not in _PACKAGED_DATASETS_MODULES:
+        # Add a nice name to the configuratiom
+        if name is None:
+            name = path.split("/")[-1].split(os.sep)[-1]
+        # Resolve the data files
+        allowed_extensions = [
+            extension
+            for extension in _EXTENSION_TO_MODULE
+            if _EXTENSION_TO_MODULE[extension] == camelcase_to_snakecase(builder_cls.__name__)
+        ]
+        data_files = data_files if data_files is not None else "*"
+        if base_path.startswith(config.HF_ENDPOINT):
+            dataset_info = HfApi(config.HF_ENDPOINT).dataset_info(path, revision=script_version, token=use_auth_token)
+            data_files = _resolve_data_files_in_dataset_repository(
+                dataset_info, data_files, allowed_extensions=allowed_extensions
+            )
+        else:  # local dir
+            data_files = _resolve_data_files_locally_or_by_urls(
+                path, data_files, allowed_extensions=allowed_extensions
+            )
+    elif path in _PACKAGED_DATASETS_MODULES:
+        if data_files is None:
+            error_msg = f"Please specify the data files to load for the {path} dataset builder."
+            example_extensions = [
+                extension for extension in _EXTENSION_TO_MODULE if _EXTENSION_TO_MODULE[extension] == path
+            ]
+            if example_extensions:
+                error_msg += f'\nFor example `data_files={{"train": "path/to/data/train/*.{example_extensions[0]}"}}`'
+            raise ValueError(error_msg)
+        data_files = _resolve_data_files_locally_or_by_urls(".", data_files)
 
     # Instantiate the dataset builder
     builder_instance: DatasetBuilder = builder_cls(
@@ -717,7 +969,7 @@ def load_dataset(
     path: str,
     name: Optional[str] = None,
     data_dir: Optional[str] = None,
-    data_files: Union[Dict, List] = None,
+    data_files: Optional[Union[Dict, List, str]] = None,
     split: Optional[Union[str, Split]] = None,
     cache_dir: Optional[str] = None,
     features: Optional[Features] = None,
@@ -755,17 +1007,38 @@ def load_dataset(
 
         3. Return a dataset built from the requested splits in ``split`` (default: all).
 
+    It also allows to load a dataset from a local directory or a dataset repository on the Hugging Face Hub without dataset script.
+    In this case, it automatically loads all the data files from the directory or the dataset repository.
+
     Args:
 
-        path (:obj:`str`): Path to the dataset processing script with the dataset builder. Can be either:
+        path (:obj:`str`): Path or name of the dataset.
+            Depending on ``path``, the dataset builder that is returned id either generic dataset builder (csv, json, text etc.) or a dataset builder defined defined a dataset script (a python file).
+
+            For local datasets:
 
-            - a local path to processing script or the directory containing the script (if the script has the same name as the directory),
+            - if ``path`` is a local directory (but doesn't contain a dataset script)
+              -> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
+              e.g. ``'./path/to/directory/with/my/csv/data'``.
+            - if ``path`` is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory):
+              -> load the dataset builder from the dataset script
               e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'``.
-            - a dataset identifier in the HuggingFace Datasets Hub (list all available datasets and ids with ``datasets.list_datasets()``)
-              e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'``.
+
+            For datasets on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``)
+
+            - if ``path`` is a canonical dataset on the HF Hub (ex: `glue`, `squad`)
+              -> load the dataset builder from the dataset script in the github repository at huggingface/datasets
+              e.g. ``'squad'`` or ``'glue'``.
+            - if ``path`` is a dataset repository on the HF hub (without a dataset script)
+              -> load a generic dataset builder (csv, text etc.) based on the content of the repository
+              e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing your data files.
+            - if ``path`` is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
+              -> load the dataset builder from the dataset script in the dataset repository
+              e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing a dataset script `'dataset_name.py'`.
+
         name (:obj:`str`, optional): Defining the name of the dataset configuration.
         data_dir (:obj:`str`, optional): Defining the data_dir of the dataset configuration.
-        data_files (:obj:`str`, optional): Defining the data_files of the dataset configuration.
+        data_files (:obj:`Union[Dict, List, str]`, optional): Defining the data_files of the dataset configuration.
         split (:class:`Split` or :obj:`str`): Which split of the data to load.
             If None, will return a `dict` with all splits (typically `datasets.Split.TRAIN` and `datasets.Split.TEST`).
             If given, will return a single Dataset.
@@ -810,23 +1083,22 @@ def load_dataset(
     if streaming:
         if not config.AIOHTTP_AVAILABLE:
             raise ImportError(
-                f"To be able to use dataset streaming, you need to install dependencies like aiohttp "
-                f'using "pip install \'datasets[streaming]\'" or "pip install aiohttp" for instance'
+                "To be able to use dataset streaming, you need to install dependencies like aiohttp "
+                'using "pip install \'datasets[streaming]\'" or "pip install aiohttp" for instance'
             )
-    # Download/copy dataset processing script
 
     # Create a dataset builder
     builder_instance = load_dataset_builder(
-        path,
-        name,
-        data_dir,
-        data_files,
-        cache_dir,
-        features,
-        download_config,
-        download_mode,
-        script_version,
-        use_auth_token,
+        path=path,
+        name=name,
+        data_dir=data_dir,
+        data_files=data_files,
+        cache_dir=cache_dir,
+        features=features,
+        download_config=download_config,
+        download_mode=download_mode,
+        script_version=script_version,
+        use_auth_token=use_auth_token,
         **config_kwargs,
     )
 
diff --git a/src/datasets/packaged_modules/__init__.py b/src/datasets/packaged_modules/__init__.py
index 097aa5d3852..91b115cd49b 100644
--- a/src/datasets/packaged_modules/__init__.py
+++ b/src/datasets/packaged_modules/__init__.py
@@ -31,3 +31,12 @@ def hash_python_lines(lines: List[str]) -> str:
     "parquet": (parquet.__name__, hash_python_lines(inspect.getsource(parquet).splitlines())),
     "text": (text.__name__, hash_python_lines(inspect.getsource(text).splitlines())),
 }
+
+_EXTENSION_TO_MODULE = {
+    "csv": "csv",
+    "tsv": "csv",
+    "json": "json",
+    "jsonl": "json",
+    "parquet": "parquet",
+    "txt": "text",
+}
diff --git a/src/datasets/utils/download_manager.py b/src/datasets/utils/download_manager.py
index 33b40ebe995..a1f4c0110d7 100644
--- a/src/datasets/utils/download_manager.py
+++ b/src/datasets/utils/download_manager.py
@@ -214,6 +214,7 @@ def download(self, url_or_urls):
         return downloaded_path_or_paths.data
 
     def _download(self, url_or_filename: str, download_config: DownloadConfig) -> str:
+        url_or_filename = str(url_or_filename)
         if is_relative_path(url_or_filename):
             # append the relative path to the base_path
             url_or_filename = url_or_path_join(self._base_path, url_or_filename)
diff --git a/src/datasets/utils/file_utils.py b/src/datasets/utils/file_utils.py
index 6035bc69ff5..90a7a6c96b7 100644
--- a/src/datasets/utils/file_utils.py
+++ b/src/datasets/utils/file_utils.py
@@ -172,7 +172,7 @@ def hf_hub_url(path: str, name: str, version: Optional[str] = None) -> str:
 
 def url_or_path_join(base_name: str, *pathnames: str) -> str:
     if is_remote_url(base_name):
-        return posixpath.join(base_name, *pathnames)
+        return posixpath.join(base_name, *(str(pathname).lstrip("/") for pathname in pathnames))
     else:
         return Path(base_name, *pathnames).as_posix()
 
diff --git a/src/datasets/utils/streaming_download_manager.py b/src/datasets/utils/streaming_download_manager.py
index f208bd90449..57e3f97b42c 100644
--- a/src/datasets/utils/streaming_download_manager.py
+++ b/src/datasets/utils/streaming_download_manager.py
@@ -15,6 +15,7 @@
 
 logger = get_logger(__name__)
 BASE_KNOWN_EXTENSIONS = ["txt", "csv", "json", "jsonl", "tsv", "conll"]
+UNSUPPORTED_ARCHIVE_EXTENSIONS_FOR_STREAMING = ["tar", "xz", "rar", "zst"]
 
 
 def xjoin(a, *p):
@@ -108,6 +109,7 @@ def download(self, url_or_urls):
         return url_or_urls
 
     def _download(self, url_or_filename):
+        url_or_filename = str(url_or_filename)
         if is_relative_path(url_or_filename):
             # append the relative path to the base_path
             url_or_filename = url_or_path_join(self._base_path, url_or_filename)
@@ -118,6 +120,7 @@ def extract(self, path_or_paths):
         return urlpaths
 
     def _extract(self, urlpath):
+        urlpath = str(urlpath)
         protocol = self._get_extraction_protocol(urlpath)
         if protocol is None:
             # no extraction
@@ -129,14 +132,16 @@ def _extract(self, urlpath):
             return f"{protocol}://::{urlpath}"
 
     def _get_extraction_protocol(self, urlpath) -> Optional[str]:
-        path = urlpath.split("::")[0]
+        path = str(urlpath).split("::")[0]
         if path.split(".")[-1] in BASE_KNOWN_EXTENSIONS:
             return None
         elif path.endswith(".gz") and not path.endswith(".tar.gz"):
             return "gzip"
         elif path.endswith(".zip"):
             return "zip"
-        raise NotImplementedError(f"Extraction protocol for file at {urlpath} is not implemented yet")
+        elif any(ext in path.split(".") for ext in UNSUPPORTED_ARCHIVE_EXTENSIONS_FOR_STREAMING):
+            raise NotImplementedError(f"Extraction protocol for file at {urlpath} is not implemented yet")
+        return None
 
     def download_and_extract(self, url_or_urls):
         return self.extract(self.download(url_or_urls))
diff --git a/tests/test_load.py b/tests/test_load.py
index d9dbcc3465c..fdd2f381f76 100644
--- a/tests/test_load.py
+++ b/tests/test_load.py
@@ -6,11 +6,13 @@
 import time
 from functools import partial
 from hashlib import sha256
+from pathlib import Path, PurePath
 from unittest import TestCase
 from unittest.mock import patch
 
 import pytest
 import requests
+from huggingface_hub.hf_api import DatasetInfo
 
 import datasets
 from datasets import SCRIPTS_VERSION, load_dataset, load_from_disk
@@ -19,8 +21,12 @@
 from datasets.dataset_dict import DatasetDict, IterableDatasetDict
 from datasets.features import Features, Value
 from datasets.iterable_dataset import IterableDataset
-from datasets.load import prepare_module
-from datasets.utils.file_utils import DownloadConfig
+from datasets.load import (
+    _resolve_data_files_in_dataset_repository,
+    _resolve_data_files_locally_or_by_urls,
+    prepare_module,
+)
+from datasets.utils.file_utils import DownloadConfig, is_remote_url
 
 from .utils import (
     OfflineSimulationMode,
@@ -28,6 +34,7 @@
     assert_arrow_memory_increases,
     offline,
     require_streaming,
+    set_current_working_directory_to_temp_dir,
 )
 
 
@@ -58,12 +65,14 @@ def _generate_examples(self, filepath, **kwargs):
 """
 
 SAMPLE_DATASET_IDENTIFIER = "lhoestq/test"
+SAMPLE_DATASET_IDENTIFIER2 = "lhoestq/test2"
 SAMPLE_NOT_EXISTING_DATASET_IDENTIFIER = "lhoestq/_dummy"
+SAMPLE_DATASET_NAME_THAT_DOESNT_EXIST = "_dummy"
 
 
 @pytest.fixture
 def data_dir(tmp_path):
-    data_dir = tmp_path / "data"
+    data_dir = tmp_path / "data_dir"
     data_dir.mkdir()
     with open(data_dir / "train.txt", "w") as f:
         f.write("foo\n" * 10)
@@ -72,6 +81,22 @@ def data_dir(tmp_path):
     return str(data_dir)
 
 
+@pytest.fixture
+def complex_data_dir(tmp_path):
+    data_dir = tmp_path / "complex_data_dir"
+    data_dir.mkdir()
+    (data_dir / "data").mkdir()
+    with open(data_dir / "data" / "train.txt", "w") as f:
+        f.write("foo\n" * 10)
+    with open(data_dir / "data" / "test.txt", "w") as f:
+        f.write("bar\n" * 10)
+    with open(data_dir / "README.md", "w") as f:
+        f.write("This is a readme")
+    with open(data_dir / ".dummy", "w") as f:
+        f.write("this is a dummy file that is not a data file")
+    return str(data_dir)
+
+
 @pytest.fixture
 def dataset_loading_script_dir(tmp_path):
     script_name = DATASET_LOADING_SCRIPT_NAME
@@ -188,7 +213,7 @@ def test_load_dataset_users(self):
         with self.assertRaises(FileNotFoundError) as context:
             datasets.load_dataset("lhoestq/_dummy")
         self.assertIn(
-            "https://huggingface.co/datasets/lhoestq/_dummy/resolve/main/_dummy.py",
+            "lhoestq/_dummy",
             str(context.exception),
         )
         for offline_simulation_mode in list(OfflineSimulationMode):
@@ -196,18 +221,79 @@ def test_load_dataset_users(self):
                 with self.assertRaises(ConnectionError) as context:
                     datasets.load_dataset("lhoestq/_dummy")
                 self.assertIn(
-                    "https://huggingface.co/datasets/lhoestq/_dummy/resolve/main/_dummy.py",
+                    "lhoestq/_dummy",
                     str(context.exception),
                 )
 
 
-def test_load_dataset_builder(dataset_loading_script_dir, data_dir):
+def test_load_dataset_builder_for_absolute_script_dir(dataset_loading_script_dir, data_dir):
     builder = datasets.load_dataset_builder(dataset_loading_script_dir, data_dir=data_dir)
     assert isinstance(builder, DatasetBuilder)
     assert builder.name == DATASET_LOADING_SCRIPT_NAME
     assert builder.info.features == Features({"text": Value("string")})
 
 
+def test_load_dataset_builder_for_relative_script_dir(dataset_loading_script_dir, data_dir):
+    with set_current_working_directory_to_temp_dir():
+        relative_script_dir = DATASET_LOADING_SCRIPT_NAME
+        shutil.copytree(dataset_loading_script_dir, relative_script_dir)
+        builder = datasets.load_dataset_builder(relative_script_dir, data_dir=data_dir)
+        assert isinstance(builder, DatasetBuilder)
+        assert builder.name == DATASET_LOADING_SCRIPT_NAME
+        assert builder.info.features == Features({"text": Value("string")})
+
+
+def test_load_dataset_builder_for_script_path(dataset_loading_script_dir, data_dir):
+    builder = datasets.load_dataset_builder(
+        os.path.join(dataset_loading_script_dir, DATASET_LOADING_SCRIPT_NAME + ".py"), data_dir=data_dir
+    )
+    assert isinstance(builder, DatasetBuilder)
+    assert builder.name == DATASET_LOADING_SCRIPT_NAME
+    assert builder.info.features == Features({"text": Value("string")})
+
+
+def test_load_dataset_builder_for_absolute_data_dir(complex_data_dir):
+    builder = datasets.load_dataset_builder(complex_data_dir)
+    assert isinstance(builder, DatasetBuilder)
+    assert builder.name == "text"
+    assert builder.config.name == Path(complex_data_dir).name
+    assert isinstance(builder.config.data_files, list)
+    assert len(builder.config.data_files) > 0
+
+
+def test_load_dataset_builder_for_relative_data_dir(complex_data_dir):
+    with set_current_working_directory_to_temp_dir():
+        relative_data_dir = "relative_data_dir"
+        shutil.copytree(complex_data_dir, relative_data_dir)
+        builder = datasets.load_dataset_builder(relative_data_dir)
+        assert isinstance(builder, DatasetBuilder)
+        assert builder.name == "text"
+        assert builder.config.name == relative_data_dir
+        assert isinstance(builder.config.data_files, list)
+        assert len(builder.config.data_files) > 0
+
+
+def test_load_dataset_builder_for_community_dataset_with_script():
+    builder = datasets.load_dataset_builder(SAMPLE_DATASET_IDENTIFIER)
+    assert isinstance(builder, DatasetBuilder)
+    assert builder.name == SAMPLE_DATASET_IDENTIFIER.split("/")[-1]
+    assert builder.info.features == Features({"text": Value("string")})
+
+
+def test_load_dataset_builder_for_community_dataset_without_script():
+    builder = datasets.load_dataset_builder(SAMPLE_DATASET_IDENTIFIER2)
+    assert isinstance(builder, DatasetBuilder)
+    assert builder.name == "text"
+    assert builder.config.name == SAMPLE_DATASET_IDENTIFIER2.split("/")[-1]
+    assert isinstance(builder.config.data_files, list)
+    assert len(builder.config.data_files) > 0
+
+
+def test_load_dataset_builder_fail():
+    with pytest.raises(FileNotFoundError):
+        datasets.load_dataset_builder("blabla")
+
+
 @pytest.mark.parametrize("keep_in_memory", [False, True])
 def test_load_dataset_local(dataset_loading_script_dir, data_dir, keep_in_memory, caplog):
     with assert_arrow_memory_increases() if keep_in_memory else assert_arrow_memory_doesnt_increase():
@@ -224,9 +310,13 @@ def test_load_dataset_local(dataset_loading_script_dir, data_dir, keep_in_memory
             assert len(dataset) == 2
             assert "Using the latest cached version of the module" in caplog.text
     with pytest.raises(FileNotFoundError) as exc_info:
-        datasets.load_dataset("_dummy")
-    m_combined_path = re.search(fr"\S*{re.escape(os.path.join('_dummy', '_dummy.py'))}\b", str(exc_info.value))
-    assert m_combined_path is not None and os.path.isabs(m_combined_path.group())
+        datasets.load_dataset(SAMPLE_DATASET_NAME_THAT_DOESNT_EXIST)
+    m_combined_path = re.search(
+        fr"http\S*{re.escape(SAMPLE_DATASET_NAME_THAT_DOESNT_EXIST + '/' + SAMPLE_DATASET_NAME_THAT_DOESNT_EXIST + '.py')}\b",
+        str(exc_info.value),
+    )
+    assert m_combined_path is not None and is_remote_url(m_combined_path.group())
+    assert os.path.abspath(SAMPLE_DATASET_NAME_THAT_DOESNT_EXIST) in str(exc_info.value)
 
 
 @require_streaming
@@ -363,3 +453,98 @@ def test_load_dataset_deletes_extracted_files(deleted, jsonl_gz_path, tmp_path):
         ds = load_dataset("json", split="train", data_files=data_files, cache_dir=cache_dir)
     assert ds[0] == {"col_1": "0", "col_2": 0, "col_3": 0.0}
     assert (sorted((cache_dir / "downloads" / "extracted").iterdir()) == []) is deleted
+
+
+@pytest.mark.parametrize(
+    "pattern,size", [("*", 2), ("**/*", 2), ("*.txt", 2), ("data/*", 2), ("**/*.txt", 2), ("**/train.txt", 1)]
+)
+def test_resolve_data_files_locally_or_by_urls(complex_data_dir, pattern, size):
+    resolved_data_files = _resolve_data_files_locally_or_by_urls(complex_data_dir, pattern)
+    files_to_ignore = {".dummy", "README.md"}
+    expected_resolved_data_files = [
+        path for path in Path(complex_data_dir).rglob(pattern) if path.name not in files_to_ignore and path.is_file()
+    ]
+    assert len(resolved_data_files) == size
+    assert sorted(resolved_data_files) == sorted(expected_resolved_data_files)
+    assert all(isinstance(path, Path) for path in resolved_data_files)
+    assert all(path.is_file() for path in resolved_data_files)
+
+
+def test_resolve_data_files_locally_or_by_urls_with_absolute_path(tmp_path, complex_data_dir):
+    abs_path = os.path.join(complex_data_dir, "data", "train.txt")
+    resolved_data_files = _resolve_data_files_locally_or_by_urls(str(tmp_path / "blabla"), abs_path)
+    assert len(resolved_data_files) == 1
+
+
+@pytest.mark.parametrize("pattern,size,extensions", [("*", 2, ["txt"]), ("*", 2, None), ("*", 0, ["blablabla"])])
+def test_resolve_data_files_locally_or_by_urls_with_extensions(complex_data_dir, pattern, size, extensions):
+    if size > 0:
+        resolved_data_files = _resolve_data_files_locally_or_by_urls(
+            complex_data_dir, pattern, allowed_extensions=extensions
+        )
+        assert len(resolved_data_files) == size
+    else:
+        with pytest.raises(FileNotFoundError):
+            _resolve_data_files_locally_or_by_urls(complex_data_dir, pattern, allowed_extensions=extensions)
+
+
+def test_fail_resolve_data_files_locally_or_by_urls(complex_data_dir):
+    with pytest.raises(FileNotFoundError):
+        _resolve_data_files_locally_or_by_urls(complex_data_dir, "blablabla")
+
+
+@pytest.mark.parametrize(
+    "pattern,size", [("*", 2), ("**/*", 2), ("*.txt", 2), ("data/*", 2), ("**/*.txt", 2), ("**/train.txt", 1)]
+)
+def test_resolve_data_files_in_dataset_repository(complex_data_dir, pattern, size):
+    dataset_info = DatasetInfo(
+        siblings=[
+            {"rfilename": path.relative_to(complex_data_dir).as_posix()}
+            for path in Path(complex_data_dir).rglob("*")
+            if path.is_file()
+        ]
+    )
+    resolved_data_files = _resolve_data_files_in_dataset_repository(dataset_info, pattern)
+    files_to_ignore = {".dummy", "README.md"}
+    expected_resolved_data_files = [
+        path.relative_to(complex_data_dir)
+        for path in Path(complex_data_dir).rglob(pattern)
+        if path.name not in files_to_ignore and path.is_file()
+    ]
+    assert len(resolved_data_files) == size
+    assert sorted(resolved_data_files) == sorted(expected_resolved_data_files)
+    assert all(isinstance(path, PurePath) for path in resolved_data_files)
+    assert all((Path(complex_data_dir) / path).is_file() for path in resolved_data_files)
+
+
+@pytest.mark.parametrize("pattern,size,extensions", [("*", 2, ["txt"]), ("*", 2, None), ("*", 0, ["blablabla"])])
+def test_resolve_data_files_in_dataset_repository_with_extensions(complex_data_dir, pattern, size, extensions):
+    dataset_info = DatasetInfo(
+        siblings=[
+            {"rfilename": path.relative_to(complex_data_dir).as_posix()}
+            for path in Path(complex_data_dir).rglob("*")
+            if path.is_file()
+        ]
+    )
+    if size > 0:
+        resolved_data_files = _resolve_data_files_in_dataset_repository(
+            dataset_info, pattern, allowed_extensions=extensions
+        )
+        assert len(resolved_data_files) == size
+    else:
+        with pytest.raises(FileNotFoundError):
+            resolved_data_files = _resolve_data_files_in_dataset_repository(
+                dataset_info, pattern, allowed_extensions=extensions
+            )
+
+
+def test_fail_resolve_data_files_in_dataset_repository(complex_data_dir):
+    dataset_info = DatasetInfo(
+        siblings=[
+            {"rfilename": path.relative_to(complex_data_dir).as_posix()}
+            for path in Path(complex_data_dir).rglob("*")
+            if path.is_file()
+        ]
+    )
+    with pytest.raises(FileNotFoundError):
+        _resolve_data_files_in_dataset_repository(dataset_info, "blablabla")


Code review comments:
diff_hunk: @@ -158,22 +158,52 @@ Apart from :obj:`name` and :obj:`split`, the :func:`datasets.load_dataset` metho
 
 The use of these arguments is discussed in the :ref:`load_dataset_cache_management` section below. You can also find the full details on these arguments on the package reference page for :func:`datasets.load_dataset`.
 
+From a community dataset on the Hugging Face Hub
+-----------------------------------------------------------
+
+The community shares hundreds of datasets on the Hugging Face Hub using **dataset repositories**.
+A dataset repository is a versioned repository of data files.
+Everyone can create a dataset repository on the Hugging Face Hub and upload their data.
+
+For example we have created a demo dataset at https://huggingface.co/datasets/lhoestq/demo1.
+In this dataset repository we uploaded some csv files, and you can load the dataset with:
+
+.. code-block::
+
+    >>> from datasets import load_dataset
+    >>> dataset = load_dataset('lhoestq/demo1')
+
+You can even choose which files to use in a dataset repository.
+For example you can load a subset of the **C4 dataset for language modeling**, hosted by AllenAI on the Hub.
+You can browse the dataset repository at https://huggingface.co/datasets/allenai/c4
+
+In the following example we specify which subset of the files to use with the ``data_files`` parameter:
+
+.. code-block::
+
+    >>> from datasets import load_dataset
+    >>> c4_subset = load_dataset('allenai/c4', data_files='en/c4-train.0000*-of-01024.json.gz')
+
+It will use all the files that match the pattern passed in ``data_files``.
+If you don't specify which data files to use, it will use all the data files (here all C4 is about 13TB of data).
+
 
 .. _loading-from-local-files:
 
-From local files
+From local or remote files
 -----------------------------------------------------------
 
-It's also possible to create a dataset from local files.
+It's also possible to create a dataset from your own local or remote files.
 
 Generic loading scripts are provided for:
 
 - CSV files (with the :obj:`csv` script),
 - JSON files (with the :obj:`json` script),
 - text files (read as a line-by-line dataset with the :obj:`text` script),
+- parquet files (with the :obj:`parquet` script).
 - pandas pickled dataframe (with the :obj:`pandas` script).
 
-If you want to control better how your files are loaded, or if you have a file format exactly reproducing the file format for one of the datasets provided on the `HuggingFace Hub <https://huggingface.co/datasets>`__, it can be more flexible and simpler to create **your own loading script**, from scratch or by adapting one of the provided loading scripts. In this case, please go check the :doc:`add_dataset` chapter.
+If you want to control better how your files are loaded, or if you have a file format exactly reproducing the file format for one of the datasets provided on the `Hugging Face Hub <https://huggingface.co/datasets>`__, it can be more flexible and simpler to create **your own loading script**, from scratch or by adapting one of the provided loading scripts. In this case, please go check the :doc:`add_dataset` chapter.
path: docs/source/loading_datasets.rst
text: @user1:
```suggestion
If you want more fine-grained control on how your files are loaded or if you have a file format that matches the format for one of the datasets provided on the `Hugging Face Hub <https://huggingface.co/datasets>`__, it can be more  simpler to create **your own loading script**, from scratch or by adapting one of the provided loading scripts. In this case, please go check the :doc:`add_dataset` section.
```

diff_hunk: @@ -902,8 +919,8 @@ def as_streaming_dataset(
             raise ValueError(f"Builder {self.name} is not streamable.")
         if not config.AIOHTTP_AVAILABLE:
             raise ImportError(
-                f"To be able to use dataset streaming, you need to install dependencies like aiohttp "
-                f'using "pip install \'datasets[streaming]\'" or "pip install aiohttp" for instance'
+                "To be able to use dataset streaming, you need to install dependencies like aiohttp "
+                'using \'datasets[streaming]\'" or "pip install aiohttp" for instance'
path: src/datasets/builder.py
text: @user1:
I think you can move `aiohttp` to required dep since we are now pushing for the streaming feature a lot

diff_hunk: @@ -15,6 +15,7 @@
 
 logger = get_logger(__name__)
 BASE_KNOWN_EXTENSIONS = ["txt", "csv", "json", "jsonl", "tsv", "conll"]
+UNSUPPORTED_ARCHIVE_EXTENSIONS_FOR_STREAMING = ["tar", "xz", "rar", "zst"]
path: src/datasets/utils/streaming_download_manager.py
text: @user1:
Here also we should have the list of supported (and unsupported) extension/file-format clearly stated in the doc

@user1:
I'm a bit surprised you do a black-list instead of a white-list here

@author:
This is because most scripts call `download_and_extract` even for files that don't need extraction. We shouldn't raise an error for an uncompressed file. If we had a white list and you passed un uncompressed file, it wouldn't find any supported compression extension in the filename, so it would fail.

@author:
in the future we'll fallback on reading the header of the file to know if it's a compressed file or not