Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

enhancement

New feature or request

generic discussion

Generic discussion on the library

I open this PR to have a public discussion about this topic and make a decision.

As previously discussed, once we have the metadata in the dataset card (README file, containing both Markdown info and YAML tags), what is the point of having also the JSON metadata (dataset_infos.json file)?

On the other hand, the dummy data is necessary for testing (in our CI suite) that the canonical dataset loads correctly. However:
- the dataset preview feature is already an indirect test that the dataset loads correctly (it also tests it is streamable though)
- we are migrating canonical datasets to the Hub
Do we really need to continue testing them in out CI?

Also note that for generating both (dataset_infos.json file and dummy data), the entire dataset needs being downloaded. This can be an issue for huge datasets (like WIT, with 400 GB of data).

Feel free to ping other people for the discussion.

CC: @lhoestq @mariosasko @thomwolf @julien-c @patrickvonplaten @anton-l @LysandreJik @yjernite @nateraw


Allows DatasetDict.filter to have batching option

- Related to: #3244
- Fixes: #3503

We extends `.filter( ... batched: bool)` support to DatasetDict.


cast_column function not working with map function in streaming mode for Audio features

bug

Something isn't working

## Describe the bug
I am trying to use Audio class for loading audio features using custom dataset. I am able to cast 'audio' feature into 'Audio' format with cast_column function. On using map function, I am not getting 'Audio' casted feature but getting path of audio file only.
I am getting features of 'audio' of string type with load_dataset call. After using cast_column 'audio' feature is converted into 'Audio' type. But in map function I am not able to get Audio type for audio feature & getting string type data containing path of file only. So I am not able to use processor in encode function.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset, Audio
from transformers import Wav2Vec2Processor

def encode(batch, processor):
    print("Audio: ",batch['audio'])
    batch["input_values"] = processor(batch["audio"]['array'], sampling_rate=16000).input_values
    return batch

def print_ds(ds):
    iterator = iter(ds)
    for d in iterator:
        print("Data: ",d)
        break

processor = Wav2Vec2Processor.from_pretrained(pretrained_model_path)

dataset = load_dataset("custom_dataset.py","train",data_files={'train':'train_path.txt'},
                                data_dir="data", streaming=True, split="train")

print("Features: ",dataset.features)

print_ds(dataset)

dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print("Features: ",dataset.features)

print_ds(dataset)

dataset = dataset.map(lambda x: encode(x,processor))

print("Features: ",dataset.features)

print_ds(dataset)

```

## Expected results

map function not printing Audio type features be used with processor function and getting error in processor call due to this.

## Actual results

# after load_dataset call
Features:  {'sentence': Value(dtype='string', id=None), 'audio': Value(dtype='string', id=None)}
Data:  {'sentence': 'और अपने पेट को माँ की स्वादिष्ट गरमगरम जलेबियाँ हड़पते\n', 'audio': 'data/0116_003.wav'}

# after cast_column call
Features:  {'sentence': Value(dtype='string', id=None), 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None)}
Data:  {'sentence': 'और अपने पेट को माँ की स्वादिष्ट गरमगरम जलेबियाँ हड़पते\n', 'audio': {'path': 'data/0116_003.wav', 'array': array([ 1.2662281e-06,  1.0264218e-06, -1.3615092e-06, ...,
        1.3017889e-02,  1.0085563e-02,  4.8155054e-03], dtype=float32), 'sampling_rate': 16000}}

# after map call
Features:  None
Audio:  data/0116_003.wav

Traceback (most recent call last):
  File "demo2.py", line 36, in <module>
    print_ds(dataset)
  File "demo2.py", line 11, in print_ds
    for d in iterator:
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 192, in __iter__
    yield key, self.function(example)
  File "demo2.py", line 32, in <lambda>
    dataset = dataset.map(lambda x: batch_encode(x,processor))
  File "demo2.py", line 6, in batch_encode
    batch["input_values"] = processor(batch["audio"]['array'], sampling_rate=16000).input_values
TypeError: string indices must be integers
## Environment info

- `datasets` version: 1.17.0
- Platform: Linux-4.14.243 with-debian-bullseye-sid
- Python version: 3.7.9
- PyArrow version: 6.0.1


Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Something isn't working

dataset bug

A bug in a dataset script provided in the library

## Describe the bug
I am unable to download the PubMed dataset from the link provided in the [Hugging Face Course (Chapter 5 Section 4)](https://huggingface.co/course/chapter5/4?fw=pt).

https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset

# This takes a few minutes to run, so go grab a tea or coffee while you wait :)
data_files = "https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst"
pubmed_dataset = load_dataset("json", data_files=data_files, split="train")
pubmed_dataset
```

I also tried with `wget` as follows.
```
wget https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst
```

## Expected results
I expect to be able to download this file.

## Actual results
Traceback
```
---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
/usr/lib/python3/dist-packages/urllib3/connection.py in _new_conn(self)
    158         try:
--> 159             conn = connection.create_connection(
    160                 (self._dns_host, self.port), self.timeout, **extra_kw

/usr/lib/python3/dist-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     83     if err is not None:
---> 84         raise err
     85

/usr/lib/python3/dist-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     73                 sock.bind(source_address)
---> 74             sock.connect(sa)
     75             return sock

timeout: timed out

During handling of the above exception, another exception occurred:

ConnectTimeoutError                       Traceback (most recent call last)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    664             # Make the request on the httplib connection object.
--> 665             httplib_response = self._make_request(
    666                 conn,

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    375         try:
--> 376             self._validate_conn(conn)
    377         except (SocketTimeout, BaseSSLError) as e:

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    995         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 996             conn.connect()
    997

/usr/lib/python3/dist-packages/urllib3/connection.py in connect(self)
    313         # Add certificate verification
--> 314         conn = self._new_conn()
    315         hostname = self.host

/usr/lib/python3/dist-packages/urllib3/connection.py in _new_conn(self)
    163         except SocketTimeout:
--> 164             raise ConnectTimeoutError(
    165                 self,

ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)')

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
/usr/lib/python3/dist-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    438             if not chunked:
--> 439                 resp = conn.urlopen(
    440                     method=request.method,

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    718
--> 719             retries = retries.increment(
    720                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]

/usr/lib/python3/dist-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    435         if new_retry.is_exhausted():
--> 436             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    437

MaxRetryError: HTTPSConnectionPool(host='the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)'))

During handling of the above exception, another exception occurred:

ConnectTimeout                            Traceback (most recent call last)
/tmp/ipykernel_15104/606583593.py in <module>
      3 # This takes a few minutes to run, so go grab a tea or coffee while you wait :)
      4 data_files = "https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst"
----> 5 pubmed_dataset = load_dataset("json", data_files=data_files, split="train")
      6 pubmed_dataset

~/.local/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1655
   1656     # Create a dataset builder
-> 1657     builder_instance = load_dataset_builder(
   1658         path=path,
   1659         name=name,

~/.local/lib/python3.8/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, script_version, **config_kwargs)
   1492         download_config = download_config.copy() if download_config else DownloadConfig()
   1493         download_config.use_auth_token = use_auth_token
-> 1494     dataset_module = dataset_module_factory(
   1495         path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
   1496     )

~/.local/lib/python3.8/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_files, **download_kwargs)
   1116     # Try packaged
   1117     if path in _PACKAGED_DATASETS_MODULES:
-> 1118         return PackagedDatasetModuleFactory(
   1119             path, data_files=data_files, download_config=download_config, download_mode=download_mode
   1120         ).get_module()

~/.local/lib/python3.8/site-packages/datasets/load.py in get_module(self)
    773             else get_patterns_locally(str(Path().resolve()))
    774         )
--> 775         data_files = DataFilesDict.from_local_or_remote(patterns, use_auth_token=self.downnload_config.use_auth_token)
    776         module_path, hash = _PACKAGED_DATASETS_MODULES[self.name]
    777         builder_kwargs = {"hash": hash, "data_files": data_files}

~/.local/lib/python3.8/site-packages/datasets/data_files.py in from_local_or_remote(cls, patterns, base_path, allowed_extensions, use_auth_token)
    576         for key, patterns_for_key in patterns.items():
    577             out[key] = (
--> 578                 DataFilesList.from_local_or_remote(
    579                     patterns_for_key,
    580                     base_path=base_path,

~/.local/lib/python3.8/site-packages/datasets/data_files.py in from_local_or_remote(cls, patterns, base_path, allowed_extensions, use_auth_token)
    545         base_path = base_path if base_path is not None else str(Path().resolve())
    546         data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
--> 547         origin_metadata = _get_origin_metadata_locally_or_by_urls(data_files, use_auth_token=use_auth_token)
    548         return cls(data_files, origin_metadata)
    549

~/.local/lib/python3.8/site-packages/datasets/data_files.py in _get_origin_metadata_locally_or_by_urls(data_files, max_workers, use_auth_token)
    492     data_files: List[Union[Path, Url]], max_workers=64, use_auth_token: Optional[Union[bool, str]] = None
    493 ) -> Tuple[str]:
--> 494     return thread_map(
    495         partial(_get_single_origin_metadata_locally_or_by_urls, use_auth_token=use_auth_token),
    496         data_files,

~/.local/lib/python3.8/site-packages/tqdm/contrib/concurrent.py in thread_map(fn, *iterables, **tqdm_kwargs)
     92     """
     93     from concurrent.futures import ThreadPoolExecutor
---> 94     return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
     95
     96

~/.local/lib/python3.8/site-packages/tqdm/contrib/concurrent.py in _executor_map(PoolExecutor, fn, *iterables, **tqdm_kwargs)
     74             map_args.update(chunksize=chunksize)
     75         with PoolExecutor(**pool_kwargs) as ex:
---> 76             return list(tqdm_class(ex.map(fn, *iterables, **map_args), **kwargs))
     77
     78

~/.local/lib/python3.8/site-packages/tqdm/notebook.py in __iter__(self)
    252     def __iter__(self):
    253         try:
--> 254             for obj in super(tqdm_notebook, self).__iter__():
    255                 # return super(tqdm...) will not catch exception
    256                 yield obj

~/.local/lib/python3.8/site-packages/tqdm/std.py in __iter__(self)
   1171         # (note: keep this check outside the loop for performance)
   1172         if self.disable:
-> 1173             for obj in iterable:
   1174                 yield obj
   1175             return

/usr/lib/python3.8/concurrent/futures/_base.py in result_iterator()
    617                     # Careful not to keep a reference to the popped future
    618                     if timeout is None:
--> 619                         yield fs.pop().result()
    620                     else:
    621                         yield fs.pop().result(end_time - time.monotonic())

/usr/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    442                     raise CancelledError()
    443                 elif self._state == FINISHED:
--> 444                     return self.__get_result()
    445                 else:
    446                     raise TimeoutError()

/usr/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    387         if self._exception:
    388             try:
--> 389                 raise self._exception
    390             finally:
    391                 # Break a reference cycle with the exception in self._exception

/usr/lib/python3.8/concurrent/futures/thread.py in run(self)
     55
     56         try:
---> 57             result = self.fn(*self.args, **self.kwargs)
     58         except BaseException as exc:
     59             self.future.set_exception(exc)

~/.local/lib/python3.8/site-packages/datasets/data_files.py in _get_single_origin_metadata_locally_or_by_urls(data_file, use_auth_token)
    483     if isinstance(data_file, Url):
    484         data_file = str(data_file)
--> 485         return (request_etag(data_file, use_auth_token=use_auth_token),)
    486     else:
    487         data_file = str(data_file.resolve())

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in request_etag(url, use_auth_token)
    489 def request_etag(url: str, use_auth_token: Optional[Union[str, bool]] = None) -> Optional[str]:
    490     headers = get_authentication_headers_for_url(url, use_auth_token=use_auth_token)
--> 491     response = http_head(url, headers=headers, max_retries=3)
    492     response.raise_for_status()
    493     etag = response.headers.get("ETag") if response.ok else None

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in http_head(url, proxies, headers, cookies, allow_redirects, timeout, max_retries)
    474     headers = copy.deepcopy(headers) or {}
    475     headers["user-agent"] = get_datasets_user_agent(user_agent=headers.get("user-agent"))
--> 476     response = _request_with_retry(
    477         method="HEAD",
    478         url=url,

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in _request_with_retry(method, url, max_retries, base_wait_time, max_wait_time, timeout, **params)
    407         except (requests.exceptions.ConnectTimeout, requests.exceptions.ConnectionError) as err:
    408             if tries > max_retries:
--> 409                 raise err
    410             else:
    411                 logger.info(f"{method} request to {url} timed out, retrying... [{tries/max_retries}]")

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in _request_with_retry(method, url, max_retries, base_wait_time, max_wait_time, timeout, **params)
    403         tries += 1
    404         try:
--> 405             response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
    406             success = True
    407         except (requests.exceptions.ConnectTimeout, requests.exceptions.ConnectionError) as err:

/usr/lib/python3/dist-packages/requests/api.py in request(method, url, **kwargs)
     58     # cases, and look like a memory leak in others.
     59     with sessions.Session() as session:
---> 60         return session.request(method=method, url=url, **kwargs)
     61
     62

/usr/lib/python3/dist-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    531         }
    532         send_kwargs.update(settings)
--> 533         resp = self.send(prep, **send_kwargs)
    534
    535         return resp

/usr/lib/python3/dist-packages/requests/sessions.py in send(self, request, **kwargs)
    644
    645         # Send the request
--> 646         r = adapter.send(request, **kwargs)
    647
    648         # Total elapsed time of the request (approximately)

/usr/lib/python3/dist-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    502                 # TODO: Remove this in 3.0.0: see #2811
    503                 if not isinstance(e.reason, NewConnectionError):
--> 504                     raise ConnectTimeout(e, request=request)
    505
    506             if isinstance(e.reason, ResponseError):

ConnectTimeout: HTTPSConnectionPool(host='the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)'))
```

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.11.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 6.0.1


Batched in filter throws error

bug

Something isn't working

I hope this is really a bug, I could not find it among the open issues

## Describe the bug
using `batched=False` in DataSet.filter throws error
```python
TypeError: filter() got an unexpected keyword argument 'batched'
```
but in the docs it is lister as an argument.

## Steps to reproduce the bug
```python
task = "mnli"
max_length = 128
tokenizer = AutoTokenizer.from_pretrained("./pretrained_models/pretrained_models_drozd/sl250.m.gsic.titech.ac.jp:8000/21.11.17_06.30.32_roberta-base_a0057/checkpoints/smpl_400M/hf/")

dataset = load_dataset("glue", task)

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

##### tokenization_parameters
sentence1_key, sentence2_key = task_to_keys[task]
def preprocess_function(examples, max_length):
    if sentence2_key is None:
        return tokenizer(
            examples[sentence1_key], truncation=True, max_length=max_length
        )
    return tokenizer(
        examples[sentence1_key],
        examples[sentence2_key],
        truncation=False,
        padding="max_length",
        max_length=max_length,
    )

encoded_dataset = dataset.map(
    lambda x: preprocess_function(x, max_length=max_length), batched=False
)

encoded_dataset.filter(lambda x: len(x['input_ids']) <= max_length, batched=False)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1, 1.17.0
- Platform: ubuntu
- Python version: 3.8.12


Add QuALITY

Fixes #3441.


Update pib dataset card

Related to #3496


Docs: Add VCTK dataset description

This PR is a very minor followup to #1837, with only docs changes (single comment string).


Adjusting chunk size for streaming datasets

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
I want to use mc4 which I cannot save locally, so I stream it. However, I want to process the entire dataset and filter some documents from it. With the current chunk size of around 1000 documents (right?) I hit a performance bottleneck because of the frequent decompressing.

**Describe the solution you'd like**
I would appreciate a parameter in the load_dataset function, that allows me to set the chunksize myself (to a value like 100'000 in my case). Like that, I hope to improve the processing time.


update `pretty_name` for first 200 datasets

I made a script some time back to fetch `pretty_names` from `papers_with_code` dataset along with some other rules incase that dataset wasn't available on `papers_with_code`. Updating them in the `README` of `datasets`. Took only the first 200 datasets into consideration and after some eyeballing, most of them were looking good to me!


Changing sampling rate in audio dataset and subsequently mapping with `num_proc > 1` leads to weird bug

bug

Something isn't working

Running:

```python
from datasets import load_dataset, DatasetDict
import datasets
from transformers import AutoFeatureExtractor

raw_datasets = DatasetDict()

raw_datasets["train"] = load_dataset("common_voice", "ab", split="train")

feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")

raw_datasets = raw_datasets.cast_column(
    "audio", datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)
)

num_workers = 16

def prepare_dataset(batch):
    sample = batch["audio"]

    inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])
    return batch

raw_datasets.map(
    prepare_dataset,
    remove_columns=next(iter(raw_datasets.values())).column_names,
    num_proc=16,
    desc="preprocess datasets",
)
```

gives

```bash
  File "/home/patrick/experiments/run_bug.py", line 25, in <module>
    raw_datasets.map(
  File "/home/patrick/python_bin/datasets/dataset_dict.py", line 492, in map
    {
  File "/home/patrick/python_bin/datasets/dataset_dict.py", line 493, in <dictcomp>
    k: dataset.map(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2139, in map
    shards = [
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2140, in <listcomp>
    self.shard(num_shards=num_proc, index=rank, contiguous=True, keep_in_memory=keep_in_memory)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 3164, in shard
    return self.select(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 485, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/patrick/python_bin/datasets/fingerprint.py", line 411, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2756, in select
    return self._new_dataset_with_indices(indices_buffer=buf_writer.getvalue(), fingerprint=new_fingerprint)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2667, in _new_dataset_with_indices
    return Dataset(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 659, in __init__
    raise ValueError(
ValueError: External features info don't match the dataset:
Got
{'client_id': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None), 'down_votes': Value(dtype='int64', id=None), 'age': Value(dtype='string', id=None), 'gender': Value(dtype='string', id=None), 'accent': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None)}
with type
struct<client_id: string, path: string, audio: string, sentence: string, up_votes: int64, down_votes: int64, age: string, gender: string, accent: string, locale: string, segment: string>

but expected something like
{'client_id': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'audio': {'path': Value(dtype='string', id=None), 'bytes': Value(dtype='binary', id=None)}, 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None), 'down_votes': Value(dtype='int64', id=None), 'age': Value(dtype='string', id=None), 'gender': Value(dtype='string', id=None), 'accent': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None)}
with type
struct<client_id: string, path: string, audio: struct<path: string, bytes: binary>, sentence: string, up_votes: int64, down_votes: int64, age: string, gender: string, accent: string, locale: string, segment: string>
```

Versions:

```python
- `datasets` version: 1.16.2.dev0
- Platform: Linux-5.15.8-76051508-generic-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 6.0.1
```

and `transformers`:

```
- `transformers` version: 4.16.0.dev0
- Platform: Linux-5.15.8-76051508-generic-x86_64-with-glibc2.33
- Python version: 3.9.7
```


Update version of pib dataset and make it streamable

This PR:
- Updates version of pib dataset: from 0.0.0 to 1.3.0
- Makes the dataset streamable

Fix #3491.

CC: @severo


Add VoxLingua107

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** 107 languages, totaling 6628 hours for the train split.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Clone full repo to detect new tags when mirroring datasets on the Hub

The new releases of `datasets` were not detected because the shallow clone in the CI wasn't getting the git tags.

By cloning the full repository we can properly detect a new release, and tag all the dataset repositories accordingly
cc @SBrandeis


Fix VCTK encoding

utf-8 encoding was missing in the VCTK dataset builder added in #3351


Add `gzip` for `to_json`

(Partially) closes #3480. I have added `gzip` compression for `to_json`. I realised we can run into this compression problem with `to_csv` as well. `IOHandler` can be used for `to_csv` too. Please let me know if any changes are required.


Update version of pib dataset

Requesting to add a new dataset

On the Hub we have v0, while there exists v1.3.

Related to bigscience-workshop/data_tooling#130


Does datasets support load text from HDFS?

The raw text data is stored on HDFS due to the dataset's size is too large to store on my develop machine,
so I wander does datasets support read data from hdfs?


Avoid unnecessary list creations

Like in `join([... for s in ...])`. Also changed other things that I saw:
* Use a `with` statement for many `open` that missed them, so the files don't remain open.
* Remove unused variables.
* Many HTTP links converted into HTTPS (verified).
* Remove unnecessary "r" mode arg in `open` (double-checked it was actually the default in each case).
* Remove Python 2 style of using `super`.
* Run `pyupgrade $(find . -name "*.py" -type f) --py36-plus` (which already does some of the previous points).
* Run `dos2unix $(find . -name "*.py" -type f)` (CRLF to LF line endings).
* Fix typos.


URL query parameters are set as path in the compression hop for fsspec

bug

Something isn't working

## Describe the bug
There is an ssue with `StreamingDownloadManager._extract`.

I don't know how the test `test_streaming_gg_drive_gzipped` passes:

For
```python
TEST_GG_DRIVE_GZIPPED_URL = "https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz"
urlpath = StreamingDownloadManager().download_and_extract(TEST_GG_DRIVE_GZIPPED_URL)
```
gives `urlpath`:
```python
'gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz::https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz'
```

The gzip path makes no sense: `gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz`

## Steps to reproduce the bug
```python
from datasets.utils.streaming_download_manager import StreamingDownloadManager

dl_manager = StreamingDownloadManager()
urlpath = dl_manager.extract("https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz")
print(urlpath)
```

## Expected results
The query parameters should not be set as part of the path.


Update ADD_NEW_DATASET.md

fixed make style prompt for Windows Terminal


Fix weird spacing in ManualDownloadError message

`textwrap.dedent` works based on the spaces at the beginning. Before this change, there wasn't any space.


enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
When using `dataset.set_format("torch")`, I must make sure every columns in datasets can convert to `torch`, however, sometimes I want to keep some string columns.

**Describe the solution you'd like**
skip columns which cannot set to specific format when set_format instead of raise an error.


make shape verification to use ArrayXD instead of nested lists for map

New feature or request

As describe in https://github.com/huggingface/datasets/issues/2005#issuecomment-793716753 and mentioned by @mariosasko in [image feature example](https://colab.research.google.com/drive/1mIrTnqTVkWLJWoBzT1ABSe-LFelIep1c#scrollTo=ow3XHDvf2I0B&line=1&uniqifier=1), IMO make shape verifcaiton to use ArrayXD instead of nested lists for map can help user reduce unnecessary cast. I notice datasets have done something special for `input_ids` and `attention_mask` which is also unnecessary after this feature added.


Remove unused phony rule from Makefile


Fix duplicate keys in NewsQA

* Fix duplicate keys in NewsQA when loading from CSV files.
* Fix s/narqa/newsqa/ in the download manually error message.
* Make the download manually error message show nicely when printed. Otherwise, is hard to read due to spacing issues.
* Fix the format of the license text.
* Reformat the code to make it simpler.


Fix overriding of filesystem info

Previously, `BaseCompressedFileFileSystem.info` was overridden and transformed from function to dict.

This generated a bug for filesystem methods that use `self.info()`, like e.g. `fs.isfile()`.

This PR:
- Adds tests for `fs.isfile` (that use `fs.info`).
- Fixes custom `BaseCompressedFileFileSystem.info` by removing its overriding.


the compression format requested when saving a dataset in json format is not respected

bug

Something isn't working

## Describe the bug
In the documentation of the `to_json` method, it is stated in the parameters that
> **to_json_kwargs – Parameters to pass to pandas’s pandas.DataFrame.to_json.

however when we pass for example `compression="gzip"`, the saved file is not compressed.

Would you also have expected compression to be applied? :relaxed:

## Steps to reproduce the bug
```python
my_dict = {"a": [1, 2, 3], "b": [1, 2, 3]}
```
### Result with datasets
```python
from datasets import Dataset

dataset = Dataset.from_dict(my_dict)
dataset.to_json("dic_with_datasets.jsonl.gz", compression="gzip")
!cat dic_with_datasets.jsonl.gz
```
output
```
{"a":1,"b":1}
{"a":2,"b":2}
{"a":3,"b":3}
```
Note: I would expected to see binary data here

### Result with pandas
```python
import pandas as pd

df = pd.DataFrame(my_dict)
df.to_json("dic_with_pandas.jsonl.gz", lines=True, orient="records", compression="gzip")
!cat dic_with_pandas.jsonl.gz
```
output
```
4��a�dic_with_pandas.jsonl��VJT�2�QJ��\� ��g��yƵ���������)���
```
Note: It looks like binary data

## Expected results

I would have expected that the saved result with datasets would also be a binary file

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyArrow version: 5.0.0


Dataset preview is not available (I think for all Hugging Face datasets)

Something isn't working

dataset-viewer

Related to the dataset viewer on huggingface.co

## Dataset viewer issue for '*french_book_reviews*'

**Link:** https://huggingface.co/datasets/Abirate/french_book_reviews

**short description of the issue**
For my dataset, the dataset preview is no longer functional (it used to work: The dataset had been added the day before and it was fine...)
And, after looking over the datasets, I discovered that this issue affects all Hugging Face datasets (as of yesterday, December 23, 2021, around 10 p.m. (CET)).
**Am I the one who added this dataset** :  Yes

**Note**: here a screenshot showing the issue
![Dataset preview is not available for my dataset](https://user-images.githubusercontent.com/66887439/147333078-60734578-420d-4e91-8691-a90afeaa8948.jpg)

**And here for glue dataset :**

![Dataset preview is not available for other Hugging Face datasets(glue)](https://user-images.githubusercontent.com/66887439/147333492-26fa530c-befd-4992-8361-70c51397a25a.jpg)


Extend support for streaming datasets that use os.walk

This PR extends the support in streaming mode for datasets that use `os.walk`, by patching that function.

This PR adds support for streaming mode to datasets:
1. autshumato
1. code_x_glue_cd_code_to_text
1. code_x_glue_tc_nl_code_search_adv
1. nchlt

CC: @severo


Use `iter_files` instead of `str(Path(...)` in image dataset

Use `iter_files` in the `beans` and the `cats_vs_dogs` dataset scripts as suggested by @albertvillanova.

Additional changes:
* Fix `iter_files` in `MockDownloadManager` (see this [CI error](https://app.circleci.com/pipelines/github/huggingface/datasets/9247/workflows/2657ff8a-b531-4fd9-a9fc-6541a72e8d83/jobs/57028))
* Add support for `os.path.isdir` and `os.path.isfile` in streaming (`os.path.isfile` is needed in `StreamingDownloadManager`'s `iter_files` to make `cats_vs_dogs` streamable)

TODO:
- [ ] add tests for `xisdir` and `xisfile`


Extend support for streaming datasets that use ET.parse

This PR extends the support in streaming mode for datasets that use `ET.parse`, by patching the function.

This PR adds support for streaming mode to datasets:
1. ami
1. assin
1. assin2
1. counter
1. enriched_web_nlg
1. europarl_bilingual
1. hyperpartisan_news_detection
1. polsum
1. qa4mre
1. quail
1. ted_talks_iwslt
1. udhr
1. web_nlg
1. winograd_wsc

CC: @severo


The rotten_tomatoes dataset of movie reviews contains some reviews in Spanish

bug

Something isn't working

## Describe the bug
See title. I don't think this is intentional and they probably should be removed. If they stay the dataset description should be at least updated to make it clear to the user.

## Steps to reproduce the bug
Go to the [dataset viewer](https://huggingface.co/datasets/viewer/?dataset=rotten_tomatoes) for the dataset, set the offset to 4160 for the train dataset, and scroll through the results. I found ones at index 4166 and 4173. There's others too (e.g. index 2888) but those two are easy to find like that.

## Expected results
English movie reviews only.

## Actual results
Example of a Spanish movie review (4173):
> "É uma pena que , mais tarde , o próprio filme abandone o tom de paródia e passe a utilizar os mesmos clichês que havia satirizado "


Decode images when iterating

If I iterate over a vision dataset, the images are not decoded, and the dictionary with the bytes is returned.

This PR enables image decoding in `Dataset.__iter__`

Close https://github.com/huggingface/datasets/issues/3473


Iterating over a vision dataset doesn't decode the images

Something isn't working

vision

Vision datasets

## Describe the bug

If I load `mnist` and I iterate over the dataset, the images are not decoded, and the dictionary with the bytes is returned.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import PIL

mnist = load_dataset("mnist", split="train")

first_image = mnist[0]["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # passes
first_image = next(iter(mnist))["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # fails
```

## Expected results

The image should be decoded, as a PIL Image

## Actual results

We get a dictionary
```
{'bytes': b'\x89PNG\r\n\x1a\n\x00..., 'path': None}
```

## Environment info

- `datasets` version: 1.17.1.dev0
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyArrow version: 6.0.0

The bug also exists in 1.17.0

## Investigation

I think the issue is that decoding is disabled in `__iter__`:

https://github.com/huggingface/datasets/blob/dfe5b73387c5e27de6a16b0caeb39d3b9ded66d6/src/datasets/arrow_dataset.py#L1651-L1661

Do you remember why it was disabled in the first place @albertvillanova ?
Also cc @mariosasko @NielsRogge


Fix `str(Path(...))` conversion in streaming on Linux. This should fix the streaming of the `beans` and `cats_vs_dogs` datasets.


Fix Tashkeela dataset to yield stripped text

This PR:
- Yields stripped text
- Fix path for Windows
- Adds license
- Adds more info in dataset card

Close bigscience-workshop/data_tooling#279


Fix rendering of docs

Minor fix in docs.

Currently, `ClassLabel` docstring rendering is not right.


Fix METEOR missing NLTK's omw-1.4

NLTK 3.6.6 now requires `omw-1.4` to be downloaded for METEOR to work.
This should fix the CI on master


Add COCO dataset

This PR adds the MS COCO dataset. Compared to the [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/object_detection/coco.py) script, this implementation adds 8 additional configs to cover the tasks other than object detection.

Some notes:
* the data exposed by TFDS is contained in the `2014`, `2015`, `2017` and `2017_panoptic_segmentation` configs here
* I've updated `encode_nested_example` for easier handling of missing values (cc @lhoestq @albertvillanova; will add tests if you are OK with the changes in `features.py`)
* this implementation should fix https://github.com/huggingface/datasets/pull/3377#issuecomment-985559427

TODOs:
- [x] dataset card
- [ ] dummy data

cc @merveenoyan

Closes #2526


Push dataset infos.json to Hub

When doing `push_to_hub`, the feature types are lost (see issue https://github.com/huggingface/datasets/issues/3394).
This PR fixes this by also pushing a `dataset_infos.json` file to the Hub, that stores the feature types.

Other minor changes:
- renamed the `___` separator to `--`, since `--` is now disallowed in a name in the back-end.

I tested this feature with datasets like conll2003 that has feature types like `ClassLabel` that were previously lost.

Close https://github.com/huggingface/datasets/issues/3394

I would like to include this in today's release (though not mandatory), so feel free to comment/suggest changes


Add CRASS dataset

Added crass dataset


Unable to load 'cnn_dailymail' dataset

bug

Something isn't working

duplicate

This issue or pull request already exists

## Describe the bug
I wanted to load cnn_dailymail dataset from huggingface datasets on Google Colab, but I am getting an error while loading it.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('cnn_dailymail', '3.0.0', ignore_verifications = True)
```

## Expected results
Expecting to load 'cnn_dailymail' dataset.

## Actual results
`NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0


struct.error: 'i' format requires -2147483648 <= number <= 2147483647

bug

Something isn't working

## Describe the bug
A clear and concise description of what the bug is.

using latest datasets=datasets-1.16.1-py3-none-any.whl
process my own multilingual dataset by following codes, and the number of rows in all dataset is 306000, the max_length of each sentence is 256:
![image](https://user-images.githubusercontent.com/30341159/146865779-3d25d011-1f42-4026-9e1b-76f6e1d172e9.png)

then I get this error:
![image](https://user-images.githubusercontent.com/30341159/146865844-e60a404c-5f3a-403c-b2f1-acd943b5cdb8.png)

I have seen the issue in #2134 and #2150, so I don't understand why latest repo still can't deal with big dataset.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux docker
- Python version: 3.6


Update swahili_news dataset

Update dataset with latest verion data files.

Fix #3462.
Close bigscience-workshop/data_tooling#107


Update swahili_news dataset

dataset request

Requesting to add a new dataset

Please note also: the HuggingFace version at https://huggingface.co/datasets/swahili_news is outdated. An updated version, with deduplicated text and official splits, can be found at https://zenodo.org/record/5514203.

## Adding a Dataset
- **Name:** swahili_news

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Related to:
- bigscience-workshop/data_tooling#107


Fix links in metrics description

Remove Markdown syntax for links in metrics description, as it is not properly rendered.

Related to #3437.


Don't encode lists as strings when using `Value("string")`

Following https://github.com/huggingface/datasets/pull/3456#event-5792250497 it looks like `datasets` can silently convert lists to strings using `str()`, instead of raising an error.

This PR fixes this and should fix the issue with WER showing low values if the input format is not right.


dataset.filter overwriting previously set dataset._indices values, resulting in the wrong elements being selected.

bug

Something isn't working

## Describe the bug
When using dataset.select to select a subset of a dataset, dataset._indices are set to indicate which elements are now considered in the dataset.
The same thing happens when you shuffle the dataset; dataset._indices are set to indicate what the new order of the data is.
However, if you then use a dataset.filter, that filter interacts with those dataset._indices values in a non-intuitive manner.
https://huggingface.co/docs/datasets/_modules/datasets/arrow_dataset.html#Dataset.filter

Effectively, it looks like the original set of _indices were discared and overwritten by the set created during the filter operation.

I think this is actually an issue with how the map function handles dataset._indices. Ideally it should use the _indices it gets passed, and then return an updated _indices which reflect the map transformation applied to the starting _indices.

## Steps to reproduce the bug
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print("initial 10 elements")
print(dataset['label'])   # -> [1, 1, 0, 1, 0, 0, 0, 1, 0, 0]

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)
print("filtered 10 elements looking for label 0")
print(dataset['label'])  # -> [1, 1, 1, 1, 1, 1]
```

## Actual results
```
$ python indices_bug.py
initial 10 elements
[1, 1, 0, 1, 0, 0, 0, 1, 0, 0]
filtered 10 elements looking for label 0
[1, 1, 1, 1, 1, 1]
```

This code block first shuffles the dataset (to get a mix of label 0 and label 1).
Then it selects just the first 10 elements (the number of elements does not matter, 10 is just easy to visualize). The important part is that you select some subset of the dataset.
Finally, a filter is applied to pull out just the elements with `label == 0`.

The bug is that you cannot combine any dataset operation which sets the dataset._indices with filter.
In this case I have 2, shuffle and subset.

If you just use a single dataset._indices operation (in this case shuffle) the bug still shows up.

The shuffle sets the dataset._indices and then filter uses those indices in the map, then overwrites dataset._indices with the filter results.
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print(dataset['label']) # -> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

## Expected results
In an ideal world, the dataset filter would respect any dataset._indices values which had previously been set.

If you use dataset.filter with the base dataset (where dataset._indices has not been set) then the filter command works as expected.

## Environment info
Here are the commands required to rebuild the conda environment from scratch.
```
# create a virtual environment
conda create -n dataset_indices python=3.8 -y

# activate the virtual environment
conda activate dataset_indices

# install huggingface datasets
conda install datasets
```

<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.11.0-41-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 3.0.0

### Full Conda Environment
```
$ conda env export
name: dasaset_indices
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - abseil-cpp=20210324.2=h2531618_0
  - aiohttp=3.8.1=py38h7f8727e_0
  - aiosignal=1.2.0=pyhd3eb1b0_0
  - arrow-cpp=3.0.0=py38h6b21186_4
  - attrs=21.2.0=pyhd3eb1b0_0
  - aws-c-common=0.4.57=he6710b0_1
  - aws-c-event-stream=0.1.6=h2531618_5
  - aws-checksums=0.1.9=he6710b0_0
  - aws-sdk-cpp=1.8.185=hce553d0_0
  - bcj-cffi=0.5.1=py38h295c915_0
  - blas=1.0=mkl
  - boost-cpp=1.73.0=h27cfd23_11
  - bottleneck=1.3.2=py38heb32a55_1
  - brotli=1.0.9=he6710b0_2
  - brotli-python=1.0.9=py38heb0550a_2
  - brotlicffi=1.0.9.2=py38h295c915_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.10.26=h06a4308_2
  - certifi=2021.10.8=py38h06a4308_0
  - cffi=1.14.6=py38h400218f_0
  - conllu=4.4.1=pyhd3eb1b0_0
  - cryptography=36.0.0=py38h9ce1e76_0
  - dataclasses=0.8=pyh6d0b6a4_7
  - dill=0.3.4=pyhd3eb1b0_0
  - double-conversion=3.1.5=he6710b0_1
  - et_xmlfile=1.1.0=py38h06a4308_0
  - filelock=3.4.0=pyhd3eb1b0_0
  - frozenlist=1.2.0=py38h7f8727e_0
  - gflags=2.2.2=he6710b0_0
  - glog=0.5.0=h2531618_0
  - gmp=6.2.1=h2531618_2
  - grpc-cpp=1.39.0=hae934f6_5
  - huggingface_hub=0.0.17=pyhd3eb1b0_0
  - icu=58.2=he6710b0_3
  - idna=3.3=pyhd3eb1b0_0
  - importlib-metadata=4.8.2=py38h06a4308_0
  - importlib_metadata=4.8.2=hd3eb1b0_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - krb5=1.19.2=hac12032_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libboost=1.73.0=h3ff78a5_11
  - libcurl=7.80.0=h0b77cf5_0
  - libedit=3.1.20210910=h7f8727e_0
  - libev=4.33=h7f8727e_1
  - libevent=2.1.8=h1ba5d50_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libnghttp2=1.46.0=hce63b2e_0
  - libprotobuf=3.17.2=h4ff587b_1
  - libssh2=1.9.0=h1ba5d50_1
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - libthrift=0.14.2=hcc01f38_0
  - libxml2=2.9.12=h03d6c58_0
  - libxslt=1.1.34=hc22bd24_0
  - lxml=4.6.3=py38h9120a33_0
  - lz4-c=1.9.3=h295c915_1
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py38h7f8727e_0
  - mkl_fft=1.3.1=py38hd3c417c_0
  - mkl_random=1.2.2=py38h51133e4_0
  - multiprocess=0.70.12.2=py38h7f8727e_0
  - multivolumefile=0.2.3=pyhd3eb1b0_0
  - ncurses=6.3=h7f8727e_2
  - numexpr=2.7.3=py38h22e1b3c_1
  - numpy=1.21.2=py38h20f2e39_0
  - numpy-base=1.21.2=py38h79a1101_0
  - openpyxl=3.0.9=pyhd3eb1b0_0
  - openssl=1.1.1l=h7f8727e_0
  - orc=1.6.9=ha97a36c_3
  - packaging=21.3=pyhd3eb1b0_0
  - pip=21.2.4=py38h06a4308_0
  - py7zr=0.16.1=pyhd3eb1b0_1
  - pycparser=2.21=pyhd3eb1b0_0
  - pycryptodomex=3.10.1=py38h27cfd23_1
  - pyopenssl=21.0.0=pyhd3eb1b0_1
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pyppmd=0.16.1=py38h295c915_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.12=h12debd9_0
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python-xxhash=2.0.2=py38h7f8727e_0
  - pyzstd=0.14.4=py38h7f8727e_3
  - re2=2020.11.01=h2531618_1
  - readline=8.1=h27cfd23_0
  - requests=2.26.0=pyhd3eb1b0_0
  - setuptools=58.0.4=py38h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - snappy=1.1.8=he6710b0_0
  - sqlite=3.36.0=hc218d9a_0
  - texttable=1.6.4=pyhd3eb1b0_0
  - tk=8.6.11=h1ccaba5_0
  - typing_extensions=3.10.0.2=pyh06a4308_0
  - uriparser=0.9.3=he6710b0_1
  - utf8proc=2.6.1=h27cfd23_0
  - wheel=0.37.0=pyhd3eb1b0_1
  - xxhash=0.8.0=h7f8727e_3
  - xz=5.2.5=h7b6447c_0
  - zipp=3.6.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7f8727e_4
  - zstd=1.4.9=haebb681_0
  - pip:
    - async-timeout==4.0.2
    - charset-normalizer==2.0.9
    - datasets==1.16.1
    - fsspec==2021.11.1
    - huggingface-hub==0.2.1
    - multidict==5.2.0
    - pandas==1.3.5
    - pyarrow==6.0.1
    - pytz==2021.3
    - pyyaml==6.0
    - tqdm==4.62.3
    - typing-extensions==4.0.1
    - urllib3==1.26.7
    - yarl==1.7.2

```


Fix duplicated tag in wikicorpus dataset card


Add CMU Graphics Lab Motion Capture dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** CMU Graphics Lab Motion Capture database
- **Description:** The database contains free motions which you can download and use.
- **Data:** http://mocap.cs.cmu.edu/
- **Motivation:** Nice motion capture dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


[WER] Better error message for wer

Currently we have the following problem when using the WER. When the input format to the WER metric is wrong, instead of throwing an error message a word-error-rate is computed which is incorrect. E.g. when doing the following:

```python
from datasets import load_metric

wer = load_metric("wer")

target_str = ["hello this is nice", "hello the weather is bloomy"]
pred_str = [["hello it's nice"], ["hello it's the weather"]]

print("Wrong:", wer.compute(predictions=pred_str, references=target_str))
print("Correct", wer.compute(predictions=[x[0] for x in pred_str], references=target_str))
```
We get:
```
Wrong: 1.0
Correct 0.5555555555555556
```

meaning that we get a word-error rate for incorrectly passed input formats. We should raise an error here instead so that people don't spent hours fixing a model while it's their incorrect evaluation metric is the problem for a low WER.


Easier information editing

enhancement

New feature or request

generic discussion

Generic discussion on the library

**Is your feature request related to a problem? Please describe.**
It requires a lot of effort to improve a datasheet.

**Describe the solution you'd like**
UI or at least a link to the place where the code that needs to be edited is (and an easy way to edit this code directly from the site, without cloning, branching, makefile etc.)

**Describe alternatives you've considered**
The current Ux is to have the 8 steps for contribution while One just wishes to change a line a type etc.


Fix iter_archive generator

This PR:
- Adds tests to DownloadManager and StreamingDownloadManager `iter_archive` for both path and file inputs
- Fixes bugs in `iter_archive` introduced in:
  - #3443

Fix #3453.


ValueError while iter_archive

bug

Something isn't working

## Describe the bug
After the merge of:
- #3443

the method `iter_archive` throws a ValueError:
```
ValueError: read of closed file
```

## Steps to reproduce the bug
```python
for path, file in dl_manager.iter_archive(archive_path):
    pass
```


enhancement

New feature or request

good second issue

Issues a bit more difficult than "Good First" issues

why the stratify option is omitted from test_train_split function?

is there any other way implement the stratify option while splitting the dataset? as it is important point to be considered while splitting the dataset.


[Staging] Update dataset repos automatically on the Hub

Let's have a script that updates the dataset repositories on staging for now. This way we can make sure it works fine before going in prod.

Related to https://github.com/huggingface/datasets/issues/3341

The script runs on each commit on `master`. It checks the datasets that were changed, and it pushes the changes to the corresponding repositories on the Hub.

If there's a new dataset, then a new repository is created.

If the commit is a new release of `datasets`, it also pushes the tag to all the repositories.


Unexpected behavior doing Split + Filter

bug

Something isn't working

## Describe the bug
I observed unexpected behavior when applying 'train_test_split' followed by 'filter' on dataset. Elements of the training dataset eventually end up in the test dataset (after applying the 'filter')

## Steps to reproduce the bug
```
from datasets import Dataset
import pandas as pd
dic = {'x': [1,2,3,4,5,6,7,8,9], 'y':['q','w','e','r','t','y','u','i','o']}
df = pd.DataFrame.from_dict(dic)
dataset = Dataset.from_pandas(df)
split_dataset = dataset.train_test_split(test_size=0.5, shuffle=False, seed=42)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]
eval_dataset_2 = eval_dataset.filter(lambda example: example['x'] % 2 == 0)
print( eval_dataset['x'])
print(eval_dataset_2['x'])
```
One observes that elements in eval_dataset2 are actually coming from the training dataset...

## Expected results
The expected results would be that the filtered eval dataset would only contain elements from the original eval dataset.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Windows 10
- Python version: 3.7
- PyArrow version: 5.0.0


Add `__add__()`, `__iadd__()` and similar to `Dataset` class

enhancement

New feature or request

generic discussion

Generic discussion on the library

**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
I would like to be able to concatenate datasets as follows:
```python
>>> dataset["train"] += dataset["validation"]
```

... instead of using `concatenate_datasets()`:
```python
>>> raw_datasets["train"] = concatenate_datasets([raw_datasets["train"], raw_datasets["validation"]])
>>> del raw_datasets["validation"]
```

**Describe alternatives you've considered**
Well, I have considered `concatenate_datasets()` 😀

**Additional context**
N.a.


JSONDecodeError with HuggingFace dataset viewer

## Dataset viewer issue for 'pubmed_neg'

**Link:** https://huggingface.co/datasets/IGESML/pubmed_neg

I am getting the error:
Status code:   400
Exception:     JSONDecodeError
Message:       Expecting property name enclosed in double quotes: line 61 column 2 (char 1202)

I have checked all files - I am not using single quotes anywhere. Not sure what is causing this issue.

Am I the one who added this dataset ? Yes


HF_DATASETS_OFFLINE=1 didn't stop datasets.builder from downloading

bug

Something isn't working

## Describe the bug
According to https://huggingface.co/docs/datasets/loading_datasets.html#loading-a-dataset-builder, setting HF_DATASETS_OFFLINE to 1 should make datasets to "run in full offline mode". It didn't work for me. At the very beginning, datasets still tried to download "custom data configuration" for JSON, despite I have run the program once and cached all data into the same --cache_dir.

"Downloading" is not an issue when running with local disk, but crashes often with cloud storage because (1) multiply GPU processes try to access the same file, AND (2) FileLocker fails to synchronize all processes, due to storage throttling. 99% of times, when the main process releases FileLocker, the file is not actually ready for access in cloud storage and thus triggers "FileNotFound" errors for all other processes. Well, another way to resolve the problem is to investigate super reliable cloud storage, but that's out of scope here.

## Steps to reproduce the bug
```
export HF_DATASETS_OFFLINE=1
python run_clm.py --model_name_or_path=models/gpt-j-6B --train_file=trainpy.v2.train.json --validation_file=trainpy.v2.eval.json --cache_dir=datacache/trainpy.v2
```

## Expected results
datasets should stop all "downloading" behavior but reuse the cached JSON configuration. I think the problem here is part of the cache directory path, "default-471372bed4b51b53", is randomly generated, and it could change if some parameters changed. And I didn't find a way to use a fixed path to ensure datasets to reuse cached data every time.

## Actual results
The logging shows datasets are still downloading into "datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426".
```
12/16/2021 10:25:59 - WARNING - datasets.builder - Using custom data configuration default-471372bed4b51b53
12/16/2021 10:25:59 - INFO - datasets.builder - Generating dataset json (datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426)
Downloading and preparing dataset json/default to  datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|██████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 17623.13it/s]
12/16/2021 10:25:59 - INFO - datasets.utils.download_manager - Downloading took 0.0 min
12/16/2021 10:26:00 - INFO - datasets.utils.download_manager - Checksum Computation took 0.0 min
100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1206.99it/s]
12/16/2021 10:26:00 - INFO - datasets.utils.info_utils - Unable to verify checksums.
12/16/2021 10:26:00 - INFO - datasets.builder - Generating split train
12/16/2021 10:26:01 - INFO - datasets.builder - Generating split validation
12/16/2021 10:26:02 - INFO - datasets.utils.info_utils - Unable to verify splits sizes.
Dataset json downloaded and prepared to  datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 53.54it/s]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux
- Python version:  3.8.10
- PyArrow version: 6.0.1


Remove redundant local path information in audio/image datasets

Remove the redundant path information in the audio/image dataset as discussed in https://github.com/huggingface/datasets/pull/3430#issuecomment-994734828

TODOs:
* [ ] merge https://github.com/huggingface/datasets/pull/3430
* [ ] merge https://github.com/huggingface/datasets/pull/3364
* [ ] re-generate the info files of the updated audio datasets

cc:  @patrickvonplaten @anton-l @nateraw (I expect this to break the audio/vision examples in Transformers; after this change you'll be able to access underlying paths as follows `dset = dset.cast_column("audio", Audio(..., decode=False)); path = dset[0]["audio"]`)


question

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Align the Dataset and IterableDataset processing API

enhancement

New feature or request

generic discussion

Generic discussion on the library

## Intro

Currently the two classes have two distinct API for processing:

### The `.map()` method

Both have those parameters in common: function, batched, batch_size

- IterableDataset is missing those parameters:
with_indices, with_rank, input_columns, drop_last_batch, remove_columns, features, disable_nullable, fn_kwargs, num_proc

- Dataset also has additional parameters that are exclusive, due to caching:
keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, suffix_template, new_fingerprint

- There is also an important difference in terms of behavior:
**Dataset.map adds new columns** (with dict.update)
BUT
**IterableDataset discards previous columns** (it overwrites the dict)
IMO the two methods should have the same behavior. This would be an important breaking change though.

- Dataset.map is eager while IterableDataset.map is lazy

### The `.shuffle()` method

- Both have an optional seed parameter, but IterableDataset requires a mandatory parameter buffer_size to control the size of the local buffer used for approximate shuffling.

- IterableDataset is missing the parameter generator

- Also Dataset has exclusive parameters due to caching: keep_in_memory, load_from_cache_file, indices_cache_file_name, writer_batch_size, new_fingerprint

### The `.with_format()` method

- IterableDataset only supports "torch" (it misses tf, jax, pandas, arrow) and is missing the parameters: columns, output_all_columns and format_kwargs

### Other methods

- Both have the same `remove_columns` method
- IterableDataset is missing: cast, cast_column, filter, rename_column, rename_columns, class_encode_column, flatten, prepare_for_task, train_test_split, shard
- Some other methods are missing but we can discuss them: set_transform, formatted_as, with_transform
- And others don't really make sense for an iterable dataset: select, sort, add_column, add_item
- Dataset is missing skip and take, that IterableDataset implements.

## Questions

I think it would be nice to be able to switch between streaming and regular dataset easily, without changing the processing code significantly.

1. What should be aligned and what shouldn't between those two APIs ?

IMO the minimum is to align the main processing methods.

It would mean aligning breaking the current `Iterable.map` to have the same behavior as `Dataset.map` (add columns with dict.update), and add multiprocessing as well as the missing parameters.

It would also mean implementing the missing methods: cast, cast_column, filter, rename_column, rename_columns, class_encode_column, flatten, prepare_for_task, train_test_split, shard

2. What are the breaking changes for IterableDataset ?

The main breaking change would be the change of behavior of `IterableDataset.map`, because currently it discards all the previous columns instead of keeping them.

3. Shall we also do some changes for regular datasets ?

I agree the simplest would be to have the exact same methods for both Dataset and IterableDataset. However this is probably not a good idea because it would prevent users from using the best benefits of them. That's why we can keep some aspects of regular datasets as they are:
- keep the eager Dataset.map with caching
- keep the with_transform method for lazy processing
- keep Dataset.select (it could also be added to IterableDataset even though it's not recommended)

We could have a completely aligned `map` method if both methods were lazy by default, but this is a very big breaking change so I'm not sure we can consider doing that.

For information, TFDS does lazy map by default, and has an additional `.cache()` method.

## Opinions ?

I'd love to gather some opinions about this here. If the two APIs are more aligned it would be awesome for the examples in `transformers`, and it would create a satisfactory experience for users that want to switch from one mode to the other.

cc @mariosasko @albertvillanova @thomwolf @patrickvonplaten @sgugger


Extend iter_archive to support file object input

This PR adds support to passing a file object to `[Streaming]DownloadManager.iter_archive`.

With this feature, we can iterate over a tar file inside another tar file.


Extend text to support yielding lines, paragraphs or documents

Add `config.row` option to `text` module to allow yielding lines (default, current case), paragraphs or documents.

Feel free to comment on the name of the config parameter `row`:
- Currently, the docs state datasets are made of rows and columns
- Other names I considered: `example`, `item`


Add QuALITY dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** QuALITY
- **Description:** A challenging question answering with very long contexts (Twitter [thread](https://twitter.com/sleepinyourhat/status/1471225421794529281?s=20))
- **Paper:** No ArXiv link yet, but draft is [here](https://github.com/nyu-mll/quality/blob/main/quality_preprint.pdf)
- **Data:** GitHub repo [here](https://github.com/nyu-mll/quality)
- **Motivation:** This dataset would serve as a nice way to benchmark long-range Transformer models like BigBird, Longformer and their descendants. In particular, it would be very interesting to see how the S4 model fares on this given it's impressive performance on the Long Range Arena

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


datasets keeps reading from cached files, although I disabled it

bug

Something isn't working

## Describe the bug
Hi,
I am trying to avoid dataset library using cached files, I get the following bug when this tried to read the cached files. I tried to do the followings:
```
from datasets import set_caching_enabled
set_caching_enabled(False)
```
also force redownlaod:
```
 download_mode='force_redownload'
```
but none worked so far, this is on a cluster and on some of the machines this reads from the cached files, I really appreciate any idea on how to fully remove caching @lhoestq
many thanks

```
File "run_clm.py", line 496, in <module>
    main()
  File "run_clm.py", line 419, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 943, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/transformers/trainer.py", line 1445, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 172, in evaluate
    output = self.eval_loop(
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 241, in eval_loop
    metrics = self.compute_pet_metrics(eval_datasets, model, self.extra_info[metric_key_prefix], task=task)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 268, in compute_pet_metrics
    centroids = self._compute_per_token_train_centroids(model, task=task)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 353, in _compute_per_token_train_centroids
    data = get_label_samples(self.get_per_task_train_dataset(task), label)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 350, in get_label_samples
    return dataset.filter(lambda example: int(example['labels']) == label)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 470, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2519, in filter
    indices = self.map(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2036, in map
    return self._map_single(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 503, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 470, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2248, in _map_single
    return Dataset.from_file(cache_file_name, info=info, split=self.split)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 654, in from_file
    return cls(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 593, in __init__
    self.info.features = self.info.features.reorder_fields_as(inferred_features)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/features/features.py", line 1092, in reorder_fields_as
    return Features(recursive_reorder(self, other))
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/features/features.py", line 1081, in recursive_reorder
    raise ValueError(f"Keys mismatch: between {source} and {target}" + stack_position)
ValueError: Keys mismatch: between {'indices': Value(dtype='uint64', id=None)} and {'candidates_ids': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None), 'labels': Value(dtype='int64', id=None), 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None), 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), 'extra_fields': {}, 'task': Value(dtype='string', id=None)}

```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 6.0.1


Add `cast_column` to `IterableDataset`

Closes #3369.

cc: @patrickvonplaten


Update supported versions of Python in setup.py

Update the list of supported versions of Python in `setup.py` to keep the PyPI project description updated.


Update BLEURT hyperlink

The description of BLEURT on the hf.co website has a strange use of URL hyperlinking. This PR attempts to fix this, although I am not 100% sure Markdown syntax is allowed on the frontend or not.

![Screen Shot 2021-12-15 at 17 31 27](https://user-images.githubusercontent.com/26859204/146226432-c83cbdaf-f57d-4999-b53c-85da718ff7fb.png)


Add the OneStopQa dataset

Adding OneStopQA,  a multiple choice reading comprehension dataset annotated according to the STARC (Structured Annotations for Reading Comprehension) scheme.


Improve Wikipedia Loading Script

* More structured approach to detecting redirects
* Remove redundant template filter code (covered by strip_code)
* Add language-specific lists of additional media namespace aliases for filtering
* Add language-specific lists of category namespace aliases for new link text cleaning step
* Remove magic words (parser directions like __TOC__ that occasionally occur in text)

Fix #3400

With support from @albertvillanova
CC @yjernite


Add The People's Speech

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** The People's Speech
- **Description:** a massive English-language dataset of audio transcriptions of full sentences.
- **Paper:** https://openreview.net/pdf?id=R8CwidgJ0yT
- **Data:** https://mlcommons.org/en/peoples-speech/
- **Motivation:** With over 30,000 hours of speech, this dataset is the largest and most diverse freely available English speech recognition corpus today.

[The article](https://thegradient.pub/new-datasets-to-democratize-speech-recognition-technology-2/) which may be useful when working on the dataset.

cc: @anton-l

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add Multilingual Spoken Words dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Multilingual Spoken Words
- **Description:** Multilingual Spoken Words Corpus is a large and growing audio dataset of spoken words in 50 languages for academic research and commercial applications in keyword spotting and spoken term search, licensed under CC-BY 4.0. The dataset contains more than 340,000 keywords, totaling 23.4 million 1-second spoken examples (over 6,000 hours).

  Read more: https://mlcommons.org/en/news/spoken-words-blog/
- **Paper:** https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/fe131d7f5a6b38b23cc967316c13dae2-Paper-round2.pdf
- **Data:** https://mlcommons.org/en/multilingual-spoken-words/
- **Motivation:**

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Correctly indent builder config in dataset script docs


Unable to resolve any data file after loading once

when I rerun my program, it occurs this error
" Unable to resolve any data file that matches '['**train*']' at /data2/whr/lzy/open_domain_data/retrieval/wiki_dpr with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'zip']", so how could i deal with this problem?
thx.
And below is my code .
![image](https://user-images.githubusercontent.com/84694183/146023446-d75fdec8-65c1-484f-80d8-6c20ff5e994b.png)


Make decoding of Audio and Image feature optional

Add the `decode` argument (`True` by default) to the `Audio` and the `Image` feature to make it possible to toggle on/off decoding of these features.

Even though we've discussed that on Slack, I'm not removing the `_storage_dtype` argument of the Audio feature in this PR to avoid breaking the Audio feature tests.


Make cast cacheable (again) on Windows

`cast` currently emits the following warning when called on Windows:
```
Parameter 'function'=<function Dataset.cast.<locals>.<lambda> at 0x000001C930571EA0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting
and caching to work. If you reuse this transform, the caching mechanism will consider it to be different
from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
```

It seems like the issue stems from the `config.PYARROW_VERSION` object not being serializable on Windows (tested with `dumps(lambda: config.PYARROW_VERSION)`), so I'm fixing this by capturing `config.PYARROW_VERSION.major` before the lambda definition.


Clean squad dummy data

Some unused files were remaining, this PR removes them. We just need to keep the dummy_data.zip file


Add The Pile Enron Emails subset

Add:
- Enron Emails subset of The Pile: "enron_emails" config

Close bigscience-workshop/data_tooling#310.

CC: @StellaAthena


Update disaster_response_messages download urls (+ add validation split)

Fixes #3240, fixes #3416


Getting configs names takes too long

bug

Something isn't working

## Steps to reproduce the bug
```python
from datasets import get_dataset_config_names
get_dataset_config_names("allenai/c4")
```

## Expected results

I would expect to get the answer quickly, at least in less than 10s

## Actual results

It takes about 45s on my environment

## Environment info

- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Add RedCaps dataset

Add the RedCaps dataset. I'm not adding the generated `dataset_infos.json` file for now due to its size (11 MB).

TODOs:
- [x] dummy data
- [x] dataset card

Close #3316


data duplicate when setting num_works > 1 with streaming data

Something isn't working

## Describe the bug
The data is repeated num_works times when we load_dataset with streaming and set num_works > 1 when construct dataloader

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import pandas as pd
import numpy as np
import os

from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm
import shutil

NUM_OF_USER = 1000000
NUM_OF_ACTION = 50000
NUM_OF_SEQUENCE = 10000
NUM_OF_FILES = 32
NUM_OF_WORKERS = 16

if __name__ == "__main__":
    shutil.rmtree("./dataset")
    for i in range(NUM_OF_FILES):
        sequence_data = pd.DataFrame(
            {
                "imei": np.random.randint(1, NUM_OF_USER, size=NUM_OF_SEQUENCE),
                "sequence": np.random.randint(1, NUM_OF_ACTION, size=NUM_OF_SEQUENCE)
            }
        )

        if not os.path.exists("./dataset"):
            os.makedirs("./dataset")

        sequence_data.to_csv(f"./dataset/sequence_data_{i}.csv",

 index=False)

    dataset = load_dataset("csv",
                           data_files=[os.path.join("./dataset",file) for file in os.listdir("./dataset") if file.endswith(".csv")],
                           split="train",
                           streaming=True).with_format("torch")
    data_loader = DataLoader(dataset,
                             batch_size=1024,
                             num_workers=NUM_OF_WORKERS)

    result = pd.DataFrame()
    for i, batch in tqdm(enumerate(data_loader)):
        result = pd.concat([result,
                           pd.DataFrame(batch)],
                           axis=0)
    result.to_csv(f"num_work_{NUM_OF_WORKERS}.csv", index=False)

```

## Expected results
data do not duplicate

## Actual results
data duplicate NUM_OF_WORKERS = 16
![image](https://user-images.githubusercontent.com/16486492/145748707-9d2df25b-2f4f-4d7b-a83e-242be4fc8934.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:datasets==1.14.0
- Platform:transformers==4.11.3
- Python version:3.8
- PyArrow version:


Error about load_metric

Something isn't working

## Describe the bug
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1371, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable

## Steps to reproduce the bug
```python
metric = load_metric("glue", "sst2")
```

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 6.0.1


Adding mMARCO dataset

Adding mMARCO (v1.1) to HF datasets.


Add eli5_category dataset

This pull request adds a categorized Long-form question answering dataset `ELI5_Category`. It's a new variant of the [ELI5](https://huggingface.co/datasets/eli5) dataset that uses the Reddit tags to alleviate the training/validation overlapping in the origin ELI5 dataset.

A [report](https://celeritasml.netlify.app/posts/2021-12-01-eli5c/)(Section 2) on this dataset.


`.to_json` is extremely slow after `.select`

Something isn't working

## Describe the bug
Saving a dataset to JSON with `to_json` is extremely slow after using `.select` on the original dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset

original = load_dataset("squad", split="train")
original.to_json("from_original.json")  # Takes 0 seconds

selected_subset1 = original.select([i for i in range(len(original))])
selected_subset1.to_json("from_select1.json")  # Takes 212 seconds

selected_subset2 = original.select([i for i in range(int(len(original) / 2))])
selected_subset2.to_json("from_select2.json")  # Takes 90 seconds
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master (https://github.com/huggingface/datasets/commit/6090f3cfb5c819f441dd4a4bb635e037c875b044)
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 6.0.0


Add Wikisource dataset

Add loading script for Wikisource dataset.

Fix #3399.

CC: @geohci, @yjernite


Fix type of bridge field in QED

Use `Value("string")` instead of `Value("bool")` for the feature type of the `"bridge"` field in the QED dataset. If the value is `False`, set to `None`.

The following paragraph in the QED repo explains the purpose of this field:
>Each annotation in referential_equalities is a pair of spans, the question_reference and the sentence_reference, corresponding to an entity mention in the question and the selected_sentence respectively. As described in the paper, sentence_references can be "bridged in", in which case they do not correspond with any actual span in the selected_sentence. Hence, sentence_reference spans contain an additional field, bridge, which is a prepositional phrase when a reference is bridged, and is False otherwise. Prepositional phrases serve to link bridged references to an anchoring phrase in the selected_sentence. In the case a sentence_reference is bridged, the start and end, as well as the span string, map to such an anchoring phrase in the selected_sentence.

Fix #3346

cc @VictorSanh


disaster_response_messages unavailable

## Dataset viewer issue for '* disaster_response_messages*'

**Link:** https://huggingface.co/datasets/disaster_response_messages

Dataset unavailable. Link dead: https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv

Am I the one who added this dataset ?No


Non-deterministic tests: CI tests randomly fail

Something isn't working

## Describe the bug
Some CI tests fail randomly.

1. In https://github.com/huggingface/datasets/pull/3375/commits/c10275fe36085601cb7bdb9daee9a8f1fc734f48, there were 3 failing tests, only on Linux:
   ```
   =========================== short test summary info ============================
   FAILED tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol[https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh-zip]
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive - Fi...
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
   = 3 failed, 3553 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 192.79s (0:03:12) =
   ```

2. After re-running the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/57bfe1f342cd3c59d2510b992d5f06a0761eb147, there was only 1 failing test (one on Linux and a different one on Windows):
   - On Linux:
      ```
      =========================== short test summary info ============================
      FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
      = 1 failed, 3555 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 199.76s (0:03:19) =
      ```
   - On Windows:
      ```
      =========================== short test summary info ===========================
      FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_without_script
      = 1 failed, 3551 passed, 2954 skipped, 2 xfailed, 1 xpassed, 121 warnings in 478.58s (0:07:58) =
      ```

   The test `tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped` passes locally.

3. After re-running again the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/39f32f2119cf91b86867216bb5c356c586503c6a, ALL the tests passed.


Skip None encoding (line deleted by accident in #3195)

Return the line deleted by accident in #3195 while [resolving merge conflicts](https://github.com/huggingface/datasets/pull/3195/commits/8b0ed15be08559056b817836a07d47acda0c4510).

Fix #3181 (finally :))


Add WIDER FACE dataset

Adds the WIDER FACE face detection benchmark.

TODOs:
* [x] dataset card
* [x] dummy data


Fix flaky test again for s3 serialization

Following https://github.com/huggingface/datasets/pull/3388 that wasn't enough (see CI error [here](https://app.circleci.com/pipelines/github/huggingface/datasets/9080/workflows/b971fb27-ff20-4220-9416-c19acdfdf6f4/jobs/55985))


[chinese wwm] load_datasets behavior not as expected when using run_mlm_wwm.py script

Something isn't working

## Describe the bug

Model I am using (Bert, XLNet ...): bert-base-chinese

The problem arises when using:
* [https://github.com/huggingface/transformers/blob/master/examples/research_projects/mlm_wwm/run_mlm_wwm.py] the official example scripts: `rum_mlm_wwm.py`

The tasks I am working on is: pretraining whole word masking with my own dataset and ref.json file
I tried follow the run_mlm_wwm.py procedure to do whole word masking on pretraining task. my file is in .txt form, where one line represents one sample, with `9,264,784` chinese lines in total. the ref.json file is also contains 9,264,784 lines of whole word masking reference data for my chinese corpus. but when I try to adapt the run_mlm_wwm.py script, it shows that somehow after
`datasets["train"] = load_dataset(...`
`len(datasets["train"])` returns `9,265,365`
then, after `tokenized_datasets = datasets.map(...`
`len(tokenized_datasets["train"])` returns `9,265,279`
I'm really confused and tried to trace code by myself but can't know what happened after a week trial.

I want to know what happened in the `load_dataset()` function and `datasets.map` here and how did I get more lines of data than I input. so I'm here to ask.
## To reproduce
Sorry that I can't provide my data here since it did not belong to me. but I'm sure I remove the blank lines.

## Expected behavior
I expect the code run as it should. but the AssertionError in line 167 keeps raise as the line of reference json and datasets['train'] differs.

Thanks for your patient reading!

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 3.0.0


Fix dependencies conflicts in Windows CI after conda update to 4.11

For some reason the CI wasn't using python 3.6 but python 3.7 after the update to conda 4.11


Pass new_fingerprint in multiprocessing

Following https://github.com/huggingface/datasets/pull/3045

Currently one can pass `new_fingerprint` to `.map()` to use a custom fingerprint instead of the one computed by hashing the map transform. However it's ignored if `num_proc>1`.

In this PR I fixed that by passing `new_fingerprint` to `._map_single()` when `num_proc>1`.
More specifically, `new_fingerprint` with a suffix based on the process `rank` is passed, so that each process has a different `new_fingerprint`

cc @TevenLeScao @vlievin


Typo in Dataset viewer error message

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

When creating an empty dataset repo, the Dataset Preview provides a helpful message that no files were found. There is a tiny typo in that message: "ressource" should be "resource"

![Screen Shot 2021-12-09 at 15 31 31](https://user-images.githubusercontent.com/26859204/145415725-9cd728f0-c2c8-4b4e-a8e1-4f4d7841c94a.png)

Am I the one who added this dataset ?

N/A


Use max number of data files to infer module

When inferring the module for datasets without script, set a maximum number of iterations over data files.

This PR fixes the issue of taking too long when hundred of data files present.

Please, feel free to agree on both numbers:
```
# Datasets without script
DATA_FILES_MAX_NUMBER = 10
ARCHIVED_DATA_FILES_MAX_NUMBER = 5
```

Fix #3404.


Fix module inference for archive with a directory

Fix module inference for an archive file that contains files within a directory.

Fix #3405.


ZIP format inference does not work when files located in a dir inside the archive

Something isn't working

## Describe the bug
When a zipped file contains archived files within a directory, the function `infer_module_for_data_files_in_archives` does not work.

It only works for files located in the root directory of the ZIP file.

## Steps to reproduce the bug
```python
infer_module_for_data_files_in_archives(["path/to/zip/file.zip"], False)
```


Optimize ZIP format inference

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
When hundreds of ZIP files are present in a dataset, format inference takes too long.

See: https://github.com/bigscience-workshop/data_tooling/issues/232#issuecomment-986685497

**Describe the solution you'd like**
Iterate over a maximum number of files.

CC: @lhoestq


Cannot import name 'maybe_sync'

bug

Something isn't working

## Describe the bug
Cannot seem to import datasets when running run_summarizer.py script on a VM set up on ovhcloud

## Steps to reproduce the bug
```python
from datasets import load_dataset
```

## Expected results
No error

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 48, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import (
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/audio.py", line 8, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 16, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/__init__.py", line 13, in <module>
    from .s3filesystem import S3FileSystem  # noqa: F401
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/s3filesystem.py", line 1, in <module>
    import s3fs
  File "/opt/conda/lib/python3.7/site-packages/s3fs/__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "/opt/conda/lib/python3.7/site-packages/s3fs/core.py", line 11, in <module>
    from fsspec.asyn import AsyncFileSystem, sync, sync_wrapper, maybe_sync
ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.0
- Platform: OVH Cloud Tesla V100 Machine
- Python version: 3.7.9
- PyArrow version: 6.0.1


More robust first elem check in encode/cast example

Fix #3306


Add Wikimedia pre-processed datasets

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Add pre-processed data to:
  - *wikimedia/wikipedia*: https://huggingface.co/datasets/wikimedia/wikipedia
  - *wikimedia/wikisource*: https://huggingface.co/datasets/wikimedia/wikisource
- **Description:** Add pre-processed data to the Hub for all languages
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** This will be very useful for the NLP community, as the pre-processing has a high cost for lot of researchers (both in computation and in knowledge)

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite


Improve Wikipedia loading script

Requesting to add a new dataset

As reported by @geohci, the "wikipedia" processing/loading script could be improved by some additional small suggested processing functions:
- _extract_content(filepath):
  - Replace .startswith("#redirect") with more structured approach: if elem.find(f"./{namespace}redirect") is None: continue
- _parse_and_clean_wikicode(raw_content, parser):
  - Remove rm_template from cleaning -- this is redundant with .strip_code() from mwparserformhell
  - Build a language-specific list of namespace prefixes to filter out per below get_namespace_prefixes
  - Optional: strip prefixes like categories -- e.g., Category:Towns in Tianjin becomes Towns in Tianjin
  - Optional: strip magic words


Add Wikisource dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *wikisource*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** Additional high quality textual data, besides Wikipedia.

Add loading script as "canonical" dataset (as it is the case for ""wikipedia").

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite


Add URL field to Wikimedia dataset instances: wikipedia,...

Requesting to add a new dataset

As reported by @geohci, once we will host pre-processed data in the Hub, we should add the full URL to data instances (new field "url") in order to conform to proper attribution from license requirement. See, e.g.: https://fair-trec.github.io/docs/Fair_Ranking_2021_Participant_Instructions.pdf#subsection.3.2

This should be done for all pre-processed datasets under "wikimedia" org in the Hub: https://huggingface.co/wikimedia


add BNL newspapers

This pull request adds the BNL's [processed newspaper collections](https://data.bnl.lu/data/historical-newspapers/) as a dataset. This is partly done to support BigScience see: https://github.com/bigscience-workshop/data_tooling/issues/192.

The Datacard is more sparse than I would like but I plan to make a separate pull request to try and make this more complete at a later date.

I had to manually add the `dummy_data` but I believe I've done this correctly (the tests pass locally).


Install Audio dependencies to support audio decoding

## Dataset viewer issue for '*openslr*', '*projecte-aina/parlament_parla*'

**Link:** *https://huggingface.co/datasets/openslr*
**Link:** *https://huggingface.co/datasets/projecte-aina/parlament_parla*

Error:
```
Status code:   400
Exception:     ImportError
Message:       To support decoding audio files, please install 'librosa'.
```

Am I the one who added this dataset ? Yes-No
- openslr: No
- projecte-aina/parlament_parla: Yes


Fix formatting in IterableDataset.map docs

Fix formatting in the recently added `Map` section of the streaming docs.


Preserve all feature types when saving a dataset on the Hub with `push_to_hub`

bug

Something isn't working

Currently, if one of the dataset features is of type `ClassLabel`, saving the dataset with `push_to_hub` and reloading the dataset with `load_dataset` will return the feature of type `Value`. To fix this, we should do something similar to `save_to_disk` (which correctly preserves the types) and not only push the parquet files in `push_to_hub`, but also the dataset `info` (stored in a JSON file).


Common Voice Belarusian Dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Common Voice Belarusian Dataset*
- **Description:** *[commonvoice.mozilla.org/be](https://commonvoice.mozilla.org/be)*
- **Data:** *[commonvoice.mozilla.org/be/datasets](https://commonvoice.mozilla.org/be/datasets)*
- **Motivation:** *It has more than 7GB of data, so it will be great to have it in this package so anyone can try to train something for Belarusian language.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


## Dataset viewer issue for `dansbecker/hackernews_hiring_posts`

**Link:** https://huggingface.co/datasets/dansbecker/hackernews_hiring_posts

*short description of the issue*

Dataset preview not showing for uploaded DatasetDict. See https://discuss.huggingface.co/t/dataset-preview-not-showing-for-uploaded-datasetdict/12603

Am I the one who added this dataset ?

No -> @dansbecker


method to select columns

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

* There is currently no way to select some columns of a dataset. In pandas, one can use `df[['col1', 'col2']]` to select columns, but in `datasets`, it results in error.

**Describe the solution you'd like**

* A new method that can be used to create a new dataset with only a list of specified columns.

**Describe alternatives you've considered**

`.remove_columns(self, columns: Union[str, List[str]], inverse: bool = False)`
Or
`.select(self,  indices: Iterable = None, columns: List[str] = None)`