Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

enhancement

New feature or request

IMO, the data streaming test is good enough of a test that the dataset works correctly (assuming that we can more or less ensure that if streaming works then the non-streaming case will also work), so that for datasets that have a working dataset preview, we can remove the dummy data IMO. On the other hand, it seems like not all datasets have streaming enabled yet and for those datasets (if they are used a lot), I think it would be nice to continue testing some dummy data.

I don't really have an opinion regarding the JSON metadata as I don't know enough about it.

I open this PR to have a public discussion about this topic and make a decision.

As previously discussed, once we have the metadata in the dataset card (README file, containing both Markdown info and YAML tags), what is the point of having also the JSON metadata (dataset_infos.json file)?

On the other hand, the dummy data is necessary for testing (in our CI suite) that the canonical dataset loads correctly. However:
- the dataset preview feature is already an indirect test that the dataset loads correctly (it also tests it is streamable though)
- we are migrating canonical datasets to the Hub
Do we really need to continue testing them in out CI?

Also note that for generating both (dataset_infos.json file and dummy data), the entire dataset needs being downloaded. This can be an issue for huge datasets (like WIT, with 400 GB of data).

Feel free to ping other people for the discussion.

CC: @lhoestq @mariosasko @thomwolf @julien-c @patrickvonplaten @anton-l @LysandreJik @yjernite @nateraw


Allows DatasetDict.filter to have batching option

- Related to: #3244
- Fixes: #3503

We extends `.filter( ... batched: bool)` support to DatasetDict.


cast_column function not working with map function in streaming mode for Audio features

bug

Something isn't working

## Describe the bug
I am trying to use Audio class for loading audio features using custom dataset. I am able to cast 'audio' feature into 'Audio' format with cast_column function. On using map function, I am not getting 'Audio' casted feature but getting path of audio file only.
I am getting features of 'audio' of string type with load_dataset call. After using cast_column 'audio' feature is converted into 'Audio' type. But in map function I am not able to get Audio type for audio feature & getting string type data containing path of file only. So I am not able to use processor in encode function.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset, Audio
from transformers import Wav2Vec2Processor

def encode(batch, processor):
    print("Audio: ",batch['audio'])
    batch["input_values"] = processor(batch["audio"]['array'], sampling_rate=16000).input_values
    return batch

def print_ds(ds):
    iterator = iter(ds)
    for d in iterator:
        print("Data: ",d)
        break

processor = Wav2Vec2Processor.from_pretrained(pretrained_model_path)

dataset = load_dataset("custom_dataset.py","train",data_files={'train':'train_path.txt'},
                                data_dir="data", streaming=True, split="train")

print("Features: ",dataset.features)

print_ds(dataset)

dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print("Features: ",dataset.features)

print_ds(dataset)

dataset = dataset.map(lambda x: encode(x,processor))

print("Features: ",dataset.features)

print_ds(dataset)

```

## Expected results

map function not printing Audio type features be used with processor function and getting error in processor call due to this.

## Actual results

# after load_dataset call
Features:  {'sentence': Value(dtype='string', id=None), 'audio': Value(dtype='string', id=None)}
Data:  {'sentence': 'और अपने पेट को माँ की स्वादिष्ट गरमगरम जलेबियाँ हड़पते\n', 'audio': 'data/0116_003.wav'}

# after cast_column call
Features:  {'sentence': Value(dtype='string', id=None), 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None)}
Data:  {'sentence': 'और अपने पेट को माँ की स्वादिष्ट गरमगरम जलेबियाँ हड़पते\n', 'audio': {'path': 'data/0116_003.wav', 'array': array([ 1.2662281e-06,  1.0264218e-06, -1.3615092e-06, ...,
        1.3017889e-02,  1.0085563e-02,  4.8155054e-03], dtype=float32), 'sampling_rate': 16000}}

# after map call
Features:  None
Audio:  data/0116_003.wav

Traceback (most recent call last):
  File "demo2.py", line 36, in <module>
    print_ds(dataset)
  File "demo2.py", line 11, in print_ds
    for d in iterator:
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 192, in __iter__
    yield key, self.function(example)
  File "demo2.py", line 32, in <lambda>
    dataset = dataset.map(lambda x: batch_encode(x,processor))
  File "demo2.py", line 6, in batch_encode
    batch["input_values"] = processor(batch["audio"]['array'], sampling_rate=16000).input_values
TypeError: string indices must be integers
## Environment info

- `datasets` version: 1.17.0
- Platform: Linux-4.14.243 with-debian-bullseye-sid
- Python version: 3.7.9
- PyArrow version: 6.0.1


Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Something isn't working

## Describe the bug
I am unable to download the PubMed dataset from the link provided in the [Hugging Face Course (Chapter 5 Section 4)](https://huggingface.co/course/chapter5/4?fw=pt).

https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset

# This takes a few minutes to run, so go grab a tea or coffee while you wait :)
data_files = "https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst"
pubmed_dataset = load_dataset("json", data_files=data_files, split="train")
pubmed_dataset
```

I also tried with `wget` as follows.
```
wget https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst
```

## Expected results
I expect to be able to download this file.

## Actual results
Traceback
```
---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
/usr/lib/python3/dist-packages/urllib3/connection.py in _new_conn(self)
    158         try:
--> 159             conn = connection.create_connection(
    160                 (self._dns_host, self.port), self.timeout, **extra_kw

/usr/lib/python3/dist-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     83     if err is not None:
---> 84         raise err
     85

/usr/lib/python3/dist-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     73                 sock.bind(source_address)
---> 74             sock.connect(sa)
     75             return sock

timeout: timed out

During handling of the above exception, another exception occurred:

ConnectTimeoutError                       Traceback (most recent call last)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    664             # Make the request on the httplib connection object.
--> 665             httplib_response = self._make_request(
    666                 conn,

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    375         try:
--> 376             self._validate_conn(conn)
    377         except (SocketTimeout, BaseSSLError) as e:

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    995         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 996             conn.connect()
    997

/usr/lib/python3/dist-packages/urllib3/connection.py in connect(self)
    313         # Add certificate verification
--> 314         conn = self._new_conn()
    315         hostname = self.host

/usr/lib/python3/dist-packages/urllib3/connection.py in _new_conn(self)
    163         except SocketTimeout:
--> 164             raise ConnectTimeoutError(
    165                 self,

ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)')

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
/usr/lib/python3/dist-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    438             if not chunked:
--> 439                 resp = conn.urlopen(
    440                     method=request.method,

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    718
--> 719             retries = retries.increment(
    720                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]

/usr/lib/python3/dist-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    435         if new_retry.is_exhausted():
--> 436             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    437

MaxRetryError: HTTPSConnectionPool(host='the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)'))

During handling of the above exception, another exception occurred:

ConnectTimeout                            Traceback (most recent call last)
/tmp/ipykernel_15104/606583593.py in <module>
      3 # This takes a few minutes to run, so go grab a tea or coffee while you wait :)
      4 data_files = "https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst"
----> 5 pubmed_dataset = load_dataset("json", data_files=data_files, split="train")
      6 pubmed_dataset

~/.local/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1655
   1656     # Create a dataset builder
-> 1657     builder_instance = load_dataset_builder(
   1658         path=path,
   1659         name=name,

~/.local/lib/python3.8/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, script_version, **config_kwargs)
   1492         download_config = download_config.copy() if download_config else DownloadConfig()
   1493         download_config.use_auth_token = use_auth_token
-> 1494     dataset_module = dataset_module_factory(
   1495         path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
   1496     )

~/.local/lib/python3.8/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_files, **download_kwargs)
   1116     # Try packaged
   1117     if path in _PACKAGED_DATASETS_MODULES:
-> 1118         return PackagedDatasetModuleFactory(
   1119             path, data_files=data_files, download_config=download_config, download_mode=download_mode
   1120         ).get_module()

~/.local/lib/python3.8/site-packages/datasets/load.py in get_module(self)
    773             else get_patterns_locally(str(Path().resolve()))
    774         )
--> 775         data_files = DataFilesDict.from_local_or_remote(patterns, use_auth_token=self.downnload_config.use_auth_token)
    776         module_path, hash = _PACKAGED_DATASETS_MODULES[self.name]
    777         builder_kwargs = {"hash": hash, "data_files": data_files}

~/.local/lib/python3.8/site-packages/datasets/data_files.py in from_local_or_remote(cls, patterns, base_path, allowed_extensions, use_auth_token)
    576         for key, patterns_for_key in patterns.items():
    577             out[key] = (
--> 578                 DataFilesList.from_local_or_remote(
    579                     patterns_for_key,
    580                     base_path=base_path,

~/.local/lib/python3.8/site-packages/datasets/data_files.py in from_local_or_remote(cls, patterns, base_path, allowed_extensions, use_auth_token)
    545         base_path = base_path if base_path is not None else str(Path().resolve())
    546         data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
--> 547         origin_metadata = _get_origin_metadata_locally_or_by_urls(data_files, use_auth_token=use_auth_token)
    548         return cls(data_files, origin_metadata)
    549

~/.local/lib/python3.8/site-packages/datasets/data_files.py in _get_origin_metadata_locally_or_by_urls(data_files, max_workers, use_auth_token)
    492     data_files: List[Union[Path, Url]], max_workers=64, use_auth_token: Optional[Union[bool, str]] = None
    493 ) -> Tuple[str]:
--> 494     return thread_map(
    495         partial(_get_single_origin_metadata_locally_or_by_urls, use_auth_token=use_auth_token),
    496         data_files,

~/.local/lib/python3.8/site-packages/tqdm/contrib/concurrent.py in thread_map(fn, *iterables, **tqdm_kwargs)
     92     """
     93     from concurrent.futures import ThreadPoolExecutor
---> 94     return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
     95
     96

~/.local/lib/python3.8/site-packages/tqdm/contrib/concurrent.py in _executor_map(PoolExecutor, fn, *iterables, **tqdm_kwargs)
     74             map_args.update(chunksize=chunksize)
     75         with PoolExecutor(**pool_kwargs) as ex:
---> 76             return list(tqdm_class(ex.map(fn, *iterables, **map_args), **kwargs))
     77
     78

~/.local/lib/python3.8/site-packages/tqdm/notebook.py in __iter__(self)
    252     def __iter__(self):
    253         try:
--> 254             for obj in super(tqdm_notebook, self).__iter__():
    255                 # return super(tqdm...) will not catch exception
    256                 yield obj

~/.local/lib/python3.8/site-packages/tqdm/std.py in __iter__(self)
   1171         # (note: keep this check outside the loop for performance)
   1172         if self.disable:
-> 1173             for obj in iterable:
   1174                 yield obj
   1175             return

/usr/lib/python3.8/concurrent/futures/_base.py in result_iterator()
    617                     # Careful not to keep a reference to the popped future
    618                     if timeout is None:
--> 619                         yield fs.pop().result()
    620                     else:
    621                         yield fs.pop().result(end_time - time.monotonic())

/usr/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    442                     raise CancelledError()
    443                 elif self._state == FINISHED:
--> 444                     return self.__get_result()
    445                 else:
    446                     raise TimeoutError()

/usr/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    387         if self._exception:
    388             try:
--> 389                 raise self._exception
    390             finally:
    391                 # Break a reference cycle with the exception in self._exception

/usr/lib/python3.8/concurrent/futures/thread.py in run(self)
     55
     56         try:
---> 57             result = self.fn(*self.args, **self.kwargs)
     58         except BaseException as exc:
     59             self.future.set_exception(exc)

~/.local/lib/python3.8/site-packages/datasets/data_files.py in _get_single_origin_metadata_locally_or_by_urls(data_file, use_auth_token)
    483     if isinstance(data_file, Url):
    484         data_file = str(data_file)
--> 485         return (request_etag(data_file, use_auth_token=use_auth_token),)
    486     else:
    487         data_file = str(data_file.resolve())

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in request_etag(url, use_auth_token)
    489 def request_etag(url: str, use_auth_token: Optional[Union[str, bool]] = None) -> Optional[str]:
    490     headers = get_authentication_headers_for_url(url, use_auth_token=use_auth_token)
--> 491     response = http_head(url, headers=headers, max_retries=3)
    492     response.raise_for_status()
    493     etag = response.headers.get("ETag") if response.ok else None

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in http_head(url, proxies, headers, cookies, allow_redirects, timeout, max_retries)
    474     headers = copy.deepcopy(headers) or {}
    475     headers["user-agent"] = get_datasets_user_agent(user_agent=headers.get("user-agent"))
--> 476     response = _request_with_retry(
    477         method="HEAD",
    478         url=url,

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in _request_with_retry(method, url, max_retries, base_wait_time, max_wait_time, timeout, **params)
    407         except (requests.exceptions.ConnectTimeout, requests.exceptions.ConnectionError) as err:
    408             if tries > max_retries:
--> 409                 raise err
    410             else:
    411                 logger.info(f"{method} request to {url} timed out, retrying... [{tries/max_retries}]")

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in _request_with_retry(method, url, max_retries, base_wait_time, max_wait_time, timeout, **params)
    403         tries += 1
    404         try:
--> 405             response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
    406             success = True
    407         except (requests.exceptions.ConnectTimeout, requests.exceptions.ConnectionError) as err:

/usr/lib/python3/dist-packages/requests/api.py in request(method, url, **kwargs)
     58     # cases, and look like a memory leak in others.
     59     with sessions.Session() as session:
---> 60         return session.request(method=method, url=url, **kwargs)
     61
     62

/usr/lib/python3/dist-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    531         }
    532         send_kwargs.update(settings)
--> 533         resp = self.send(prep, **send_kwargs)
    534
    535         return resp

/usr/lib/python3/dist-packages/requests/sessions.py in send(self, request, **kwargs)
    644
    645         # Send the request
--> 646         r = adapter.send(request, **kwargs)
    647
    648         # Total elapsed time of the request (approximately)

/usr/lib/python3/dist-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    502                 # TODO: Remove this in 3.0.0: see #2811
    503                 if not isinstance(e.reason, NewConnectionError):
--> 504                     raise ConnectTimeout(e, request=request)
    505
    506             if isinstance(e.reason, ResponseError):

ConnectTimeout: HTTPSConnectionPool(host='the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)'))
```

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.11.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 6.0.1


Batched in filter throws error

Something isn't working

I hope this is really a bug, I could not find it among the open issues

## Describe the bug
using `batched=False` in DataSet.filter throws error
```python
TypeError: filter() got an unexpected keyword argument 'batched'
```
but in the docs it is lister as an argument.

## Steps to reproduce the bug
```python
task = "mnli"
max_length = 128
tokenizer = AutoTokenizer.from_pretrained("./pretrained_models/pretrained_models_drozd/sl250.m.gsic.titech.ac.jp:8000/21.11.17_06.30.32_roberta-base_a0057/checkpoints/smpl_400M/hf/")

dataset = load_dataset("glue", task)

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

##### tokenization_parameters
sentence1_key, sentence2_key = task_to_keys[task]
def preprocess_function(examples, max_length):
    if sentence2_key is None:
        return tokenizer(
            examples[sentence1_key], truncation=True, max_length=max_length
        )
    return tokenizer(
        examples[sentence1_key],
        examples[sentence2_key],
        truncation=False,
        padding="max_length",
        max_length=max_length,
    )

encoded_dataset = dataset.map(
    lambda x: preprocess_function(x, max_length=max_length), batched=False
)

encoded_dataset.filter(lambda x: len(x['input_ids']) <= max_length, batched=False)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1, 1.17.0
- Platform: ubuntu
- Python version: 3.8.12


Add QuALITY

Fixes #3441.


Update pib dataset card

Related to #3496


Docs: Add VCTK dataset description

This PR is a very minor followup to #1837, with only docs changes (single comment string).


Adjusting chunk size for streaming datasets

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
I want to use mc4 which I cannot save locally, so I stream it. However, I want to process the entire dataset and filter some documents from it. With the current chunk size of around 1000 documents (right?) I hit a performance bottleneck because of the frequent decompressing.

**Describe the solution you'd like**
I would appreciate a parameter in the load_dataset function, that allows me to set the chunksize myself (to a value like 100'000 in my case). Like that, I hope to improve the processing time.


update `pretty_name` for all datasets

I made a script some time back to fetch `pretty_names` from `papers_with_code` dataset along with some other rules. Updating them in the `README` of `datasets`. Took only first 200 datasets in consideration and after some eye balling most of them were looking good to me!


Changing sampling rate in audio dataset and subsequently mapping with `num_proc > 1` leads to weird bug

bug

Something isn't working

Same error occures when using max samples with https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py

Running:

```python
from datasets import load_dataset, DatasetDict
import datasets
from transformers import AutoFeatureExtractor

raw_datasets = DatasetDict()

raw_datasets["train"] = load_dataset("common_voice", "ab", split="train")

feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")

raw_datasets = raw_datasets.cast_column(
    "audio", datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)
)

num_workers = 16

def prepare_dataset(batch):
    sample = batch["audio"]

    inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])
    return batch

raw_datasets.map(
    prepare_dataset,
    remove_columns=next(iter(raw_datasets.values())).column_names,
    num_proc=16,
    desc="preprocess datasets",
)
```

gives

```bash
  File "/home/patrick/experiments/run_bug.py", line 25, in <module>
    raw_datasets.map(
  File "/home/patrick/python_bin/datasets/dataset_dict.py", line 492, in map
    {
  File "/home/patrick/python_bin/datasets/dataset_dict.py", line 493, in <dictcomp>
    k: dataset.map(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2139, in map
    shards = [
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2140, in <listcomp>
    self.shard(num_shards=num_proc, index=rank, contiguous=True, keep_in_memory=keep_in_memory)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 3164, in shard
    return self.select(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 485, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/patrick/python_bin/datasets/fingerprint.py", line 411, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2756, in select
    return self._new_dataset_with_indices(indices_buffer=buf_writer.getvalue(), fingerprint=new_fingerprint)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2667, in _new_dataset_with_indices
    return Dataset(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 659, in __init__
    raise ValueError(
ValueError: External features info don't match the dataset:
Got
{'client_id': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None), 'down_votes': Value(dtype='int64', id=None), 'age': Value(dtype='string', id=None), 'gender': Value(dtype='string', id=None), 'accent': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None)}
with type
struct<client_id: string, path: string, audio: string, sentence: string, up_votes: int64, down_votes: int64, age: string, gender: string, accent: string, locale: string, segment: string>

but expected something like
{'client_id': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'audio': {'path': Value(dtype='string', id=None), 'bytes': Value(dtype='binary', id=None)}, 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None), 'down_votes': Value(dtype='int64', id=None), 'age': Value(dtype='string', id=None), 'gender': Value(dtype='string', id=None), 'accent': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None)}
with type
struct<client_id: string, path: string, audio: struct<path: string, bytes: binary>, sentence: string, up_votes: int64, down_votes: int64, age: string, gender: string, accent: string, locale: string, segment: string>
```

Versions:

```python
- `datasets` version: 1.16.2.dev0
- Platform: Linux-5.15.8-76051508-generic-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 6.0.1
```

and `transformers`:

```
- `transformers` version: 4.16.0.dev0
- Platform: Linux-5.15.8-76051508-generic-x86_64-with-glibc2.33
- Python version: 3.9.7
```


Update version of pib dataset and make it streamable

It seems like there is still an error: `Message:       'TarContainedFile' object has no attribute 'readable'`

https://huggingface.co/datasets/pib/viewer

@severo I was wondering about that...

It works fine when I run it in streaming mode in my terminal:
```python
In [3]: from datasets import load_dataset; ds = load_dataset("pib", "gu-pa", split="train", streaming=True); item = next(iter(ds))

In [4]: item
Out[4]:
{'translation': {'gu': 'એવો નિર્ણય લેવાયો હતો કે ખંતપૂર્વકની કામગીરી હાથ ધરવા, કાયદેસર અને ટેકનિકલ મૂલ્યાંકન કરવા, વેન્ચર કેપિટલ ઇન્વેસ્ટમેન્ટ સમિતિની બેઠક યોજવા વગેરે એઆઇએફને કરવામાં આવેલ પ્રતિબદ્ધતાના 0.50 ટકા સુધી અને બાકીની રકમ એફએફએસને પૂર્ણ કરવામાં આવશે.',
  'pa': 'ਇਹ ਵੀ ਫੈਸਲਾ ਕੀਤਾ ਗਿਆ ਕਿ ਐੱਫਆਈਆਈ ਅਤੇ ਬਕਾਏ ਲਈ ਕੀਤੀਆਂ ਗਈਆਂ ਵਚਨਬੱਧਤਾਵਾਂ ਦੇ 0.50 % ਦੀ ਸੀਮਾ ਤੱਕ ਐੱਫਈਐੱਸ ਨੂੰ ਮਿਲਿਆ ਜਾਏਗਾ, ਇਸ ਨਾਲ ਉੱਦਮ ਪੂੰਜੀ ਨਿਵੇਸ਼ ਕਮੇਟੀ ਦੀ ਬੈਠਕ ਦਾ ਆਯੋਜਨ ਉਚਿਤ ਸਾਵਧਾਨੀ, ਕਾਨੂੰਨੀ ਅਤੇ ਤਕਨੀਕੀ ਮੁੱਲਾਂਕਣ ਲਈ ਸੰਚਾਲਨ ਖਰਚ ਆਦਿ ਦੀ ਪੂਰਤੀ ਹੋਵੇਗੀ।'}}
```

This PR:
- Updates version of pib dataset: from 0.0.0 to 1.3.0
- Makes the dataset streamable

Fix #3491.

CC: @severo


Add VoxLingua107

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** 107 languages, totaling 6628 hours for the train split.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Clone full repo to detect new tags when mirroring datasets on the Hub

The CI fail is unrelated to this PR and fixed on master, merging :)

The new releases of `datasets` were not detected because the shallow clone in the CI wasn't getting the git tags.

By cloning the full repository we can properly detect a new release, and tag all the dataset repositories accordingly
cc @SBrandeis


Fix VCTK encoding

utf-8 encoding was missing in the VCTK dataset builder added in #3351


Add `gzip` for `to_json`

(Partially) closes #3480. I have added `gzip` compression for `to_json`. I realised we can run into this compression problem with `to_csv` as well. `IOHandler` can be used for `to_csv` too. Please let me know if any changes are required.


Update version of pib dataset

Requesting to add a new dataset

On the Hub we have v0, while there exists v1.3.

Related to bigscience-workshop/data_tooling#130


Does datasets support load text from HDFS?

The raw text data is stored on HDFS due to the dataset's size is too large to store on my develop machine,
so I wander does datasets support read data from hdfs?


Avoid unnecessary list creations

Like in `join([... for s in ...])`. Also changed other things that I saw:
* Use a `with` statement for many `open` that missed them, so the files don't remain open.
* Remove unused variables.
* Many HTTP links converted into HTTPS (verified).
* Remove unnecessary "r" mode arg in `open` (double-checked it was actually the default in each case).
* Remove Python 2 style of using `super`.
* Run `pyupgrade $(find . -name "*.py" -type f) --py36-plus` (which already does some of the previous points).
* Run `dos2unix $(find . -name "*.py" -type f)` (CRLF to LF line endings).
* Fix typos.


URL query parameters are set as path in the compression hop for fsspec

bug

Something isn't working

## Describe the bug
There is an ssue with `StreamingDownloadManager._extract`.

I don't know how the test `test_streaming_gg_drive_gzipped` passes:

For
```python
TEST_GG_DRIVE_GZIPPED_URL = "https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz"
urlpath = StreamingDownloadManager().download_and_extract(TEST_GG_DRIVE_GZIPPED_URL)
```
gives `urlpath`:
```python
'gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz::https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz'
```

The gzip path makes no sense: `gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz`

## Steps to reproduce the bug
```python
from datasets.utils.streaming_download_manager import StreamingDownloadManager

dl_manager = StreamingDownloadManager()
urlpath = dl_manager.extract("https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz")
print(urlpath)
```

## Expected results
The query parameters should not be set as part of the path.


Update ADD_NEW_DATASET.md

fixed make style prompt for Windows Terminal


Fix weird spacing in ManualDownloadError message

`textwrap.dedent` works based on the spaces at the beginning. Before this change, there wasn't any space.


enhancement

New feature or request

You can add columns that you wish to set into `torch` format using `dataset.set_format("torch", ['id', 'abc'])` so that input batch of the transform only contains those columns

Sorry, I miss `output_all_columns` args and thought after `dataset.set_format("torch", columns=columns)` I can only get specific columns I assigned.

**Is your feature request related to a problem? Please describe.**
When using `dataset.set_format("torch")`, I must make sure every columns in datasets can convert to `torch`, however, sometimes I want to keep some string columns.

**Describe the solution you'd like**
skip columns which cannot set to specific format when set_format instead of raise an error.


make shape verification to use ArrayXD instead of nested lists for map

New feature or request

As describe in https://github.com/huggingface/datasets/issues/2005#issuecomment-793716753 and mentioned by @mariosasko in [image feature example](https://colab.research.google.com/drive/1mIrTnqTVkWLJWoBzT1ABSe-LFelIep1c#scrollTo=ow3XHDvf2I0B&line=1&uniqifier=1), IMO make shape verifcaiton to use ArrayXD instead of nested lists for map can help user reduce unnecessary cast. I notice datasets have done something special for `input_ids` and `attention_mask` which is also unnecessary after this feature added.


Remove unused phony rule from Makefile


Fix duplicate keys in NewsQA

* Fix duplicate keys in NewsQA when loading from CSV files.
* Fix s/narqa/newsqa/ in the download manually error message.
* Make the download manually error message show nicely when printed. Otherwise, is hard to read due to spacing issues.
* Fix the format of the license text.
* Reformat the code to make it simpler.


Fix overriding of filesystem info

Previously, `BaseCompressedFileFileSystem.info` was overridden and transformed from function to dict.

This generated a bug for filesystem methods that use `self.info()`, like e.g. `fs.isfile()`.

This PR:
- Adds tests for `fs.isfile` (that use `fs.info`).
- Fixes custom `BaseCompressedFileFileSystem.info` by removing its overriding.


the compression format requested when saving a dataset in json format is not respected

bug

Something isn't working

Thanks for reporting @SaulLu.

At first sight I think the problem is caused because `pandas` only takes into account the `compression` parameter if called with a non-null file path or buffer. And in our implementation, we call pandas `to_json` with `None` `path_or_buf`.

We should fix this:
- either handling directly the `compression` parameter ourselves
- or refactoring to pass non-null path or buffer to pandas

CC: @lhoestq

I was thinking if we can handle the `compression` parameter by ourselves? Compression types will be similar to what `pandas` offer. Initially, we can try this with 2-3 compression types and see how good/bad it is? Let me know if it sounds good, I can raise a PR for this next week

Hi ! Thanks for your help @bhavitvyamalik :)
Maybe let's start with `gzip` ? I think it's the most common use case, then if we're fine with it we can add other compression methods

## Describe the bug
In the documentation of the `to_json` method, it is stated in the parameters that
> **to_json_kwargs – Parameters to pass to pandas’s pandas.DataFrame.to_json.

however when we pass for example `compression="gzip"`, the saved file is not compressed.

Would you also have expected compression to be applied? :relaxed:

## Steps to reproduce the bug
```python
my_dict = {"a": [1, 2, 3], "b": [1, 2, 3]}
```
### Result with datasets
```python
from datasets import Dataset

dataset = Dataset.from_dict(my_dict)
dataset.to_json("dic_with_datasets.jsonl.gz", compression="gzip")
!cat dic_with_datasets.jsonl.gz
```
output
```
{"a":1,"b":1}
{"a":2,"b":2}
{"a":3,"b":3}
```
Note: I would expected to see binary data here

### Result with pandas
```python
import pandas as pd

df = pd.DataFrame(my_dict)
df.to_json("dic_with_pandas.jsonl.gz", lines=True, orient="records", compression="gzip")
!cat dic_with_pandas.jsonl.gz
```
output
```
4��a�dic_with_pandas.jsonl��VJT�2�QJ��\� ��g��yƵ���������)���
```
Note: It looks like binary data

## Expected results

I would have expected that the saved result with datasets would also be a binary file

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyArrow version: 5.0.0


Dataset preview is not available (I think for all Hugging Face datasets)

Something isn't working

dataset-viewer

Related to the dataset viewer on huggingface.co

You're right, we have an issue today with the datasets preview. We're investigating.

It should be fixed now. Thanks for reporting.

## Dataset viewer issue for '*french_book_reviews*'

**Link:** https://huggingface.co/datasets/Abirate/french_book_reviews

**short description of the issue**
For my dataset, the dataset preview is no longer functional (it used to work: The dataset had been added the day before and it was fine...)
And, after looking over the datasets, I discovered that this issue affects all Hugging Face datasets (as of yesterday, December 23, 2021, around 10 p.m. (CET)).
**Am I the one who added this dataset** :  Yes

**Note**: here a screenshot showing the issue
![Dataset preview is not available for my dataset](https://user-images.githubusercontent.com/66887439/147333078-60734578-420d-4e91-8691-a90afeaa8948.jpg)

**And here for glue dataset :**

![Dataset preview is not available for other Hugging Face datasets(glue)](https://user-images.githubusercontent.com/66887439/147333492-26fa530c-befd-4992-8361-70c51397a25a.jpg)


Extend support for streaming datasets that use os.walk

Nice. I'll update the dataset viewer once merged, and test on these four datasets

This PR extends the support in streaming mode for datasets that use `os.walk`, by patching that function.

This PR adds support for streaming mode to datasets:
1. autshumato
1. code_x_glue_cd_code_to_text
1. code_x_glue_tc_nl_code_search_adv
1. nchlt

CC: @severo


Use `iter_files` instead of `str(Path(...)` in image dataset

`iter_archive` is about to support ZIP archives. I think we should use this no ?

see #3347 https://github.com/huggingface/datasets/pull/3379

I was interested in the support for isfile/dir in remote.

Anyway, `iter_files` will be available for community users.

I'm not a big fan of having two functions that do the same thing. What do you think ?

They do not do the same thing:
- One iterates over files in a directory
- The other I guess will iterate over the members of an archive

Makes sense ! Sounds good then - sorry for my misunderstanding

Note that `iter_archive` will be more performant for data streaming that `iter_files` thanks to the buffering so maybe in the future we can `iter_archive` for some of these datasets

Yes, @lhoestq I agree with you: once `iter_archive` supports zip files, it will be more suitable than `iter_files` for these 2 datasets.

Anyway, this PR also implements `isfile`/`isdir` in streaming mode, besides fixing `iter_files`. And I'm interested in having those in master.

Maybe, could we merge this PR into master and take note to refactor the datasets to use `iter_archive` once zip is supported?
Other option could be to split this PR into 2...

Use `iter_files` in the `beans` and the `cats_vs_dogs` dataset scripts as suggested by @albertvillanova.

Additional changes:
* Fix `iter_files` in `MockDownloadManager` (see this [CI error](https://app.circleci.com/pipelines/github/huggingface/datasets/9247/workflows/2657ff8a-b531-4fd9-a9fc-6541a72e8d83/jobs/57028))
* Add support for `os.path.isdir` and `os.path.isfile` in streaming (`os.path.isfile` is needed in `StreamingDownloadManager`'s `iter_files` to make `cats_vs_dogs` streamable)

TODO:
- [ ] add tests for `xisdir` and `xisfile`


Extend support for streaming datasets that use ET.parse

This PR extends the support in streaming mode for datasets that use `ET.parse`, by patching the function.

This PR adds support for streaming mode to datasets:
1. ami
1. assin
1. assin2
1. counter
1. enriched_web_nlg
1. europarl_bilingual
1. hyperpartisan_news_detection
1. polsum
1. qa4mre
1. quail
1. ted_talks_iwslt
1. udhr
1. web_nlg
1. winograd_wsc

CC: @severo


The rotten_tomatoes dataset of movie reviews contains some reviews in Spanish

bug

Something isn't working

Hi @puzzler10, thanks for reporting.

Please note this dataset is not hosted on Hugging Face Hub. See:
https://github.com/huggingface/datasets/blob/c8f914473b041833fd47178fa4373cdcb56ac522/datasets/rotten_tomatoes/rotten_tomatoes.py#L42

If there are issues with the source data of a dataset, you should contact the data owners/creators instead. In the homepage associated with this dataset (http://www.cs.cornell.edu/people/pabo/movie-review-data/), you can find the authors of the dataset and how to contact them:
> If you have any questions or comments regarding this site, please send email to Bo Pang or Lillian Lee.

P.S.: Please also note that the example you gave of non-English review is in Portuguese (not Spanish). ;)

Maybe best to just put a quick sentence in the dataset description that highlights this?

## Describe the bug
See title. I don't think this is intentional and they probably should be removed. If they stay the dataset description should be at least updated to make it clear to the user.

## Steps to reproduce the bug
Go to the [dataset viewer](https://huggingface.co/datasets/viewer/?dataset=rotten_tomatoes) for the dataset, set the offset to 4160 for the train dataset, and scroll through the results. I found ones at index 4166 and 4173. There's others too (e.g. index 2888) but those two are easy to find like that.

## Expected results
English movie reviews only.

## Actual results
Example of a Spanish movie review (4173):
> "É uma pena que , mais tarde , o próprio filme abandone o tom de paródia e passe a utilizar os mesmos clichês que havia satirizado "


Decode images when iterating

If I iterate over a vision dataset, the images are not decoded, and the dictionary with the bytes is returned.

This PR enables image decoding in `Dataset.__iter__`

Close https://github.com/huggingface/datasets/issues/3473


Iterating over a vision dataset doesn't decode the images

Something isn't working

vision

Vision datasets

As discussed, I remember I set `decoded=False` here to avoid decoding just by iterating over examples of dataset. We wanted to decode only if the "audio" field (for Audio feature) was accessed.

> I set decoded=False here to avoid decoding just by iterating over examples of dataset. We wanted to decode only if the "audio" field (for Audio feature) was accessed

https://github.com/huggingface/datasets/pull/3430 will add more control to decoding, so I think it's OK to enable decoding in `__iter__` for now. After we merge the linked PR, the user can easily disable it again.

@mariosasko I wonder why there is no issue in `Audio` feature with decoding disabled in `__iter__`, whereas there is in `Image` feature.

Enabling decoding in `__iter__` will make fail Audio regressions tests: https://github.com/huggingface/datasets/runs/4608657230?check_suite_focus=true
```
=========================== short test summary info ============================
FAILED tests/features/test_audio.py::test_dataset_with_audio_feature_map_is_not_decoded
FAILED tests/features/test_audio.py::test_dataset_with_audio_feature_map_is_decoded
========================= 2 failed, 15 passed in 8.37s =========================

Please also note that the regression tests were implemented in accordance with the specifications:
- when doing a `map` (wich calls `__iter__`) of a function that doesn't access the audio field, the decoding should be disabled; this is why the decoding is disabled in `__iter__` (and only enabled in `__getitem__`).

>  I wonder why there is no issue in Audio feature with decoding disabled in __iter__, whereas there is in Image feature.

@albertvillanova Not sure if I understand this part. Currently, both the Image and the Audio feature don't decode data in `__iter__`, so their behavior is aligned there.

Therefore, this is not an issue, neither for Audio nor Image feature.

Could you please elaborate more on the expected use case? @lhoestq @NielsRogge

The expected use cases (in accordance with the specs: see #2324):
- decoding should be enabled when accessing a specific item (`__getitem__`)
- decoding should be disabled while iterating (`__iter__`) to allow preprocessing of non-audio/image features (like label or text, for example) using `.map`
-  decoding should be enabled in a `.map` only if the `.map` function accesses the audio/image feature (implemented using `LazyDict`)

For me it's not an issue, actually. I just (mistakenly) tried to iterate over a PyTorch Dataset instead of a PyTorch DataLoader,

i.e. I did this:

`batch = next(iter(train_ds)) `

whereas I actually wanted to do

`batch = next(iter(train_dataloader))`

and then it turned out that in the first case, the image was a string of bytes rather than a Pillow image, hence Quentin opened an issue.

Thanks @NielsRogge for the context.

So IMO everything is working as expected.

I'm closing this issue. Feel free to reopen it again if further changes of the specs should be addressed.

Thanks for the details :)

I still think that it's unexpected to get different results when doing
```python
for i in range(len(dataset)):
    sample = dataset[i]
```
and
```python
for sample in dataset:
    pass
```
even though I understand that if you don't need to decode the data, then decoding image or audio data when iterating is a waste of time and resources.

But in this case users can still drop the column that need decoding to get the full speed back no ?

## Describe the bug

If I load `mnist` and I iterate over the dataset, the images are not decoded, and the dictionary with the bytes is returned.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import PIL

mnist = load_dataset("mnist", split="train")

first_image = mnist[0]["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # passes
first_image = next(iter(mnist))["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # fails
```

## Expected results

The image should be decoded, as a PIL Image

## Actual results

We get a dictionary
```
{'bytes': b'\x89PNG\r\n\x1a\n\x00..., 'path': None}
```

## Environment info

- `datasets` version: 1.17.1.dev0
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyArrow version: 6.0.0

The bug also exists in 1.17.0

## Investigation

I think the issue is that decoding is disabled in `__iter__`:

https://github.com/huggingface/datasets/blob/dfe5b73387c5e27de6a16b0caeb39d3b9ded66d6/src/datasets/arrow_dataset.py#L1651-L1661

Do you remember why it was disabled in the first place @albertvillanova ?
Also cc @mariosasko @NielsRogge


Fix `str(Path(...))` conversion in streaming on Linux. This should fix the streaming of the `beans` and `cats_vs_dogs` datasets.


Fix Tashkeela dataset to yield stripped text

This PR:
- Yields stripped text
- Fix path for Windows
- Adds license
- Adds more info in dataset card

Close bigscience-workshop/data_tooling#279


Fix rendering of docs

Minor fix in docs.

Currently, `ClassLabel` docstring rendering is not right.


Fix METEOR missing NLTK's omw-1.4

I also modified the doctest call to raise the exception that doctest may catch, instead of `doctest.UnexpectedException`.
This will make debugging easier if it happens again

NLTK 3.6.6 now requires `omw-1.4` to be downloaded for METEOR to work.
This should fix the CI on master


Add COCO dataset

The CI failures other than a missing dummy data file and missing fields in the card are unrelated to this PR.

Thanks a lot for this great work and fixing TFDS based script @mariosasko 🤗 will generate the dummy dataset and write the model card tomorrow!

@mariosasko I added the dataset card, I'm on the dummy data rn.

This PR adds the MS COCO dataset. Compared to the [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/object_detection/coco.py) script, this implementation adds 8 additional configs to cover the tasks other than object detection.

Some notes:
* the data exposed by TFDS is contained in the `2014`, `2015`, `2017` and `2017_panoptic_segmentation` configs here
* I've updated `encode_nested_example` for easier handling of missing values (cc @lhoestq @albertvillanova; will add tests if you are OK with the changes in `features.py`)
* this implementation should fix https://github.com/huggingface/datasets/pull/3377#issuecomment-985559427

TODOs:
- [x] dataset card
- [ ] dummy data

cc @merveenoyan

Closes #2526


Push dataset infos.json to Hub

The change from `___` to `--` was allowed by https://github.com/huggingface/moon-landing/pull/1657

When doing `push_to_hub`, the feature types are lost (see issue https://github.com/huggingface/datasets/issues/3394).
This PR fixes this by also pushing a `dataset_infos.json` file to the Hub, that stores the feature types.

Other minor changes:
- renamed the `___` separator to `--`, since `--` is now disallowed in a name in the back-end.

I tested this feature with datasets like conll2003 that has feature types like `ClassLabel` that were previously lost.

Close https://github.com/huggingface/datasets/issues/3394

I would like to include this in today's release (though not mandatory), so feel free to comment/suggest changes


Add CRASS dataset

Hi Albert,
Thank you for your comments.
I hope I have uploaded my local git repo to include the dummy files and style reworkings.
Added YAML in Readme as well.

Please check again.

Hope it works now :)

Added crass dataset


Unable to load 'cnn_dailymail' dataset

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi @talha1503, thanks for reporting.

It seems there is an issue with one of the data files hosted at Google Drive:
```
Google Drive - Quota exceeded

Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
```

As you probably know, Hugging Face does not host the data, and in this case the data owner decided to host their data at Google Drive, which has quota limits.

Is there anything we could do, @lhoestq @mariosasko?

This looks related to https://github.com/huggingface/datasets/issues/996

## Describe the bug
I wanted to load cnn_dailymail dataset from huggingface datasets on Google Colab, but I am getting an error while loading it.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('cnn_dailymail', '3.0.0', ignore_verifications = True)
```

## Expected results
Expecting to load 'cnn_dailymail' dataset.

## Actual results
`NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0


struct.error: 'i' format requires -2147483648 <= number <= 2147483647

bug

Something isn't working

## Describe the bug
A clear and concise description of what the bug is.

using latest datasets=datasets-1.16.1-py3-none-any.whl
process my own multilingual dataset by following codes, and the number of rows in all dataset is 306000, the max_length of each sentence is 256:
![image](https://user-images.githubusercontent.com/30341159/146865779-3d25d011-1f42-4026-9e1b-76f6e1d172e9.png)

then I get this error:
![image](https://user-images.githubusercontent.com/30341159/146865844-e60a404c-5f3a-403c-b2f1-acd943b5cdb8.png)

I have seen the issue in #2134 and #2150, so I don't understand why latest repo still can't deal with big dataset.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux docker
- Python version: 3.6


Update swahili_news dataset

Update dataset with latest verion data files.

Fix #3462.
Close bigscience-workshop/data_tooling#107


Update swahili_news dataset

dataset request

Requesting to add a new dataset

Please note also: the HuggingFace version at https://huggingface.co/datasets/swahili_news is outdated. An updated version, with deduplicated text and official splits, can be found at https://zenodo.org/record/5514203.

## Adding a Dataset
- **Name:** swahili_news

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Related to:
- bigscience-workshop/data_tooling#107


Fix links in metrics description

Remove Markdown syntax for links in metrics description, as it is not properly rendered.

Related to #3437.


Don't encode lists as strings when using `Value("string")`

Following https://github.com/huggingface/datasets/pull/3456#event-5792250497 it looks like `datasets` can silently convert lists to strings using `str()`, instead of raising an error.

This PR fixes this and should fix the issue with WER showing low values if the input format is not right.


dataset.filter overwriting previously set dataset._indices values, resulting in the wrong elements being selected.

bug

Something isn't working

I think this is a duplicate of [#3190](https://github.com/huggingface/datasets/issues/3190)?

Upgrading the datasets version as per #3190 fixes this bug.
I'm Marking as closed.

## Describe the bug
When using dataset.select to select a subset of a dataset, dataset._indices are set to indicate which elements are now considered in the dataset.
The same thing happens when you shuffle the dataset; dataset._indices are set to indicate what the new order of the data is.
However, if you then use a dataset.filter, that filter interacts with those dataset._indices values in a non-intuitive manner.
https://huggingface.co/docs/datasets/_modules/datasets/arrow_dataset.html#Dataset.filter

Effectively, it looks like the original set of _indices were discared and overwritten by the set created during the filter operation.

I think this is actually an issue with how the map function handles dataset._indices. Ideally it should use the _indices it gets passed, and then return an updated _indices which reflect the map transformation applied to the starting _indices.

## Steps to reproduce the bug
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print("initial 10 elements")
print(dataset['label'])   # -> [1, 1, 0, 1, 0, 0, 0, 1, 0, 0]

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)
print("filtered 10 elements looking for label 0")
print(dataset['label'])  # -> [1, 1, 1, 1, 1, 1]
```

## Actual results
```
$ python indices_bug.py
initial 10 elements
[1, 1, 0, 1, 0, 0, 0, 1, 0, 0]
filtered 10 elements looking for label 0
[1, 1, 1, 1, 1, 1]
```

This code block first shuffles the dataset (to get a mix of label 0 and label 1).
Then it selects just the first 10 elements (the number of elements does not matter, 10 is just easy to visualize). The important part is that you select some subset of the dataset.
Finally, a filter is applied to pull out just the elements with `label == 0`.

The bug is that you cannot combine any dataset operation which sets the dataset._indices with filter.
In this case I have 2, shuffle and subset.

If you just use a single dataset._indices operation (in this case shuffle) the bug still shows up.

The shuffle sets the dataset._indices and then filter uses those indices in the map, then overwrites dataset._indices with the filter results.
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print(dataset['label']) # -> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

## Expected results
In an ideal world, the dataset filter would respect any dataset._indices values which had previously been set.

If you use dataset.filter with the base dataset (where dataset._indices has not been set) then the filter command works as expected.

## Environment info
Here are the commands required to rebuild the conda environment from scratch.
```
# create a virtual environment
conda create -n dataset_indices python=3.8 -y

# activate the virtual environment
conda activate dataset_indices

# install huggingface datasets
conda install datasets
```

<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.11.0-41-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 3.0.0

### Full Conda Environment
```
$ conda env export
name: dasaset_indices
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - abseil-cpp=20210324.2=h2531618_0
  - aiohttp=3.8.1=py38h7f8727e_0
  - aiosignal=1.2.0=pyhd3eb1b0_0
  - arrow-cpp=3.0.0=py38h6b21186_4
  - attrs=21.2.0=pyhd3eb1b0_0
  - aws-c-common=0.4.57=he6710b0_1
  - aws-c-event-stream=0.1.6=h2531618_5
  - aws-checksums=0.1.9=he6710b0_0
  - aws-sdk-cpp=1.8.185=hce553d0_0
  - bcj-cffi=0.5.1=py38h295c915_0
  - blas=1.0=mkl
  - boost-cpp=1.73.0=h27cfd23_11
  - bottleneck=1.3.2=py38heb32a55_1
  - brotli=1.0.9=he6710b0_2
  - brotli-python=1.0.9=py38heb0550a_2
  - brotlicffi=1.0.9.2=py38h295c915_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.10.26=h06a4308_2
  - certifi=2021.10.8=py38h06a4308_0
  - cffi=1.14.6=py38h400218f_0
  - conllu=4.4.1=pyhd3eb1b0_0
  - cryptography=36.0.0=py38h9ce1e76_0
  - dataclasses=0.8=pyh6d0b6a4_7
  - dill=0.3.4=pyhd3eb1b0_0
  - double-conversion=3.1.5=he6710b0_1
  - et_xmlfile=1.1.0=py38h06a4308_0
  - filelock=3.4.0=pyhd3eb1b0_0
  - frozenlist=1.2.0=py38h7f8727e_0
  - gflags=2.2.2=he6710b0_0
  - glog=0.5.0=h2531618_0
  - gmp=6.2.1=h2531618_2
  - grpc-cpp=1.39.0=hae934f6_5
  - huggingface_hub=0.0.17=pyhd3eb1b0_0
  - icu=58.2=he6710b0_3
  - idna=3.3=pyhd3eb1b0_0
  - importlib-metadata=4.8.2=py38h06a4308_0
  - importlib_metadata=4.8.2=hd3eb1b0_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - krb5=1.19.2=hac12032_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libboost=1.73.0=h3ff78a5_11
  - libcurl=7.80.0=h0b77cf5_0
  - libedit=3.1.20210910=h7f8727e_0
  - libev=4.33=h7f8727e_1
  - libevent=2.1.8=h1ba5d50_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libnghttp2=1.46.0=hce63b2e_0
  - libprotobuf=3.17.2=h4ff587b_1
  - libssh2=1.9.0=h1ba5d50_1
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - libthrift=0.14.2=hcc01f38_0
  - libxml2=2.9.12=h03d6c58_0
  - libxslt=1.1.34=hc22bd24_0
  - lxml=4.6.3=py38h9120a33_0
  - lz4-c=1.9.3=h295c915_1
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py38h7f8727e_0
  - mkl_fft=1.3.1=py38hd3c417c_0
  - mkl_random=1.2.2=py38h51133e4_0
  - multiprocess=0.70.12.2=py38h7f8727e_0
  - multivolumefile=0.2.3=pyhd3eb1b0_0
  - ncurses=6.3=h7f8727e_2
  - numexpr=2.7.3=py38h22e1b3c_1
  - numpy=1.21.2=py38h20f2e39_0
  - numpy-base=1.21.2=py38h79a1101_0
  - openpyxl=3.0.9=pyhd3eb1b0_0
  - openssl=1.1.1l=h7f8727e_0
  - orc=1.6.9=ha97a36c_3
  - packaging=21.3=pyhd3eb1b0_0
  - pip=21.2.4=py38h06a4308_0
  - py7zr=0.16.1=pyhd3eb1b0_1
  - pycparser=2.21=pyhd3eb1b0_0
  - pycryptodomex=3.10.1=py38h27cfd23_1
  - pyopenssl=21.0.0=pyhd3eb1b0_1
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pyppmd=0.16.1=py38h295c915_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.12=h12debd9_0
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python-xxhash=2.0.2=py38h7f8727e_0
  - pyzstd=0.14.4=py38h7f8727e_3
  - re2=2020.11.01=h2531618_1
  - readline=8.1=h27cfd23_0
  - requests=2.26.0=pyhd3eb1b0_0
  - setuptools=58.0.4=py38h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - snappy=1.1.8=he6710b0_0
  - sqlite=3.36.0=hc218d9a_0
  - texttable=1.6.4=pyhd3eb1b0_0
  - tk=8.6.11=h1ccaba5_0
  - typing_extensions=3.10.0.2=pyh06a4308_0
  - uriparser=0.9.3=he6710b0_1
  - utf8proc=2.6.1=h27cfd23_0
  - wheel=0.37.0=pyhd3eb1b0_1
  - xxhash=0.8.0=h7f8727e_3
  - xz=5.2.5=h7b6447c_0
  - zipp=3.6.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7f8727e_4
  - zstd=1.4.9=haebb681_0
  - pip:
    - async-timeout==4.0.2
    - charset-normalizer==2.0.9
    - datasets==1.16.1
    - fsspec==2021.11.1
    - huggingface-hub==0.2.1
    - multidict==5.2.0
    - pandas==1.3.5
    - pyarrow==6.0.1
    - pytz==2021.3
    - pyyaml==6.0
    - tqdm==4.62.3
    - typing-extensions==4.0.1
    - urllib3==1.26.7
    - yarl==1.7.2

```


Fix duplicated tag in wikicorpus dataset card

CI is failing just because of empty sections - merging


Add CMU Graphics Lab Motion Capture dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** CMU Graphics Lab Motion Capture database
- **Description:** The database contains free motions which you can download and use.
- **Data:** http://mocap.cs.cmu.edu/
- **Motivation:** Nice motion capture dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


[WER] Better error message for wer

Hi ! I don't think this would solve this issue.
Currently it looks like there's a bug that converts the list `["hello it's nice"]` to a string `'["hello it's nice"]'` since this is what the metric expects as input. The conversion is done before the data are passed to `_compute()`.

This is `Value("string").encode_example` that is called to do the conversion. Since `str()` encoding is too permissive we should consider raising an error if the example is not a string (even though it can be converted to string).

I just opened https://github.com/huggingface/datasets/pull/3460 to fix that. It now raises an error instead of computing the wrong WER

Currently we have the following problem when using the WER. When the input format to the WER metric is wrong, instead of throwing an error message a word-error-rate is computed which is incorrect. E.g. when doing the following:

```python
from datasets import load_metric

wer = load_metric("wer")

target_str = ["hello this is nice", "hello the weather is bloomy"]
pred_str = [["hello it's nice"], ["hello it's the weather"]]

print("Wrong:", wer.compute(predictions=pred_str, references=target_str))
print("Correct", wer.compute(predictions=[x[0] for x in pred_str], references=target_str))
```
We get:
```
Wrong: 1.0
Correct 0.5555555555555556
```

meaning that we get a word-error rate for incorrectly passed input formats. We should raise an error here instead so that people don't spent hours fixing a model while it's their incorrect evaluation metric is the problem for a low WER.


Easier information editing

enhancement

New feature or request

generic discussion

Generic discussion on the library

Hi ! I guess you are talking about the dataset cards that are in this repository on github ?

I think github allows to submit a PR even for 1 line though the `Edit file` button on the page of the dataset card.

Maybe let's mention this in `CONTRIBUTING.md` ?

**Is your feature request related to a problem? Please describe.**
It requires a lot of effort to improve a datasheet.

**Describe the solution you'd like**
UI or at least a link to the place where the code that needs to be edited is (and an easy way to edit this code directly from the site, without cloning, branching, makefile etc.)

**Describe alternatives you've considered**
The current Ux is to have the 8 steps for contribution while One just wishes to change a line a type etc.


Fix iter_archive generator

This PR:
- Adds tests to DownloadManager and StreamingDownloadManager `iter_archive` for both path and file inputs
- Fixes bugs in `iter_archive` introduced in:
  - #3443

Fix #3453.


ValueError while iter_archive

bug

Something isn't working

## Describe the bug
After the merge of:
- #3443

the method `iter_archive` throws a ValueError:
```
ValueError: read of closed file
```

## Steps to reproduce the bug
```python
for path, file in dl_manager.iter_archive(archive_path):
    pass
```


enhancement

New feature or request

why the stratify option is omitted from test_train_split function?

is there any other way implement the stratify option while splitting the dataset? as it is important point to be considered while splitting the dataset.


[Staging] Update dataset repos automatically on the Hub

do keep us updated on how it's going in staging! cc @SBrandeis

Sure ! For now it works smoothly. We'll also do a new release today.

I can send you some repos to explore on staging, in case you want to see how they look like after being updated.
For example [swahili_news](https://moon-staging.huggingface.co/datasets/swahili_news/tree/main)

Let's have a script that updates the dataset repositories on staging for now. This way we can make sure it works fine before going in prod.

Related to https://github.com/huggingface/datasets/issues/3341

The script runs on each commit on `master`. It checks the datasets that were changed, and it pushes the changes to the corresponding repositories on the Hub.

If there's a new dataset, then a new repository is created.

If the commit is a new release of `datasets`, it also pushes the tag to all the repositories.


Unexpected behavior doing Split + Filter

bug

Something isn't working

Hi ! This is an issue with `datasets` 1.12. Sorry for the inconvenience. Can you update to `>=1.13` ?
see https://github.com/huggingface/datasets/issues/3190

Maybe we should also backport the bug fix to `1.12` (in a new version `1.12.2`)

## Describe the bug
I observed unexpected behavior when applying 'train_test_split' followed by 'filter' on dataset. Elements of the training dataset eventually end up in the test dataset (after applying the 'filter')

## Steps to reproduce the bug
```
from datasets import Dataset
import pandas as pd
dic = {'x': [1,2,3,4,5,6,7,8,9], 'y':['q','w','e','r','t','y','u','i','o']}
df = pd.DataFrame.from_dict(dic)
dataset = Dataset.from_pandas(df)
split_dataset = dataset.train_test_split(test_size=0.5, shuffle=False, seed=42)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]
eval_dataset_2 = eval_dataset.filter(lambda example: example['x'] % 2 == 0)
print( eval_dataset['x'])
print(eval_dataset_2['x'])
```
One observes that elements in eval_dataset2 are actually coming from the training dataset...

## Expected results
The expected results would be that the filtered eval dataset would only contain elements from the original eval dataset.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Windows 10
- Python version: 3.7
- PyArrow version: 5.0.0


Add `__add__()`, `__iadd__()` and similar to `Dataset` class

enhancement

New feature or request

generic discussion

Generic discussion on the library

I was going through the codebase, and I believe the implementation of __add__() and __iadd__() will be similar to concatenate_datasets() after the elimination of code for arguments other than the list of datasets (info, split, axis).
(Assuming elimination of axis means concatenating over axis 1.)

**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
I would like to be able to concatenate datasets as follows:
```python
>>> dataset["train"] += dataset["validation"]
```

... instead of using `concatenate_datasets()`:
```python
>>> raw_datasets["train"] = concatenate_datasets([raw_datasets["train"], raw_datasets["validation"]])
>>> del raw_datasets["validation"]
```

**Describe alternatives you've considered**
Well, I have considered `concatenate_datasets()` 😀

**Additional context**
N.a.


JSONDecodeError with HuggingFace dataset viewer

Hi ! I think the issue comes from the dataset_infos.json file: it has the "flat" field twice.

Can you try deleting this file and regenerating it please ?

Thanks! That fixed that, but now I am getting:
Server Error
Status code:   400
Exception:     KeyError
Message:       'feature'

I checked the dataset_infos.json and pubmed_neg.py script, I don't use 'feature' anywhere as a key. Is the dataset viewer expecting that I do?

It seems that the `feature` key is missing from some feature type definition in your dataset_infos.json:
```json
			"tokens": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
			},
			"tags": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
			}
```
They should be
```json
			"tokens": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
                                 "feature": {"dtype": "string", "id": null, "_type": "Value"}
			},
			"tags": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence",
                                 "feature": {"num_classes": 5, "names": ["-", "S", "H", "N", "C"], "names_file": null, "id": null, "_type": "ClassLabel"}
			}
```

Note that you can generate the dataset_infos.json automatically to avoid mistakes:
```bash
datasets-cli test ./path/to/dataset --save_infos
```

## Dataset viewer issue for 'pubmed_neg'

**Link:** https://huggingface.co/datasets/IGESML/pubmed_neg

I am getting the error:
Status code:   400
Exception:     JSONDecodeError
Message:       Expecting property name enclosed in double quotes: line 61 column 2 (char 1202)

I have checked all files - I am not using single quotes anywhere. Not sure what is causing this issue.

Am I the one who added this dataset ? Yes


HF_DATASETS_OFFLINE=1 didn't stop datasets.builder from downloading

bug

Something isn't working

Hi ! Indeed it says "downloading and preparing" but in your case it didn't need to download anything since you used local files (it would have thrown an error otherwise). I think we can improve the logging to make it clearer in this case

@lhoestq Thank you for explaining. I am sorry but I was not clear about my intention. I didn't want to kill internet traffic; I wanted to kill all write activity. In other words, you can imagine that my storage has only read access but crashes on write.

When run_clm.py is invoked with the same parameters, the hash in the cache directory "datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/..." doesn't change, and my job can load cached data properly. This is great.

Unfortunately, when params change (which happens sometimes), the hash changes and the old cache is invalid. datasets builder would create a new cache directory with the new hash and create JSON builder there, even though every JSON builder is the same. I didn't find a way to avoid such behavior.

This problem can be resolved when using datasets.map() for tokenizing and grouping text. This function allows me to specify output filenames with --cache_file_names, so that the cached files are always valid.

This is the code that I used to freeze cache filenames for tokenization. I wish I could do the same to datasets.load_dataset()
```
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on dataset",
            cache_file_names={k: os.path.join(model_args.cache_dir, f'{k}-tokenized') for k in raw_datasets},
        )
```

Hi ! `load_dataset` may re-generate your dataset if some parameters changed indeed. If you want to freeze a dataset loaded with `load_dataset`, I think the best solution is just to save it somewhere on your disk with `.save_to_disk(my_dataset_dir)` and reload it with `load_from_disk(my_dataset_dir)`. This way you will be able to reload the dataset without having to run `load_dataset`

## Describe the bug
According to https://huggingface.co/docs/datasets/loading_datasets.html#loading-a-dataset-builder, setting HF_DATASETS_OFFLINE to 1 should make datasets to "run in full offline mode". It didn't work for me. At the very beginning, datasets still tried to download "custom data configuration" for JSON, despite I have run the program once and cached all data into the same --cache_dir.

"Downloading" is not an issue when running with local disk, but crashes often with cloud storage because (1) multiply GPU processes try to access the same file, AND (2) FileLocker fails to synchronize all processes, due to storage throttling. 99% of times, when the main process releases FileLocker, the file is not actually ready for access in cloud storage and thus triggers "FileNotFound" errors for all other processes. Well, another way to resolve the problem is to investigate super reliable cloud storage, but that's out of scope here.

## Steps to reproduce the bug
```
export HF_DATASETS_OFFLINE=1
python run_clm.py --model_name_or_path=models/gpt-j-6B --train_file=trainpy.v2.train.json --validation_file=trainpy.v2.eval.json --cache_dir=datacache/trainpy.v2
```

## Expected results
datasets should stop all "downloading" behavior but reuse the cached JSON configuration. I think the problem here is part of the cache directory path, "default-471372bed4b51b53", is randomly generated, and it could change if some parameters changed. And I didn't find a way to use a fixed path to ensure datasets to reuse cached data every time.

## Actual results
The logging shows datasets are still downloading into "datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426".
```
12/16/2021 10:25:59 - WARNING - datasets.builder - Using custom data configuration default-471372bed4b51b53
12/16/2021 10:25:59 - INFO - datasets.builder - Generating dataset json (datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426)
Downloading and preparing dataset json/default to  datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|██████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 17623.13it/s]
12/16/2021 10:25:59 - INFO - datasets.utils.download_manager - Downloading took 0.0 min
12/16/2021 10:26:00 - INFO - datasets.utils.download_manager - Checksum Computation took 0.0 min
100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1206.99it/s]
12/16/2021 10:26:00 - INFO - datasets.utils.info_utils - Unable to verify checksums.
12/16/2021 10:26:00 - INFO - datasets.builder - Generating split train
12/16/2021 10:26:01 - INFO - datasets.builder - Generating split validation
12/16/2021 10:26:02 - INFO - datasets.utils.info_utils - Unable to verify splits sizes.
Dataset json downloaded and prepared to  datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 53.54it/s]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux
- Python version:  3.8.10
- PyArrow version: 6.0.1


Remove redundant local path information in audio/image datasets

Cool, I'm in favor of this PR. Our official examples in speech already make use of `"audio"` so no need to change anything there. It would be great if we could prominently feature how one can get the audio path without decoding in the docs.

@patrickvonplaten Yes, I agree.

cc @stevhliu we should add an example where decoding is disabled (to read paths) to  [this section](https://github.com/huggingface/datasets/blob/master/docs/source/audio_process.rst#audio-datasets) in the docs and remove the mentions of `path`/`file` (if we merge this PR).

Remove the redundant path information in the audio/image dataset as discussed in https://github.com/huggingface/datasets/pull/3430#issuecomment-994734828

TODOs:
* [ ] merge https://github.com/huggingface/datasets/pull/3430
* [ ] merge https://github.com/huggingface/datasets/pull/3364
* [ ] re-generate the info files of the updated audio datasets

cc:  @patrickvonplaten @anton-l @nateraw (I expect this to break the audio/vision examples in Transformers; after this change you'll be able to access underlying paths as follows `dset = dset.cast_column("audio", Audio(..., decode=False)); path = dset[0]["audio"]`)


question

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Align the Dataset and IterableDataset processing API

enhancement

New feature or request

generic discussion

Generic discussion on the library

Yes I agree, these should be as aligned as possible. Maybe we can also check the feedback in the survey at http://hf.co/oss-survey and see if people mentioned related things on the API (in particular if we go the breaking change way, it would be good to be sure we are taking the right direction for the community).

I like this proposal.

> There is also an important difference in terms of behavior:
Dataset.map adds new columns (with dict.update)
BUT
IterableDataset discards previous columns (it overwrites the dict)
IMO the two methods should have the same behavior. This would be an important breaking change though.

> The main breaking change would be the change of behavior of IterableDataset.map, because currently it discards all the previous columns instead of keeping them.

Yes, this behavior of `IterableDataset.map` was surprising to me the first time I used it because I was expecting the same behavior as `Dataset.map`, so I'm OK with the breaking change here.

> IterableDataset only supports "torch" (it misses tf, jax, pandas, arrow) and is missing the parameters: columns, output_all_columns and format_kwargs

\+  it's also missing the actual formatting code (we return unformatted tensors)
> We could have a completely aligned map method if both methods were lazy by default, but this is a very big breaking change so I'm not sure we can consider doing that.

> For information, TFDS does lazy map by default, and has an additional .cache() method.

If I understand this part correctly, the idea would be for `Dataset.map` to behave similarly to `Dataset.with_transform` (lazy processing) and to have an option to cache processed data (with `.cache()`). This idea is really nice because it can also be applied to `IterableDataset` to fix https://github.com/huggingface/datasets/issues/3142 (again we get the aligned APIs). However, this change would break a lot of things, so I'm still not sure if this is a step in the right direction (maybe it's OK for Datasets 2.0?)
> If the two APIs are more aligned it would be awesome for the examples in transformers, and it would create a satisfactory experience for users that want to switch from one mode to the other.

Yes, it would be amazing to have an option to easily switch between these two modes.

I agree with the rest.

> If I understand this part correctly, the idea would be for Dataset.map to behave similarly to Dataset.with_transform (lazy processing) and to have an option to cache processed data (with .cache()). This idea is really nice because it can also be applied to IterableDataset to fix #3142 (again we get the aligned APIs). However, this change would break a lot of things, so I'm still not sure if this is a step in the right direction (maybe it's OK for Datasets 2.0?)

Yea this is too big of a change in my opinion. Anyway it's fine as it is right now with streaming=lazy and regular=eager.

## Intro

Currently the two classes have two distinct API for processing:

### The `.map()` method

Both have those parameters in common: function, batched, batch_size

- IterableDataset is missing those parameters:
with_indices, with_rank, input_columns, drop_last_batch, remove_columns, features, disable_nullable, fn_kwargs, num_proc

- Dataset also has additional parameters that are exclusive, due to caching:
keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, suffix_template, new_fingerprint

- There is also an important difference in terms of behavior:
**Dataset.map adds new columns** (with dict.update)
BUT
**IterableDataset discards previous columns** (it overwrites the dict)
IMO the two methods should have the same behavior. This would be an important breaking change though.

- Dataset.map is eager while IterableDataset.map is lazy

### The `.shuffle()` method

- Both have an optional seed parameter, but IterableDataset requires a mandatory parameter buffer_size to control the size of the local buffer used for approximate shuffling.

- IterableDataset is missing the parameter generator

- Also Dataset has exclusive parameters due to caching: keep_in_memory, load_from_cache_file, indices_cache_file_name, writer_batch_size, new_fingerprint

### The `.with_format()` method

- IterableDataset only supports "torch" (it misses tf, jax, pandas, arrow) and is missing the parameters: columns, output_all_columns and format_kwargs

### Other methods

- Both have the same `remove_columns` method
- IterableDataset is missing: cast, cast_column, filter, rename_column, rename_columns, class_encode_column, flatten, prepare_for_task, train_test_split, shard
- Some other methods are missing but we can discuss them: set_transform, formatted_as, with_transform
- And others don't really make sense for an iterable dataset: select, sort, add_column, add_item
- Dataset is missing skip and take, that IterableDataset implements.

## Questions

I think it would be nice to be able to switch between streaming and regular dataset easily, without changing the processing code significantly.

1. What should be aligned and what shouldn't between those two APIs ?

IMO the minimum is to align the main processing methods.

It would mean aligning breaking the current `Iterable.map` to have the same behavior as `Dataset.map` (add columns with dict.update), and add multiprocessing as well as the missing parameters.

It would also mean implementing the missing methods: cast, cast_column, filter, rename_column, rename_columns, class_encode_column, flatten, prepare_for_task, train_test_split, shard

2. What are the breaking changes for IterableDataset ?

The main breaking change would be the change of behavior of `IterableDataset.map`, because currently it discards all the previous columns instead of keeping them.

3. Shall we also do some changes for regular datasets ?

I agree the simplest would be to have the exact same methods for both Dataset and IterableDataset. However this is probably not a good idea because it would prevent users from using the best benefits of them. That's why we can keep some aspects of regular datasets as they are:
- keep the eager Dataset.map with caching
- keep the with_transform method for lazy processing
- keep Dataset.select (it could also be added to IterableDataset even though it's not recommended)

We could have a completely aligned `map` method if both methods were lazy by default, but this is a very big breaking change so I'm not sure we can consider doing that.

For information, TFDS does lazy map by default, and has an additional `.cache()` method.

## Opinions ?

I'd love to gather some opinions about this here. If the two APIs are more aligned it would be awesome for the examples in `transformers`, and it would create a satisfactory experience for users that want to switch from one mode to the other.

cc @mariosasko @albertvillanova @thomwolf @patrickvonplaten @sgugger


Extend iter_archive to support file object input

This PR adds support to passing a file object to `[Streaming]DownloadManager.iter_archive`.

With this feature, we can iterate over a tar file inside another tar file.


Extend text to support yielding lines, paragraphs or documents

> The parameter can also be named `split_by` with values "line", "paragraph" or "document" (no 's' at the end)

@lhoestq @mariosasko I would avoid the term `split` in this context and keep it only for "train", "validation" and "test" splits.
- https://huggingface.co/docs/datasets/process.html#split
  > datasets.Dataset.train_test_split() creates train and test splits, if your dataset doesn’t already have them.
- https://huggingface.co/docs/datasets/process.html#process-multiple-splits
  > Many datasets have splits that you can process simultaneously with datasets.DatasetDict.map().

Please note that in the documentation, one of the terms more frequently used in this context is **"row"**:
- https://huggingface.co/docs/datasets/access.html#features-and-columns
  > A dataset is a table of rows and typed columns.

  > Return the number of rows and columns with the following standard attributes:
  > dataset.num_columns
  > 4
  > dataset.num_rows
  > 3668

- https://huggingface.co/docs/datasets/access.html#rows-slices-batches-and-columns
  > Get several rows of your dataset at a time with slice notation or a list of indices:
- https://huggingface.co/docs/datasets/process.html#map
  > This function can even create new rows and columns.

Other of the terms more frequently used in the docs (in the code as well) is **"example"**:
- https://huggingface.co/docs/datasets/process.html#map
  > It allows you to apply a processing function to each example in a dataset, independently or in batches.
- https://huggingface.co/docs/datasets/process.html#batch-processing
  > datasets.Dataset.map() also supports working with batches of examples.
- https://huggingface.co/docs/datasets/process.html#split-long-examples
  > When your examples are too long, you may want to split them
- https://huggingface.co/docs/datasets/process.html#data-augmentation
  > With batch processing, you can even augment your dataset with additional examples.

Less frequently used: **"item"**:
- https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.add_item
  > Add item to Dataset.

Other term used in the docs (although less frequently) is **"sample"**. The advantage of this word is that it is also a verb, so we can use the parameter: "sample_by" (if you insist on using a verb instead of a noun).

In summary, these proposals:
- config.row
- config.example
- config.item
- config.sample
- config.sample_by

I like `sample_by`. Another idea I had was `separate_by`.

It could also be `sampling`, `sampling_method`, `separation_method`.

Not a big fan of the proposed nouns alone since they are very generic, that's why I tried to have something more specific.

I also agree that we actually should avoid `split` to avoid any confusion

Thanks for the analysis of the used terms. I also like `sample_by` (`separate_by` is good too).

Add `config.row` option to `text` module to allow yielding lines (default, current case), paragraphs or documents.

Feel free to comment on the name of the config parameter `row`:
- Currently, the docs state datasets are made of rows and columns
- Other names I considered: `example`, `item`


Add QuALITY dataset

dataset request

Requesting to add a new dataset

I'll take this one if no one hasn't yet!

## Adding a Dataset
- **Name:** QuALITY
- **Description:** A challenging question answering with very long contexts (Twitter [thread](https://twitter.com/sleepinyourhat/status/1471225421794529281?s=20))
- **Paper:** No ArXiv link yet, but draft is [here](https://github.com/nyu-mll/quality/blob/main/quality_preprint.pdf)
- **Data:** GitHub repo [here](https://github.com/nyu-mll/quality)
- **Motivation:** This dataset would serve as a nice way to benchmark long-range Transformer models like BigBird, Longformer and their descendants. In particular, it would be very interesting to see how the S4 model fares on this given it's impressive performance on the Long Range Arena

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


datasets keeps reading from cached files, although I disabled it

bug

Something isn't working

Hi ! What version of `datasets` are you using ? Can you also provide the logs you get before it raises the error ?

## Describe the bug
Hi,
I am trying to avoid dataset library using cached files, I get the following bug when this tried to read the cached files. I tried to do the followings:
```
from datasets import set_caching_enabled
set_caching_enabled(False)
```
also force redownlaod:
```
 download_mode='force_redownload'
```
but none worked so far, this is on a cluster and on some of the machines this reads from the cached files, I really appreciate any idea on how to fully remove caching @lhoestq
many thanks

```
File "run_clm.py", line 496, in <module>
    main()
  File "run_clm.py", line 419, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 943, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/transformers/trainer.py", line 1445, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 172, in evaluate
    output = self.eval_loop(
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 241, in eval_loop
    metrics = self.compute_pet_metrics(eval_datasets, model, self.extra_info[metric_key_prefix], task=task)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 268, in compute_pet_metrics
    centroids = self._compute_per_token_train_centroids(model, task=task)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 353, in _compute_per_token_train_centroids
    data = get_label_samples(self.get_per_task_train_dataset(task), label)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 350, in get_label_samples
    return dataset.filter(lambda example: int(example['labels']) == label)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 470, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2519, in filter
    indices = self.map(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2036, in map
    return self._map_single(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 503, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 470, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2248, in _map_single
    return Dataset.from_file(cache_file_name, info=info, split=self.split)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 654, in from_file
    return cls(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 593, in __init__
    self.info.features = self.info.features.reorder_fields_as(inferred_features)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/features/features.py", line 1092, in reorder_fields_as
    return Features(recursive_reorder(self, other))
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/features/features.py", line 1081, in recursive_reorder
    raise ValueError(f"Keys mismatch: between {source} and {target}" + stack_position)
ValueError: Keys mismatch: between {'indices': Value(dtype='uint64', id=None)} and {'candidates_ids': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None), 'labels': Value(dtype='int64', id=None), 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None), 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), 'extra_fields': {}, 'task': Value(dtype='string', id=None)}

```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 6.0.1


Add `cast_column` to `IterableDataset`

Closes #3369.

cc: @patrickvonplaten


Update supported versions of Python in setup.py

Update the list of supported versions of Python in `setup.py` to keep the PyPI project description updated.


Update BLEURT hyperlink

seems like a very very low-prio improvement :)

@albertvillanova thanks for the feedback! I removed the formatting altogether since I think this is a bit simpler tor read than non-rendered Markdown

The description of BLEURT on the hf.co website has a strange use of URL hyperlinking. This PR attempts to fix this, although I am not 100% sure Markdown syntax is allowed on the frontend or not.

![Screen Shot 2021-12-15 at 17 31 27](https://user-images.githubusercontent.com/26859204/146226432-c83cbdaf-f57d-4999-b53c-85da718ff7fb.png)


Add the OneStopQa dataset

Adding OneStopQA,  a multiple choice reading comprehension dataset annotated according to the STARC (Structured Annotations for Reading Comprehension) scheme.


Improve Wikipedia Loading Script

I wanted to flag a change from since we discussed this: I initially wrote a function for using the Wikimedia APIs to collect namespace aliases, but decided that adding in more http requests to the script wasn't a great idea so instead used that code to build a static list that I just added directly to the code.

Also, an FYI that python library dependencies weren't working on my local end so I wasn't able to directly test the code. I tested a copy with the problematic elements stripped (beam etc.) that worked fine, but someone with a working local copy may want to test just to make sure I didn't accidentally break anything.

Also, while I would argue more strongly for some of the changes in this code, they are five distinct changes so not so hard to remove one or two if other folks think they aren't worth the overhead etc.

* More structured approach to detecting redirects
* Remove redundant template filter code (covered by strip_code)
* Add language-specific lists of additional media namespace aliases for filtering
* Add language-specific lists of category namespace aliases for new link text cleaning step
* Remove magic words (parser directions like __TOC__ that occasionally occur in text)

Fix #3400

With support from @albertvillanova
CC @yjernite


Add The People's Speech

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** The People's Speech
- **Description:** a massive English-language dataset of audio transcriptions of full sentences.
- **Paper:** https://openreview.net/pdf?id=R8CwidgJ0yT
- **Data:** https://mlcommons.org/en/peoples-speech/
- **Motivation:** With over 30,000 hours of speech, this dataset is the largest and most diverse freely available English speech recognition corpus today.

[The article](https://thegradient.pub/new-datasets-to-democratize-speech-recognition-technology-2/) which may be useful when working on the dataset.

cc: @anton-l

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add Multilingual Spoken Words dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Multilingual Spoken Words
- **Description:** Multilingual Spoken Words Corpus is a large and growing audio dataset of spoken words in 50 languages for academic research and commercial applications in keyword spotting and spoken term search, licensed under CC-BY 4.0. The dataset contains more than 340,000 keywords, totaling 23.4 million 1-second spoken examples (over 6,000 hours).

  Read more: https://mlcommons.org/en/news/spoken-words-blog/
- **Paper:** https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/fe131d7f5a6b38b23cc967316c13dae2-Paper-round2.pdf
- **Data:** https://mlcommons.org/en/multilingual-spoken-words/
- **Motivation:**

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Correctly indent builder config in dataset script docs


Unable to resolve any data file after loading once

Hi ! `load_dataset` accepts as input either a local dataset directory or a dataset name from the Hugging Face Hub.

So here you are getting this error the second time because it tries to load the local `wiki_dpr` directory, instead of `wiki_dpr` from the Hub. It doesn't work since it's a **cache** directory, not a **dataset** directory in itself.

To fix that you can use another cache directory like `cache_dir="/data2/whr/lzy/open_domain_data/retrieval/cache"`

when I rerun my program, it occurs this error
" Unable to resolve any data file that matches '['**train*']' at /data2/whr/lzy/open_domain_data/retrieval/wiki_dpr with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'zip']", so how could i deal with this problem?
thx.
And below is my code .
![image](https://user-images.githubusercontent.com/84694183/146023446-d75fdec8-65c1-484f-80d8-6c20ff5e994b.png)