Loading JSON throws ArrowNotImplementedError

bug

Something isn't working

Hi @lewtun, thanks for reporting.

Apparently, `pyarrow.json` tries to cast timestamp-like fields in your JSON file to pyarrow timestamp type, and it fails with `ArrowNotImplementedError`.

I will investigate if there is a way to tell pyarrow not to try that timestamp casting.

I think the issue is more complex than that...

I just took one of your JSON lines and pyarrow.json read it without problem.

> I just took one of your JSON lines an pyarrow.json read it without problem.

yes, and for some peculiar reason the error is non-deterministic (i was eventually able to load the whole dataset by just re-running the `load_dataset` cell multiple times 🤔)

thanks for looking into this 🙏 !

I think the error is generated by the `pyarrow.json.read()` option: `read_options=paj.ReadOptions(block_size=block_size)`...
cc: @lhoestq

The code works fine on my side.
Not sure what's going on here :/

I remember we did a few changes in the JSON loader in #2638 , did you do an update `datasets` when debugging this ?

OK after upgrading `datasets` to v1.12.1 the issue seems to have gone away. Closing this now :)

Oops, I spoke too soon 😓

After deleting the cache and trying the above code snippet again I am hitting the same error. You can also reproduce it in the Colab notebook I linked to in the issue description.

@albertvillanova @lhoestq I noticed the same issue using datasets v1.12.1. Is there an update on when this could be fixed?

Apparently it's possible to make it work by increasing the `block_size`, let me open a PR

I just opened a PR with a fix, feel free to install `datasets` from source from source and let me know if it helps

@zijwang did PR #3000 solve the problem for you? It did for me, so it all is good on your end we can close this issue. Thanks again to @lhoestq for the pyarrow magic 🤯

## Describe the bug
I have created a [dataset](https://huggingface.co/datasets/lewtun/github-issues-test) of GitHub issues in line-separated JSON format and am finding that I cannot load it with the `json` loading script (see stack trace below).

Curiously, there is no problem loading the dataset with `pandas` which suggests some incorrect type inference is being made on the `datasets` side. For example, the stack trace indicates that some URL fields are being parsed as timestamps.

You can find a Colab notebook which reproduces the error [here](https://colab.research.google.com/drive/1YUCM0j1vx5ZrouQbYSzal6RwB4-Aoh4o?usp=sharing).

**Edit:** If one repeatedly tries to load the dataset, it _eventually_ works but I think it would still be good to understand why it fails in the first place :)

## Steps to reproduce the bug
```python
from datasets import load_dataset
from huggingface_hub import hf_hub_url
import pandas as pd

# returns https://huggingface.co/datasets/lewtun/github-issues-test/resolve/main/issues-datasets.jsonl
data_files = hf_hub_url(repo_id="lewtun/github-issues-test", filename="issues-datasets.jsonl", repo_type="dataset")
# throws ArrowNotImplementedError
dset = load_dataset("json", data_files=data_files, split="test")
# no problem with pandas ...
df = pd.read_json(data_files, orient="records", lines=True)
df.head()
```

## Expected results
I can load any line-separated JSON file, similar to `pandas`.

## Actual results
```
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
<ipython-input-7-5b8e82b6c3a2> in <module>()
----> 1 dset = load_dataset("json", data_files=data_files, split="test")

9 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: JSON conversion to struct<url: timestamp[s], html_url: timestamp[s], labels_url: timestamp[s], id: int64, node_id: timestamp[s], number: int64, title: timestamp[s], description: timestamp[s], creator: struct<login: timestamp[s], id: int64, node_id: timestamp[s], avatar_url: timestamp[s], gravatar_id: timestamp[s], url: timestamp[s], html_url: timestamp[s], followers_url: timestamp[s], following_url: timestamp[s], gists_url: timestamp[s], starred_url: timestamp[s], subscriptions_url: timestamp[s], organizations_url: timestamp[s], repos_url: timestamp[s], events_url: timestamp[s], received_events_url: timestamp[s], type: timestamp[s], site_admin: bool>, open_issues: int64, closed_issues: int64, state: timestamp[s], created_at: timestamp[s], updated_at: timestamp[s], due_on: timestamp[s], closed_at: timestamp[s]> is not supported
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyArrow version: 3.0.0


Fix streaming zip files

Hi ! I don't fully understand this change @albertvillanova
The `_extract` method used to return the compound URL that points to the root of the inside of the archive.
This way users can use the usual os.path.join or other functions to point to the relevant files. I don't see why you're using a glob pattern ?

This change is to allow this:
```python
data_files = f"https://huggingface.co/datasets/albertvillanova/datasets-tests-compression/resolve/main/sample.zip"
ds = load_dataset("json", split="train", data_files=data_files, streaming=True)
assert isinstance(ds, IterableDataset)
```
Note that in this case the user will not call os.path.join.

Before this PR it gave error because pointing to the root, without any subsequent join, gives error:
```python
fsspec.open("zip://::https://huggingface.co/datasets/albertvillanova/datasets-tests-compression/resolve/main/sample.zip")
```

Currently, streaming remote zip data files gives `FileNotFoundError` message:
```python
data_files = f"https://huggingface.co/datasets/albertvillanova/datasets-tests-compression/resolve/main/sample.zip"
ds = load_dataset("json", split="train", data_files=data_files, streaming=True)
next(iter(ds))
```

This PR fixes it by adding a glob string.

The corresponding test is implemented in PR #2786.


Make creating/editing dataset cards easier, by editing on site and dumping info from test command.

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

Creating and editing dataset cards should be but not that easy
- If other else know Some information I don't know (bias of dataset, dataset curation, supported dataset, ...), he/she should know the description on hf.co comes from README.md under github huggingface/datasets/datasets/the dataset, and willing to make a pr to add or fix information.
- Many information is also saved in `dataset_info.json` (citaion, description), but still need to write it down to README.md again.
- Contributor need to pip install and start a local server just for tagging the dataset's size. And contributor may be creating the dataset on lab's server, which can't open a browser.
- if any one proposes a new tag, it doesn't show in the list that another creator see. (a stackoverflow way may be ideal)
- dataset card generator web app doesn't generate the necessary subsecion `Contributions` for us.

**Describe the solution you'd like**
- Everyone (or at least the author/contributor) can edit the description, information, tags of the dataset, on hf.co website. Just like wikipedia+stackoverflow
-  We can infer the actual data size, citation, data instance, ... from `dataset_info.json` and `dataset.arrow`  via `dataset-cli test`


add cedr dataset

> Hi ! Thanks a lot for adding this one :)
>
> Good job with the dataset card and the dataset script !
>
> I left a few suggestions

Thank you very much for your helpful suggestions. I have tried to carry them all out.


Warnings and documentation about pickling incorrect

bug

Something isn't working

## Describe the bug

I have a docs bug and a closely related docs enhancement suggestion!

### Bug

The warning and documentation say "either `dill` or `pickle`" for fingerprinting. But it seems that `dill`, which is installed by `datasets` by default, _must_ work, or else the fingerprinting fails.

Warning:

https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L262

Docs:

> For a transform to be hashable, it needs to be pickleable using dill or pickle.
> – [docs](https://huggingface.co/docs/datasets/processing.html#fingerprinting)

For my code, `pickle` works, but `dill` fails. The `dill` failure has already been reported in https://github.com/huggingface/datasets/issues/2643. However, the `dill` failure causes a hashing failure in the datasets library, without any backing off to `pickle`. This implies that it's not the case that either `dill` **or** `pickle` can work, but that `dill` must work if it is installed. I think this is more accurate wording, since it is installed and used by default:

https://github.com/huggingface/datasets/blob/c93525dc291346e54212567fa72d7d607befe937/setup.py#L83

... and the hashing will fail if it fails.

### Enhancement

I think it'd be very helpful to add to the documentation how to debug hashing failures. It took me a while to figure out how to diagnose this. There is a very nice two-liner by @lhoestq in https://github.com/huggingface/datasets/issues/2516#issuecomment-865173139:

```python
from datasets.fingerprint import Hasher
Hasher.hash(my_object)
```

I think add this to the docs will help future users quickly debug any hashing troubles of their own :-)

## Steps to reproduce the bug

`dill` but not `pickle` hashing failure in https://github.com/huggingface/datasets/issues/2643

## Expected results
If either `dill` or `pickle` can successfully hash, the hashing will succeed.

## Actual results
If `dill` or `pickle` cannot hash, the hashing fails.

## Environment info

- `datasets` version: 1.9.0
- Platform: Linux-5.8.0-1038-gcp-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Fix type hint for data_files

Fix type hint for `data_files` in signatures and docstrings.


Update: GooAQ - add train/val/test splits

@albertvillanova my tests are failing here:
```
dataset_name = 'gooaq'

    def test_load_dataset(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True, use_local_dummy_data=True)

tests/test_dataset_common.py:234:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:187: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
```
When I try loading dataset on local machine it works fine. Any suggestions on how can I avoid this error?

Thanks for the help, @albertvillanova! All tests are passing now.

[GooAQ](https://github.com/allenai/gooaq) dataset was recently updated after splits were added for the same. This PR contains new updated GooAQ with train/val/test splits and updated README as well.


Fix typo in cnn_dailymail


Fix typo in test_dataset_common


Updated dataset description of DaNE

Thanks for finishing it @albertvillanova


How to sample every file in a list of files making up a split in a dataset when loading?

Hi ! This is not possible just with `load_dataset`.

You can do something like this instead:
```python
seed=42
data_files_dict = {
    "train": [train_file1, train_file2],
    "test": [test_file1, test_file2],
    "val": [val_file1, val_file2]
}
dataset = datasets.load_dataset(
    "csv",
    data_files=data_files_dict,
).shuffle(seed=seed)

sample_dataset = {splitname: split.select(range(8)) for splitname, split in dataset.items()}
```

Another alternative is loading each file separately with `split="train[:8]"` and then use `concatenate_datasets` to merge the sample of each file.

I am loading a dataset with multiple train, test, and validation files like this:

```
data_files_dict = {
    "train": [train_file1, train_file2],
    "test": [test_file1, test_file2],
    "val": [val_file1, val_file2]
}
dataset = datasets.load_dataset(
    "csv",
    data_files=data_files_dict,
    split=['train[:8]', 'test[:8]', 'val[:8]']
)

```

However, this only selects the first 8 rows from train_file1, test_file1, val_file1, since they are the first files in the lists.

I'm trying to formulate a split argument that can sample from each file specified in my list of files that make up each split.

Is this type of splitting supported? If so, how can I do it?


Something isn't working

the bug code locate in ：
    if data_args.task_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset("glue", data_args.task_name, cache_dir=model_args.cache_dir)

Hi @jinec,

From time to time we get this kind of `ConnectionError` coming from the github.com website: https://raw.githubusercontent.com

Normally, it should work if you wait a little and then retry.

Could you please confirm if the problem persists?

cannot connect，even by Web browser，please check that  there is some  problems。

> I can access https://raw.githubusercontent.com/huggingface/datasets/1.7.0/datasets/glue/glue.py without problem...

I can not access https://raw.githubusercontent.com/huggingface/datasets either,   I am in China

> Finally i can access it, by the superfast software. Thanks

Excuse me, I have the same problem as you, could you please tell me how to solve it？

Hello,
I am trying to run run_glue.py and it gives me this error -

Traceback (most recent call last):
  File "E:/BERT/pytorch_hugging/transformers/examples/pytorch/text-classification/run_glue.py", line 546, in <module>
    main()
  File "E:/BERT/pytorch_hugging/transformers/examples/pytorch/text-classification/run_glue.py", line 250, in main
    datasets = load_dataset("glue", data_args.task_name, cache_dir=model_args.cache_dir)
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\load.py", line 718, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\load.py", line 320, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\utils\file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "C:\install\Anaconda3\envs\huggingface\lib\site-packages\datasets\utils\file_utils.py", line 623, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.7.0/datasets/glue/glue.py

Trying to do python run_glue.py  --model_name_or_path
bert-base-cased
--task_name
mrpc
--do_train
--do_eval
--max_seq_length
128
--per_device_train_batch_size
32
--learning_rate
2e-5
--num_train_epochs
3
--output_dir
./tmp/mrpc/

Is this something on my end? From what I can tell, this was re-fixeded by @fullyz a few months ago.
Thank you!


Support streaming compressed files

Add support to stream compressed files (current options in fsspec):
- bz2
- lz4
- xz
- zstd

cc: @lewtun


Add KS task to SUPERB

thanks a lot for implementing this @anton-l !!

i won't have time to review this while i'm away, so happy for @albertvillanova and @patrickvonplaten to decide when to merge :)

@albertvillanova thanks! Everything should be ready now :)

> The _background_noise_/_silence_ audio files are much longer than others, so they require some sort of slicing for downstream training. I decided to leave the implementation of that up to the users, since TFDS and s3prl take different approaches (either slicing wavs deterministically, or subsampling randomly at runtime)

@anton-l I was thinking that maybe we could give some hints in the dataset card (in a Usage section); something similar as for diarization: https://github.com/huggingface/datasets/blob/master/datasets/superb/README.md#example-of-usage
Note that for diarization it is not yet finished: we have to test it and then provide an end-to-end example: https://github.com/huggingface/datasets/pull/2661/files#r680224909

@albertvillanova yeah, I'm not sure how to best implement it in pure `datasets` yet. It's something like this, where `sample_noise()` needs to be called from a pytorch batch collator or other framework-specific variant:

```python
def map_to_array(example):
    import soundfile as sf

    speech_array, sample_rate = sf.read(example["file"])
    example["speech"] = speech_array
    example["sample_rate"] = sample_rate
    return example

def sample_noise(example):
    # Use a version of this function in a stateless way to extract random 1 sec slices of background noise
    # on each epoch
    from random import randint

    # _silence_ audios are longer than 1 sec
    if example["label"] == "_silence_":
        random_offset = randint(0, len(example["speech"]) - example["sample_rate"] - 1)
        example["speech"] = example["speech"][random_offset : random_offset + example["sample_rate"]]

    return example
```

I see... Yes, not trivial indeed. Maybe for the moment you could add those functions above to the README (as it is the case for now in diarization)? What do you think?

Add the KS (keyword spotting) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051).

- [s3prl instructions](https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/README.md#ks-keyword-spotting)
- [s3prl implementation](https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/speech_commands/dataset.py)
- [TFDS implementation](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/speech_commands.py)

Some notable quirks:
- The dataset is originally single-archive (train+val+test all in one), but the test set has a "canonical" distribution in a separate archive, which is also used here (see `_split_ks_files()`).
- The `_background_noise_`/`_silence_` audio files are much longer than others, so they require some sort of slicing for downstream training. I decided to leave the implementation of that up to the users, since TFDS and s3prl take different approaches (either slicing wavs deterministically, or subsampling randomly at runtime)

Related to #2619.


Fix renaming of corpus_bleu args

Last `sacrebleu` release (v2.0.0) has renamed `sacrebleu.corpus_bleu` args from `(sys_stream, ref_streams)` to `(hipotheses, references)`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR passes the args without parameter names, so that it is valid for all versions of `sacrebleu`.

This is a partial hotfix of #2781.

Close #2781.


Latest v2.0.0 release of sacrebleu has broken some metrics

Something isn't working

## Describe the bug
After `sacrebleu` v2.0.0 release (see changes here: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15), some of `datasets` metrics are broken:
- Default tokenizer `sacrebleu.DEFAULT_TOKENIZER` no longer exists:
  - #2739
  - #2778
- Bleu tokenizers are no longer accessible with `sacrebleu.TOKENIZERS`:
  - #2779
- `corpus_bleu` args have been renamed from `(sys_stream, ref_streams)` to `(hipotheses, references)`:
  - #2782


VIVOS dataset for Vietnamese ASR


Fix sacrebleu tokenizers

Last `sacrebleu` release (v2.0.0) has removed `sacrebleu.TOKENIZERS`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR makes a hot fix of the bug by using a private function in `sacrebleu`: `sacrebleu.metrics.bleu._get_tokenizer()`.

Eventually, this should be further fixed in order to use only public functions.

This is a partial hotfix of #2781.


Do not pass tokenize to sacrebleu

Last `sacrebleu` release (v2.0.0) has removed `sacrebleu.DEFAULT_TOKENIZER`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR does not pass `tokenize` to `sacrebleu` (note that the user cannot pass it anyway) and `sacrebleu` will use its default, no matter where it is and how it is called.

Related to #2739.

This is a partial hotfix of #2781.


Use packaging to handle versions

Use packaging module to handle/validate/check versions of Python packages.

Related to #2769.


document `config.HF_DATASETS_OFFLINE` and precedence

enhancement

New feature or request

https://github.com/huggingface/datasets/pull/1976 implemented `HF_DATASETS_OFFLINE`, but:
1. `config.HF_DATASETS_OFFLINE` is not documented
2. the precedence is not documented (env, config)

I'm thinking it probably should be similar to what it says https://huggingface.co/docs/datasets/loading_datasets.html#from-the-huggingface-hub about `datasets.config.IN_MEMORY_MAX_SIZE`:

Quote:
> The default in 🤗 Datasets is to memory-map the dataset on disk unless you set datasets.config.IN_MEMORY_MAX_SIZE different from 0 bytes (default). In that case, the dataset will be copied in-memory if its size is smaller than datasets.config.IN_MEMORY_MAX_SIZE bytes, and memory-mapped otherwise. This behavior can be enabled by setting either the configuration option datasets.config.IN_MEMORY_MAX_SIZE (higher precedence) or the environment variable HF_DATASETS_IN_MEMORY_MAX_SIZE (lower precedence) to nonzero.

Context: trying to use `config.HF_DATASETS_OFFLINE` here:
https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/48
but are uncertain if it's safe, since it's not documented as a public API.

Thank you!

@lhoestq, @albertvillanova


`generate_random_fingerprint()` deterministic with 🤗Transformers' `set_seed()`

bug

Something isn't working

I dug into what I believe is the root of this issue and added a repro in my comment. If this is better addressed as a cross-team issue, let me know and I can open an issue in the Transformers repo

Hi !

IMO we shouldn't try to modify `set_seed` from transformers but maybe make `datasets` have its own RNG just to generate random fingerprints.

Any opinion on this @LysandreJik ?

## Describe the bug

**Update:** I dug into this to try to reproduce the underlying issue, and I believe it's that `set_seed()` from the `transformers` library makes the "random" fingerprint identical each time. I believe this is still a bug, because `datasets` is used exactly this way in `transformers` after `set_seed()` has been called, and I think that using `set_seed()` is a standard procedure to aid reproducibility. I've added more details to reproduce this below.

Hi there! I'm using my own local dataset and custom preprocessing function. My preprocessing function seems to be unpickle-able, perhaps because it is from a closure (will debug this separately). I get this warning, which is expected:

https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L260-L265

However, what's not expected is that the `datasets` actually _does_ seem to cache and reuse this dataset between runs! After that line, the next thing that's logged looks like:

```text
 Loading cached processed dataset at /home/xxx/.cache/huggingface/datasets/csv/default-xxx/0.0.0/xxx/cache-xxx.arrow
```

The path is exactly the same each run (e.g., last 26 runs).

This becomes a problem because I'll pass in the `--max_eval_samples` flag to the HuggingFace example script I'm running off of ([run_swag.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/multiple-choice/run_swag.py)).  The fact that the cached dataset is reused means this flag gets ignored. I'll try to load 100 examples, and it will load the full cached 1,000,000.

I think that

https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L248

... is actually consistent because randomness is being controlled in HuggingFace/Transformers for reproducibility. I've added a demo of this below.

## Steps to reproduce the bug

```python
# Contents of print_fingerprint.py
from transformers import set_seed
from datasets.fingerprint import generate_random_fingerprint
set_seed(42)
print(generate_random_fingerprint())
```

```bash
for i in {0..10}; do
    python print_fingerprint.py
done

1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
```

## Expected results
After the "random hash" warning is emitted, a random hash is generated, and no outdated cached datasets are reused.

## Actual results
After the "random hash" warning is emitted, an identical hash is generated each time, and an outdated cached dataset is reused each run.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.0
- Platform: Linux-5.8.0-1038-gcp-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Prevent .map from using multiprocessing when loading from cache

I'm guessing tests are failling, because this was pushed before https://github.com/huggingface/datasets/pull/2779 was merged? cc @albertvillanova

Hi @thomasw21, yes you are right: those failing tests were fixed with #2779.

Would you mind to merge current upstream master branch and push again?
```
git checkout sequential_map_when_cached
git fetch upstream master
git merge upstream/master
git push -u origin sequential_map_when_cached
```

Thanks for working on this ! I'm sure we can figure something out ;)

Currently `map` starts a process to apply the map function on each shard. If the shard has already been processed, then the process that has been spawned loads the processed shard from the cache and returns it.

I think we should be able to simply not start a process if a shard is already processed and cached.
This way:
- you won't need to specify `sequential=True`
- it won't create new processes if the dataset is already processed and cached
- it will properly reload each processed shard that is cached

To know if we have to start a new process for a shard you can use the function `update_fingerprint` from fingerprint.py to know the expected fingerprint of the processed shard.
Then, if the shard has already been processed, there will be a cache file named `cached-<new_fingerprint>.arrow` and you can load it with
```
Dataset.from_file(path_to_cache_file, info=self.info, split=self.split)
```

Let me know if that makes sense !

Yes that makes total sense, I tried to initially do that, except the way fingerprint is handled doesn't allow to easily manipulate such a field. Typically the fingerprinting is handled in `@fingerprint_transform` which has a bunch of params that aren't quite easy to extract. Those params are used to manipulate args, kwargs in fancy ways in order to finally obtain a dictionary used for fingerprint. I could duplicate everything, but this look like a very risky thing to do. I'll take a look if I can make something work with `inspect` if I can make a very simple wrapper.

A much more simpler solution I think is adding an optional `shard: Optional[int] = None` parameter. If None, we use the number of proc as the number of shards, otherwise we pass down the expected number of shards and use either sequential/multiprocessing (with arbitrary number of workers) to load the shards? This would allow the weird case where one wants a large number of shards with a limited amount of processes. Not the smartest thing to do, but it's not an absurd behaviour. Would this be acceptable?

@lhoestq friendly ping as I feel it's up for review.

The CI error is unrelated to the changes of this PR - it looks like an SSL issue with conda

## Context

On our setup, we use different setup to train vs proprocessing datasets. Usually we are able to obtain a high number of cpus to preprocess, which allows us to use `num_proc` however we can't use as many during training phase. Currently if we use `num_proc={whatever the preprocessing value was}` we load from cache, but we get:

```
Traceback (most recent call last):
  File "lib/python3.8/site-packages/multiprocess/pool.py", line 131, in worker
    put((job, i, result))
  File "lib/python3.8/site-packages/multiprocess/queues.py", line 371, in put
    self._writer.send_bytes(obj)
  File "lib/python3.8/site-packages/multiprocess/connection.py", line 203, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "lib/python3.8/site-packages/multiprocess/connection.py", line 414, in _send_bytes
    self._send(header + buf)
  File "lib/python3.8/site-packages/multiprocess/connection.py", line 371, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
```

Our current guess, is that we're spawning too many processes compared to the number of cpus available, and it's running OOM.  Also we're loading this in DDP setting which means that for each gpu, I need to spawn a high number of processes to match the preprocessing fingerprint.

Instead what we suggest:
 - Allow loading shard sequentially, sharing the same fingerprint as the multiprocessed one, in order to leverage multiprocessing when we actually generate the cache, and remove it when loading from cache.

## Current issues

~I'm having a hard time making fingerprints match. For some reason, the multiprocessing and the sequential version generate two different hash.~

**EDIT**: Turns out multiprocessing and sequential have different `transform` value for fingerprinting (check `fingerprint_transform`) when running `_map_single`:
 - sequential : `datasets.arrow_dataset.Dataset._map_single`
 - multiprocessing: `datasets.arrow_dataset._map_single`

 This discrepancy is caused by multiprocessing pickling the transformer function, it doesn't seem to keep the `Dataset` hierarchy. I'm still unclear on why `func.__qual_name__` isn't handled correctly in multiprocessing. But replacing `__qualname__` by `__name__` fixes the issue.

## What was done

~We try to prevent the usage of multiprocessing when loading a dataset. Instead we load all cached shards sequentially.~

I couldn't find a nice way to obtain the cached_file_name and check they all exist before deciding to use the multiprocessing flow or not. Instead I expose an optional boolean `sequential` in `map` method.

## TODO
 - [x] Check that the multiprocessed version and the sequential version output the same output
 - [x] Check that sequential can load multiprocessed
 - [x] Check that multiprocessed can load sequential

 ## Test

```python
from datasets import load_dataset
from multiprocessing import Pool
import random

def process(batch, rng):
  length = len(batch["text"])
  return {**batch, "processed_text": [f"PROCESSED {rng.random()}" for _ in range(length)]}

dataset = load_dataset("stas/openwebtext-10k", split="train")
print(dataset.column_names)
print(type(dataset))

rng = random.Random(42)
dataset1 = dataset.map(process, batched=True, batch_size=50, num_proc=4, fn_kwargs={"rng": rng})

# This one should be loaded from cache
rng = random.Random(42)
dataset2 = dataset.map(process, batched=True, batch_size=50, num_proc=4, fn_kwargs={"rng": rng}, sequential=True)

# Just to check that the random generator was correct
print(dataset1[-1]["processed_text"])
print(dataset2[-1]["processed_text"])
```

 ## Other solutions

I chose to load everything sequentially, but we can probably find a way to load shards in parallel using another number of workers (essentially this would be an argument not used for fingerprinting, allowing to allow `m` shards using `n` processes, which would be very useful when same dataset have to be loaded on two different setup, and we still want to leverage cache).

Also we can use a env variable similarly to `TOKENIZERS_PARALLELISM` as this seems generally setup related (though this changes slightly if we use multiprocessing).
cc @lhoestq (since I had asked you previously on `num_proc` being used for fingerprinting). Don't know if this is acceptable.


Remove dataset_infos.json

enhancement

New feature or request

generic discussion

Generic discussion on the library

**Is your feature request related to a problem? Please describe.**
As discussed, there are infos in the `dataset_infos.json` which are redundant and we could have them only in the README file.

Others could be migrated to the README, like: "dataset_size", "size_in_bytes", "download_size", "splits.split_name.[num_bytes, num_examples]",...

However, there are others that do not seem too meaningful in the README, like the checksums.

**Describe the solution you'd like**
Open a discussion to decide what to do with the `dataset_infos.json` files: which information to be migrated and/or which information to be kept.

cc: @julien-c @lhoestq


Remove returned feature constrain

enhancement

New feature or request

In the current version, the returned value of the map function has to be list or ndarray. However, this makes it unsuitable for many tasks. In NLP, many features are sparse like verb words, noun chunks, if we want to assign different values to different words, which will result in a large sparse matrix if we only score useful words like verb words.

Mostly, when using it on large scale, saving it as a whole takes a lot of disk storage and making it hard to read,  the normal method is saving it in sparse form. However, the NumPy does not support sparse, therefore I have to use PyTorch or scipy to transform a matrix into special sparse form, which is not a form that can be transformed into list or ndarry. This violates the feature constraints of the map function.

I do appreciate the convenience of Datasets package, but I do not think the compulsory datatype constrain is necessary, in some cases, we just cannot transform it into a list or ndarray due to some reasons.  Any way to fix this? Or what I can do to disable the compulsory datatype constrain?


[WIP][Common Voice 7] Add common voice 7.0

Hi ! I think the name `common_voice_7` is fine :)
Moreover if the dataset_infos.json is missing I'm pretty sure you don't need to specify `ignore_verifications=True`

Hi, how about to add a new parameter "version" in the function load_dataset, something like:
`load_dataset("common_voice", "lg", version="7.0") `
This is to avoid creating a new common_voice_? dataset (with almost the same code) every time
Mozilla updates their Common Voice dataset.

This PR allows to load the new common voice dataset manually as explained when doing:

```python
from datasets import load_dataset

ds = load_dataset("./datasets/datasets/common_voice_7", "ab")
```

=>

```
                   Please follow the manual download instructions:

                     You need to manually the dataset from `https://commonvoice.mozilla.org/en/datasets`.
                     Make sure you choose the version `Common Voice Corpus 7.0`.
                     Choose a language of your choice and find the corresponding language-id, *e.g.*, `Abkhaz` with language-id `ab`. The following language-ids are available:

                     ['ab', 'ar', 'as', 'az', 'ba', 'bas', 'be', 'bg', 'br', 'ca', 'cnh', 'cs', 'cv', 'cy', 'de', 'dv', 'el', 'en', 'eo', 'es', 'et', 'eu', 'fa', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gl', 'gn', 'ha', 'hi', 'hsb', 'hu', 'hy-AM', 'ia', 'id', 'it', 'ja', 'ka', 'kab', 'kk', 'kmr', 'ky', 'lg', 'lt', 'lv', 'mn', 'mt', 'nl', 'or', 'pa-IN', 'pl', 'pt', 'rm-sursilv', 'rm-vallader', 'ro', 'ru', 'rw', 'sah', 'sk', 'sl', 'sr', 'sv-SE', 'ta', 'th', 'tr', 'tt', 'ug', 'uk', 'ur', 'uz', 'vi', 'vot', 'zh-CN', 'zh-HK', 'zh-TW']

                     Next, you will have to enter your email address to download the dataset in the `tar.gz` format. Save the file under <path-to-file>.
                     The file should then be extracted with: ``tar -xvzf <path-to-file>`` which will extract a folder called ``cv-corpus-7.0-2021-07-21``.
                     The dataset can then be loaded with `datasets.load_dataset("common_voice", <language-id>, data_dir="<path-to-'cv-corpus-7.0-2021-07-21'-folder>", ignore_verifications=True).
```

Having followed those instructions one can then download the data as follows:

```python
from datasets import load_dataset

ds = load_dataset("./datasets/datasets/common_voice_7", "ab", data_dir="./cv-corpus-7.0-2021-07-21/", ignore_verifications=True)
```

## TODO
- [ ] Discuss naming. Is the name ok here "common_voice_7"? The dataset script differs only really in one point from `common_voice.py` in that all the metadata is different (more hours etc...) and that it has to use manual data dir for now
- [ ] Ideally we should get a bundled download link. For `common_voice.py` there is a bundled download link: `https://voice-prod-bundler-ee1969a6ce8178826482b88e843c335139bd3fb4.s3.amazonaws.com/cv-corpus-6.1-2020-12-11/{}.tar.gz` that allows one to directly download the data. However such a link is missing for Common Voice 7. I guess we should try to contact common voice about it and ask whether we could host the data or help otherwise somehow. See: https://github.com/common-voice/common-voice-bundler/issues/15 cc @yjernite
- [ ] I did not compute the dataset.json and it would mean that I'd have to download 76 datasets totalling around 1TB manually before running the checksum command. This just takes too much time. For now the user will have to add a `ignore_verifications=True` to download the data. This step would also be much easier if we could get a bundled link
- [ ] Add dummy data


Add support for fast tokenizer in BertScore

This PR adds support for a fast tokenizer in BertScore, which has been added recently to the lib.
Fixes #2765


Allow PyArrow from source

When installing pyarrow from source the version is:

```python
>>> import pyarrow; pyarrow.__version__
'2.1.0.dev612'
```

-> however this breaks the install check at init of `datasets`. This PR makes sure that everything coming after the last `'.'` is removed.


`ArrowInvalid: Added column's length must match table's length.` after using `select`

bug

Something isn't working

Hi,

the `select` method creates an indices mapping and doesn't modify the underlying PyArrow table by default for better performance. To modify the underlying table after the `select` call, call `flatten_indices` on the dataset object as follows:
```python
from datasets import load_dataset

ds = load_dataset("tweets_hate_speech_detection")['train'].select(range(128))
ds = ds.flatten_indices()
ds = ds.add_column('ones', [1]*128)
```

Thanks for the question @lvwerra. And thanks for the answer @mariosasko. ^^

## Describe the bug
I would like to add a column to a downsampled dataset. However I get an error message saying the length don't match with the length of the unsampled dataset indicated. I suspect that the dataset size is not updated when calling `select`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

ds = load_dataset("tweets_hate_speech_detection")['train'].select(range(128))
ds = ds.add_column('ones', [1]*128)
```

## Expected results
I would expect a new column named `ones` filled with `1`. When I check the length of `ds` it says `128`. Interestingly, it works when calling `ds = ds.map(lambda x: x)` before adding the column.

## Actual results
Specify the actual results or traceback.
```python
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
/var/folders/l4/2905jygx4tx5jv8_kn03vxsw0000gn/T/ipykernel_6301/868709636.py in <module>
      1 from datasets import load_dataset
      2 ds = load_dataset("tweets_hate_speech_detection")['train'].select(range(128))
----> 3 ds = ds.add_column('ones', [0]*128)

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    183         }
    184         # apply actual function
--> 185         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    186         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    187         # re-apply format to the output

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/arrow_dataset.py in add_column(self, name, column, new_fingerprint)
   2965         column_table = InMemoryTable.from_pydict({name: column})
   2966         # Concatenate tables horizontally
-> 2967         table = ConcatenationTable.from_tables([self._data, column_table], axis=1)
   2968         # Update features
   2969         info = self.info.copy()

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in from_tables(cls, tables, axis)
    715             table_blocks = to_blocks(table)
    716             blocks = _extend_blocks(blocks, table_blocks, axis=axis)
--> 717         return cls.from_blocks(blocks)
    718
    719     @property

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in from_blocks(cls, blocks)
    663             return cls(table, blocks)
    664         else:
--> 665             table = cls._concat_blocks_horizontally_and_vertically(blocks)
    666             return cls(table, blocks)
    667

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in _concat_blocks_horizontally_and_vertically(cls, blocks)
    623             if not tables:
    624                 continue
--> 625             pa_table_horizontally_concatenated = cls._concat_blocks(tables, axis=1)
    626             pa_tables_to_concat_vertically.append(pa_table_horizontally_concatenated)
    627         return cls._concat_blocks(pa_tables_to_concat_vertically, axis=0)

~/git/semantic-clustering/env/lib/python3.8/site-packages/datasets/table.py in _concat_blocks(blocks, axis)
    612                 else:
    613                     for name, col in zip(table.column_names, table.columns):
--> 614                         pa_table = pa_table.append_column(name, col)
    615             return pa_table
    616         else:

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.append_column()

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.add_column()

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/git/semantic-clustering/env/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Added column's length must match table's length. Expected length 31962 but got length 128
```

## Environment info
- `datasets` version: 1.11.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 5.0.0


equal operation to perform unbatch for huggingface datasets

Something isn't working

Hi @lhoestq
Maybe this is clearer to explain like this, currently map function, map one example to "one" modified one, lets assume we want to map one example to "multiple" examples, in which we do not know in advance how many examples they would be per each entry. I greatly appreciate telling me how I can handle this operation, thanks a lot

Hi,
this is also my question on how to perform similar operation as "unbatch" in tensorflow in great huggingface dataset library.
thanks.

Hi,

`Dataset.map` in the batched mode allows you to map a single row to multiple rows. So to perform "unbatch", you can do the following:
```python
import collections

def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {ex['passage']}"
        new_batch["inputs"].extend([inputs] * len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"])
    return new_batch

dset = dset.map(unbatch, batched=True, remove_columns=dset.column_names)
```

Dear @mariosasko
First, thank you very much for coming back to me on this, I appreciate it a lot. I tried this solution,  I am getting errors, do you mind
giving me one test example to be able to run your code, to understand better the format of the inputs to your function?
in this function https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L952 they copy each example to the number of "answers", do you mean one should not do the copying part and use directly your function?

thank you very much for your help and time.

Hi @mariosasko
I think finally I got this, I think you mean to do things in one step, here is the full example for completeness:

```
def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        # updates the passage.
        passage = ex['passage']
        passage = re.sub(r'(\.|\?|\!|\"|\')\n@highlight\n', r'\1 ', passage)
        passage = re.sub(r'\n@highlight\n', '. ', passage)
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {passage}"
        # duplicates the samples based on  number of answers.
        num_answers = len(ex["answers"])
        num_duplicates = np.maximum(1, num_answers)
        new_batch["inputs"].extend([inputs] * num_duplicates) #len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"] if num_answers > 0 else ["<unk>"])
    return new_batch

data = datasets.load_dataset('super_glue', 'record', split="train", script_version="master")
data = data.map(unbatch, batched=True, remove_columns=data.column_names)
```

Thanks a lot again, this was a super great way to do it.

Hi
I need to use "unbatch" operation in tensorflow on a huggingface dataset, I could not find this operation, could you kindly direct me how I can do it, here is the problem I am trying to solve:

I am considering "record" dataset in SuperGlue and I need to replicate each entery of the dataset for each answer, to make it similar to what T5 originally did:

https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L925

Here please find an example:

  For example, a typical example from ReCoRD might look like
  {
      'passsage': 'This is the passage.',
      'query': 'A @placeholder is a bird.',
      'entities': ['penguin', 'potato', 'pigeon'],
      'answers': ['penguin', 'pigeon'],
  }
  and I need a prosessor which would turn this example into the following two examples:
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'penguin',
  }
  and
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'pigeon',
  }

For doing this, one need unbatch, as each entry can map to multiple samples depending on the number of answers, I am not sure how to perform this operation with  huggingface datasets library and greatly appreciate your help

@lhoestq

Thank you very much.


fix typo (ShuffingConfig -> ShufflingConfig)

pretty straightforward, it should be Shuffling instead of Shuffing


BERTScore Error

Something isn't working

Hi,

The `use_fast_tokenizer` argument has been recently added to the bert-score lib. I've opened a PR with the fix. In the meantime, you can try to downgrade the version of bert-score with the following command to make the code work:
```
pip uninstall bert-score
pip install "bert-score<0.3.10"
```

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
predictions = ["hello there", "general kenobi"]
references = ["hello there", "general kenobi"]
bert = load_metric('bertscore')
bert.compute(predictions=predictions, references=references,lang='en')
```

# Bug
`TypeError: get_hash() missing 1 required positional argument: 'use_fast_tokenizer'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Colab
- Python version:
- PyArrow version:


Add DER metric for SUPERB speaker diarization task


English wikipedia datasets is not clean

Something isn't working

Hi ! Certain users might need these data (for training or simply to explore/index the dataset).

Feel free to implement a map function that gets rid of these paragraphs and process the wikipedia dataset with it before training

## Describe the bug
Wikipedia english dumps contain many wikipedia paragraphs like "References", "Category:" and "See Also" that should not be used for training.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset
w = load_dataset('wikipedia', '20200501.en')
print(w['train'][0]['text'])
```

> 'Yangliuqing () is a market town in Xiqing District, in the western suburbs of Tianjin, People\'s Republic of China. Despite its relatively small size, it has been named since 2006 in the "famous historical and cultural market towns in China".\n\nIt is best known in China for creating nianhua or Yangliuqing nianhua. For more than 400 years, Yangliuqing has in effect specialised in the creation of these woodcuts for the New Year.  wood block prints using vivid colourschemes to portray traditional scenes of children\'s games often interwoven with auspiciouse objects.\n\n, it had 27 residential communities () and 25 villages under its administration.\n\nShi Family Grand Courtyard\n\nShi Family Grand Courtyard (Tiānjīn Shí Jiā Dà Yuàn, 天津石家大院) is situated in Yangliuqing Town of Xiqing District, which is the former residence of wealthy merchant Shi Yuanshi - the 4th son of Shi Wancheng, one of the eight great masters in Tianjin. First built in 1875, it covers over 6,000 square meters, including large and small yards and over 200 folk houses, a theater and over 275 rooms that served as apartments and places of business and worship for this powerful family. Shifu Garden, which finished its expansion in October 2003, covers 1,200 square meters, incorporates the elegance of imperial garden and delicacy of south garden. Now the courtyard of Shi family covers about 10,000 square meters, which is called the first mansion in North China. Now it serves as the folk custom museum in Yangliuqing, which has a large collection of folk custom museum in Yanliuqing, which has a large collection of folk art pieces like Yanliuqing New Year pictures, brick sculpture.\n\nShi\'s ancestor came from Dong\'e County in Shandong Province, engaged in water transport of grain. As the wealth gradually accumulated, the Shi Family moved to Yangliuqing and bought large tracts of land and set up their residence. Shi Yuanshi came from the fourth generation of the family, who was a successful businessman and a good household manager, and the residence was thus enlarged for several times until it acquired the present scale. It is believed to be the first mansion in the west of Tianjin.\n\nThe residence is symmetric based on the axis formed by a passageway in the middle, on which there are four archways. On the east side of the courtyard, there are traditional single-story houses with rows of rooms around the four sides, which was once the living area for the Shi Family. The rooms on north side were the accountants\' office. On the west are the major constructions including the family hall for worshipping Buddha, theater and the south reception room. On both sides of the residence are side yard rooms for maids and servants.\n\nToday, the Shi mansion, located in the township of Yangliuqing to the west of central Tianjin, stands as a surprisingly well-preserved monument to China\'s pre-revolution mercantile spirit. It also serves as an on-location shoot for many of China\'s popular historical dramas. Many of the rooms feature period furniture, paintings and calligraphy, and the extensive Shifu Garden.\n\nPart of the complex has been turned into the Yangliuqing Museum, which includes displays focused on symbolic aspects of the courtyards\'  construction, local folk art and customs, and traditional period furnishings and crafts.\n\n**See also \n\nList of township-level divisions of Tianjin\n\nReferences \n\n http://arts.cultural-china.com/en/65Arts4795.html\n\nCategory:Towns in Tianjin'**

## Expected results
I expect no junk in the data.

## Actual results
Specify the actual results or traceback.

## Environment info
- `datasets` version: 1.10.2
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 3.0.0


Add RVL-CDIP dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

[labels_only.tar.gz](https://docs.google.com/uc?authuser=0&id=0B0NKIRwUL9KYcXo3bV9LU0t3SGs&export=download) on the RVL-CDIP website does not work for me.

> 404. That’s an error. The requested URL was not found on this server.

I contacted the author ( Adam Harley) regarding this, and he told me that the link works for him. Not sure what the issue is. But Adam shared the file (labels_only.tar.gz) with me as an attachment.

Are we allowed to host this file(labels_only.tar.gz) elsewhere and use that link instead ?

Thank you.

## Adding a Dataset
- **Name:** RVL-CDIP
- **Description:** The RVL-CDIP (Ryerson Vision Lab Complex Document Information Processing) dataset consists of 400,000 grayscale images in 16 classes, with 25,000 images per class. There are 320,000 training images, 40,000 validation images, and 40,000 test images. The images are sized so their largest dimension does not exceed 1000 pixels.
- **Paper:** https://www.cs.cmu.edu/~aharley/icdar15/
- **Data:** https://www.cs.cmu.edu/~aharley/rvl-cdip/
- **Motivation:** I'm currently adding LayoutLMv2 and LayoutXLM to HuggingFace Transformers. LayoutLM (v1) already exists in the library. This dataset has a large value for document image classification (i.e. classifying scanned documents). LayoutLM models obtain SOTA on this dataset, so would be great to directly use it in notebooks.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Error loading C4 realnewslike dataset

bug

Something isn't working

Hi @danshirron,
`c4` was updated few days back by @lhoestq. The new configs are `['en', 'en.noclean', 'en.realnewslike', 'en.webtextlike'].` You'll need to remove any older version of this dataset you previously downloaded and then run `load_dataset` again with new configuration.

@bhavitvyamalik @lhoestq , just tried the above and got:
>>> a=datasets.load_dataset('c4','en.realnewslike')
Downloading: 3.29kB [00:00, 1.66MB/s]
Downloading: 2.40MB [00:00, 12.6MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 819, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 701, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 1049, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 268, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 360, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig en.realnewslike not found. Available: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']
>>>

datasets version is 1.11.0

I think I had an older version of datasets installed and that's why I commented the old configurations in my last comment, my bad! I re-checked and updated it to latest version (`datasets==1.11.0`) and it's showing `available configs: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']`.

I tried `raw_datasets = load_dataset('c4', 'realnewslike')` and the download started. Make sure you don't have any old copy of this dataset and you download it fresh using the latest version of datasets. Sorry for the mix up!

It works. I probably had some issue with the cache. after cleaning it im able to download the dataset. Thanks

## Describe the bug
Error loading C4 realnewslike dataset. Validation part mismatch

## Steps to reproduce the bug
```python
 raw_datasets = load_dataset('c4', 'realnewslike', cache_dir=model_args.cache_dir)
## Expected results
success on data loading
## Actual results
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15.3M/15.3M [00:00<00:00, 28.1MB/s]Traceback (most recent call last):
  File "run_mlm_tf.py", line 794, in <module>
    main()
  File "run_mlm_tf.py", line 425, in main
    raw_datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir)                                                                                File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 843, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 608, in download_and_prepare
    self._download_and_prepare(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 698, in _download_and_prepare                                                                                verify_splits(self.info.splits, split_dict)                                                                                File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=38165657946, num_examples=13799838, dataset_name='c4'), 'recorded': SplitInfo(name='validation', num_bytes=37875873, num_examples=13863, dataset_name='c4')}]

## Environment info
- `datasets` version: 1.10.2
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1


Add Nuswide dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** *NUSWIDE*
- **Description:** *[A Real-World Web Image Dataset from National University of Singapore](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html)*
- **Paper:** *[here](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/nuswide-civr2009.pdf)*
- **Data:** *[here](https://github.com/wenting-zhao/nuswide)*
- **Motivation:** *This dataset is a benchmark in the Text Retrieval task.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Raise ManualDownloadError when loading a dataset that requires previous manual download

This PR implements the raising of a `ManualDownloadError` when loading a dataset that requires previous manual download, and this is missing.

The `ManualDownloadError` is raised whether the dataset is loaded in normal or streaming mode.

Close #2749.

cc: @severo


Unexpected type after `concatenate_datasets`

bug

Something isn't working

Hi @JulesBelveze, thanks for your question.

Note that 🤗  `datasets` internally store their data in Apache Arrow format.

However, when accessing dataset columns, by default they are returned as native Python objects (lists in this case).

If you would like their columns to be returned in a more suitable format for your use case (torch arrays), you can use the method `set_format()`:
```python
concat_dataset.set_format(type="torch")
```

You have detailed information in our docs:
- [Using a Dataset with PyTorch/Tensorflow](https://huggingface.co/docs/datasets/torch_tensorflow.html)
- [Dataset.set_format()](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.set_format)

Thanks @albertvillanova it indeed did the job 😃
Thanks for your answer!

## Describe the bug
I am trying to concatenate two `Dataset` using `concatenate_datasets` but it turns out that after concatenation the features are casted from `torch.Tensor` to `list`.
It then leads to a weird tensors when trying to convert it to a `DataLoader`. However, if I use each `Dataset` separately everything behave as expected.

## Steps to reproduce the bug
```python
>>> featurized_teacher
Dataset({
    features: ['t_labels', 't_input_ids', 't_token_type_ids', 't_attention_mask'],
    num_rows: 502
})
>>> for f in featurized_teacher.features:
     print(featurized_teacher[f].shape)
torch.Size([502])
torch.Size([502, 300])
torch.Size([502, 300])
torch.Size([502, 300])

>>> featurized_student
Dataset({
    features: ['s_features', 's_labels'],
    num_rows: 502
})
>>> for f in featurized_student.features:
     print(featurized_student[f].shape)
torch.Size([502, 64])
torch.Size([502])
```
The shapes seem alright to me. Then the results after concatenation are as follow:
```python
>>> concat_dataset = datasets.concatenate_datasets([featurized_student, featurized_teacher], axis=1)
>>> type(concat_dataset["t_labels"])
<class 'list'>
```
One would expect to obtain the same type as the one before concatenation.

Am I doing something wrong here? Any idea on how to fix this unexpected behavior?

## Environment info
- `datasets` version: 1.9.0
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.9.5
- PyArrow version: 3.0.0


Fix metadata JSON for ubuntu_dialogs_corpus dataset

Related to #2743.


Fix metadata JSON for turkish_movie_sentiment dataset

Related to #2743.


Generate metadata JSON for telugu_books dataset

Related to #2743.


Generate metadata JSON for reclor dataset

Related to #2743.


Generate metadata JSON for lm1b dataset

Related to #2743.


Update metadata for wikihow dataset

Update metadata for wikihow dataset:
- Remove leading new line character in description and citation
- Update metadata JSON
- Remove no longer necessary `urls_checksums/checksums.txt` file

Related to #2748.


Second concatenation of datasets produces errors

Something isn't working

Hi @Aktsvigun, thanks for reporting.

I'm investigating this.

Hi @albertvillanova ,
any update on this? Can I probably help in some way?

Hi @Aktsvigun! We are planning to address this issue before our next release, in a couple of weeks at most. 😅

In the meantime, if you would like to contribute, feel free to open a Pull Request. You are welcome. Here you can find more information: [How to contribute to Datasets?](CONTRIBUTING.md)

I can't reproduce the bug on master. I believe this issue was fixed by https://github.com/huggingface/datasets/pull/3551.

Hi,

I am need to concatenate my dataset with others several times, and after I concatenate it for the second time, the features of features (e.g. tags names) are collapsed. This hinders, for instance, the usage of tokenize function with `data.map`.

```
from datasets import load_dataset, concatenate_datasets

data = load_dataset('trec')['train']
concatenated = concatenate_datasets([data, data])
concatenated_2 = concatenate_datasets([concatenated, concatenated])
print('True features of features:', concatenated.features)
print('\nProduced features of features:', concatenated_2.features)
```
outputs

```
True features of features: {'label-coarse': ClassLabel(num_classes=6, names=['DESC', 'ENTY', 'ABBR', 'HUM', 'NUM', 'LOC'], names_file=None, id=None), 'label-fine': ClassLabel(num_classes=47, names=['manner', 'cremat', 'animal', 'exp', 'ind', 'gr', 'title', 'def', 'date', 'reason', 'event', 'state', 'desc', 'count', 'other', 'letter', 'religion', 'food', 'country', 'color', 'termeq', 'city', 'body', 'dismed', 'mount', 'money', 'product', 'period', 'substance', 'sport', 'plant', 'techmeth', 'volsize', 'instru', 'abb', 'speed', 'word', 'lang', 'perc', 'code', 'dist', 'temp', 'symbol', 'ord', 'veh', 'weight', 'currency'], names_file=None, id=None), 'text': Value(dtype='string', id=None)}

Produced features of features: {'label-coarse': Value(dtype='int64', id=None), 'label-fine': Value(dtype='int64', id=None), 'text': Value(dtype='string', id=None)}
```

I am using `datasets` v.1.11.0


Raise a proper exception when trying to stream a dataset that requires to manually download files

Something isn't working

Hi @severo, thanks for reporting.

As discussed, datasets requiring manual download should be:
- programmatically identifiable
- properly handled with more clear error message when trying to load them with streaming

In relation with programmatically identifiability, note that for datasets requiring manual download, their builder have a property `manual_download_instructions` which is not None:
```python
# Dataset requiring manual download:
builder.manual_download_instructions is not None
```

## Describe the bug

At least for 'reclor', 'telugu_books', 'turkish_movie_sentiment', 'ubuntu_dialogs_corpus', 'wikihow', trying to `load_dataset` in streaming mode raises a `TypeError` without any detail about why it fails.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("reclor", streaming=True)
```

## Expected results

Ideally: raise a specific exception, something like `ManualDownloadError`.

Or at least give the reason in the message, as when we load in normal mode:

```python
from datasets import load_dataset
dataset = load_dataset("reclor")
```

```
AssertionError: The dataset reclor with config default requires manual data.
 Please follow the manual download instructions:   to use ReClor you need to download it manually. Please go to its homepage (http://whyu.me/reclor/) fill the google
  form and you will receive a download link and a password to extract it.Please extract all files in one folder and use the path folder in datasets.load_dataset('reclor', data_dir='path/to/folder/folder_name')
  .
 Manual data can be loaded with `datasets.load_dataset(reclor, data_dir='<path/to/manual/data>')
```

## Actual results

```
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

## Environment info

- `datasets` version: 1.11.0
- Platform: macOS-11.5-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


Generate metadata JSON for wikihow dataset

Related to #2743.


add multi-proc in `to_json`

Thank you for working on this, @bhavitvyamalik

10% is not solving the issue, we want 5-10x faster on a machine that has lots of resources, but limited processing time.

So let's benchmark it on an instance with many more cores, I can test with 12 on my dev box and 40 on JZ.

Could you please share the test I could run with both versions?

Should we also test the sharded version I shared in https://github.com/huggingface/datasets/issues/2663#issue-946552273 so optionally 3 versions to test.

Since I was facing `OSError: [Errno 12] Cannot allocate memory` in CircleCI tests, I've added `num_proc` option instead of always using full `cpu_count`. You can test both v1 and v2 through this branch (some redundancy needs to be removed).

Update: I was able to convert into json which took 50% less time as compared to v1 on `ascent_kb` dataset. Will post the benchmarking script with results here.

Here are the benchmarks with the current branch for both v1 and v2 (dataset: `ascent_kb`, 8.9M samples):
| batch_size | time (in sec) | time (in sec) |
|------------|---------------|---------------|
|            | num_proc = 1  | num_proc = 4  |
| 10k        | 185.56        | 170.11        |
| 50k        | 175.79        | 86.84         |
| **100k**       | 191.09        | **78.35**         |
| 125k       | 198.28        | 90.89         |

Increasing the batch size on my machine helped in making v2 around 50% faster as compared to v1. Timings may vary depending on the machine. I'm including the benchmarking script as well. CircleCI errors are unrelated (something related to `bertscore`)
```
import time
from datasets import load_dataset
import pathlib
import os
from pathlib import Path
import shutil
import gc

batch_sizes = [10_000, 50_000, 100_000, 125_000]
num_procs = [1, 4]  # change this according to your machine

SAVE_LOC = "./new_dataset.json"

for batch in batch_sizes:
    for num in num_procs:
        dataset = load_dataset("ascent_kb")

        local_start = time.time()
        ans = dataset['train'].to_json(SAVE_LOC, batch_size=batch, num_proc=num)
        local_end = time.time() - local_start

        print(f"Time taken on {num} num_proc and {batch} batch_size: ", local_end)

        # remove that dataset and its contents from cache and newly generated json
        new_json = pathlib.Path(SAVE_LOC)
        new_json.unlink()

        try:
            shutil.rmtree(os.path.join(str(Path.home()), ".cache", "huggingface"))
        except OSError as e:
            print("Error: %s - %s." % (e.filename, e.strerror))

        gc.collect()
```
This will download the dataset in every iteration and run `to_json`. I didn't do multiple iterations here for `to_json` (for a specific batch_size and num_proc) and took average time as I found that v1 got faster after 1st iteration (maybe it's caching somewhere). Since you'll be doing this operation only once, I thought it'll be better to report how both v1 and v2 performed in single iteration only.

Important: Benchmarking script will delete the newly generated json and `~/.cache/huggingface/` after every iteration so that it doesn't end up using any cached data (just to be on a safe side)

Thank you for sharing the benchmark, @bhavitvyamalik. Your results look promising.

But if I remember correctly the sharded version at https://github.com/huggingface/datasets/issues/2663#issue-946552273 was much faster. So we probably should compare to it as well? And if it's faster than at least document that manual sharding version?

-------

That's a dangerous benchmark as it'd wipe out many other HF things. Why not wipe out:
```
~/.cache/huggingface/datasets/ascent_kb/
```

Running the benchmark now.

Weird, I tried to adapt your benchmark to using shards and the program no longer works. It instead quickly uses up all available RAM and hangs. Has something changed recently in `datasets`? You can try:

```
import time
from datasets import load_dataset
import pathlib
import os
from pathlib import Path
import shutil
import gc
from multiprocessing import cpu_count, Process, Queue

batch_sizes = [10_000, 50_000, 100_000, 125_000]
num_procs = [1, 8]  # change this according to your machine

DATASET_NAME = ("ascent_kb")
num_shards = [1, 8]
for batch in batch_sizes:
    for shards in num_shards:
        dataset = load_dataset(DATASET_NAME)["train"]
        #print(dataset)

        def process_shard(idx):
            print(f"Sharding {idx}")
            ds_shard = dataset.shard(shards, idx, contiguous=True)
            # ds_shard = ds_shard.shuffle() # remove contiguous=True above if shuffling
            print(f"Saving {DATASET_NAME}-{idx}.jsonl")
            ds_shard.to_json(f"{DATASET_NAME}-{idx}.jsonl", orient="records", lines=True, force_ascii=False)

        local_start = time.time()
        queue = Queue()
        processes = [Process(target=process_shard, args=(idx,)) for idx in range(shards)]
        for p in processes:
            p.start()

        for p in processes:
            p.join()
        local_end = time.time() - local_start

        print(f"Time taken on {shards} shards and {batch} batch_size: ", local_end)
```

Just careful, so that it won't crash your compute environment. As it almost crashed mine.

So this part seems to no longer work:
```
        dataset = load_dataset("ascent_kb")["train"]
        ds_shard = dataset.shard(1, 0, contiguous=True)
        ds_shard.to_json("ascent_kb-0.jsonl", orient="records", lines=True, force_ascii=False)
```

If you are using `to_json` without any `num_proc`or `num_proc=1` then essentially it'll fall back to v1 only and I've kept it as it is (the tests were passing as well)

> That's a dangerous benchmark as it'd wipe out many other HF things. Why not wipe out:

That's because some dataset related files were still left inside `~/.cache/huggingface/datasets` folder. You can wipe off datasets folder inside your cache maybe

> dataset = load_dataset("ascent_kb")["train"]
> ds_shard = dataset.shard(1, 0, contiguous=True)
> ds_shard.to_json("ascent_kb-0.jsonl", orient="records", lines=True, force_ascii=False)

I tried this `lama` dataset (1.3M) and it worked fine. Trying it with `ascent_kb` currently, will update it here.

I don't think the issue has anything to do with your work, @bhavitvyamalik. I forgot to mention I tested to see the same problem with the latest datasets release.

Interesting, I tried your suggestion. This:
```
python -c 'import datasets; ds="lama"; dataset = datasets.load_dataset(ds)["train"]; \
dataset.shard(1, 0, contiguous=True).to_json(f"{ds}-0.jsonl", orient="records", lines=True, force_ascii=False)'
```
works fine and takes just a few GBs to complete.

this on the other hand blows up memory-wise:
```
python -c 'import datasets; ds="ascent_kb"; dataset = datasets.load_dataset(ds)["train"]; \
dataset.shard(1, 0, contiguous=True).to_json(f"{ds}-0.jsonl", orient="records", lines=True, force_ascii=False)'
```
and I have to kill it before it uses up all RAM. (I have 128GB of it, so it should be more than enough)

> That's because some dataset related files were still left inside ~/.cache/huggingface/datasets folder. You can wipe off datasets folder inside your cache maybe

I think recent datasets added a method that will print out the path for all the different components for a given dataset, I can't recall the name though. It was when we were discussing a janitor program to clear up space selectively.

> and I have to kill it before it uses up all RAM. (I have 128GB of it, so it should be more than enough)

Same thing just happened on my machine too. Memory leak somewhere maybe? Even if you were to load this dataset in your memory it shouldn't take more than 4GB. You were earlier doing this for `oscar` dataset. Is it working fine for that?

Hmm, looks like `datasets` has changed and won't accept my currently cached oscar-en (crashes), so I'd rather not download 0.5TB again.

Were you able to reproduce the memory blow up with `ascent_kb`? It's should be a much quicker task to verify.

But yes, oscar worked just fine with `.shard()` which is what I used to process it fast.

What I tried is:
```
HF_DATASETS_OFFLINE=1 HF_DATASETS_CACHE=cache python -c 'import datasets; ds="oscar"; \
dataset = datasets.load_dataset(ds, "unshuffled_deduplicated_en")["train"]; \
dataset.shard(1000000, 0, contiguous=True).to_json(f"{ds}-0.jsonl", orient="records", lines=True, force_ascii=False)'
```
and got:
```
Using the latest cached version of the module from /gpfswork/rech/six/commun/modules/datasets_modules/datasets/oscar/e4f06cecc7ae02f7adf85640b4019bf476d44453f251a1d84aebae28b0f8d51d (last modified on Fri Aug  6 01:52:35 2021) since it couldn't be found locally at oscar/oscar.py or remotely (OfflineModeIsEnabled).
Reusing dataset oscar (cache/oscar/unshuffled_deduplicated_en/1.0.0/e4f06cecc7ae02f7adf85640b4019bf476d44453f251a1d84aebae28b0f8d51d)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/load.py", line 755, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 737, in as_dataset
    datasets = utils.map_nested(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 203, in map_nested
    mapped = [
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 764, in _build_single_dataset
    ds = self._as_dataset(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 834, in _as_dataset
    dataset_kwargs = ArrowReader(self._cache_dir, self.info).read(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 217, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 238, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 173, in _read_files
    pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 308, in _get_table_from_filename
    table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 327, in read_table
    return table_cls.from_file(filename)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/table.py", line 450, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/table.py", line 43, in _memory_mapped_arrow_table_from_file
    memory_mapped_stream = pa.memory_map(filename)
  File "pyarrow/io.pxi", line 782, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 743, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```

> Were you able to reproduce the memory blow up with ascent_kb? It's should be a much quicker task to verify.

Yes, this blows up memory-wise on my machine too.

I found that a [similar error](https://discuss.huggingface.co/t/saving-memory-with-run-mlm-py-with-wikipedia-datasets/4160) was posted on the forum on 5th March. Since you already knew how much time [#2663 comment](https://github.com/huggingface/datasets/issues/2663#issue-946552273) took, can you try benchmarking v1 and v2 for now maybe until we have a fix for this memory blow up?

OK, so I benchmarked using "lama" though it's too small for this kind of test, since the sharding is much slower than one thread here.

Results: https://gist.github.com/stas00/dc1597a1e245c5915cfeefa0eee6902c

So sharding does really bad there, and your json over procs is doing great!

Any suggestions to a somewhat bigger dataset, but not too big? say 10 times of lama?

Looks great! I had a few questions/suggestions related to `benchmark-datasets-to_json.py`:

1. You have used only 10_000 and 100_000 batch size. Including more batch sizes may help you find the perfect batch size for your machine and even give you some extra speed-up.
For eg, I found `load_dataset("cc100", lang="eu")` with batch size 125_000 took less time as compared to batch size 100_000 (71.16 sec v/s 67.26 sec) since this dataset has 2 fields only `['id', 'text']`, so that's why we can go for higher batch size here.

2. Why have you used `num_procs` 1 and 4 only?

You can use:
1. `dataset = load_dataset("cc100", lang="af")`. Even though it has only 2 fields but there are around 9.9 mil samples. (lama had around 1.3 mil samples)
2. `dataset = load_dataset("cc100", lang="eu")` -> 16 mil samples. (if you want something more than 9.9 mil)
3. `dataset = load_dataset("neural_code_search", 'search_corpus')` -> 4.7 mil samples

Thank you, @bhavitvyamalik

My apologies, at the moment I have not found time to do more benchmark with the proposed other datasets. I will try to do it later, but I don't want it to hold your PR, it's definitely a great improvement based on the benchmarks I did run! And the comparison to sharded is really just of interest to me to see if it's on par or slower.

So if other reviewers are happy, this definitely looks like a great improvement to me and addresses the request I made in the first place.

> Why have you used num_procs 1 and 4 only?

Oh, no particular reason, I was just comparing to 4 shards on my desktop. Typically it's sufficient to go from 1 to 2-4 to see whether the distributed approach is faster or not. Once hit larger numbers you often run into bottlenecks like IO, and then numbers can be less representative. I hope it makes sense.

Tested it with a larger dataset (`srwac`) and memory utilisation remained constant with no swap memory used. @lhoestq should I also add test for the same? Last time I tried this, I got `OSError: [Errno 12] Cannot allocate memory` in CircleCI tests

Closes #2663. I've tried adding multiprocessing in `to_json`. Here's some benchmarking I did to compare the timings of current version (say v1) and multi-proc version (say v2). I did this with `cpu_count` 4 (2015 Macbook Air)

1. Dataset name: `ascent_kb` - 8.9M samples (all samples were used, reporting this for a single run)
v1- ~225 seconds for converting whole dataset to json
v2- ~200 seconds for converting whole dataset to json

2. Dataset name: `lama` - 1.3M samples (all samples were used, reporting this for 2 runs)
v1- ~26 seconds for converting whole dataset to json
v2- ~23.6 seconds for converting whole dataset to json

I think it's safe to say that v2 is 10% faster as compared to v1. Timings may improve further with better configuration.

The only bottleneck I feel is writing to file from the output list. If we can improve that aspect then timings may improve further.

Let me know if any changes/improvements can be done in this @stas00, @lhoestq, @albertvillanova. @lhoestq even suggested to extend this work with other export methods as well like `csv` or `parquet`.


Cannot load `few-nerd` dataset

Something isn't working

Hi @Mehrad0711,

I'm afraid there is no "canonical" Hugging Face dataset named "few-nerd".

There are 2 kinds of datasets hosted at the Hugging Face Hub:
- canonical datasets (their identifier contains no slash "/"): we, the Hugging Face team, supervise their implementation and we make sure they work correctly by means of our test suite
- community datasets (their identifier contains a slash "/", where before the slash it is the username or the organization name): those datasets are uploaded to the Hub by the community, and we, the Hugging Face team, do not supervise them; it is the responsibility of the user/organization implementing them properly if they want them to be used by other users.

In this specific case, there is no "canonical" dataset named "few-nerd". On the other hand, there are two "community" datasets named "few-nerd":
- ["nbroad/few-nerd"](https://huggingface.co/datasets/nbroad/few-nerd)
- ["dfki-nlp/few-nerd"](https://huggingface.co/datasets/dfki-nlp/few-nerd)

If they were properly implemented, you should be able to load them this way:
```python
# "nbroad/few-nerd" community dataset
ds = load_dataset("nbroad/few-nerd", "supervised")

# "dfki-nlp/few-nerd" community dataset
ds = load_dataset("dfki-nlp/few-nerd", "supervised")
```

However, they are not correctly implemented and both of them give errors:
- "nbroad/few-nerd":
  ```
  TypeError: expected str, bytes or os.PathLike object, not dict
  ```
- "dfki-nlp/few-nerd":
  ```
  ConnectionError: Couldn't reach https://cloud.tsinghua.edu.cn/f/09265750ae6340429827/?dl=1
  ```

You could try to contact their users/organizations to inform them about their bugs and ask them if they are planning to fix them. Alternatively you could try to implement your own script for this dataset.

Thanks @albertvillanova for your detailed explanation! I will resort to my own scripts for now.

Hello, @Mehrad0711; Hi, @albertvillanova  !
I am the maintainer of the `dfki/few-nerd" dataset script, sorry for the very late reply and hope this message finds you well!
We should use
```
dataset = load_dataset("dfki-nlp/few-nerd", name="supervised")
```
instead of not specifying the "name" argument, where name is from `["supervised", "inter", "intra"]`. Otherwise the method just treats "supervised" as `split`, which we reserve after specifying the name, since for each name, there are three splits: train, dev and test.

Also we use Tsinghua server source to download data files since it is the official source referred in the paper where the dataset is released (even though it is cc-by-sa-4.0 licensed, means we can copy the data anywhere after mentioning the license
). Sometimes the server just runs down due to high pressure, kinda weird (we encountered the same server problem serveral times a month when we conducted experiments on Few-NERD XD). I tried the script just now and it works perfectly!
```
>> dataset
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 131767
    })
    validation: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 18824
    })
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 37648
    })
})
>>> dataset["train"]
Dataset({
    features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
    num_rows: 131767
})
>>> dataset["train"][0]
{'id': '0', 'tokens': ['Paul', 'International', 'airport', '.'], 'ner_tags': [0, 0, 0, 0], 'fine_ner_tags': [0, 0, 0, 0]}
```
Anyways if you cannot stand the pain with the server and its slow download speed, you can also download the `dfki/few-nerd.py` script from HF and change the `_URLs` to your personal drive (after you once successfully download the data and upload to your cloud drive), and then load the .py script locally.

Hope this reply can still be any help. If you still have problems with it, feel free to ask here and I am glad to help!
Best wishes.

Hi @chen-yuxuan, thanks for your answer.

Just a few comments:

- Please, note that as we use `datasets.load_dataset` implementation, we can pass the configuration name as the second positional argument (no need to pass explicitly `name=`) and it downloads the 3 splits:
```python
  In [4]: ds = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5k/11.5k [00:00<00:00, 2.85MB/s]
Downloading and preparing dataset few_nerd/supervised to .cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e40882b71f037a4a1f232025899170fbe8113cd2f4a26dddd2add7222a077255...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6M/14.6M [01:16<00:00, 190kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.9M/11.9M [01:14<00:00, 160kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.0M/12.0M [01:04<00:00, 186kB/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:58<00:00, 79.45s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.11it/s]
```

- On the other hand, please note that your script does not work on Windows machines, because you call `open()` without passing the encoding parameter:
```
~\.cache\huggingface\modules\datasets_modules\datasets\dfki-nlp___few-nerd\e40882b71f037a4a1f232025899170fbe8113cd2f4a26dddd2add7222a077255\few-nerd.py in <genexpr>(.0)
    276         assert filepath[-4:] == ".txt"
    277
--> 278         num_lines = sum(1 for _ in open(filepath))
    279         id = 0
    280

.venv\lib\encodings\cp1252.py in decode(self, input, final)
     21 class IncrementalDecoder(codecs.IncrementalDecoder):
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24
     25 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 5238: character maps to <undefined>
```

If you would like your script to be usable on Windows machines, you should pass `encoding="utf-8"` to every `open()` function:
- line 278: `num_lines = sum(1 for _ in open(filepath, encoding="utf-8"))`
- line 281: `with open(filepath, "r", encoding="utf-8")`

Thank you @albertvillanova for your detailed feedback!

> no need to pass explicitly `name=`

Good catch! I thought `split` stands before `name` in the argument list... but now it is all clear to me, sounds cool! Thanks for the explanation.

Anyways in our old code it still looks bit confusing if we only want one split but the function downloads all, so to allow efficient downloading, I optimized the code a bit so that only the specified split data is downloaded. now we get
```
>>> x = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading and preparing dataset few_nerd/supervised to /home/user/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/8e7ab598946cd5b395dcec6ea239123c8dff5b58b8e1c03b0c595b540248a885...
Downloading: 100%|███████████████████████████████████████████████████████████████████| 14.6M/14.6M [01:01<00:00, 238kB/s]
100%|██████████████████████████████████████████████████████████████████████| 3359329/3359329 [00:12<00:00, 275462.84it/s]
100%|████████████████████████████████████████████████████████████████████████| 482037/482037 [00:01<00:00, 278633.64it/s]
100%|████████████████████████████████████████████████████████████████████████| 958765/958765 [00:03<00:00, 267472.83it/s]
Dataset few_nerd downloaded and prepared to /home/user/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/8e7ab598946cd5b395dcec6ea239123c8dff5b58b8e1c03b0c595b540248a885. Subsequent calls will reuse this data.
```
where only one progress bar indicates downloading, and the three others just indicate pre-processing for the train, dev, test set.

For the encoding issue, I have made corresponding changes for the two lines you pointed out. However, I have no windows machine at hand, I would really appreciate it if you could help test on your end.

All the updates are uploaded to HF under `dfki-nlp` account where I am working for.
Thank you again for your kind help!

Hi @chen-yuxuan,

I have tested on Windows and now it works perfectly, after the fixing of the encoding issue:
```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5k/11.5k [00:00<?, ?B/s]
Downloading and preparing dataset few_nerd/supervised to C:\Users\username\.cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e1ceeaee82073fea12206e4461c7cfcd67e68c8f3ebeca179bddcacee00c4511...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3359329/3359329 [00:25<00:00, 129427.23it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 482037/482037 [00:03<00:00, 134513.66it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 958765/958765 [00:06<00:00, 143152.35it/s]
Dataset few_nerd downloaded and prepared to C:\Users\username\.cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e1ceeaee82073fea12206e4461c7cfcd67e68c8f3ebeca179bddcacee00c4511. Subsequent calls will reuse this data.765 [00:06<00:00, 139045.03it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 174.71it/s]

In [3]: ds
Out[3]:
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 131767
    })
    validation: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 18824
    })
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 37648
    })
})
```

## Describe the bug

Cannot load `few-nerd` dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset('few-nerd', 'supervised')
```

## Actual results

Executing above code will give the following error:

```
Using the latest cached version of the module from /Users/Mehrad/.cache/huggingface/modules/datasets_modules/datasets/few-nerd/62464ace912a40a0f33a11a8310f9041c9dc3590ff2b3c77c14d83ca53cfec53 (last modified on Wed Jun  2 11:34:25 2021) since it couldn't be found locally at /Users/Mehrad/Documents/GitHub/genienlp/few-nerd/few-nerd.py, or remotely (FileNotFoundError).
Downloading and preparing dataset few_nerd/supervised (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/Mehrad/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/62464ace912a40a0f33a11a8310f9041c9dc3590ff2b3c77c14d83ca53cfec53...
Traceback (most recent call last):
  File "/Users/Mehrad/opt/anaconda3/lib/python3.7/site-packages/datasets/builder.py", line 693, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/Mehrad/opt/anaconda3/lib/python3.7/site-packages/datasets/builder.py", line 1107, in _prepare_split
    disable=bool(logging.get_verbosity() == logging.NOTSET),
  File "/Users/Mehrad/opt/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/Users/Mehrad/.cache/huggingface/modules/datasets_modules/datasets/few-nerd/62464ace912a40a0f33a11a8310f9041c9dc3590ff2b3c77c14d83ca53cfec53/few-nerd.py", line 196, in _generate_examples
    with open(filepath, encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/Mehrad/.cache/huggingface/datasets/downloads/supervised/train.json'
```
The bug is probably in identifying and downloading the dataset. If I download the json splits directly from [link](https://github.com/nbroad1881/few-nerd/tree/main/uncompressed) and put them under the downloads directory, they will be processed into arrow format correctly.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Python version: 3.8
- PyArrow version: 1.0.1


added semeval18_emotion_classification dataset

For training the multilabel classifier, I would combine the labels into a list, for example for the English dataset:

```
dfpre=pd.read_csv(path+"2018-E-c-En-train.txt",sep="\t")
dfpre['list'] = dfpre[dfpre.columns[2:]].values.tolist()
df = dfpre[['Tweet', 'list']].copy()
df.rename(columns={'list': 'labels'}, inplace=True)
```

Hi @maxpel , have you had a chance to take my comments into account ?

Let me know if you have questions or if I can help :)

Hi @lhoestq ! I did take your comments into account, changed the naming and tried to add dummy data (manually). I am not sure if the dummy data is correct, maybe you can take a look at that.
The model card is still missing as I am currently very busy.

Thanks ! The dummy data looks all good, good job :)

The CI error can be fixed by merging `master` into your branch
```bash
git fetch upstream
git merge upstream/master
```

Hi! I just added the model card and I did the merge you showed above. Should I then add and commit again? The CI error is still there right now.

@lhoestq Unfortunately, I discovered a problem with the test data sets on the competion page (train and dev is fine). They still contain NONE labels for each of the emotions, for example for English: http://saifmohammad.com/WebDocs/AIT-2018/AIT2018-DATA/AIT2018-TEST-DATA/semeval2018englishtestfiles/2018-E-c-En-test.zip
Luckily, a zip file with all data of the competition contains the correct labels also for the test set:
http://saifmohammad.com/WebDocs/AIT-2018/AIT2018-DATA/SemEval2018-Task1-all-data.zip
What's the best way to correct this?

Hi ! I think we can edit the sem_eval_2018_task_1.py file to use this URL instead, and maybe update the `os.path.join` calls to the new paths to the text data in the new ZIP file. Would you like to try to make this work ?

I added the data set of SemEval 2018 Task 1 (Subtask 5) for emotion detection in three languages.

```
datasets-cli test datasets/semeval18_emotion_classification/ --save_infos --all_configs

RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_semeval18_emotion_classification
```
Both commands ran successfully.

I couldn't create the dummy data (the files are tsvs but have .txt ending, maybe that's the problem?) and therefore the test on the dummy data fails, maybe someone can help here.

I also formatted the code:
```
black --line-length 119 --target-version py36 datasets/semeval18_emotion_classification/
isort datasets/semeval18_emotion_classification/
flake8 datasets/semeval18_emotion_classification/
```
That's the publication for reference:

Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018). SemEval-2018 task 1: Affect in tweets. Proceedings of the 12th International Workshop on Semantic Evaluation, 1–17. https://doi.org/10.18653/v1/S18-1001


Fix key by recreating metadata JSON for journalists_questions dataset

Close #2743.