devopsmarc/my-issues-dataset huggingface dataset:
Is there a way to download only NQ dev? 
 Maybe I missed that in the docs, but is there a way to only download the dev set of natural questions (~1 GB)? 
As we want to benchmark QA models on different datasets, I would like to avoid downloading the 41GB of training data. 

I tried
```
dataset = nlp.load_dataset('natural_questions', split="validation", beam_runner="DirectRunner")
```
But this still triggered a big download of presumably the whole dataset. Is there any way of doing this or are splits / slicing options only available after downloading?

Thanks! 
 Ok, got it. I think this could be a valuable feature - especially for large datasets like NQ, but potentially also others. 
For us, it will in this case make the difference of using the library or keeping the old downloads of the raw dev datasets.  
However, I don't know if that fits into your plans with the library and can also understand if you don't want to support this.
Is there a way to download only NQ dev? 
 Maybe I missed that in the docs, but is there a way to only download the dev set of natural questions (~1 GB)? 
As we want to benchmark QA models on different datasets, I would like to avoid downloading the 41GB of training data. 

I tried
```
dataset = nlp.load_dataset('natural_questions', split="validation", beam_runner="DirectRunner")
```
But this still triggered a big download of presumably the whole dataset. Is there any way of doing this or are splits / slicing options only available after downloading?

Thanks! 
 I don't think we could force this behavior generally since the dataset script authors are free to organize the file download as they want (sometimes the mapping between split and files can be very much nontrivial) but we can add an additional configuration for Natural Question indeed as @lhoestq indicate.
MissingBeamOptions for Wikipedia 20200501.en 
 There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
``` 
 Fixed. Could you try again @mitchellgordon95 ?
It was due a file not being updated on S3.

We need to make sure all the datasets scripts get updated properly @julien-c
MissingBeamOptions for Wikipedia 20200501.en 
 There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
``` 
 I found the same issue with almost any language other than English. (For English, it works). Will someone need to update the file on S3 again?
MissingBeamOptions for Wikipedia 20200501.en 
 There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
``` 
 This is because only some languages are already preprocessed (en, de, fr, it) and stored on our google storage.
We plan to have a systematic way to preprocess more wikipedia languages in the future.

For the other languages you have to process them on your side using apache beam. That's why the lib asks for a Beam runner.
Faster Shuffling? 
 Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping. 

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.) 
 I think the slowness here probably come from the fact that we are copying from and to python.

@lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?
Faster Shuffling? 
 Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping. 

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.) 
 > @lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?

I just tried with `writer.write_table` with tables of 1000 elements and it's slower that the solution in #405 

On my side (select 10 000 examples):
- Original implementation: 12s
- Batched solution: 100ms
- solution using arrow tables: 350ms

I'll try with arrays and record batches to see if we can make it work.
Faster Shuffling? 
 Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping. 

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.) 
 Shuffling is now significantly faster thanks to #513 
Feel free to play with it now :)

Closing this one, but feel free to re-open if you have other questions
Faster Shuffling? 
 Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping. 

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.) 
 > Shuffling is now significantly faster thanks to #513 Feel free to play with it now :)
> 
> Closing this one, but feel free to re-open if you have other questions

I have a similar issue. My code is 

```
    for batch_num in range(num_batches):
        print(f'--> {batch_num=}\n') if verbose else None
        # - Get batch
        shuffled_dataset = dataset.shuffle(buffer_size=buffer_size, seed=seed)
        raw_text_batch = shuffled_dataset.take(batch_size)
        tokenized_batch = map(raw_text_batch)
        if verbose:
            time_start = time.time()
            print(f'{raw_text_batch=}')
            print(f'{tokenized_batch=}')
            print(f'{next(iter(raw_text_batch))=}')
            print(f'{next(iter(tokenized_batch))=}')
            print(f'Time it took: {time.time() - time_start} seconds \a\n')
   ```

without the suffle it takes 4.5 secs with it takes 87.1 secs. Is this difference expected? my dataset version is:
```
(beyond_scale) brando9@ampere1:~/beyond-scale-language-data-diversity$ pip list | grep dataset
datasets                             2.14.3
```

@lhoestq thoughts?
Faster Shuffling? 
 Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping. 

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.) 
 Shuffling leads to doing random access in many different locations on disk which is slower than reading contiguous data.

There is a super fast approximate shuffling algorithm implemented for iterable datasets though:

```python
iterable_dataset = dataset.to_iterable_dataset(num_shards=1024)
shuffled_dataset = iterable_dataset.shuffle(buffer_size=1000)
```

(the first batch might be a bit slow to get because the algorithm first fills a buffer before returning the first batch, see the [docs](https://huggingface.co/docs/datasets/v2.14.4/en/stream#shuffle) for more info)
🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17 
 1. I try downloading `wmt14`, `wmt15`, `wmt17`, `wmt19` with the following code:
```
nlp.load_dataset('wmt14','de-en')
nlp.load_dataset('wmt15','de-en')
nlp.load_dataset('wmt17','de-en')
nlp.load_dataset('wmt19','de-en')
```
The code runs but the download speed is **extremely slow**, the same behaviour is not observed on `wmt16` and `wmt18`

2. When trying to download `wmt17 zh-en`, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz 
 similar slow download speed here for nlp.load_dataset('wmt14', 'fr-en')
`
Downloading: 100%|██████████████████████████████████████████████████████████| 658M/658M [1:00:42<00:00, 181kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████| 918M/918M [1:39:38<00:00, 154kB/s]
Downloading:   2%|▉                                                       | 40.9M/2.37G [04:48<5:03:06, 128kB/s]
`
Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?
🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17 
 1. I try downloading `wmt14`, `wmt15`, `wmt17`, `wmt19` with the following code:
```
nlp.load_dataset('wmt14','de-en')
nlp.load_dataset('wmt15','de-en')
nlp.load_dataset('wmt17','de-en')
nlp.load_dataset('wmt19','de-en')
```
The code runs but the download speed is **extremely slow**, the same behaviour is not observed on `wmt16` and `wmt18`

2. When trying to download `wmt17 zh-en`, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz 
 > The code runs but the download speed is extremely slow, the same behaviour is not observed on wmt16 and wmt18

The original source for the files may provide slow download speeds.
We can probably host these files ourselves.

> When trying to download wmt17 zh-en, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

Looks like the file`UNv1.0.en-zh.tar.gz` is missing, or the url changed. We need to fix that

> Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

Right now I don't think it's possible. Maybe @patrickvonplaten knows more about it
🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17 
 1. I try downloading `wmt14`, `wmt15`, `wmt17`, `wmt19` with the following code:
```
nlp.load_dataset('wmt14','de-en')
nlp.load_dataset('wmt15','de-en')
nlp.load_dataset('wmt17','de-en')
nlp.load_dataset('wmt19','de-en')
```
The code runs but the download speed is **extremely slow**, the same behaviour is not observed on `wmt16` and `wmt18`

2. When trying to download `wmt17 zh-en`, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz 
 Yeah, the download speed is sadly always extremely slow :-/. 
I will try to check out the `wmt17 zh-en` bug :-)
Conversion through to_pandas output numpy arrays for lists instead of python objects 
 In a related question, the conversion through to_pandas output numpy arrays for the lists instead of python objects.

Here is an example:
```python
>>> dataset._data.slice(key, 1).to_pandas().to_dict("list")
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [array([  101,  7277,  2180,  5303,  4806,  1117,  1711,   117,  2292,
        1119,  1270,   107,  1103,  7737,   107,   117,  1104,  9938,
        4267, 12223, 21811,  1117,  2554,   119,   102])], 'token_type_ids': [array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0])], 'attention_mask': [array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1])]}
>>> type(dataset._data.slice(key, 1).to_pandas().to_dict("list")['input_ids'][0])
<class 'numpy.ndarray'>
>>> dataset._data.slice(key, 1).to_pydict()
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [[101, 7277, 2180, 5303, 4806, 1117, 1711, 117, 2292, 1119, 1270, 107, 1103, 7737, 107, 117, 1104, 9938, 4267, 12223, 21811, 1117, 2554, 119, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
``` 
 To convert from arrow type we have three options: to_numpy, to_pandas and to_pydict/to_pylist.

- to_numpy and to_pandas return numpy arrays instead of lists but are very fast.
- to_pydict/to_pylist can be 100x slower and become the bottleneck for reading data, but at least they return lists.

Maybe we can have to_pydict/to_pylist as the default and use to_numpy or to_pandas when the format (set by `set_format`) is 'numpy' or 'pandas'
[dataset] Structure of MLQA seems unecessary nested 
 The features of the MLQA dataset comprise several nested dictionaries with a single element inside (for `questions` and `ids`): https://github.com/huggingface/nlp/blob/master/datasets/mlqa/mlqa.py#L90-L97

Should we keep this @mariamabarham @patrickvonplaten? Was this added for compatibility with tfds?

```python
            features=nlp.Features(
                {
                    "context": nlp.Value("string"),
                    "questions": nlp.features.Sequence({"question": nlp.Value("string")}),
                    "answers": nlp.features.Sequence(
                        {"text": nlp.Value("string"), "answer_start": nlp.Value("int32"),}
                    ),
                    "ids": nlp.features.Sequence({"idx": nlp.Value("string")})
``` 
 Same for the RACE dataset: https://github.com/huggingface/nlp/blob/master/datasets/race/race.py

Should we scan all the datasets to remove this pattern of un-necessary nesting?
TypeError when computing bertscore 
 Hi, 

I installed nlp 0.3.0 via pip, and my python version is 3.7.
When I tried to compute bertscore with the code:
```
import nlp 
bertscore = nlp.load_metric('bertscore')  
# load hyps and refs 
...
print (bertscore.compute(hyps, refs, lang='en'))
```

I got the following error.
```
Traceback (most recent call last):
  File "bert_score_evaluate.py", line 16, in <module>
    print (bertscore.compute(hyps, refs, lang='en'))
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metric.py", line 200, in compute
    output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metrics/bertscore/fb176889831bf0ce995ed197edc94b2e9a83f647a869bb8c9477dbb2d04d0f08/bertscore.py", line 105, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() takes 3 positional arguments but 4 were given
```

It seems like there is something wrong with get_hash() function? 
 I am not able to reproduce this issue on my side.
Could you give us more details about the inputs you used ?

I do get another error though:
```
~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/bert_score/utils.py in bert_cos_score_idf(model, refs, hyps, tokenizer, idf_dict, verbose, batch_size, device, all_layers)
    371         return sorted(list(set(l)), key=lambda x: len(x.split(" ")))
    372 
--> 373     sentences = dedup_and_sort(refs + hyps)
    374     embs = []
    375     iter_range = range(0, len(sentences), batch_size)

ValueError: operands could not be broadcast together with shapes (0,) (2,)
```
That's because it gets numpy arrays as input and not lists. See #387
Segmentation fault when loading local JSON dataset as of #372 
 The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault. 

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.) 
 Yes, deleting the directory solves the error whenever I try to rerun.

By replacing the json-loader, you mean the cached file in my `site-packages` directory? e.g. `/home/XXX/.cache/lib/python3.7/site-packages/nlp/datasets/json/(...)/json.py` 

When I was testing this out before the #372 PR was merged I had issues installing it properly locally. Since the `json.py` script was downloaded instead of actually using the one provided in the local install. Manually updating that file seemed to solve it, but it didn't seem like a proper solution. Especially when having to run this on a remote compute cluster with no access to that directory.
Segmentation fault when loading local JSON dataset as of #372 
 The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault. 

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.) 
 I see, diving in the JSON file for SQuAD it's a pretty complex structure.

The best solution for you, if you have a dataset really similar to SQuAD would be to copy and modify the SQuAD data processing script. We will probably add soon an option to be able to specify file path to use instead of the automatic URL encoded in the script but in the meantime you can:
- copy the [squad script](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py)  in a new script for your dataset
- in the new script replace [these `urls_to_download `](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py#L99-L102) by `urls_to_download=self.config.data_files`
- load the dataset with `dataset = load_dataset('path/to/your/new/script', data_files={nlp.Split.TRAIN: "./datasets/train-v2.0.json"})`

This way you can reuse all the processing logic of the SQuAD loading script.
Segmentation fault when loading local JSON dataset as of #372 
 The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault. 

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.) 
 This seems like a more sensible solution! Thanks, @thomwolf. It's been a little daunting to understand what these scripts actually do, due to the level of abstraction and central documentation.

Am I correct in assuming that the `_generate_examples()` function is the actual procedure for how the data is loaded from file? Meaning that essentially with a file containing another format, that is the only function that requires re-implementation? I'm working with a lot of datasets that, due to licensing and privacy, cannot be published. As this library is so neatly integrated with the transformers library and gives easy access to public sets such as SQUAD and increased performance, it is very neat to be able to load my private sets as well. As of now, I have just been working on scripts for translating all my data into the SQUAD-format before using the json script, but I see that it might not be necessary after all.
Segmentation fault when loading local JSON dataset as of #372 
 The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault. 

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.) 
 Yes `_generate_examples()` is the main entry point. If you change the shape of the returned dictionary you also need to update the `features` in the `_info`.

I'm currently writing the doc so it should be easier soon to use the library and know how to add your datasets.
Segmentation fault when loading local JSON dataset as of #372 
 The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault. 

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.) 
 Could you try to update pyarrow to >=0.17.0 @vegarab ?
I don't have any segmentation fault with my version of pyarrow (0.17.1)

I tested with
```python
import nlp
s = nlp.load_dataset("json", data_files="train-v2.0.json", field="data", split="train")
s[0]
# {'title': 'Normans', 'paragraphs': [{'qas': [{'question': 'In what country is Normandy located?', 'id':...
```
Segmentation fault when loading local JSON dataset as of #372 
 The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault. 

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.) 
 Also if you want to have your own dataset script, we now have a new documentation !
See here:
https://huggingface.co/nlp/add_dataset.html
can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries 
 Trying to load a local SQuAD-formatted dataset (from a JSON file, about 60MB):
```
dataset = nlp.load_dataset(path='json', data_files={nlp.Split.TRAIN: ["./path/to/file.json"]})
```
causes
```
Traceback (most recent call last):
  File "dataloader.py", line 9, in <module>
    ["./path/to/file.json"]})
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 719, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False):
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/datasets/json/88c1bc5c68489f7eda549ed05a5a738527c613b3e7a4ee3524d9d233353a949b/json.py", line 53, in _generate_tables
    file, read_options=self.config.pa_read_options, parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 191, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

I haven't been able to find any reports of this specific pyarrow error here or elsewhere.  
 I am able to reproduce this with the official SQuAD `train-v2.0.json` file downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/
can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries 
 Trying to load a local SQuAD-formatted dataset (from a JSON file, about 60MB):
```
dataset = nlp.load_dataset(path='json', data_files={nlp.Split.TRAIN: ["./path/to/file.json"]})
```
causes
```
Traceback (most recent call last):
  File "dataloader.py", line 9, in <module>
    ["./path/to/file.json"]})
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 719, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False):
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/datasets/json/88c1bc5c68489f7eda549ed05a5a738527c613b3e7a4ee3524d9d233353a949b/json.py", line 53, in _generate_tables
    file, read_options=self.config.pa_read_options, parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 191, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

I haven't been able to find any reports of this specific pyarrow error here or elsewhere.  
 I am facing this issue in transformers library 3.0.2 while reading a csv using datasets.
Is this fixed in latest version? 
I updated the latest version 4.0.1 but still getting this error. What could cause this error?
load_metric can't acquire lock anymore 
 I can't load metric (glue) anymore after an error in a previous run. I even removed the whole cache folder `/home/XXX/.cache/huggingface/`, and the issue persisted. What are the steps to fix this?

    Traceback (most recent call last):
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 101, in __init__
        self.filelock.acquire(timeout=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/filelock.py", line 278, in acquire
        raise Timeout(self._lock_file)
    filelock.Timeout: The file lock '/home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock' could not be acquired.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "examples_huggingface_nlp.py", line 268, in <module>
        main()
      File "examples_huggingface_nlp.py", line 242, in main
        dataset, metric = get_dataset_metric(glue_task)
      File "examples_huggingface_nlp.py", line 77, in get_dataset_metric
        metric = nlp.load_metric('glue', glue_config, experiment_id=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/load.py", line 440, in load_metric
        **metric_init_kwargs,
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 104, in __init__
        "Cannot acquire lock, caching file might be used by another process, "
    ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id' for this run.
    I0709 15:54:41.008838 139854118430464 filelock.py:318] Lock 139852058030936 released on /home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock
 
 I found that, in the same process (or the same interactive session), if I do

import nlp

m1 = nlp.load_metric('glue', 'mrpc')
m2 = nlp.load_metric('glue', 'sst2')

I will get the same error `ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id'`.