noamaanMulla-03/datasets-issues huggingface dataset:
Fix text delimiter

I changed the delimiter in the `text` dataset script.
It should fix the `pyarrow.lib.ArrowInvalid: CSV parse error` from #622 

I changed the delimiter to an unused ascii character that is not present in text files : `\b`


Which OS are you using ?@abhi1nandy2

> Which OS are you using ?

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Do you mind sharing the data you used (or part of it), so I can try to reproduce ?
Or at least some info about the text file you're using ? (size, n of lines, encoding)

Lot of data, difficult to share. There are 46 shards, each having about 256000 lines. using `file` command gives this - `ASCII text, with very long lines`.

Ok I see, no problem :) 
I'll see what I can do

Could you just test with one single dummy text file with a few lines to see if you're having the issue ?
Also which version of `datasets` do you have ?
The process cannot access the file because it is being used by another process (windows)

Hi, I consistently get the following error when developing in my PC (windows 10):

```
    train_dataset = train_dataset.map(convert_to_features, batched=True)
  File "C:\Users\saareliad\AppData\Local\Continuum\miniconda3\envs\py38\lib\site-packages\nlp\arrow_dataset.py", line 970, in map
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\saareliad\AppData\Local\Continuum\miniconda3\envs\py38\lib\shutil.py", line 803, in move
    os.unlink(src)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\saareliad\\.cache\\huggingface\\datasets\\squad\\plain_text\\1.0.0\\408a8fa46a1e2805445b793f1022e743428ca739a34809fce872f0c7f17b44ab\\tmpsau1bep1'

```


Hi, which version of `nlp` are you using?

By the way we'll be releasing today a significant update fixing many issues (but also comprising a few breaking changes).
You can see more informations here #545 and try it by installing from source from the master branch.

I'm using version 0.4.0.



Ok, it's probably fixed on master. Otherwise if you can give me a fully self-contained exemple to reproduce the error, I can try to investigate.

I get the same behavior, on Windows, when `map`ping a function to a loaded dataset. 
The error doesn't occur if I re-run the cell a second time though! 
I'm on version 1.0.1.

This is going to be fixed by #644 

@saareliad I got the same issue that troubled me quite a while. Unfortunately, there are no good answers to this issue online, I tried it on Linux and that's absolutely fine. After hacking the source code, I solved this problem as follows.

In the source code file: arrow_dataset.py -> _map_single(...)

change
```python
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```
to
```python
tmp_file.close()
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```

Then it works without needing multiple times runs to avoid the permission error.
I know this solution is unusual since it changes the source code. Hopefully, the lib's contributors can have better solutions in the future.


@wangcongcong123 thanks  for sharing.
(BTW I also solved it locally on windows by putting the problematic line under try except and not using cache... On windows I just needed 1% of the dataset anyway)
The download instructions for c4 datasets are not contained in the error message

The manual download instructions are not clear 
```The dataset c4 with config en requires manual data. 
 Please follow the manual download instructions: <bound method C4.manual_download_instructions of <datasets_modules.datasets.c4.830b0c218bd41fed439812c8dd19dbd4767d2a3faa385eb695cf8666c982b1b3.c4.C4 object at 0x7ff8c5969760>>. 
 Manual data can be loaded with `datasets.load_dataset(c4, data_dir='<path/to/manual/data>')
```

Either `@property`  could be added to C4.manual_download_instrcutions (or make it a real property), or the manual_download_instructions function needs to be called I think.

Let me know if you want a PR for this, but I'm not sure which possible fix is the correct one.


Good catch !
Indeed the `@property` is missing.

Feel free to open a PR :)

Also not that C4 is a dataset that needs an Apache Beam runtime to be generated.
For example Dataflow, Spark, Flink etc.

Usually we generate the dataset on our side once and for all, but we haven't done it for C4 yet.
More info about beam datasets [here](https://huggingface.co/docs/datasets/beam_dataset.html)

Let me know if you have any questions
OverflowError when slicing with an array containing negative ids

```python
from datasets import Dataset

d = ds.Dataset.from_dict({"a": range(10)})

print(d[0])
# {'a': 0}

print(d[-1])
# {'a': 9}

print(d[[0, -1]])
# OverflowError
```
results in
```
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-5-863dc3555598> in <module>
----> 1 d[[0, -1]]

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in __getitem__(self, key)
   1070             format_columns=self._format_columns,
   1071             output_all_columns=self._output_all_columns,
-> 1072             format_kwargs=self._format_kwargs,
   1073         )
   1074 

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1025                 indices = key
   1026 
-> 1027             indices_array = pa.array([int(i) for i in indices], type=pa.uint64())
   1028 
   1029             # Check if we need to convert indices

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

OverflowError: can't convert negative value to unsigned int
```
[BUG] No such file or directory

This happens when both
1. Huggingface datasets cache dir does not exist
2. Try to load a local dataset script

builder.py throws an error when trying to create a filelock in a directory (cache/datasets) that does not exist
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L177

Tested on v1.0.2

@lhoestq
Use multiprocess from pathos for multiprocessing

[Multiprocess](https://github.com/uqfoundation/multiprocess) (from the [pathos](https://github.com/uqfoundation/pathos) project) allows to use lambda functions in multiprocessed map.
It was suggested to use it by @kandorm.

We're already using dill which is its only dependency.


We can just install multiprocess actually, I'll change that

Just an FYI: I remember that I wanted to try pathos a couple of years back and I ran into issues considering cross-platform; the code would just break on Windows. If I can verify this PR by running CPU tests on Windows, let me know!

That's good to know thanks
I guess we can just wait for #644 to be merged first. I'm working on fixing the tests for windows

Looks like all the CI jobs on windows passed !
I also tested locally on my windows and it works great :) 

I think this is ready to merge, let me know if you have any remarks @thomwolf @BramVanroy
Fix bug related to boolean in GAP dataset.

### Why I did
The value in `row["A-coref"]` and `row["B-coref"]` is `'TRUE'` or `'FALSE'`.
This type is `string`, then `bool('FALSE')` is equal to `True` in Python.
So, both rows are transformed into `True` now.

So, I modified this problem.

### What I did
I modified `bool(row["A-coref"])` and `bool(row["B-coref"])` to `row["A-coref"] == "TRUE"` and `row["B-coref"] == "TRUE"`.

Thank you!


Hi !

Good catch, thanks for creating this PR :)

Could you also regenerate the metadata for this dataset using 
```
datasets-cli test ./datasets/gap --save_infos --all_configs
```

That'd be awesome

@lhoestq Thank you for your revieing!!!

I've performed it and have read CONTRIBUTING.md now!
Fix column order issue in cast

Previously, the order of the columns in the features passes to `cast_` mattered.
However even though features passed to `cast_` had the same order as the dataset features, it could fail because the schema that was built was always in alphabetical order.
This issue was reported by @lewtun  in #623 

To fix that I fixed the schema to follow the order of the arrow table columns.
I also added the possibility to give features that are not ordered the same way as the dataset features.
Fix SQuAD metric kwargs description

The `answer_start` field was missing in the kwargs docstring.

This should fix #657 

FYI another fix was proposed by @tshrjn in #658 and suggests to remove this field.
However IMO `answer_start` is useful to match the squad dataset format for consistency, even though it is not used in the metric computation. I think it's better to keep it this way, so that you can just give references=squad["answers"] to .compute(). 

Let me know what sounds the best for you
Switch to pandas reader for text dataset

Following the discussion in #622 , it appears that there's no appropriate ways to use the payrrow csv reader to read text files because of the separator.

In this PR I switched to pandas to read the file.

Moreover pandas allows to read the file by chunk, which means that you can build the arrow dataset from a text file that is bigger than RAM (we used to have to shard text files an mentioned in https://github.com/huggingface/datasets/issues/610#issuecomment-691672919)

From a test that I did locally on a 1GB text file, the pyarrow reader used to run in 150ms while the new one takes 650ms (multithreading off for pyarrow). This is probably due to chunking since I am having the same speed difference by calling `read()` and calling `read(chunksize)` + `readline()` to read the text file.


If the windows tests in the CI pass, today will be a happy day
Disable tokenizers parallelism in multiprocessed map

It was reported in #620 that using multiprocessing with a tokenizers shows this message:
```
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
```
This message is shown when TOKENIZERS_PARALLELISM is unset.
Moreover if it is set to `true`, then the program just hangs.

To hide the message (if TOKENIZERS_PARALLELISM is unset) and avoid hanging (if TOKENIZERS_PARALLELISM is `true`), then I set TOKENIZERS_PARALLELISM to `false` when forking the process. After forking is gets back to its original value.

Also I added a warning if TOKENIZERS_PARALLELISM was `true` and is set to `false`:
```
Setting TOKENIZERS_PARALLELISM=false for forked processes.
```

cc @n1t0
XNLI dataset: NonMatchingChecksumError

Hi,
I tried to download "xnli" dataset in colab using 
`xnli = load_dataset(path='xnli')`
but got 'NonMatchingChecksumError' error

`NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-27-a87bedc82eeb> in <module>()
----> 1 xnli = load_dataset(path='xnli')

3 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']`

The same code worked well several days ago in colab but stopped working now. Thanks!


Thanks for reporting.
The data file must have been updated by the host.
I'll update the checksum with the new one.

Well actually it looks like the link isn't working anymore :(

The new link is https://cims.nyu.edu/~sbowman/xnli/XNLI-1.0.zip
I'll update the dataset script

I'll do a release in the next few days to make the fix available for everyone.
In the meantime you can load `xnli` with
```
xnli = load_dataset('xnli', script_version="master")
```
This will use the latest version of the xnli script (available on master branch), instead of the old one.

That's awesome! Thanks a lot!
Add hotpot QA

Added the [HotpotQA](https://github.com/hotpotqa/hotpot) multi-hop question answering dataset.




Awesome :) 

Don't pay attention to the RemoteDatasetTest error, I'm fixing it right now

You can rebase from master to fix the CI test :)

If we're lucky we can even include this dataset in today's release

Just thinking since `type` can only be `comparison` or `bridge` and `level` can only be `easy`, `medium`, `hard` should they be `ClassLabel`?

> Just thinking since `type` can only be `comparison` or `bridge` and `level` can only be `easy`, `medium`, `hard` should they be `ClassLabel`?

I think it's more a tag than a label. I guess a string is fine
load_dataset() won't download in Windows

I don't know if this is just me or Windows. Maybe other Windows users can chime in if they don't have this problem. I've been trying to get some of the tutorials working on Windows, but when I use the load_dataset() function, it just stalls and the script keeps running indefinitely without downloading anything. I've waited upwards of 18 hours to download the 'multi-news' dataset (which isn't very big), and still nothing. I've tried running it through different IDE's and the command line, but it had the same behavior. I've also tried it with all virus and malware protection turned off. I've made sure python and all IDE's are exceptions to the firewall and all the requisite permissions are enabled.

Additionally, I checked to see if other packages could download content such as an nltk corpus, and they could. I've also run the same script using Ubuntu and it downloaded fine (and quickly). When I copied the downloaded datasets from my Ubuntu drive to my Windows .cache folder it worked fine by reusing the already-downloaded dataset, but it's cumbersome to do that for every dataset I want to try in my Windows environment.

Could this be a bug, or is there something I'm doing wrong or not thinking of?

Thanks.


I have the same issue. Tried to download a few of them and not a single one is downloaded successfully.

This is the output:
```
>>> dataset = load_dataset('blended_skill_talk', split='train')
Using custom data configuration default               <-- This step never ends
```

This was fixed in #644 
I'll do a new release soon :)

In the meantime you can run it by installing from source

Closing since version 1.1.0 got released with Windows support :) 
Let me know if it works for you now
Add CoNLL-2003 shared task dataset

Please consider adding CoNLL-2003 shared task dataset as it's beneficial for token classification tasks. The motivation behind this PR is the [PR](https://github.com/huggingface/transformers/pull/7041) in the transformers project. This dataset would be not only useful for the usual run-of-the-mill NER tasks but also for syntactic chunking and part-of-speech (POS) tagging. 


I think we should somewhere mention, that is the dataset in IOB2 tagging scheme, whereas the original dataset uses IOB1 :)

Indeed this is something we want to mention.

If would want to add more details about the IOB1->2 change, feel free to ignore my suggestions and edit the description + update the dataset_info

@lhoestq do you want me to update it or you'll update it. I am ok either way

The best would be to mention this change in the description and then update the dataset_info.json file.
Could you do that if you don't mind ?

Then it should be ready to merge :)

Thanks again for adding this dataset !

No problem @lhoestq I'll do the update

@lhoestq please check if 847addf is exactly what we want

Is the German task also part of this? If not, can it be accessed via the Datasets library?
Use python read for text dataset

As mentioned in #622 the pandas reader used for text dataset doesn't work properly when there are \r characters in the text file.

Instead I switched to pure python using `open` and `read`.
From my benchmark on a 100MB text file, it's the same speed as the previous pandas reader.


One thing though, could we try to read the files in parallel?

We could but I'm not sure this would help a lot since the bottleneck is the drive IO if the files are big enough.
It could make sense for very small files.

Looks like windows is not a big fan of this approach
I'm working on a fix

I remember issue https://github.com/huggingface/datasets/issues/546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

I think it's worth testing multiprocessing. It could also be something we add to our speed benchmarks

> I remember issue #546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

It still would be interesting I think, especially in scenarios where IO is less of an issue (SSDs particularly) and where there are many smaller files. Wrapping this function in a `pool.map` is perhaps an easy thing to try. 

Merging this one for now for the patch release
Error in the notebooks/Overview.ipynb notebook

Hi,

I got the following error in **cell number 3** while exploring the **Overview.ipynb** notebook in google colab. I used the [link ](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb) provided in the main README file to open it in colab. 

```python
# You can access various attributes of the datasets before downloading them
squad_dataset = list_datasets()[datasets.index('squad')]

pprint(squad_dataset.__dict__)  # It's a simple python dataclass
```

Error message
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-8dc805c4949c> in <module>()
      2 squad_dataset = list_datasets()[datasets.index('squad')]
      3 
 ----> 4 pprint(squad_dataset.__dict__)  # It's a simple python dataclass
    
AttributeError: 'str' object has no attribute '__dict__'
```

The object `squad_dataset` is a `str` not a `dataclass` .


Do this:
``` python
squad_dataset = list_datasets(with_details=True)[datasets.index('squad')]
pprint(squad_dataset.__dict__)  # It's a simple python dataclass
```

Thanks! This worked. I have created a PR to fix this in the notebook.
`metric.compute` throws `ArrowInvalid` error

I get the following error with `rouge.compute`. It happens only with distributed training, and it occurs randomly I can't easily reproduce it. This is using `nlp==0.4.0`

```
  File "/home/beltagy/trainer.py", line 92, in validation_step
    rouge_scores = rouge.compute(predictions=generated_str, references=gold_str, rouge_types=['rouge2', 'rouge1', 'rougeL'])
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/metric.py", line 224, in compute
    self.finalize(timeout=timeout)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/metric.py", line 213, in finalize
    self.data = Dataset(**reader.read_files(node_files))
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 217, in read_files
    dataset_kwargs = self._read_files(files=files, info=self._info, original_instructions=original_instructions)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 162, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 276, in _get_dataset_from_filename
    f = pa.ipc.open_stream(mmap)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/pyarrow/ipc.py", line 173, in open_stream
    return RecordBatchStreamReader(source)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/pyarrow/ipc.py", line 64, in __init__
    self._open(source)
  File "pyarrow/ipc.pxi", line 469, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
```


Hmm might be related to what we are solving in #564

Could you try to update to `datasets>=1.0.0` (we changed the name of the library) and try again ?
If is was related to the distributed setup settings it must be fixed.
If it was related to empty metric inputs it's going to be fixed in #654 

Closing this one as it was fixed in #654 
Feel free to re-open if you have other questions
Don't use tqdm 4.50.0

tqdm 4.50.0 introduced permission errors on windows
see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111) for the error details.

For now I just added `<4.50.0` in the setup.py
Hopefully we can find what's wrong with this version soon
Fix train_test_split output format

There was an issue in the `transmit_format` wrapper that returned bad formats when using train_test_split.
This was due to `column_names` being handled as a List[str] instead of Dict[str, List[str]] when the dataset transform (train_test_split) returns a DatasetDict (one set of column names per split).

This should fix @timothyjlaurent 's issue in #620 and fix #676 

I added tests for `transmit_format` so that it doesn't happen again
train_test_split returns empty dataset item

I try to split my dataset by `train_test_split`, but after that the item in `train` and `test` `Dataset` is empty.
The codes:
```
yelp_data = datasets.load_from_disk('/home/ssd4/huanglianzhe/test_yelp')
    print(yelp_data[0])
    yelp_data = yelp_data.train_test_split(test_size=0.1)
    print(yelp_data)
    print(yelp_data['test'])
    print(yelp_data['test'][0])
```
The outputs:
```
{'stars': 2.0, 'text': 'xxxx'}
Loading cached split indices for dataset at /home/ssd4/huanglianzhe/test_yelp/cache-f9b22d8b9d5a7346.arrow and /home/ssd4/huanglianzhe/test_yelp/cache-4aa26fa4005059d1.arrow
DatasetDict({'train': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 7219009), 'test': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)})
Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)
{}    # yelp_data['test'][0] is empty
```


The problem still exists after removing the cache files.

Can you reproduce this example in a Colab so we can investigate? (or give more information on your software/hardware config)

Thanks for reporting.
I just found the issue, I'm creating a PR

We'll do a release pretty soon to include the fix :)
In the meantime you can install the lib from source if you want to
[Feature Request] Add the OpenWebText dataset

The OpenWebText dataset is an open clone of OpenAI's WebText dataset. It can be used to train ELECTRA as is specified in the [README](https://www.github.com/google-research/electra).

More information and the download link are available [here](https://skylion007.github.io/OpenWebTextCorpus/).

Labels: name: dataset request

We're experimenting with hosting the OpenWebText corpus on Zenodo for easier downloading. https://zenodo.org/record/3834942#.Xs1w8i-z2J8

Closing since it's been added in #660