NyssaW/github-issues huggingface dataset:
Fix bug related to boolean in GAP dataset.

### Why I did
The value in `row["A-coref"]` and `row["B-coref"]` is `'TRUE'` or `'FALSE'`.
This type is `string`, then `bool('FALSE')` is equal to `True` in Python.
So, both rows are transformed into `True` now.

So, I modified this problem.

### What I did
I modified `bool(row["A-coref"])` and `bool(row["B-coref"])` to `row["A-coref"] == "TRUE"` and `row["B-coref"] == "TRUE"`.

Thank you!

Comments:
Hi !

Good catch, thanks for creating this PR :)

Could you also regenerate the metadata for this dataset using 
```
datasets-cli test ./datasets/gap --save_infos --all_configs
```

That'd be awesome

@lhoestq Thank you for your revieing!!!

I've performed it and have read CONTRIBUTING.md now!
The download instructions for c4 datasets are not contained in the error message

The manual download instructions are not clear 
```The dataset c4 with config en requires manual data. 
 Please follow the manual download instructions: <bound method C4.manual_download_instructions of <datasets_modules.datasets.c4.830b0c218bd41fed439812c8dd19dbd4767d2a3faa385eb695cf8666c982b1b3.c4.C4 object at 0x7ff8c5969760>>. 
 Manual data can be loaded with `datasets.load_dataset(c4, data_dir='<path/to/manual/data>')
```

Either `@property`  could be added to C4.manual_download_instrcutions (or make it a real property), or the manual_download_instructions function needs to be called I think.

Let me know if you want a PR for this, but I'm not sure which possible fix is the correct one.

Comments:
Good catch !
Indeed the `@property` is missing.

Feel free to open a PR :)

Also not that C4 is a dataset that needs an Apache Beam runtime to be generated.
For example Dataflow, Spark, Flink etc.

Usually we generate the dataset on our side once and for all, but we haven't done it for C4 yet.
More info about beam datasets [here](https://huggingface.co/docs/datasets/beam_dataset.html)

Let me know if you have any questions
train_test_split returns empty dataset item

I try to split my dataset by `train_test_split`, but after that the item in `train` and `test` `Dataset` is empty.
The codes:
```
yelp_data = datasets.load_from_disk('/home/ssd4/huanglianzhe/test_yelp')
    print(yelp_data[0])
    yelp_data = yelp_data.train_test_split(test_size=0.1)
    print(yelp_data)
    print(yelp_data['test'])
    print(yelp_data['test'][0])
```
The outputs:
```
{'stars': 2.0, 'text': 'xxxx'}
Loading cached split indices for dataset at /home/ssd4/huanglianzhe/test_yelp/cache-f9b22d8b9d5a7346.arrow and /home/ssd4/huanglianzhe/test_yelp/cache-4aa26fa4005059d1.arrow
DatasetDict({'train': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 7219009), 'test': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)})
Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)
{}    # yelp_data['test'][0] is empty
```

Comments:
The problem still exists after removing the cache files.

Can you reproduce this example in a Colab so we can investigate? (or give more information on your software/hardware config)

Thanks for reporting.
I just found the issue, I'm creating a PR

We'll do a release pretty soon to include the fix :)
In the meantime you can install the lib from source if you want to
load_dataset() won't download in Windows

I don't know if this is just me or Windows. Maybe other Windows users can chime in if they don't have this problem. I've been trying to get some of the tutorials working on Windows, but when I use the load_dataset() function, it just stalls and the script keeps running indefinitely without downloading anything. I've waited upwards of 18 hours to download the 'multi-news' dataset (which isn't very big), and still nothing. I've tried running it through different IDE's and the command line, but it had the same behavior. I've also tried it with all virus and malware protection turned off. I've made sure python and all IDE's are exceptions to the firewall and all the requisite permissions are enabled.

Additionally, I checked to see if other packages could download content such as an nltk corpus, and they could. I've also run the same script using Ubuntu and it downloaded fine (and quickly). When I copied the downloaded datasets from my Ubuntu drive to my Windows .cache folder it worked fine by reusing the already-downloaded dataset, but it's cumbersome to do that for every dataset I want to try in my Windows environment.

Could this be a bug, or is there something I'm doing wrong or not thinking of?

Thanks.

Comments:
I have the same issue. Tried to download a few of them and not a single one is downloaded successfully.

This is the output:
```
>>> dataset = load_dataset('blended_skill_talk', split='train')
Using custom data configuration default               <-- This step never ends
```

This was fixed in #644 
I'll do a new release soon :)

In the meantime you can run it by installing from source

Closing since version 1.1.0 got released with Windows support :) 
Let me know if it works for you now
blog_authorship_corpus crashed

This is just to report that When I pick blog_authorship_corpus in 
https://huggingface.co/nlp/viewer/?dataset=blog_authorship_corpus
I get this:
![image](https://user-images.githubusercontent.com/7553188/94349542-4364f300-0013-11eb-897d-b25660a449f0.png)



Comments:
Thanks for reporting !
We'll free some memory
[BUG] No such file or directory

This happens when both
1. Huggingface datasets cache dir does not exist
2. Try to load a local dataset script

builder.py throws an error when trying to create a filelock in a directory (cache/datasets) that does not exist
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L177

Tested on v1.0.2

@lhoestq 

Comments:
Fix SQuAD metric kwargs description

The `answer_start` field was missing in the kwargs docstring.

This should fix #657 

FYI another fix was proposed by @tshrjn in #658 and suggests to remove this field.
However IMO `answer_start` is useful to match the squad dataset format for consistency, even though it is not used in the metric computation. I think it's better to keep it this way, so that you can just give references=squad["answers"] to .compute(). 

Let me know what sounds the best for you


Comments:
How to skip a example when running dataset.map

in processing func, I process examples and detect some invalid examples, which I did not want it to be added into train dataset. However I did not find how to skip this recognized invalid example when doing dataset.map. 

Comments:
Hi @xixiaoyao,
Depending on what you want to do you can:
- use a first step of `filter` to filter out the invalid examples: https://huggingface.co/docs/datasets/processing.html#filtering-rows-select-and-filter
- or directly detect the invalid examples inside the callable used with `map` and return them unchanged or even remove them at the same time if you are using `map` in batched mode. Here is an example where we use `map` in batched mode to add new rows on the fly but you can also use it to remove examples on the fly (that's what `filter` actually do under-the-hood): https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset

Closing this one.
Feel free to re-open if you have other questions

Letting finders-of-this-thread know that the new link is: https://huggingface.co/docs/datasets/process#data-augmentation
OverflowError when slicing with an array containing negative ids

```python
from datasets import Dataset

d = ds.Dataset.from_dict({"a": range(10)})

print(d[0])
# {'a': 0}

print(d[-1])
# {'a': 9}

print(d[[0, -1]])
# OverflowError
```
results in
```
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-5-863dc3555598> in <module>
----> 1 d[[0, -1]]

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in __getitem__(self, key)
   1070             format_columns=self._format_columns,
   1071             output_all_columns=self._output_all_columns,
-> 1072             format_kwargs=self._format_kwargs,
   1073         )
   1074 

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1025                 indices = key
   1026 
-> 1027             indices_array = pa.array([int(i) for i in indices], type=pa.uint64())
   1028 
   1029             # Check if we need to convert indices

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

OverflowError: can't convert negative value to unsigned int
```

Comments:
Loss not decrease with Datasets and Transformers

HI,

The following script is used to fine-tune a BertForSequenceClassification model on SST2.

The script is adapted from [this colab](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb) that presents an example of fine-tuning BertForQuestionAnswering using squad dataset. In that colab, loss works fine. When I adapt it to SST2, the loss fails to decrease as it should. I attach the adapted script below and appreciate anyone pointing out what I miss?

```python
import torch
from datasets import load_dataset
from transformers import BertForSequenceClassification
from transformers import BertTokenizerFast
# Load our training dataset and tokenizer
dataset = load_dataset("glue", 'sst2')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
del dataset["test"] # let's remove it in this demo

# Tokenize our training dataset
def convert_to_features(example_batch):
    encodings = tokenizer(example_batch["sentence"])
    encodings.update({"labels": example_batch["label"]})
    return encodings

encoded_dataset = dataset.map(convert_to_features, batched=True)
# Format our dataset to outputs torch.Tensor to train a pytorch model
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'labels']
encoded_dataset.set_format(type='torch', columns=columns)

# Instantiate a PyTorch Dataloader around our dataset
# Let's do dynamic batching (pad on the fly with our own collate_fn)
def collate_fn(examples):
    return tokenizer.pad(examples, return_tensors='pt')

dataloader = torch.utils.data.DataLoader(encoded_dataset['train'], collate_fn=collate_fn, batch_size=8)
# Now let's train our model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
# Let's load a pretrained Bert model and a simple optimizer
model = BertForSequenceClassification.from_pretrained('bert-base-cased', return_dict=True)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
model.train().to(device)
for i, batch in enumerate(dataloader):
    batch.to(device)
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    model.zero_grad()
    print(f'Step {i} - loss: {loss:.3}')


```
In case needed.

- datasets == 1.0.2
- transformers == 3.2.0

Comments:
And I tested it on T5ForConditionalGeneration, that works no problem.

Hi did you manage to fix your issue ?

If so feel free to share your fix and close this thread
load_dataset from local squad.py, raise error: TypeError: 'NoneType' object is not callable 


version: 1.0.2

```
train_dataset  = datasets.load_dataset('squad') 
```

The above code can works. However, when I download the squad.py from your server, and saved as `my_squad.py` to local. I run followings raise errors.
```
train_dataset  = datasets.load_dataset('./my_squad.py')                                                                                                
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-25a84b4d1581> in <module>
----> 1 train_dataset  = nlp.load_dataset('./my_squad.py')

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    602         hash=hash,
    603         features=features,
--> 604         **config_kwargs,
    605     )
    606 

TypeError: 'NoneType' object is not callable


Comments:
Hi !
Thanks for reporting.
It looks like no object inherits from `datasets.GeneratorBasedBuilder` (or more generally from `datasets.DatasetBuilder`) in your script.

Could you check that there exist at least one dataset builder class ?

Hi @xixiaoyao did you manage to fix your issue ?

No activity, closing

It happened when try to change the old project which use 'nlp' to new project which use 'datasets'. You should check you old 'my_squad.py' file, change the inherit class from `nlp.xxx` to `datasets.xxx`. Otherwise datasets - load.py - import_main_class() `if inspect.isclass(obj) and issubclass(obj, main_cls_type):` can not find the main_cls.
Squad Metric Description & Feature Mismatch

The [description](https://github.com/huggingface/datasets/blob/master/metrics/squad/squad.py#L39) doesn't mention `answer_start` in squad. However the `datasets.features` require [it](https://github.com/huggingface/datasets/blob/master/metrics/squad/squad.py#L68). It's also not used in the evaluation.

Comments:
Thanks for reporting !
There indeed a mismatch between the features and the kwargs description

I believe `answer_start` was added to match the squad dataset format for consistency, even though it is not used in the metric computation. I think I'd rather keep it this way, so that you can just give `references=squad["answers"]` to `.compute()`.
Maybe we can just fix the description then.

But then providing the `answer_start` becomes mandatory since the format of the features is checked against the one provided in the squad [file](https://github.com/huggingface/datasets/pull/658/files).
Use multiprocess from pathos for multiprocessing

[Multiprocess](https://github.com/uqfoundation/multiprocess) (from the [pathos](https://github.com/uqfoundation/pathos) project) allows to use lambda functions in multiprocessed map.
It was suggested to use it by @kandorm.

We're already using dill which is its only dependency.

Comments:
We can just install multiprocess actually, I'll change that

Just an FYI: I remember that I wanted to try pathos a couple of years back and I ran into issues considering cross-platform; the code would just break on Windows. If I can verify this PR by running CPU tests on Windows, let me know!

That's good to know thanks
I guess we can just wait for #644 to be merged first. I'm working on fixing the tests for windows

Looks like all the CI jobs on windows passed !
I also tested locally on my windows and it works great :) 

I think this is ready to merge, let me know if you have any remarks @thomwolf @BramVanroy
added Winogrande debiased subset

The [Winogrande](https://arxiv.org/abs/1907.10641) paper mentions a `debiased` subset that wasn't in the first release; this PR adds it.

Comments:
To fix the CI you just have to copy the dummy data to the 1.1.0 folder, and maybe create the dummy ones for the `debiased` configuration

Fixed! Thanks @lhoestq
handle data alteration when trying type

Fix #649 

The bug came from the type inference that didn't handle a weird case in Pyarrow.
Indeed this code runs without error but alters the data in arrow:
```python
import pyarrow as pa

type = pa.struct({"a": pa.struct({"b": pa.string()})})
array_with_altered_data = pa.array([{"a": {"b": "foo", "c": "bar"}}] * 10, type=type)
print(array_with_altered_data[0].as_py())
# {'a': {'b': 'foo'}} -> the sub-field "c" is missing
```
(I don't know if this is intended in pyarrow tbh)

We didn't take this case into account during type inference. By default it was keeping old features and maybe alter data.
To fix that I added a line that checks that the first element of the array is not altered.

Comments:
Problem with JSON dataset format

I have a local json dataset with the following form.

{
    'id01234': {'key1': value1, 'key2': value2, 'key3': value3},
    'id01235': {'key1': value1, 'key2': value2, 'key3': value3},
    .
    .
    .
    'id09999': {'key1': value1, 'key2': value2, 'key3': value3}
}
Note that instead of a list of records it's basically a dictionary of key value pairs with the keys being the record_ids and the values being the corresponding record.

Reading this with json:

```
data = datasets.load('json', data_files='path_to_local.json')
```
Throws an error and asks me to chose a field. What's the right way to handle this?

Comments:
Currently the `json` dataset doesn't support this format unfortunately.
However you could load it with
```python
from datasets import Dataset
import pandas as pd

df = pd.read_json("path_to_local.json", orient="index")
dataset = Dataset.from_pandas(df)
```

or you can make a custom dataset script as explained in doc here: https://huggingface.co/docs/datasets/add_dataset.html
dummy data testing can't test datasets using `dl_manager.extract` in `_split_generators`

Hi, I recently want to add a dataset whose source data is like this
```
openwebtext.tar.xz
  |__ openwebtext
         |__subset000.xz
         |     |__ ....txt
         |     |__ ....txt
         |     ...
         |__ subset001.xz
         |
         ....
```
So I wrote `openwebtext.py` like this
```
 def _split_generators(self, dl_manager):
        dl_dir = dl_manager.download_and_extract(_URL)
        owt_dir = os.path.join(dl_dir, 'openwebtext')
        subset_xzs = [
            os.path.join(owt_dir, file_name) for file_name in os.listdir(owt_dir) if file_name.endswith('xz') # filter out ...xz.lock
        ]
        ex_dirs = dl_manager.extract(subset_xzs, num_proc=round(os.cpu_count()*0.75))
        nested_txt_files = [ 
          [ 
            os.path.join(ex_dir,txt_file_name) for txt_file_name in os.listdir(ex_dir) if txt_file_name.endswith('txt')
          ] for ex_dir in ex_dirs
        ]
        txt_files = chain(*nested_txt_files)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"txt_files": txt_files}
            ),
        ]
```
All went good, I can load and use real openwebtext, except when I try to test with dummy data. The problem is  `MockDownloadManager.extract` do nothing, so `ex_dirs = dl_manager.extract(subset_xzs)` won't decompress `subset_xxx.xz`s for me.

How should I do ? Or you can modify `MockDownloadManager` to make it like a real `DownloadManager` ?

Comments:
Hi :) 
In your dummy data zip file you can just have `subset000.xz` as directories instead of compressed files.
Let me know if it helps

Thanks for your comment @lhoestq ,
Just for confirmation, changing dummy data like this won't make dummy test test the functionality to extract `subsetxxx.xz` but actually kind of circumvent it. But since we will test the real data so it is ok ?

Yes it's fine for now. We plan to add a job for slow tests.
And at one point we'll also do another pass on the dummy data handling and consider extracting files.

Thanks for the confirmation.
Also the suggestion works. Thank you.
Inconsistent behavior in map

I'm observing inconsistent behavior when applying .map(). This happens specifically when I'm incrementally adding onto a feature that is a nested dictionary. Here's a simple example that reproduces the problem.

```python
import datasets

# Dataset with a single feature called 'field' consisting of two examples
dataset = datasets.Dataset.from_dict({'field': ['a', 'b']})
print(dataset[0])
# outputs
{'field': 'a'}

# Map this dataset to create another feature called 'otherfield', which is a dictionary containing a key called 'capital'
dataset = dataset.map(lambda example: {'otherfield': {'capital': example['field'].capitalize()}})
print(dataset[0])
# output is okay
{'field': 'a', 'otherfield': {'capital': 'A'}}

# Now I want to map again to modify 'otherfield', by adding another key called 'append_x' to the dictionary under 'otherfield'
print(dataset.map(lambda example: {'otherfield': {'append_x': example['field'] + 'x'}})[0])
# printing out the first example after applying the map shows that the new key 'append_x' doesn't get added
# it also messes up the value stored at 'capital'
{'field': 'a', 'otherfield': {'capital': None}}

# Instead, I try to do the same thing by using a different mapped fn
print(dataset.map(lambda example:  {'otherfield': {'append_x': example['field'] + 'x', 'capital': example['otherfield']['capital']}})[0])
# this preserves the value under capital, but still no 'append_x'
{'field': 'a', 'otherfield': {'capital': 'A'}}

# Instead, I try to pass 'otherfield' to remove_columns
print(dataset.map(lambda example:  {'otherfield': {'append_x': example['field'] + 'x', 'capital': example['otherfield']['capital']}}, remove_columns=['otherfield'])[0])
# this still doesn't fix the problem
{'field': 'a', 'otherfield': {'capital': 'A'}}

# Alternately, here's what happens if I just directly map both 'capital' and 'append_x' on a fresh dataset.

# Recreate the dataset
dataset = datasets.Dataset.from_dict({'field': ['a', 'b']})
# Now map the entire 'otherfield' dict directly, instead of incrementally as before
print(dataset.map(lambda example:  {'otherfield': {'append_x': example['field'] + 'x', 'capital': example['field'].capitalize()}})[0])
# This looks good!
{'field': 'a', 'otherfield': {'append_x': 'ax', 'capital': 'A'}}
```

This might be a new issue, because I didn't see this behavior in the `nlp` library. 

Any help is appreciated!

Comments:
Thanks for reporting !

This issue must have appeared when we refactored type inference in `nlp`
By default the library tries to keep the same feature types when applying `map` but apparently it has troubles with nested structures. I'll try to fix that next week
Cannot download dataset_info.json

I am running my job on a cloud server where does not provide for connections from the standard compute nodes to outside resources. Hence, when I use `dataset.load_dataset()` to load data, I got an error like this:

```
ConnectionError: Couldn't reach https://storage.googleapis.com/huggingface-nlp/cache/datasets/text/default-53ee3045f07ba8ca/0.0.0/dataset_info.json
```

I tried to open this link manually, but I cannot access this file. How can I download this file and pass it through `dataset.load_dataset()` manually?

Versions:
Python version 3.7.3
PyTorch version 1.6.0
TensorFlow version 2.3.0
datasets version: 1.0.1 


Comments:
Thanks for reporting !
We should add support for servers without internet connection indeed
I'll do that early next week

Thanks, @lhoestq !
Please let me know when it is available. 

Right now the recommended way is to create the dataset on a server with internet connection and then to save it and copy the serialized dataset to the server without internet connection.

#652 should allow you to load text/json/csv/pandas datasets without an internet connection **IF** you've the dataset script locally.

Example: 
If you have `datasets/text/text.py` locally, then you can do `load_dataset("./datasets/text", data_files=...)`
Fix docs typos

This PR fixes few typos in the docs and the error in the code snippet in the set_format section in docs/source/torch_tensorflow.rst. `torch.utils.data.Dataloader` expects padded batches so it throws an error due to not being able to stack the unpadded tensors. If we follow the Quick tour from the docs where they add the `truncation=True, padding='max_length'` arguments to the tokenizer before passing data to Dataloader, we can easily fix the issue. 

Comments:
Don't use take on dataset table in pyarrow 1.0.x

Fix #615 

Comments:
I tried lower batch sizes and it didn't accelerate filter (quite the opposite actually).
The slow-down also appears for pyarrow 0.17.1 for some reason, not sure it comes from these changes

I just checked the benchmarks of other PRs and some of them had 300s (!!) for filter. This needs some investigation..

Merging this one since it's not the cause of the the slow down

@lhoestq What might be the reason for the slowdown? When I was training large batchsize, the slowdown was obvious, and my original assumption was to increase write_batch_size
Better windows support

There are a few differences in the behavior of python and pyarrow on windows.

For example there are restrictions when accessing/deleting files that are open

Fix #590 

Comments:
This PR is ready :)
It brings official support for windows.

Some tests `AWSDatasetTest` are failing.
This is because I had to fix a few datasets that were not compatible with windows.
These test will pass once they got merged on master :)
Add Polyglot-NER Dataset

Adds the [Polyglot-NER dataset](https://sites.google.com/site/rmyeid/projects/polylgot-ner) with named entity tags for 40 languages. I include separate configs for each language as well as a `combined` config which lumps them all together.

Comments:
Hi @joeddav thanks for adding this! (I did a long webarchive.org session to actually find that dataset a while ago).

One question: should we manually correct the labeling scheme to (at least) IOB1?

That means "LOC" will be converted to "I-LOC". IOB1 is not explict. mentioned in the paper, but it is used in the documentation:

https://polyglot.readthedocs.io/en/latest/NamedEntityRecognition.html

@stefan-it I went back and forth on this. My biggest problem with it is that once you are in IOB, there is the expectation that the beginning of new entities are marked with a `B-` (at least in the case of two back-to-back entities):
```
Today    O
Alice    I-PER
Bob      B-PER
and      O
I        O 
ate      O
lasagna  O
```
If we just prepend `I-` to everything, `Bob` would be incorrectly tagged `I-PER`, meaning `Bob Alice` is a single entity. The current format is bad but is at least clear that it does not contain that information.

But I could go either way if someone has a strong opinion.

Indeed I'm not sure we can convert them to IOB because of this issue. I'm fine with keeping it like that

I'll do a release later today, hopefully we can include this dataset in the release :)

Let me know if you need help with the dummy data

@lhoestq cool thanks, I think I've got it right now – just zipped them wrong. I'm running tests locally now and then will push.

@lhoestq set to merge?

@joeddav I'm fine with keeping the original labeling scheme :)
GLUE/QQP dataset: NonMatchingChecksumError

Hi @lhoestq , I know you are busy and there are also other important issues. But if this is easy to be fixed, I am shamelessly wondering if you can give me some help , so I can evaluate my models and restart with my developing cycle asap. 😚

datasets version: editable install of master at 9/17

`datasets.load_dataset('glue','qqp', cache_dir='./datasets')`

```
Downloading and preparing dataset glue/qqp (download: 57.73 MiB, generated: 107.02 MiB, post-processed: Unknown size, total: 164.75 MiB) to ./datasets/glue/qqp/1.0.0/7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4...
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
 in 
----> 1 datasets.load_dataset('glue','qqp', cache_dir='./datasets')

~/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    609         download_config=download_config,
    610         download_mode=download_mode,
--> 611         ignore_verifications=ignore_verifications,
    612     )
    613 

~/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    467                     if not downloaded_from_gcs:
    468                         self._download_and_prepare(
--> 469                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    470                         )
    471                     # Sync info

~/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    527         if verify_infos:
    528             verify_checksums(
--> 529                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    530             )
    531 

~/datasets/src/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://dl.fbaipublicfiles.com/glue/data/QQP-clean.zip']
```

Comments:
Hi ! Sure I'll take a look
Fix text delimiter

I changed the delimiter in the `text` dataset script.
It should fix the `pyarrow.lib.ArrowInvalid: CSV parse error` from #622 

I changed the delimiter to an unused ascii character that is not present in text files : `\b`

Comments:
Which OS are you using ?@abhi1nandy2

> Which OS are you using ?

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Do you mind sharing the data you used (or part of it), so I can try to reproduce ?
Or at least some info about the text file you're using ? (size, n of lines, encoding)

Lot of data, difficult to share. There are 46 shards, each having about 256000 lines. using `file` command gives this - `ASCII text, with very long lines`.

Ok I see, no problem :) 
I'll see what I can do

Could you just test with one single dummy text file with a few lines to see if you're having the issue ?
Also which version of `datasets` do you have ?
straddling object straddles two block boundaries

I am trying to read json data (it's an array with lots of dictionaries) and getting block boundaries issue as below : 

I tried calling read_json with readOptions but no luck .

```
table = json.read_json(fn)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/_json.pyx", line 246, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```


Comments:
sorry it's an apache arrow issue.
Update GLUE URLs (now hosted on FB)

NYU is switching dataset hosting from Google to FB. This PR closes https://github.com/huggingface/datasets/issues/608 and is necessary for https://github.com/jiant-dev/jiant/issues/161. This PR updates the data URLs based on changes made in https://github.com/nyu-mll/jiant/pull/1112.

Note: rebased on huggingface/datasets

Comments:
Mistakes in MLQA features names

I think the following features in MLQA shouldn't be named the way they are:
1. `questions` (should be `question`)
2. `ids` (should be `id`)
3. `start` (should be `answer_start`)

The reasons I'm suggesting these features be renamed are:
* To make them consistent with other QA datasets like SQuAD, XQuAD, TyDiQA etc. and hence make it easier to concatenate multiple QA datasets.
* The features names are not the same as the ones provided in the original MLQA datasets (it uses the names I suggested).

I know these columns can be renamed using  using `Dataset.rename_column_`, `questions` and `ids` can be easily renamed but `start` on the other hand is annoying to rename since it's nested inside the feature `answers`.


Comments:
Indeed you're right ! Thanks for reporting that

Could you open a PR to fix the features names ?
Add CoNLL-2003 shared task dataset

Please consider adding CoNLL-2003 shared task dataset as it's beneficial for token classification tasks. The motivation behind this PR is the [PR](https://github.com/huggingface/transformers/pull/7041) in the transformers project. This dataset would be not only useful for the usual run-of-the-mill NER tasks but also for syntactic chunking and part-of-speech (POS) tagging. 

Comments:
I think we should somewhere mention, that is the dataset in IOB2 tagging scheme, whereas the original dataset uses IOB1 :)

Indeed this is something we want to mention.

If would want to add more details about the IOB1->2 change, feel free to ignore my suggestions and edit the description + update the dataset_info

@lhoestq do you want me to update it or you'll update it. I am ok either way

The best would be to mention this change in the description and then update the dataset_info.json file.
Could you do that if you don't mind ?

Then it should be ready to merge :)

Thanks again for adding this dataset !

No problem @lhoestq I'll do the update

@lhoestq please check if 847addf is exactly what we want

Is the German task also part of this? If not, can it be accessed via the Datasets library?
Update GLUE URLs (now hosted on FB)

NYU is switching dataset hosting from Google to FB. This PR closes https://github.com/huggingface/datasets/issues/608 and is necessary for https://github.com/jiant-dev/jiant/issues/161. This PR updates the data URLs based on changes made in https://github.com/nyu-mll/jiant/pull/1112.

Comments:
Thanks for opening this PR :) 

We changed the name of the lib from nlp to datasets yesterday.
Could you rebase from master and re-generate the dataset_info.json file to fix the name changes ?

Rebased changes here: https://github.com/huggingface/datasets/pull/626
Don't use the old NYU GLUE dataset URLs

NYU is switching dataset hosting from Google to FB. Initial changes to `datasets` are in https://github.com/jeswan/nlp/commit/b7d4a071d432592ded971e30ef73330529de25ce. What tests do you suggest I run before opening a PR?

See: https://github.com/jiant-dev/jiant/issues/161 and https://github.com/nyu-mll/jiant/pull/1112

Comments:
Feel free to open the PR ;)
Thanks for updating the dataset_info.json file !
apply offset to indices in multiprocessed map

Fix #597 

I fixed the indices by applying an offset.
I added the case to our tests to make sure it doesn't happen again.

I also added the message proposed by @thomwolf in #597 

```python
>>> d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2, load_from_cache_file=False)
Done writing 10 indices in 80 bytes .
Testing the mapped function outputs
[0, 1]
Testing finished, running the mapping function on the dataset
Done writing 5 indices in 41 bytes .
Done writing 5 indices in 41 bytes .
Spawning 2 processes
[0, 1, 2, 3, 4]
[5, 6, 7, 8, 9]
#0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 377.90ba/s]
#1: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 378.92ba/s]
Concatenating 2 shards from multiprocessing

# Dataset(features: {'label': ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None), 'text': Value(dtype='string', id=None)}, num_rows: 10)
```

Comments:
Add MATINF dataset

@lhoestq The command to create metadata failed. I guess it's because the zip is not downloaded from a remote address? How to solve that? Also the CI fails and I don't know how to fix that :(

Comments:
Hi ! sorry for the late response

Could you try to rebase from master ? We changed the named of the library last week so you have to include this change in your code.

Can you give me more details about the error you get when running the cli command ?

Note that in case of a manual download you have to specify the directory where you downloaded the data with `--data_dir <path/to/the/directory>`

I fucked up the Git rebase lol. Closing it.
Indices incorrect with multiprocessing

When `num_proc` > 1, the indices argument passed to the map function is incorrect:

```python
d = load_dataset('imdb', split='test[:1%]')

def fn(x, inds):
    print(inds)
    return x

d.select(range(10)).map(fn, with_indices=True, batched=True)
# [0, 1]
# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2)
# [0, 1]
# [0, 1]
# [0, 1, 2, 3, 4]
# [0, 1, 2, 3, 4]
```

As you can see, the subset passed to each thread is indexed from 0 to N which doesn't reflect their positions in `d`.

Comments:
I fixed a bug that could cause this issue earlier today. Could you pull the latest version and try again ?

Still the case on master.
I guess we should have an offset in the multi-procs indeed (hopefully it's enough).

Also, side note is that we should add some logging before the "test" to say we are testing the function otherwise its confusing for the user to see two outputs I think. Proposal (see the "Testing the mapped function outputs:" lines):
```
>>> d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2)
Done writing 10 indices in 80 bytes .
Done writing 5 indices in 41 bytes .
Done writing 5 indices in 41 bytes .
Spawning 2 processes
Testing the mapped function outputs:
inds: [0, 1]
inds: [0, 1]
Testing finished, running the mapped function on the dataset:
#0:   0%|                                                                                                                                                                    | 0/1 [00:00<?, ?ba/s]
inds: [0, 1, 2, 3, 4]                                                                                                                                                                             inds: [0, 1, 2, 3, 4]                                                                                                                                                         | 0/1 [00:00<?, ?ba/s]
#0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1321.04ba/s]
#1: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1841.22ba/s]
Concatenating 2 shards from multiprocessing
Dataset(features: {'text': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None)}, num_rows: 10)
```
[style/quality] Moving to isort 5.0.0 + style/quality on datasets and metrics

Move the repo to isort 5.0.0.

Also start testing style/quality on datasets and metrics.

Specific rule: we allow F401 (unused imports) in metrics to be able to add imports to detect early on missing dependencies.
Maybe we could add this in datasets but while cleaning this I've seen many example of really unused imports in dataset so maybe it's better to have it as a line-by-line nova instead of a general rule like in metrics.

Comments:
Ready for review @lhoestq, just updated a few 156 files here
`Dataset`/`DatasetDict` has no attribute 'save_to_disk'

Hi,

As the title indicates, both `Dataset` and `DatasetDict` classes don't seem to have the `save_to_disk` method.  While the file [`arrow_dataset.py`](https://github.com/huggingface/nlp/blob/34bf0b03bfe03e7f77b8fec1cd48f5452c4fc7c1/src/nlp/arrow_dataset.py) in the repo here has the method, the file `arrow_dataset.py` which is saved after `pip install nlp -U` in my `conda` environment DOES NOT contain the `save_to_disk` method. I even tried `pip install git+https://github.com/huggingface/nlp.git ` and still no luck. Do I need to install the library in another way?

Comments:
`pip install git+https://github.com/huggingface/nlp.git` should have done the job.

Did you uninstall `nlp` before installing from github ?

> Did you uninstall `nlp` before installing from github ?

I did not. I created a new environment and installed `nlp` directly from `github` and it worked!

Thanks.
GermEval 2014: new download urls

Hi,

unfortunately, the download links for the GermEval 2014 dataset have changed: they're now located on a Google Drive.

I changed the URLs and bump version from 1.0.0 to 2.0.0.

Comments:
/cc: @vblagoje

Closing this one as #594 is merged (same changes except the dummy data hack)

Awesome @stefan-it ! @lhoestq how soon can I use the fixed GermEval dataset in HF token classification examples?

I've manually updated the script on S3, so you can actually use it right now with
```python
from nlp import load_dataset

germeval = load_dataset("germeval_14")
```

not sure if it's used in token classification examples already

Awesome. Not used yet but I am going to use it now. I've been working on an update for token classification examples and this was a missing piece. Thanks @stefan-it @lhoestq
The process cannot access the file because it is being used by another process (windows)

Hi, I consistently get the following error when developing in my PC (windows 10):

```
    train_dataset = train_dataset.map(convert_to_features, batched=True)
  File "C:\Users\saareliad\AppData\Local\Continuum\miniconda3\envs\py38\lib\site-packages\nlp\arrow_dataset.py", line 970, in map
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\saareliad\AppData\Local\Continuum\miniconda3\envs\py38\lib\shutil.py", line 803, in move
    os.unlink(src)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\saareliad\\.cache\\huggingface\\datasets\\squad\\plain_text\\1.0.0\\408a8fa46a1e2805445b793f1022e743428ca739a34809fce872f0c7f17b44ab\\tmpsau1bep1'

```

Comments:
Hi, which version of `nlp` are you using?

By the way we'll be releasing today a significant update fixing many issues (but also comprising a few breaking changes).
You can see more informations here #545 and try it by installing from source from the master branch.

I'm using version 0.4.0.



Ok, it's probably fixed on master. Otherwise if you can give me a fully self-contained exemple to reproduce the error, I can try to investigate.

I get the same behavior, on Windows, when `map`ping a function to a loaded dataset. 
The error doesn't occur if I re-run the cell a second time though! 
I'm on version 1.0.1.

This is going to be fixed by #644 

@saareliad I got the same issue that troubled me quite a while. Unfortunately, there are no good answers to this issue online, I tried it on Linux and that's absolutely fine. After hacking the source code, I solved this problem as follows.

In the source code file: arrow_dataset.py -> _map_single(...)

change
```python
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```
to
```python
tmp_file.close()
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```

Then it works without needing multiple times runs to avoid the permission error.
I know this solution is unusual since it changes the source code. Hopefully, the lib's contributors can have better solutions in the future.


@wangcongcong123 thanks  for sharing.
(BTW I also solved it locally on windows by putting the problematic line under try except and not using cache... On windows I just needed 1% of the dataset anyway)
Cannot use nlp.load_dataset text, AttributeError: module 'nlp.utils' has no attribute 'logging'


```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/nlp/load.py", line 533, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/nlp/load.py", line 61, in import_main_class
    module = importlib.import_module(module_path)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/nlp/datasets/text/5dc629379536c4037d9c2063e1caa829a1676cf795f8e030cd90a537eba20c08/text.py", line 9, in <module>
    logger = nlp.utils.logging.get_logger(__name__)
AttributeError: module 'nlp.utils' has no attribute 'logging'
```

Occurs on the following code, or any code including the load_dataset('text'):
```
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=args.block_size), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
return dataset
```

Comments:
Use github versioning

Right now dataset scripts and metrics are downloaded from S3 which is in sync with master. It means that it's not currently possible to pin the dataset/metric script version.

To fix that I changed the download url from S3 to github, and adding a `version` parameter in `load_dataset` and `load_metric` to pin a certain version of the lib, as in #562 

Comments:
I noticed that datasets like `cnn_dailymail` need the `version` parameter to be passed to its `config_kwargs`.
Shall we rename the `version` paramater in `load_dataset` ? Maybe `repo_version` or `script_version` ?
ArrowIndexError on Dataset.select

If the indices table consists in several chunks, then `dataset.select` results in an `ArrowIndexError` error for pyarrow < 1.0.0

Example:

```python
from nlp import load_dataset

mnli = load_dataset("glue", "mnli", split="train")
shuffled = mnli.shuffle(seed=42)
mnli.select(list(range(len(mnli))))
```

raises:
```python
---------------------------------------------------------------------------
ArrowIndexError                           Traceback (most recent call last)
<ipython-input-64-006a5d38d418> in <module>
----> 1 mnli.shuffle(seed=42).select(list(range(len(mnli))))

~/Desktop/hf/nlp/src/nlp/fingerprint.py in wrapper(*args, **kwargs)
    161             # Call actual function
    162 
--> 163             out = func(self, *args, **kwargs)
    164 
    165             # Update fingerprint of in-place transforms + update in-place history of transforms

~/Desktop/hf/nlp/src/nlp/arrow_dataset.py in select(self, indices, keep_in_memory, indices_cache_file_name, writer_batch_size, new_fingerprint)
   1653         if self._indices is not None:
   1654             if PYARROW_V0:
-> 1655                 indices_array = self._indices.column(0).chunk(0).take(indices_array)
   1656             else:
   1657                 indices_array = self._indices.column(0).take(indices_array)

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.Array.take()

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowIndexError: take index out of bounds
```

This is because the `take` method is only done on the first chunk which only contains 1000 elements by default (mnli has ~400 000 elements).

Shall we change that to use 
```python
pa.concat_tables(self._indices._indices.slice(i, 1) for i in indices_array)
```
instead of `take` ? @thomwolf 

Comments: