helmo/github-issues huggingface dataset:
Add sort, shuffle, test_train_split and select methods
State: closed | Author: thomwolf (MEMBER)

Add a bunch of methods to reorder/split/select rows in a dataset:
- `dataset.select(indices)`: Create a new dataset with rows selected following the list/array of indices (which can have a different size than the dataset and contain duplicated indices, the only constrain is that all the integers in the list must be smaller than the dataset size, otherwise we're indexing outside the dataset...)
- `dataset.sort(column_name)`: sort a dataset according to a column (has to be a column with a numpy compatible type)
- `dataset.shuffle(seed)`: shuffle a dataset rows
- `dataset.train_test_split(test_size, train_size)`: Return a dictionary with two random train and test subsets (`train` and `test` ``Dataset`` splits)

All these methods are **not** in-place which means they return new ``Dataset``.
This is the default behavior in the library.

Fix #147 #166 #259 

Comments: Nice !

Also it looks like we can have a train_test_split method for free:
```python
train_indices, test_indices = train_test_split(range(len(dataset)))
train = dataset.sort(indices=train_indices)
test = dataset.sort(indices=test_indices)
```

and a shuffling method for free:
```python
shuffled_indices = shuffle(range(len(dataset)))
shuffled_dataset = dataset.sort(indices=shuffled_indices)
```

Maybe we can have a specific API for train_test_split and shuffle. They are two features asked quite often (see #147, #166)

Ok, I think this one is ready to merge.

@patrickvonplaten @jplu @mariamabarham @joeddav @n1t0 @julien-c you may want to give it a look, it adds a bunch of methods to reorder/split/select rows in a dataset:
- `dataset.select(indices)`: Create a new dataset with rows selected following the list/array of indices (which can have a different size than the dataset and contain duplicated indices, the only constrain is that all the integers in the list must be smaller than the dataset size, otherwise we're indexing outside the dataset...)
- `dataset.sort(column_name)`: sort a dataset according to a column (has to be a column with a numpy compatible type)
- `dataset.shuffle(seed)`: shuffle a dataset rows
- `dataset.train_test_split(test_size, train_size)`: Return a dictionary with two random train and test subsets (`train` and `test` ``Dataset`` splits)

All these methods are **not** in-place which means they return new ``Dataset``, which is the default behavior in the library.

> Might be a solution to put 0.25 and 0.75 as default values for respectively `test_size` and `train_size`. WDYT?

According to sklearn documentation, it is indeed set to 0.25 and 0.75 if both `test_size` and `train_size` are None.
Let me add it.

I think we're good to go now :) @joeddav @thomwolf @jplu
Add pyarrow warning colab
State: closed | Author: lhoestq (MEMBER)

When a user installs `nlp` on google colab, then google colab doesn't update pyarrow, and the runtime needs to be restarted to use the updated version of pyarrow.

This is an issue because `nlp` requires the updated version to work correctly.

In this PR I added en error that is shown to the user in google colab if the user tries to `import nlp` without having restarted the runtime. The error tells the user to restart the runtime.
Fix small issues creating dataset
State: closed | Author: lhoestq (MEMBER)

Fix many small issues mentioned in #249:
- don't force to install apache beam for commands
- fix None cache dir when using `dl_manager.download_custom`
- added new extras in `setup.py` named `dev` that contains tests and quality dependencies
- mock dataset sizes when running tests with dummy data
- add a note about the naming convention of datasets (camel case - snake case) in CONTRIBUTING.md

This should help users create their datasets.
Next step is the `add_dataset.md` docs :)
Add new dataset ANLI Round 1
State: closed | Author: easonnie (CONTRIBUTOR)

Adding new dataset [ANLI](https://github.com/facebookresearch/anli/).

I'm not familiar with how to add new dataset. Let me know if there is any issue. I only include round 1 data here. There will be round 2, round 3 and more in the future with potentially different format. I think it will be better to separate them.

Comments: Hello ! Thanks for adding this one :)

This looks great, you just have to do the last steps to make the CI pass.
I can see that two things are missing:
1. the dummy data that is used to test that the script is working as expected
2. the json file with all the infos about the dataset

You can see the steps to help you create the dummy data and generate the dataset_infos.json file right [here](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset)
Why is dataset after tokenization far more larger than the orginal one ?
State: closed | Author: richarddwang (CONTRIBUTOR)

I tokenize wiki dataset by `map` and cache the results.
```
def tokenize_tfm(example):
    example['input_ids'] = hf_fast_tokenizer.convert_tokens_to_ids(hf_fast_tokenizer.tokenize(example['text']))
    return example
wiki = nlp.load_dataset('wikipedia', '20200501.en', cache_dir=cache_dir)['train']
wiki.map(tokenize_tfm, cache_file_name=cache_dir/"wikipedia/20200501.en/1.0.0/tokenized_wiki.arrow")
```
and when I see their size
```
ls -l --block-size=M
17460M  wikipedia-train.arrow
47511M  tokenized_wiki.arrow
```
The tokenized one is over 2x size of original one.
Is there something I did wrong ?

Comments: Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map`

Hi ! Thanks for your reply.

But since size of `input_ids` < size of `text`, I am wondering why
size of `input_ids` + `text` > 2x the size of `text`  🤔

Hard to tell... This is probably related to the way apache arrow compresses lists of integers, that may be different from the compression of strings.

Thanks for your point. 😀, It might be answer.
Since this is hard to know, I'll close this issue.
But if somebody knows more details, please comment below ~ 😁
[Feature request] Add a feature to dataset
State: closed | Author: sarahwie (NONE)

Is there a straightforward way to add a field to the arrow_dataset, prior to performing map?

Comments: Do you have an example of what you would like to do? (you can just add a field in the output of the unction you give to map and this will add this field in the output table)

Given another source of data loaded in, I want to pre-add it to the dataset so that it aligns with the indices of the arrow dataset prior to performing map.

E.g. 
```
new_info = list of length dataset['train']

dataset['train'] = dataset['train'].map(lambda x: some_function(x, new_info[index of x]))

def some_function(x, new_info_x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x
    return x
```
I was thinking to instead create a new field in the arrow dataset so that instance x contains all the necessary information when map function is applied (since I don't have index information to pass to map function).

This is what I have so far: 

```
import pyarrow as pa
from nlp.arrow_dataset import Dataset

aug_dataset = dataset['train'][:]
aug_dataset['new_info'] = new_info

#reformat as arrow-table
schema = dataset['train'].schema

# this line doesn't work:
schema.append(pa.field('new_info', pa.int32()))

table = pa.Table.from_pydict(
    aug_dataset,
    schema=schema
)
dataset['train'] = Dataset(table) 
```

Maybe you can use `with_indices`?

```python
new_info = list of length dataset['train']

def some_function(indice, x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x[indice]
    return x

dataset['train'] = dataset['train'].map(some_function, with_indices=True)
```

Oh great. That should work. I missed that in the documentation- thanks :)
[Feature request] Be able to remove a specific sample of the dataset
State: closed | Author: astariul (NONE)

As mentioned in #117, it's currently not possible to remove a sample of the dataset.

But it is a important use case : After applying some preprocessing, some samples might be empty for example. We should be able to remove these samples from the dataset, or at least mark them as `removed` so when iterating the dataset, we don't iterate these samples.

I think it should be a feature. What do you think ?

---

Any work-around in the meantime ?

Comments: Oh yes you can now do that with the `dataset.filter()` method that was added in #214
NonMatchingSplitsSizesError error when reading the IMDB dataset
State: closed | Author: antmarakis (NONE)

Hi!

I am trying to load the `imdb` dataset with this line:

`dataset = nlp.load_dataset('imdb', data_dir='/A/PATH', cache_dir='/A/PATH')`

but I am getting the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/load.py", line 517, in load_dataset
    save_infos=save_infos,
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 363, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 421, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=5929447, num_examples=4537, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

Am I overlooking something? Thanks!

Comments: I just tried on my side and I didn't encounter your problem.
Apparently the script doesn't generate all the examples on your side.

Can you provide the version of `nlp` you're using ?
Can you try to clear your cache and re-run the code ?

I updated it, that was it, thanks!

Hello, I am facing the same problem... how do you clear the huggingface cache?

Hi ! The cache is at ~/.cache/huggingface
You can just delete this folder if needed :)
Better access to all dataset information
State: closed | Author: thomwolf (MEMBER)

Moves all the dataset info down one level from `dataset.info.XXX` to `dataset.XXX`
This way it's easier to access `dataset.feature['label']` for instance

Also, add the original split instructions used to create the dataset in `dataset.split`
Ex:
```
from nlp import load_dataset
stsb = load_dataset('glue', name='stsb', split='train')
stsb.split
>>> NamedSplit('train')
```
Make all dataset downloads deterministic by applying `sorted` to glob and os.listdir
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

This PR makes all datasets loading deterministic by applying `sorted()` to all `glob.glob` and `os.listdir` statements.

Are there other "non-deterministic" functions apart from `glob.glob()` and `os.listdir()` that you can think of @thomwolf @lhoestq @mariamabarham @jplu ?

**Important** 
It does break backward compatibility for these datasets because
1. When loading the complete dataset the order in which the examples are saved is different now
2. When loading only part of a split, the examples themselves might be different.

@patrickvonplaten - the nlp / longformer notebook has to be updated since the examples might now be different

Comments: That's great!

I think it would be nice to test "deterministic-ness" of datasets in CI if we can do it (should be left for future PR of course)

Here is a possibility (maybe there are other ways to do it): given that we may soon have efficient and large-scale hashing (cf our discussion on versioning/tracability), we could incorporate a hash of the final Arrow Dataset to the `dataset.json` file and have a test on it as well as CI on a diversity of platform to test the hash (Win/Mac/Linux + various python/env).
What do you think @lhoestq @patrickvonplaten?

> That's great!
> 
> I think it would be nice to test "deterministic-ness" of datasets in CI if we can do it (should be left for future PR of course)
> 
> Here is a possibility (maybe there are other ways to do it): given that we may soon have efficient and large-scale hashing (cf our discussion on versioning/tracability), we could incorporate a hash of the final Arrow Dataset to the `dataset.json` file and have a test on it as well as CI on a diversity of platform to test the hash (Win/Mac/Linux + various python/env).
> What do you think @lhoestq @patrickvonplaten?

I think that's a great idea! The test should be a `RUN_SLOW` test, since it takes a considerable amount of time to download the dataset and generate the examples, but I think we should add some kind of hash test for each dataset.

Really nice!!
What is the best way to cache a dataset? 
State: closed | Author: Mistobaan (NONE)

For example if I want to use streamlit with a nlp dataset:

```
@st.cache
def load_data():
    return nlp.load_dataset('squad')
```
This code raises the error "uncachable object"

Right now I just fixed with a constant for my specific case:
```
    @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
```
But I was curious to know what is the best way in general



Comments: Everything is already cached by default in 🤗nlp (in particular dataset
loading and all the “map()” operations) so I don’t think you need to do any
specific caching in streamlit.

Tell us if you feel like it’s not the case.

On Sat, 6 Jun 2020 at 13:02, Fabrizio Milo <notifications@github.com> wrote:

> For example if I want to use streamlit with a nlp dataset:
>
> @st.cache
> def load_data():
>     return nlp.load_dataset('squad')
>
> This code raises the error "uncachable object"
>
> Right now I just fixed with a constant for my specific case:
>
>     @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
>
> But I was curious to know what is the best way in general
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/nlp/issues/246>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHKAKO7CWGX2QY55UXLRVIO3ZANCNFSM4NV333RQ>
> .
>


Closing this one. Feel free to re-open if you have other questions !
Add Allociné Dataset
State: closed | Author: TheophileBlard (CONTRIBUTOR)

This is a french binary sentiment classification dataset, which was used to train this model: https://huggingface.co/tblard/tf-allocine.

Basically, it's a french "IMDB" dataset, with more reviews.

More info on [this repo](https://github.com/TheophileBlard/french-sentiment-analysis-with-bert). 

Comments: great work @TheophileBlard 

LGTM, thanks a lot for adding dummy data tests :-) Was it difficult to create the correct dummy data folder? 

It was pretty easy actually. Documentation is on point !
UnicodeDecodeError when downloading GLUE-MNLI
State: closed | Author: patpizio (CONTRIBUTOR)

When I run
```python
dataset = nlp.load_dataset('glue', 'mnli')
```
I get an encoding error (could it be because I'm using Windows?) :
```python
# Lots of error log lines later...
~\Miniconda3\envs\nlp\lib\site-packages\tqdm\std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~\Miniconda3\envs\nlp\lib\site-packages\nlp\datasets\glue\5256cc2368cf84497abef1f1a5f66648522d5854b225162148cb8fc78a5a91cc\glue.py in _generate_examples(self, data_file, split, mrpc_files)
    529 
--> 530                 for n, row in enumerate(reader):
    531                     if is_cola_non_test:

~\Miniconda3\envs\nlp\lib\csv.py in __next__(self)
    110             self.fieldnames
--> 111         row = next(self.reader)
    112         self.line_num = self.reader.line_num

~\Miniconda3\envs\nlp\lib\encodings\cp1252.py in decode(self, input, final)
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24 

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 6744: character maps to <undefined>
```
Anyway this can be solved by specifying to decode in UTF when reading the csv file. I am proposing a PR if that's okay.

Comments: It should be good now, thanks for noticing and fixing it ! I would say that it was because you are on windows but not 100% sure

On Windows Python supports Unicode almost everywhere, but one of the notable exceptions is open() where it uses the locale encoding schema. So platform independent python scripts would always set the encoding='utf-8' in calls to open explicitly. 
In the meantime: since Python 3.7 Windows users can set the default encoding for everything including open() to Unicode by setting this environment variable: set PYTHONUTF8=1 (details can be found in [PEP 540](https://www.python.org/dev/peps/pep-0540/))

For me this fixed the problem described by the OP.
Fix empty cache dir
State: closed | Author: lhoestq (MEMBER)

If the cache dir of a dataset is empty, the dataset fails to load and throws a FileNotFounfError. We could end up with empty cache dir because there was a line in the code that created the cache dir without using a temp dir. Using a temp dir is useful as it gets renamed to the real cache dir only if the full process is successful.

So I removed this bad line, and I also reordered things a bit to make sure that we always use a temp dir. I also added warning if we still end up with empty cache dirs in the future.

This should fix #239


Comments: Looks great! Will this change force all cached datasets to be redownloaded? But even if it does, it shoud not be a big problem, I think

> Looks great! Will this change force all cached datasets to be redownloaded? But even if it does, it shoud not be a big problem, I think

No it shouldn't force to redownload
Deterministic dataset loading
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

When calling:
```python 
import nlp
dataset = nlp.load_dataset("trivia_qa", split="validation[:1%]")
```

the resulting dataset is not deterministic over different google colabs. 
After talking to @thomwolf, I suspect the reason to be the use of `glob.glob` in line:

https://github.com/huggingface/nlp/blob/2e0a8639a79b1abc848cff5c669094d40bba0f63/datasets/trivia_qa/trivia_qa.py#L180

which seems to return an ordering of files that depends on the filesystem:
https://stackoverflow.com/questions/6773584/how-is-pythons-glob-glob-ordered

I think we should go through all the dataset scripts and make sure to have deterministic behavior.

A simple solution for `glob.glob()` would be to just replace it with `sorted(glob.glob())` to have everything sorted by name. 

What do you think @lhoestq?

Comments: Yes good point !

I think using `sorted(glob.glob())` would actually solve this problem. Can you think of other reasons why dataset loading might not be deterministic? @mariamabarham @yjernite @lhoestq @thomwolf . 

I can do a sweep through the dataset scripts and fix the glob.glob() if you guys are ok with it

I'm pretty sure it would solve the problem too.

The only other dataset that is not deterministic right now is `blog_authorship_corpus` (see #215) but this is a problem related to string encodings.

I think we should do the same also for `os.list_dir`
[Metric] Bertscore : Warning : Empty candidate sentence; Setting recall to be 0.
State: closed | Author: astariul (NONE)
Labels: name: metric bug

When running BERT-Score, I'm meeting this warning :

> Warning: Empty candidate sentence; Setting recall to be 0.

Code :

```
import nlp
metric = nlp.load_metric("bertscore")
scores = metric.compute(["swag", "swags"], ["swags", "totally something different"], lang="en", device=0)
```

---

**What am I doing wrong / How can I hide this warning ?**

Comments: This print statement comes from the official implementation of bert_score (see [here](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L343)). The warning shows up only if the attention mask outputs no candidate.
Right now we want to only use official code for metrics to have fair evaluations, so I'm not sure we can do anything about it. Maybe you can try to create an issue on their [repo](https://github.com/Tiiiger/bert_score) ?
Huggingface NLP, Uploading custom dataset
State: closed | Author: Nouman97 (NONE)

Hello,

Does anyone know how we can call our custom dataset using the nlp.load command? Let's say that I have a dataset based on the same format as that of squad-v1.1, how am I supposed to load it using huggingface nlp.

Thank you!

Comments: What do you mean 'custom' ? You may want to elaborate on it when ask a question.

Anyway, there are two things you may interested
`nlp.Dataset.from_file` and `load_dataset(..., cache_dir=)`

To load a dataset you need to have a script that defines the format of the examples, the splits and the way to generate examples. As your dataset has the same format of squad, you can just copy the squad script (see the [datasets](https://github.com/huggingface/nlp/tree/master/datasets) forlder) and just replace the url to load the data to your local or remote path.

Then what you can do is `load_dataset(<path/to/your/script>)`

Also if you want to upload your script, you should be able to use the `nlp-cli`.

Unfortunately the upload feature was not shipped in the latest version 0.2.0. so right now you can either clone the repo to use it or wait for the next release. We will add some docs to explain how to upload datasets.


Since the latest release 0.2.1 you can use 
```bash
nlp-cli upload_dataset <path/to/dataset>
```
where `<path/to/dataset>` is a path to a folder containing your script (ex: `squad.py`).
This will upload the script under your namespace on our S3.

Optionally the folder can also contain `dataset_infos.json` generated using
```bash
nlp-cli test <path/to/dataset> --all_configs --save_infos
```

Then you should be able to do
```python
nlp.load_dataset("my_namespace/dataset_name")
```
Nlp cli fix endpoints
State: closed | Author: lhoestq (MEMBER)

With this PR users will be able to upload their own datasets and metrics.

As mentioned in #181, I had to use the new endpoints and revert the use of dataclasses (just in case we have changes in the API in the future).

We now distinguish commands for datasets and commands for metrics:
```bash
nlp-cli upload_dataset <path/to/dataset>
nlp-cli upload_metric <path/to/metric>
nlp-cli s3_datasets {rm, ls}
nlp-cli s3_metrics {rm, ls}
```

Does it sound good to you @julien-c @thomwolf ?

Comments: LGTM 👍
Rename dataset_infos.json to dataset_info.json
State: closed | Author: aswin-giridhar (NONE)

As the file required for the viewing in the live nlp viewer is named as  dataset_info.json

Comments: 
This was actually the right name. `dataset_infos.json` is used to have the infos of all the dataset configurations.

On the other hand `dataset_info.json` (without 's') is a cache file with the info of one specific configuration.

To fix #228, we probably just have to clear and reload the nlp-viewer cache.