SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: documentation missing how to split a dataset

Comment:
Thanks a lot, the new explanation is very helpful!

About using train_test_split from sklearn: I stumbled across the [same error message as this user ](https://github.com/huggingface/nlp/issues/147 )and thought it can't be used at the moment in this context. Will check it out again.

One of the problems is how to shuffle very large datasets, which don't fit into the memory. Well, one strategy could be shuffling data in sections. But in a case where the data is sorted by the labels you have to swap larger sections first.
Issue: documentation missing how to split a dataset

Comment:
We added a way to shuffle datasets (shuffle the indices and then reorder to make a new dataset).
You can do `shuffled_dset = dataset.shuffle(seed=my_seed)`. It shuffles the whole dataset.
There is also `dataset.train_test_split()` which if very handy (with the same signature as sklearn).

Closing this issue as we added the docs for splits and tools to split datasets. Thanks again for your feedback !
Issue: Why is dataset after tokenization far more larger than the orginal one ?

Comment:
Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map`
Issue: Why is dataset after tokenization far more larger than the orginal one ?

Comment:
Hi ! Thanks for your reply.

But since size of `input_ids` < size of `text`, I am wondering why
size of `input_ids` + `text` > 2x the size of `text`  🤔
Issue: Why is dataset after tokenization far more larger than the orginal one ?

Comment:
Hard to tell... This is probably related to the way apache arrow compresses lists of integers, that may be different from the compression of strings.
Issue: Why is dataset after tokenization far more larger than the orginal one ?

Comment:
Thanks for your point. 😀, It might be answer.
Since this is hard to know, I'll close this issue.
But if somebody knows more details, please comment below ~ 😁
Issue: Tokenizer pickling issue fix not landed in `nlp` yet?

Comment:
Yes, the new release of tokenizers solves this and should be out soon.
In the meantime, you can install it with `pip install tokenizers==0.8.0-dev2`
Issue: Tokenizer pickling issue fix not landed in `nlp` yet?

Comment:
If others run into this issue, a quick fix is to use python 3.6 instead of 3.7+. Serialization differences between the 3rd party `dataclasses` package for 3.6 and the built in `dataclasses` in 3.7+ cause the issue.

Probably a dumb fix, but it works for me.
Issue: [Feature request] Add a feature to dataset

Comment:
Do you have an example of what you would like to do? (you can just add a field in the output of the unction you give to map and this will add this field in the output table)
Issue: [Feature request] Add a feature to dataset

Comment:
Given another source of data loaded in, I want to pre-add it to the dataset so that it aligns with the indices of the arrow dataset prior to performing map.

E.g. 
```
new_info = list of length dataset['train']

dataset['train'] = dataset['train'].map(lambda x: some_function(x, new_info[index of x]))

def some_function(x, new_info_x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x
    return x
```
I was thinking to instead create a new field in the arrow dataset so that instance x contains all the necessary information when map function is applied (since I don't have index information to pass to map function).
Issue: [Feature request] Add a feature to dataset

Comment:
This is what I have so far: 

```
import pyarrow as pa
from nlp.arrow_dataset import Dataset

aug_dataset = dataset['train'][:]
aug_dataset['new_info'] = new_info

#reformat as arrow-table
schema = dataset['train'].schema

# this line doesn't work:
schema.append(pa.field('new_info', pa.int32()))

table = pa.Table.from_pydict(
    aug_dataset,
    schema=schema
)
dataset['train'] = Dataset(table) 
```
Issue: [Feature request] Add a feature to dataset

Comment:
Maybe you can use `with_indices`?

```python
new_info = list of length dataset['train']

def some_function(indice, x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x[indice]
    return x

dataset['train'] = dataset['train'].map(some_function, with_indices=True)
```
Issue: [Feature request] Be able to remove a specific sample of the dataset

Comment:
Oh yes you can now do that with the `dataset.filter()` method that was added in #214
Issue: NonMatchingSplitsSizesError error when reading the IMDB dataset

Comment:
I just tried on my side and I didn't encounter your problem.
Apparently the script doesn't generate all the examples on your side.

Can you provide the version of `nlp` you're using ?
Can you try to clear your cache and re-run the code ?
Issue: NonMatchingSplitsSizesError error when reading the IMDB dataset

Comment:
Hi ! The cache is at ~/.cache/huggingface
You can just delete this folder if needed :)
Issue: [Dataset created] some critical small issues when I was creating a dataset

Comment:
Alright I think I fixed all the problems you mentioned. Thanks again, that will be useful for many people.
There is still more work needed for point 7. but we plan to have some nice docs soon.
Issue: What is the best way to cache a dataset? 

Comment:
Everything is already cached by default in 🤗nlp (in particular dataset
loading and all the “map()” operations) so I don’t think you need to do any
specific caching in streamlit.

Tell us if you feel like it’s not the case.

On Sat, 6 Jun 2020 at 13:02, Fabrizio Milo <notifications@github.com> wrote:

> For example if I want to use streamlit with a nlp dataset:
>
> @st.cache
> def load_data():
>     return nlp.load_dataset('squad')
>
> This code raises the error "uncachable object"
>
> Right now I just fixed with a constant for my specific case:
>
>     @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
>
> But I was curious to know what is the best way in general
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/nlp/issues/246>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHKAKO7CWGX2QY55UXLRVIO3ZANCNFSM4NV333RQ>
> .
>
Issue: SST-2 test labels are all -1

Comment:
Yes, this is because the test sets for glue are hidden so the labels are
not publicly available. You can read the glue paper for more details.

On Sat, 6 Jun 2020 at 18:16, Jack Morris <notifications@github.com> wrote:

> this also happened to me with nlp.load_datasets('glue', 'mnli')
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/nlp/issues/245#issuecomment-640083980>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHMVQD2EDX2HTZUXG5DRVJTWRANCNFSM4NVG3AKQ>
> .
>
Issue: SST-2 test labels are all -1

Comment:
It should be in the datasets card (the README.md and on the hub) in my opinion. What do you think @yjernite?
Issue: SST-2 test labels are all -1

Comment:
I checked both before I got to looking at issues, so that would be fine as well.

Some additional thoughts on this: Is there a specific reason why the "test" split even has a "label" column if it isn't tagged. Shouldn't there just not be any. Seems more transparent
Issue: SST-2 test labels are all -1

Comment:
I'm a little confused with the data size.
`sst2` dataset is referenced to `Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank` and the link of the dataset in the paper is https://nlp.stanford.edu/sentiment/index.html which is often shown in GLUE/SST2 reference.
From the original data, the standard train/dev/test splits split is 6920/872/1821 for binary classification. 
Why in GLUE/SST2 the train/dev/test split is 67,349/872/1,821 ?
Issue: SST-2 test labels are all -1

Comment:
> I'm a little confused with the data size.
> `sst2` dataset is referenced to `Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank` and the link of the dataset in the paper is https://nlp.stanford.edu/sentiment/index.html which is often shown in GLUE/SST2 reference.
> From the original data, the standard train/dev/test splits split is 6920/872/1821 for binary classification.
> Why in GLUE/SST2 the train/dev/test split is 67,349/872/1,821 ?

Have you figured out this problem? AFAIK, the original sst-2 dataset is totally different from the GLUE/sst-2. Do you think so?
Issue: SST-2 test labels are all -1

Comment:
@yc1999 Sorry, I didn't solve this conflict. In the end, I just use a local data file provided by the previous work I followed(for consistent comparison), not use `datasets` package.

Related information: https://github.com/thunlp/OpenAttack/issues/146#issuecomment-766323571
Issue: SST-2 test labels are all -1

Comment:
@yc1999 I find that the original SST-2 dataset (6,920/872/1,821) can be loaded from https://huggingface.co/datasets/gpt3mix/sst2 or built with SST data and the scripts in https://github.com/prrao87/fine-grained-sentiment/tree/master/data/sst.
The GLUE/SST-2 dataset (67,349/872/1,821) should be a completely different version.
Issue: UnicodeDecodeError when downloading GLUE-MNLI

Comment:
It should be good now, thanks for noticing and fixing it ! I would say that it was because you are on windows but not 100% sure
Issue: UnicodeDecodeError when downloading GLUE-MNLI

Comment:
On Windows Python supports Unicode almost everywhere, but one of the notable exceptions is open() where it uses the locale encoding schema. So platform independent python scripts would always set the encoding='utf-8' in calls to open explicitly. 
In the meantime: since Python 3.7 Windows users can set the default encoding for everything including open() to Unicode by setting this environment variable: set PYTHONUTF8=1 (details can be found in [PEP 540](https://www.python.org/dev/peps/pep-0540/))

For me this fixed the problem described by the OP.
Issue: Deterministic dataset loading

Comment:
I think using `sorted(glob.glob())` would actually solve this problem. Can you think of other reasons why dataset loading might not be deterministic? @mariamabarham @yjernite @lhoestq @thomwolf . 

I can do a sweep through the dataset scripts and fix the glob.glob() if you guys are ok with it
Issue: Deterministic dataset loading

Comment:
I'm pretty sure it would solve the problem too.

The only other dataset that is not deterministic right now is `blog_authorship_corpus` (see #215) but this is a problem related to string encodings.
Issue: [Creating new dataset] Not found dataset_info.json

Comment:
@lhoestq - this seems to happen quite often (already the 2nd issue). Can we maybe delete this automatically?
Issue: [Creating new dataset] Not found dataset_info.json

Comment:
Hi, I rebase my local copy to `fix-empty-cache-dir`, and try to run again `python nlp-cli test datasets/bookcorpus --save_infos --all_configs`.

I got this,  
```
Traceback (most recent call last):
  File "nlp-cli", line 10, in <module>
    from nlp.commands.run_beam import RunBeamCommand
  File "/home/yisiang/nlp/src/nlp/commands/run_beam.py", line 6, in <module>
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```
And after I installed it. I got this
```
File "/home/yisiang/nlp/src/nlp/datasets/bookcorpus/aea0bd5142d26df645a8fce23d6110bb95ecb81772bb2a1f29012e329191962c/bookcorpus.py", line 88, in _split_generators
    downloaded_path_or_paths = dl_manager.download_custom(_GDRIVE_FILE_ID, download_file_from_google_drive)
  File "/home/yisiang/nlp/src/nlp/utils/download_manager.py", line 128, in download_custom
    downloaded_path_or_paths = map_nested(url_to_downloaded_path, url_or_urls)
  File "/home/yisiang/nlp/src/nlp/utils/py_utils.py", line 172, in map_nested
    return function(data_struct)
  File "/home/yisiang/nlp/src/nlp/utils/download_manager.py", line 126, in url_to_downloaded_path
    return os.path.join(self._download_config.cache_dir, hash_url_to_filename(url))
  File "/home/yisiang/miniconda3/envs/nlppr/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
```
The problem is when I print `self._download_config.cache_dir` using pdb, it is `None`.

Did I miss something ?  Or can you provide a workaround first so I can keep testing my script ?
Issue: [Metric] Bertscore : Warning : Empty candidate sentence; Setting recall to be 0.

Comment:
This print statement comes from the official implementation of bert_score (see [here](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L343)). The warning shows up only if the attention mask outputs no candidate.
Right now we want to only use official code for metrics to have fair evaluations, so I'm not sure we can do anything about it. Maybe you can try to create an issue on their [repo](https://github.com/Tiiiger/bert_score) ?
Issue: Can't download MultiNLI

Comment:
Thanks! I thought I had to use the same code displayed in the live viewer:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('multi_nli', 'plain_text')
```
Your suggestion works, even if then I got a different issue (#242).
Issue: Can't download MultiNLI

Comment:
Glad it helps !
Though I am not one of hf team, but maybe you should close this issue first.
Issue: Huggingface NLP, Uploading custom dataset

Comment:
What do you mean 'custom' ? You may want to elaborate on it when ask a question.

Anyway, there are two things you may interested
`nlp.Dataset.from_file` and `load_dataset(..., cache_dir=)`
Issue: Huggingface NLP, Uploading custom dataset

Comment:
To load a dataset you need to have a script that defines the format of the examples, the splits and the way to generate examples. As your dataset has the same format of squad, you can just copy the squad script (see the [datasets](https://github.com/huggingface/nlp/tree/master/datasets) forlder) and just replace the url to load the data to your local or remote path.

Then what you can do is `load_dataset(<path/to/your/script>)`
Issue: Huggingface NLP, Uploading custom dataset

Comment:
Also if you want to upload your script, you should be able to use the `nlp-cli`.

Unfortunately the upload feature was not shipped in the latest version 0.2.0. so right now you can either clone the repo to use it or wait for the next release. We will add some docs to explain how to upload datasets.
Issue: Huggingface NLP, Uploading custom dataset

Comment:
Since the latest release 0.2.1 you can use 
```bash
nlp-cli upload_dataset <path/to/dataset>
```
where `<path/to/dataset>` is a path to a folder containing your script (ex: `squad.py`).
This will upload the script under your namespace on our S3.

Optionally the folder can also contain `dataset_infos.json` generated using
```bash
nlp-cli test <path/to/dataset> --all_configs --save_infos
```

Then you should be able to do
```python
nlp.load_dataset("my_namespace/dataset_name")
```
Issue: Fail to download c4 english corpus

Comment:
Hello ! Thanks for noticing this bug, let me fix that.

Also for information, as specified in the changelog of the latest release, C4 currently needs to have a runtime for apache beam to work on. Apache beam is used to process this very big dataset and it can work on dataflow, spark, flink, apex, etc. You can find more info on beam datasets [here](https://github.com/huggingface/nlp/blob/master/docs/beam_dataset.md).

Our goal in the future is to make available an already-processed version of C4 (as we do for wikipedia for example) so that users without apache beam runtimes can load it.
Issue: Not able to access the XNLI dataset

Comment:
Thanks for reporting this bug !
As it seems to be just a cache problem, I closed your PR.
I think we might just need to clear and reload the `xnli` cache @srush ?
Issue: Not able to access the XNLI dataset

Comment:
Update: The dataset_info.json error is gone, but we have a new one instead:
```
ConnectionError: Couldn't reach https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip
```
I am not able to reproduce on my side unfortunately. Any idea @srush ?
Issue: [ROUGE] Different scores with `files2rouge`

Comment:
@Colanim unfortunately there are different implementations of the ROUGE metric floating around online which yield different results, and we had to chose one for the package :) We ended up including the one from the google-research repository, which does minimal post-processing before computing the P/R/F scores. If I recall correctly, files2rouge relies on the Perl, script, which among other things normalizes all numbers to a special token: in the case you presented, this should account for a good chunk of the difference.

We may end up adding in more versions of the metric, but probably not for a while (@lhoestq correct me if I'm wrong). However, feel free to take a stab at adding it in yourself and submitting a PR if you're interested!
Issue: [ROUGE] Different scores with `files2rouge`

Comment:
Thank you for your kind answer.

As a side question : Isn't it better to have a package that normalize more ?

I understand to idea of having a package that does minimal post-processing for transparency.

But it means that people using different architecture (with different tokenizers for example) will have difference in ROUGE scores even if their predictions are actually similar.  
The goal of `nlp` is to have _one package to rule them all_, right ?

I will look into it but I'm not sure I have the required skill for this ^^
Issue: [ROUGE] Different scores with `files2rouge`

Comment:
You're right, there's a pretty interesting trade-off here between robustness and sensitivity :) The flip side of your argument is that we also still want the metric to be sensitive to model mistakes. How we think about number normalization for example has evolved a fair bit since the Perl script was written: at the time, ROUGE was used mostly to evaluate short-medium text summarization systems, where there were only a few numbers in the input and it was assumed that the most popular methods in use at the time would get those right. However, as your example showcases, that assumption does not hold any more, and we do want to be able to penalize a model that generates a wrong numerical value.

Also, we think that abstracting away tokenization differences is the role of the model/tokenizer: if you use the 🤗Tokenizers library for example, it will handle that for you ;)

Finally, there is a lot of active research on developing model-powered metrics that are both more sensitive and more robust than ROUGE. Check out for example BERTscore, which is implemented in this library!
Issue: [Feature Request/Help] BLEURT model -> PyTorch

Comment:
Hitting this error when using bleurt with PyTorch ...

```
UnrecognizedFlagError: Unknown command line flag 'f'
```
... and I'm assuming because it was built for TF specifically.  Is there a way to use this metric in PyTorch?
Issue: [Feature Request/Help] BLEURT model -> PyTorch

Comment:
We currently provide a wrapper on the TensorFlow implementation: https://huggingface.co/metrics/bleurt

We have long term plans to better handle model-based metrics, but they probably won't be implemented right away

@adamwlev it would still be cool to add the BLEURT checkpoints to the transformers repo if you're interested, but that would best be discussed there :) 

closing for now
Issue: [Feature Request/Help] BLEURT model -> PyTorch

Comment:
Hi there. We ran into the same problem this year (converting BLEURT to PyTorch) and thanks to @adamwlev found his colab notebook which didn't work but served as a good starting point. Finally, we **made it work** by doing just two simple conceptual fixes: 

1. Transposing 'kernel' layers instead of 'dense' ones when copying params from the original model;
2. Taking pooler_output as a cls_state in forward function of the BleurtModel class.

Plus few minor syntactical fixes for the outdated parts. The result is still not exactly the same, but is very close to the expected one (1.0483 vs 1.0474).

Find the fixed version here (fixes are commented): https://colab.research.google.com/drive/1KsCUkFW45d5_ROSv2aHtXgeBa2Z98r03?usp=sharing
Issue: [Feature Request/Help] BLEURT model -> PyTorch

Comment:
I created a new model based on `transformers` that can load every BLEURT checkpoints released so far. https://github.com/lucadiliello/bleurt-pytorch
Issue: Colab Notebook breaks when downloading the squad dataset

Comment:
The notebook forces version 0.1.0. If I use the latest, things work, I'll run the whole notebook and create a PR.

But in the meantime, this issue gets fixed by changing:
`!pip install nlp==0.1.0`
to
`!pip install nlp`
Issue: Colab Notebook breaks when downloading the squad dataset

Comment:
When you install `nlp` for the first time on a Colab runtime, it updates the `pyarrow` library that was already on colab. This update shows this message on colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```
You just have to restart the runtime and it should be fine.
If you don't restart, then it breaks like in your first message
Issue: Colab Notebook breaks when downloading the squad dataset

Comment:
Thanks for reporting the second one ! We'll update the notebook to fix this one :)
Issue: Colab Notebook breaks when downloading the squad dataset

Comment:
This trick from @thomwolf seems to be the most reliable solution to fix this colab notebook issue:

```python
# install nlp
!pip install -qq nlp==0.2.0

# Make sure that we have a recent version of pyarrow in the session before we continue - otherwise reboot Colab to activate it
import pyarrow
if int(pyarrow.__version__.split('.')[1]) < 16:
    import os
    os.kill(os.getpid(), 9)
```
Issue: Multi-task dataset mixing

Comment:
I like this feature! I think the first question we should decide on is how to convert all datasets into the same format. In T5, the authors decided to format every dataset into a text-to-text format. If the dataset had "multiple" inputs like MNLI, the inputs were concatenated. So in MNLI the input:

> - **Hypothesis**: The St. Louis Cardinals have always won.
> 
> - **Premise**: yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals when they were there were uh a mostly a losing team but 

was flattened to a single input:

> mnli hypothesis: The St. Louis Cardinals have always won. premise:
> yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals
> when they were there were uh a mostly a losing team but.

This flattening is actually a very simple operation in `nlp` already. You would just need to do the following:

```python 
def flatten_inputs(example):
    return {"input": "mnli hypothesis: " + example['hypothesis'] + " premise: " + example['premise']}

t5_ready_mnli_ds = mnli_ds.map(flatten_inputs, remove_columns=[<all columns except output>])
```

So I guess converting the datasets into the same format can be left to the user for now. 
Then the question is how we can merge the datasets. I would probably be in favor of a simple 

```python 
dataset.add()
```

function that checks if the dataset is of the same format and if yes merges the two datasets. Finally, how should the sampling be implemented? **Examples-proportional mixing** corresponds to just merging the datasets and shuffling. For the other two sampling approaches we would need some higher-level features, maybe even a `dataset.sample()` function for merged datasets. 

What are your thoughts on this @thomwolf @lhoestq  @ghomasHudson @enzoampil ?