SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
Yes, deleting the directory solves the error whenever I try to rerun.

By replacing the json-loader, you mean the cached file in my `site-packages` directory? e.g. `/home/XXX/.cache/lib/python3.7/site-packages/nlp/datasets/json/(...)/json.py` 

When I was testing this out before the #372 PR was merged I had issues installing it properly locally. Since the `json.py` script was downloaded instead of actually using the one provided in the local install. Manually updating that file seemed to solve it, but it didn't seem like a proper solution. Especially when having to run this on a remote compute cluster with no access to that directory.
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
I see, diving in the JSON file for SQuAD it's a pretty complex structure.

The best solution for you, if you have a dataset really similar to SQuAD would be to copy and modify the SQuAD data processing script. We will probably add soon an option to be able to specify file path to use instead of the automatic URL encoded in the script but in the meantime you can:
- copy the [squad script](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py)  in a new script for your dataset
- in the new script replace [these `urls_to_download `](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py#L99-L102) by `urls_to_download=self.config.data_files`
- load the dataset with `dataset = load_dataset('path/to/your/new/script', data_files={nlp.Split.TRAIN: "./datasets/train-v2.0.json"})`

This way you can reuse all the processing logic of the SQuAD loading script.
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
This seems like a more sensible solution! Thanks, @thomwolf. It's been a little daunting to understand what these scripts actually do, due to the level of abstraction and central documentation.

Am I correct in assuming that the `_generate_examples()` function is the actual procedure for how the data is loaded from file? Meaning that essentially with a file containing another format, that is the only function that requires re-implementation? I'm working with a lot of datasets that, due to licensing and privacy, cannot be published. As this library is so neatly integrated with the transformers library and gives easy access to public sets such as SQUAD and increased performance, it is very neat to be able to load my private sets as well. As of now, I have just been working on scripts for translating all my data into the SQUAD-format before using the json script, but I see that it might not be necessary after all.
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
Yes `_generate_examples()` is the main entry point. If you change the shape of the returned dictionary you also need to update the `features` in the `_info`.

I'm currently writing the doc so it should be easier soon to use the library and know how to add your datasets.
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
Could you try to update pyarrow to >=0.17.0 @vegarab ?
I don't have any segmentation fault with my version of pyarrow (0.17.1)

I tested with
```python
import nlp
s = nlp.load_dataset("json", data_files="train-v2.0.json", field="data", split="train")
s[0]
# {'title': 'Normans', 'paragraphs': [{'qas': [{'question': 'In what country is Normandy located?', 'id':...
```
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
Also if you want to have your own dataset script, we now have a new documentation !
See here:
https://huggingface.co/nlp/add_dataset.html
Issue: Segmentation fault when loading local JSON dataset as of #372

Comment:
@lhoestq 
For some reason, I am not able to reproduce the segmentation fault, on pyarrow==0.16.0. Using the exact same environment and file.

Anyhow, I discovered that pyarrow>=0.17.0 is required to read in a JSON file where the pandas structs contain lists. Otherwise, pyarrow complains when attempting to cast the struct:
```py
import nlp
>>> s = nlp.load_dataset("json", data_files="datasets/train-v2.0.json", field="data", split="train")
Using custom data configuration default
>>> s[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 558, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 498, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pandas().to_dict("list"))
  File "pyarrow/array.pxi", line 559, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 766, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 1101, in _table_to_blocks
    list(extension_columns.keys()))
  File "pyarrow/table.pxi", line 881, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
>>> s
Dataset(schema: {'title': 'string', 'paragraphs': 'list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>'}, num_rows: 35)
```

Upgrading to >=0.17.0 provides the same dataset structure, but accessing the records is possible without the same exception.
Issue: can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries

Comment:
I am able to reproduce this with the official SQuAD `train-v2.0.json` file downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/
Issue: can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries

Comment:
I am facing this issue in transformers library 3.0.2 while reading a csv using datasets.
Is this fixed in latest version? 
I updated the latest version 4.0.1 but still getting this error. What could cause this error?
Issue: load_metric can't acquire lock anymore

Comment:
I found that, in the same process (or the same interactive session), if I do

import nlp

m1 = nlp.load_metric('glue', 'mrpc')
m2 = nlp.load_metric('glue', 'sst2')

I will get the same error `ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id'`.
Issue: How to augment data ?

Comment:
Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?
Issue: How to augment data ?

Comment:
Some samples in the dataset are too long, I want to divide them in several samples.
Issue: How to augment data ?

Comment:
Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.
Issue: How to augment data ?

Comment:
It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...
Issue: How to augment data ?

Comment:
Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq
Issue: How to augment data ?

Comment:
As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.
Issue: 🐛 [Metrics] ROUGE is non-deterministic

Comment:
> Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)
Issue: 🐛 [Metrics] ROUGE is non-deterministic

Comment:
> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
> 
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
> 
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
> 
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
> 
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_aggregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.
Issue: 🐛 [Metrics] ROUGE is non-deterministic

Comment:
Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```
Issue: [Feature request] Add dataset.ragged_map() function for many-to-many transformations

Comment:
Actually `map(batched=True)` can already change the size of the dataset.
It can accept examples of length `N` and returns a batch of length `M` (can be null or greater than `N`).

I'll make that explicit in the doc that I'm currently writing.
Issue: [Feature request] Add dataset.ragged_map() function for many-to-many transformations

Comment:
You're two steps ahead of me :) In my testing, it also works if `M` < `N`.

A batched map of different length seems to work if you directly overwrite all of the original keys, but fails if any of the original keys are preserved.

For example,
```python
# Create a dummy dataset
dset = load_dataset("wikitext", "wikitext-2-raw-v1")["test"]
dset = dset.map(lambda ex: {"length": len(ex["text"]), "foo": 1})

# Do an allreduce on each batch, overwriting both keys
dset.map(lambda batch: {"length": [sum(batch["length"])], "foo": [1]})
# Dataset(schema: {'length': 'int64', 'foo': 'int64'}, num_rows: 5)

# Now attempt an allreduce without touching the `foo` key
dset.map(lambda batch: {"length": [sum(batch["length"])]})
# This fails with the error message below
```

```bash
  File "/path/to/nlp/src/nlp/arrow_dataset.py", line 728, in map
    arrow_schema = pa.Table.from_pydict(test_output).schema
  File "pyarrow/io.pxi", line 1532, in pyarrow.lib.Codec.detect
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named foo expected length 1 but got length 2
```

Adding the `remove_columns=["length", "foo"]` argument to `map()` solves the issue. Leaving the above error for future visitors. Perfect, thank you!
Issue: ArrowBasedBuilder _prepare_split parse_schema breaks on nested structures

Comment:
Hi, it depends on what it is in your `dataset_builder.py` file. Can you share it?

If you are just loading `json` files, you can also directly use the `json` script (which will find the schema/features from your JSON structure):

```python
from nlp import load_dataset
ds = load_dataset("json", data_files=rel_datafiles)
```
Issue: ArrowBasedBuilder _prepare_split parse_schema breaks on nested structures

Comment:
The behavior I'm seeing is from the `json` script. 
I hacked this together to overcome the error with the `JSON` dataloader

```
class DatasetBuilder(hf_nlp.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = BuilderConfig

    def _info(self):
        return DatasetInfo()

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:
            if split_name in self.config.data_files:
                files = self.config.data_files[split_name]
                if isinstance(files, str):
                    files = [files]
                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))
        return splits

    def _prepare_split(self, split_generator):
        fname = "{}-{}.arrow".format(self.name, split_generator.name)
        fpath = os.path.join(self._cache_dir, fname)

        writer = ArrowWriter(path=fpath)

        generator = self._generate_tables(**split_generator.gen_kwargs)
        for key, table in utils.tqdm(generator, unit=" tables", leave=False):
            writer.write_table(table)
        num_examples, num_bytes = writer.finalize()

        split_generator.split_info.num_examples = num_examples
        split_generator.split_info.num_bytes = num_bytes
        # this is where the error is coming from
        # def parse_schema(schema, schema_dict):
        #     for field in schema:
        #         if pa.types.is_struct(field.type):
        #             schema_dict[field.name] = {}
        #             parse_schema(field.type, schema_dict[field.name])
        #         elif pa.types.is_list(field.type) and pa.types.is_struct(field.type.value_type):
        #             schema_dict[field.name] = {}
        #             parse_schema(field.type.value_type, schema_dict[field.name])
        #         else:
        #             schema_dict[field.name] = Value(str(field.type))
        # 
        # parse_schema(writer.schema, features)
        # self.info.features = Features(features)

    def _generate_tables(self, files):
        for i, file in enumerate(files):
            pa_table = paj.read_json(
                file
            )
            yield i, pa_table
```

So I basically just don't populate the `self.info.features` though this doesn't seem to cause any problems in my downstream applications. 

The other workaround I was doing was to just use pyarrow.json to build a table and then to create the Dataset with its constructor or from_table methods. `load_dataset` has nice split logic, so I'd prefer to use that.
Issue: ArrowBasedBuilder _prepare_split parse_schema breaks on nested structures

Comment:
Also noticed that if you for example in a loader script

```
from nlp import ArrowBasedBuilder

class MyBuilder(ArrowBasedBuilder):
...

```
and use that in the subclass, it will be on the module's __dict__ and will be selected before the `MyBuilder` subclass, and it will raise `NotImplementedError` on its `_generate_examples` method... In the code it check for abstract classes but Builder and ArrowBasedBuilder aren't abstract classes, they're regular classes with `@abstract_methods`.
Issue: ArrowBasedBuilder _prepare_split parse_schema breaks on nested structures

Comment:
Indeed this is part of a more general limitation which is the fact that we should generate and update the `features` from the auto-inferred Arrow schema when they are not provided (also happen when a user change the schema using `map()`, the features should be auto-generated and guessed as much as possible to keep the `features` synced with the underlying Arrow table schema).

We will try to solve this soon.
Issue: can't load SNLI dataset

Comment:
I just added the processed files of `snli` on our google storage, so that when you do `load_dataset` it can download the processed files from there :)

We are thinking about having available those processed files for more datasets in the future, because sometimes files aren't available (like for `snli`), or the download speed is too slow, or sometimes the files take time to be processed.
Issue: [Dataset requests] New datasets for Text Classification

Comment:
- `nlp` has MR! It's called `rotten_tomatoes`
- SST is part of GLUE, or is that just SST-2?
- `nlp` also has `ag_news`, a popular news classification dataset

I'd also like to see:
- the Yahoo Answers topic classification dataset
- the Kaggle Fake News classification dataset
Issue: [Dataset requests] New datasets for Text Classification

Comment:
Thanks @jxmorris12 for pointing this out. 

In glue we only have SST-2 maybe we can add separately SST-1.
Issue: [Dataset requests] New datasets for Text Classification

Comment:
This is the homepage for the Amazon dataset: https://www.kaggle.com/datafiniti/consumer-reviews-of-amazon-products

Is there an easy way to download kaggle datasets programmatically? If so, I can add this one!
Issue: [Dataset requests] New datasets for Text Classification

Comment:
Hi @jxmorris12 for now I think our `dl_manager` does not download from Kaggle.
@thomwolf , @lhoestq
Issue: [Dataset requests] New datasets for Text Classification

Comment:
Great list. Any idea if Amazon Reviews has been added?

- ~40 GB of text (sadly no emoji)
- popular MLM pre-training dataset before bigger datasets like WebText https://arxiv.org/abs/1808.01371
- turns out that binarizing the 1-5 star rating leads to great Pos/Neg/Neutral dataset, T5 paper claims to get very high accuracy (98%!) on this with small amount of finetuning https://arxiv.org/abs/2004.14546

Apologies if it's been included (great to see where) and if not, it's one of the better medium/large NLP dataset for semi-supervised learning, albeit a bit out of date. 

Thanks!! 

cc @sshleifer
Issue: [Dataset requests] New datasets for Text Classification

Comment:
On the Amazon Reviews dataset, the original UCSD website has noted these are now updated to include product reviews through 2018 -- actually quite recent compared to many other datasets. Almost certainly the largest NLP dataset out there with labels!
https://jmcauley.ucsd.edu/data/amazon/ 

Any chance someone has time to onboard this dataset in a HF way?

cc @sshleifer
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
See issues #242 and #307
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
It should be in `xtreme.py:L755`:
```python
        if self.config.name == "tydiqa" or self.config.name.startswith("MLQA") or self.config.name == "SQuAD":
            with open(filepath) as f:
                data = json.load(f)
```

Could you try to add the encoding parameter:
```python
open(filepath, encoding='utf-8')
```
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
Hello @jerryIsHere :) Did it work ?
If so we may change the dataset script to force the utf-8 encoding
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
@lhoestq sorry for being that late, I found 4 copy of xtreme.py. I did the changes as what has been told to all of them.
The problem is not solved
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
Could you provide a better error message so that we can make sure it comes from the opening of the `tydiqa`'s json files ?
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
@lhoestq 
The error message is same as before:
Exception has occurred: UnicodeDecodeError
'cp950' codec can't decode byte 0xe2 in position 111: illegal multibyte sequence
  File "D:\python\test\test.py", line 3, in <module>
    dataset = load_dataset('xtreme', 'tydiqa')

![image](https://user-images.githubusercontent.com/50871412/87748794-7c216880-c829-11ea-94f0-7caeacb4d865.png)

I said that I found 4 copy of xtreme.py and add the 「, encoding='utf-8'」 parameter to the open() function
these python script was found under this directory
C:\Users\USER\AppData\Local\Programs\Python\Python37\Lib\site-packages\nlp\datasets\xtreme
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
Hi there !
I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
I added ```encoding='UTF-8'``` to both lines that have ```open``` in ```imdb.py```  (108 and 114) and it worked for me.
Thank you !
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
> Hi there !
> I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
> I added `encoding='UTF-8'` to both lines that have `open` in `imdb.py` (108 and 114) and it worked for me.
> Thank you !

Hello !
Glad you managed to fix this issue on your side.
Do you mind opening a PR for IMDB ?
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
> This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
> Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
> See issues #242 and #307

Sorry for not responding for about a month.
I have just found that it is necessary to change / add the environment variable as what was told in #242.
Everything works after I add the new environment variable and restart my PC.

I think the encoding issue for windows isn't limited to the open() function call specific to few dataset, but actually in the entire library, depends on the machine / os you use.
Issue: 'cp950' codec error from load_dataset('xtreme', 'tydiqa')

Comment:
Since #481 we shouldn't have other issues with encodings as they need to be set to "utf-8" be default.

Closing this one, but feel free to re-open if you gave other questions
Issue: Supporting documents in ELI5

Comment:
Hi @saverymax ! For licensing reasons, the original team was unable to release pre-processed CommonCrawl documents. Instead, they provided a script to re-create them from a CommonCrawl dump, but it unfortunately requires access to a medium-large size cluster:
https://github.com/facebookresearch/ELI5#downloading-support-documents-from-the-commoncrawl

In order to make the task accessible to people who may not have access to this kind of infrastructure, we suggest to use Wikipedia as a knowledge source rather than the full CommonCrawl. The following blog post shows how you can create Wikipedia support documents and get a performance that is on par with a system that uses CommonCrawl pages.
https://yjernite.github.io/lfqa.html#task_description

Hope that helps, using ElasticSearch to index Wiki40b and create the documents should take about 4 hours. Let us know if you have any trouble with the blog post though!