SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Loading wikipedia 20200501.en throws pyarrow related error

Comment:
> Hi ! It looks like the arrow file in the folder
> `/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.
> 
> Can you take a look and check that it's 18.3GB ?
> 
> If not, then maybe you need to redownload it:
> 
> ```python
> from datasets import load_dataset
> ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
> ```

Hi Ihoestq, thanks for the reply! Actually i think my issue is i couldn't download the dataset beyond 10.7G. It feels like the whole dataset is split into different volumes and after the first one was downloaded it crashed before proceeding to the next one. I did try 'force_redownload' mode but still got the same issue.
Issue: Loading wikipedia 20200501.en throws pyarrow related error

Comment:
I just tried on my side and got no issues.
When downloading the dataset again, did it crash at 10.7GB as well ?
Issue: Loading wikipedia 20200501.en throws pyarrow related error

Comment:
> I just tried on my side and got no issues.
> When downloading the dataset again, did it crash at 10.7GB as well ?

Yes i have tried it multiple times on different machines. I am wondering if you could share the screenshot of your dependency versions and i will try to make them the same as yours?
Issue: Loading wikipedia 20200501.en throws pyarrow related error

Comment:
I tried using `datasets` from `master` on macos with python 3.7.2
I also have `requests==2.23.0` and `tqdm==4.45.0`.
Issue: TypeError when using save_to_disk in a dataset loaded with ReadInstruction split

Comment:
Hi !
I think this has been fixed recently on `master`.
Can you try again by installing `datasets` from `master` ?
```
pip install git+https://github.com/huggingface/datasets.git
```
Issue: en language data from MLQA dataset is missing

Comment:
Hi ! Indeed only the languages of the `translate-train` data are included...
I can't find a link to download the english train set on https://github.com/facebookresearch/MLQA though, do you know where we can download it ?
Issue: en language data from MLQA dataset is missing

Comment:
Hi @lhoestq 
thank you very much for coming back to me, now I see, you are right, in the link you sent I see split of  {split}-context-{context_language}-question-{question_language}.json with context_language=question_language=en, TFDS most probably has extracted english ones from these files as en language files, but translate-train/test do not have en indeed. thanks a lot for the great explanations
Issue: en language data from MLQA dataset is missing

Comment:
I close the ticket, since I do not see any en existing, they have trained on "SQuAD V1.1" instead. Thanks.
Issue: Saving large in-memory datasets with save_to_disk crashes because of pickling

Comment:
Hi !
Indeed `save_to_disk` doesn't call pickle anymore. Though the `OverflowError` can still appear for in-memory datasets bigger than 4GB. This happens when doing this for example:
```python
import pyarrow as pa
import pickle

arr = pa.array([0] * ((4 * 8 << 30) // 64))
table = pa.Table.from_arrays([a], names=["foo"])
pickle.dumps(table)  # fails with an OverflowError
pickle.dumps(table, 4)  # works !
```
We'll do the change to use `protocol=4`.

Moreover I've also seen other users complain about this error
```
struct.error: 'I' format requires 0 <= number <= 4294967295
```

It looks like something related to the 4GB limit as well but I'm not able to reproduce on my side.
Do you think you can provide a script that reproduces the issue ?
How big is your dataset ? (number of bytes, number of rows)
Issue: Saving large in-memory datasets with save_to_disk crashes because of pickling

Comment:
I just merged a fix #2150 that allows to pickle tables bigger than 4GiB
Feel free to try it on the `master` branch !
Issue: Saving large in-memory datasets with save_to_disk crashes because of pickling

Comment:
awesome! I started getting this error as well when I tried to tokenize with a longer sequence length
Issue: Saving large in-memory datasets with save_to_disk crashes because of pickling

Comment:
@prokopCerny does this fix work for you? I found that with the latest master, my container with 500GB RAM starts crashing when I try to map a large dataset using `num_proc`.

@lhoestq would it be possible to implement some logic to keep the individual cache files small (say below 100mb)? I find this helps with loading large datasets, but the "hack" I was using (increasing `num_proc` to a large number) doesn't work anymore with the latest master; my container crashes even with `num_proc=200` now
Issue: Saving large in-memory datasets with save_to_disk crashes because of pickling

Comment:
Closing since the original issue was fixed in #2150 
Feel free to reopen if you are still experiencing it.
For the other problems, please open separate issues
Issue: bug in mlqa dataset 

Comment:
If you print those questions, you get readable texts:
```python
>>> questions = [
...     "\u0645\u062a\u0649 \u0628\u062f\u0627\u062a \u0627\u0644\u0645\u062c\u0644\u0629 \u0627\u0644\u0645\u062f\u0631\u0633\u064a\u0629 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645 \u0628\u0627\u0644\u0646\u0634\u0631?",
...     "\u0643\u0645 \u0645\u0631\u0629 \u064a\u062a\u0645 \u0646\u0634\u0631\u0647\u0627 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
...     "\u0645\u0627 \u0647\u064a \u0627\u0644\u0648\u0631\u0642\u0629 \u0627\u0644\u064a\u0648\u0645\u064a\u0629 \u0644\u0644\u0637\u0644\u0627\u0628 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
...     "\u0643\u0645 \u0639\u062f\u062f \u0627\u0644\u0627\u0648\u0631\u0627\u0642 \u0627\u0644\u0627\u062e\u0628\u0627\u0631\u064a\u0629 \u0644\u0644\u0637\u0644\u0627\u0628 \u0627\u0644\u062a\u064a \u0648\u062c\u062f\u062a \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
...     "\u0641\u064a \u0627\u064a \u0633\u0646\u0629 \u0628\u062f\u0627\u062a \u0648\u0631\u0642\u0629 \u0627\u0644\u0637\u0627\u0644\u0628 \u0627\u0644\u062d\u0633 \u0627\u0644\u0633\u0644\u064a\u0645 \u0628\u0627\u0644\u0646\u0634\u0631 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?"
... ]
>>> print(questions)
['متى بدات المجلة المدرسية في نوتردام بالنشر?', 'كم مرة يتم نشرها في نوتردام?', 'ما هي الورقة اليومية للطلاب في نوتردام?', 'كم عدد الاوراق الاخبارية للطلاب التي وجدت في نوتردام?', 'في اي سنة بدات ورقة الطالب الحس السليم بالنشر في نوتردام?']
```
I don't think we can change this
Issue: bug in mlqa dataset 

Comment:
Hi @dorost1234.

In Python 3, strings are sequences of Unicode _code points_. Unicode is a specification that maps all characters (and emoji symbols) with its unique representation in terms of code points. That is what you see: Unicode code points (represented by a \u escaped sequence of 16-bit hex values).

Characters are usually represented (on screen and papers) with a graphical element called _glyph_. That is what you would like to see: glyphs. But Python does not care about glyphs: that is the job of the GUI or the terminal; glyphs are what you get with the `print` function (if your terminal is properly configured to display those glyphs).

You have more detailed information about Unicode in the Python documentation: https://docs.python.org/3/howto/unicode.html
Issue: TydiQA dataset is mixed and is not split per language 

Comment:
You can filter the languages this way:
```python
tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
```

Otherwise maybe we can have one configuration per language ?
What do you think of this for example ?

```python
load_dataset("tydiqa", "primary_task.en")
```
Issue: TydiQA dataset is mixed and is not split per language 

Comment:
Hi
thank you very much for the great response, this will be really wonderful
to have one configuration per language, as one need the dataset in majority
of case per language for cross-lingual evaluations.
This becomes also then more close to TFDS format, which is separated per
language https://www.tensorflow.org/datasets/catalog/tydi_qa which will be
really awesome to have.
thanks

On Mon, Mar 29, 2021 at 6:17 PM Quentin Lhoest ***@***.***>
wrote:

> You can filter the languages this way:
>
> tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
>
> Otherwise maybe we can have one configuration per language ?
> What do you think of this for example ?
>
> load_dataset("tydiqa", "primary_task.en")
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2132#issuecomment-809516799>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXPW2PWSQ2RHG73O7TTGCY4LANCNFSM4Z7ER7IA>
> .
>
Issue: When training with Multi-Node Multi-GPU the worker 2 has TypeError: 'NoneType' object

Comment:
Hi ! Thanks for reporting
I was able to reproduce this issue. This was caused by missing split infos if a worker reloads the cache of the other worker.

I just opened https://github.com/huggingface/datasets/pull/2137 to fix this issue
Issue: wikiann dataset is missing columns 

Comment:
Here please find TFDS format of this dataset: https://www.tensorflow.org/datasets/catalog/wikiann
where there is a span column, this is really necessary to be able to use the data, and I appreciate your help @lhoestq
Issue: wikiann dataset is missing columns 

Comment:
Hi !
Apparently you can get the spans from the NER tags using `tags_to_spans` defined here:

https://github.com/tensorflow/datasets/blob/c7096bd38e86ed240b8b2c11ecab9893715a7d55/tensorflow_datasets/text/wikiann/wikiann.py#L81-L126

It would be nice to include the `spans` field in this dataset as in TFDS. This could be a good first issue for new contributors !

The objective is to use `tags_to_spans` in the `_generate_examples` method [here](https://github.com/huggingface/nlp/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L292-L316) to create he `spans` for each example.
Issue: wikiann dataset is missing columns 

Comment:
Hi @lhoestq 
thank you very much for the help, it would be very nice to have it included, here is the full code, one need to also convert tags to string first:

```
import datasets 
from datasets import load_dataset

def tags_to_spans(tags):
  """Convert tags to spans."""
  spans = set()
  span_start = 0
  span_end = 0
  active_conll_tag = None
  for index, string_tag in enumerate(tags):
    # Actual BIO tag.
    bio_tag = string_tag[0]
    assert bio_tag in ["B", "I", "O"], "Invalid Tag"
    conll_tag = string_tag[2:]
    if bio_tag == "O":
      # The span has ended.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = None
      # We don't care about tags we are
      # told to ignore, so we do nothing.
      continue
    elif bio_tag == "B":
      # We are entering a new span; reset indices and active tag to new span.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = conll_tag
      span_start = index
      span_end = index
    elif bio_tag == "I" and conll_tag == active_conll_tag:
      # We're inside a span.
      span_end += 1
    else:
      # This is the case the bio label is an "I", but either:
      # 1) the span hasn't started - i.e. an ill formed span.
      # 2) We have IOB1 tagging scheme.
      # We'll process the previous span if it exists, but also include this
      # span. This is important, because otherwise, a model may get a perfect
      # F1 score whilst still including false positive ill-formed spans.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = conll_tag
      span_start = index
      span_end = index
  # Last token might have been a part of a valid span.
  if active_conll_tag:
    spans.add((active_conll_tag, (span_start, span_end)))
  # Return sorted list of spans
  return sorted(list(spans), key=lambda x: x[1][0])

dataset = load_dataset('wikiann', 'en', split="train")
ner_tags = {
   0:"O",
   1:"B-PER",
   2:"I-PER",
   3:"B-ORG",
   4:"I-ORG",
   5:"B-LOC",
   6:"I-LOC"
}

def get_spans(tokens, tags):
  """Convert tags to textspans."""
  spans = tags_to_spans(tags)
  text_spans = [
      x[0] + ": " + " ".join([tokens[i]
                              for i in range(x[1][0], x[1][1] + 1)])
      for x in spans
  ]
  if not text_spans:
    text_spans = ["None"]
  return text_spans


for i, d in enumerate(dataset):
   tokens = d['tokens']
   tags = d['ner_tags']
   tags = [ner_tags[i] for i in tags]
   spans = get_spans(tokens, tags)
   print("spans ", spans)
   print(d)
   if i > 10:
     break; 
```
I am not sure how to contribute to the repository and how things work, could you let me know how one can access the datasets to be able to contribute to the repository? Maybe I could do it then
thanks
Issue: wikiann dataset is missing columns 

Comment:
Cool ! Let me give you some context:

#### Contribution guide

You can find the contribution guide here:

https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md

It explains how to set up your dev environment in a few steps.

#### Dataset loading

Each Dataset is defined by a Table that have many rows (one row = one example) and columns (one column = one feature).
To change how a dataset is constructed, you have to modify its dataset script that you can find here:

https://github.com/huggingface/datasets/blob/master/datasets/wikiann/wikiann.py

It includes everything needed to load the WikiANN dataset.
You can load locally a modified version of `wikiann.py` with `load_dataset("path/to/wikiann.py")`.

#### Define a new column

Each column has a name and a type. You can see how the features of WikiANN are defined here:

https://github.com/huggingface/datasets/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L245-L263

Ideally we would have one additional feature "spans":
```python
        "spans": datasets.Sequence(datasets.Value("string")),
```

#### Compute the content of each row

To build the WikiANN rows, the _generate_examples method from [here](https://github.com/huggingface/nlp/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L292-L316) is used. This function `yield` one python dictionary for each example:
```python
yield guid_index, {"tokens": tokens, "ner_tags": ner_tags, "langs": langs}
```

The objective would be to return instead something like
```python
spans = spans = get_spans(tokens, tags)
yield guid_index, {"tokens": tokens, "ner_tags": ner_tags, "langs": langs, "spans": spans}
```

Let me know if you have questions !
Issue: How to train BERT model with next sentence prediction?

Comment:
Hi !
We're not using `TextDatasetForNextSentencePrediction` in `datasets`.
Although you can probably use the `TextDatasetForNextSentencePrediction.create_examples_from_document` on a dataset to prepare it for next sentence prediction.
Issue: How to train BERT model with next sentence prediction?

Comment:
Thanks.

Do you mean that `TextDatasetForNextSentencePrediction.create_exapmles_from_document` can be applied to dataset object other than `TextDatasetForNextSentencePrediction`  e.g. a `Dataset` object which is  loaded by  `datasets.load_dataset`?
Issue: How to train BERT model with next sentence prediction?

Comment:
It would probably require a bit of tweaking, but you can apply it to a dataset, yes.
This should give you a new dataset with sentence pairs you can train a model on.

You can find the documentation about dataset processing here:
https://huggingface.co/docs/datasets/processing.html#processing-data-with-map
Issue: Dialogue action slot name and value are reversed in MultiWoZ 2.2

Comment:
Hi
Good catch ! Thanks for reporting

If you are interested in contributing, feel free to open a PR to fix this :)
Issue: Problem downloading GEM wiki_auto_asset_turk dataset

Comment:
Hi,

sadly I can't replicate the problem on my Windows machine. Try to update the library to the newest version with:
```bash
pip install git+https://github.com/huggingface/datasets
```
Issue: Problem downloading GEM wiki_auto_asset_turk dataset

Comment:
Is there an error message ?
What stacktrace do you get if you interrupt the execution of the program while downloading ?
Issue: Problem downloading GEM wiki_auto_asset_turk dataset

Comment:
Sorry for the long time since my last comment, I tried again and don't seem to have the problem anymore, thanks for your support!
Issue: Problem downloading GEM wiki_auto_asset_turk dataset

Comment:
Great ! I'm closing the issue then. Feel free to re-open if you experience this issue again
Issue: Creating custom dataset results in error while calling the map() function

Comment:
Hi,

the `_data` attribute is missing due to `MyDataset.__init__` not calling the parent `__init__`. However, I don't think it's a good idea to subclass the `datasets.Dataset` class (e.g. it's kind of dangerous to override `datasets.Dataset.__getitem__`). Instead, it's better to follow the "association over inheritance" approach with a simple wrapper class that delegates calls to a wrapped `Dataset` (map, etc.). Btw, the library offers the `datasets.Dataset.from_pandas` class method to directly create a `datasets.Dataset` from the dataframe.
Issue: WMT19 Dataset for Kazakh-English is not formatted correctly

Comment:
Hi ! Thanks for reporting

By looking at the raw `news-commentary-v14.en-kk.tsv` file, it looks like there are at least 17 lines with this issue.
Moreover these issues are not always the same:
- L97 is only `kk` text and must be appended at the end of the `kk` text of the **next** line
- L2897 is only `kk` text and must be appended at the end of the `kk` text of the **previous** line
- L1247 and L1248 are only `kk` texts and must be inserted at the **beginning** of the `kk` text of the next line
- (and there are many others)

It would be nice to have a corrected version of this file ! The file is available in the `wmt/news-commentary` repository on the Datasets Hub here:
https://huggingface.co/datasets/wmt/news-commentary/tree/main/v14/training

Then maybe we can notify the WMT authors and host the corrected version somewhere
Issue: Request to remove S2ORC dataset

Comment:
Hello @kyleclo! Currently, we are getting the data from your bucket, so if you remove it the HF script won't work anymore :) 

Until you solve things on your end, @lhoestq suggested we just return a warning message when people try to load that dataset from HF. What would you like it to say?
Issue: Request to remove S2ORC dataset

Comment:
Hi @kyleclo, as of today, you have not removed your bucket data yet, and therefore HuggingFace can download it from there.

Is it OK? Are you planning to eventually delete it? Thank you.
Issue: Request to remove S2ORC dataset

Comment:
Hi!  Sorry I missed @yjernite 's previous message, thanks for responding!  

Is there an option where we can keep our data in our bucket, but the HF script no longer pulls data from it?
Issue: Trouble loading wiki_movies

Comment:
Hi ! `wiki_movies` was added in `datasets==1.2.0`. However it looks like you have `datasets==1.1.2`.

To use `wiki_movies`, please update `datasets` with
```
pip install --upgrade datasets
```
Issue: Trouble loading wiki_movies

Comment:
Thanks a lot! That solved it and I was able to upload a model trained on it as well :)
Issue: citation, homepage, and license fields of `dataset_info.json` are duplicated many times

Comment:
Thanks for reporting :)
Maybe we can concatenate fields only if they are different.

Currently this is done here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/info.py#L180-L196

This can be a good first contribution to the library.
Please comment if you'd like to improve this and open a PR :)
Issue: load_from_disk takes a long time to load local dataset

Comment:
Hi !
Can you share more information about the features of your dataset ? You can get them by printing `my_dataset.features`
Can you also share the code of your `map` function ?
Issue: load_from_disk takes a long time to load local dataset

Comment:
It is actually just the tokenized `wikipedia` dataset with `input_ids`, `attention_mask`, etc, with one extra column which is a list of integers. The `text` column is removed during tokenization.

```
def add_len_and_seq(example):
    end_idx = example['input_ids'].index(SEP)
    example['actual_len'] = end_idx-1
    seq_len = len(example['input_ids'])
    

    example['seq'] = [PAD_ID] + [np.uint8(example['some_integer'])]*(end_idx-1) + [PAD_ID]*(seq_len-end_idx)
    
    return example
```
Issue: load_from_disk takes a long time to load local dataset

Comment:
Is `PAD_ID` a python integer ? You need all the integers in `example['seq']` to have the same type.
Does this work if you remove the `np.uint8` and use python integers instead ?
Issue: load_from_disk takes a long time to load local dataset

Comment:
yup I casted it to `np.uint8` outside the function where it was defined. It was originally using python integers.
Issue: load_from_disk takes a long time to load local dataset

Comment:
Strangely, even when I manually created `np.arrays` of specific `dtypes`, the types in the final `dataset_info.json` that gets written are still `int64`.

Update: I tried creating lists of `int8`s and got the same result.
Issue: load_from_disk takes a long time to load local dataset

Comment:
Yes this is a known issue: #625 
We're working on making the precision kept for numpy :)
To specify the precision of the integers, currently one needs to specify the output features with `.map(..., features=output_features)`
Issue: load_from_disk takes a long time to load local dataset

Comment:
Do you know what step is taking forever in the code ?
What happens if you interrupt the execution of the dataset loading ?
Issue: load_from_disk takes a long time to load local dataset

Comment:
After a synchronous discussion, we found that the cache file sizes have an enormous effect on the loading speed: smaller cache files result in faster load times. `num_proc` controls the number of cache files that are being written and is inversely proportional to the individual file size. In other words, increase `num_proc` for smaller cache files :)

Maybe this can be highlighted somewhere in the docs.
Issue: CoNLL 2003 dataset not including German

Comment:
Hello. I've been looking for information about German Conll2003 and found your question. Official site (https://www.clips.uantwerpen.be/conll2003/ner/) mentions that organizers provide only annotation. German texts (ECI Multilingual Text Corpus) are not freely available and can be ordered from the Linguistic Data Consortium.

But maybe something has changed since 2003.
Issue: How to disable making arrow tables in load_dataset ?

Comment:
Hi ! We plan to add streaming features in the future.

This should allow to load a dataset instantaneously without generating the arrow table. The trade-off is that accessing examples from a streaming dataset must be done in an iterative way, and with an additional (but hopefully minor) overhead.
What do you think about this ?

If you have ideas or suggestions of what you expect from such features as a user, feel free to share them, this is really valuable to us !
Issue: How to disable making arrow tables in load_dataset ?

Comment:
People mainly want this feature either because it takes too much time too make arrow tables, or they occupy too much memory on the disk. I think both the problem can be solved if we provide arrow tables themselves on datasets hub. Can we do this currently @lhoestq ?
Issue: How to disable making arrow tables in load_dataset ?

Comment:
@lhoestq I think the ```try_from_hf_gcs``` provide the same functionality. What all datasets are available on HF GCS? Are all the datasets on huggingFace datasets hub are made available on GCS, automatically?