SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: dataset.shuffle() and select() resets format. Intended?

Comment:
I think it's ok to keep the format.
If we want to have this behavior for `.map` too we just have to make sure it doesn't keep a column that's been removed.
Issue: dataset.shuffle() and select() resets format. Intended?

Comment:
Since datasets 1.0.0 the format is not reset anymore.
Closing this one, but feel free to re-open if you have other questions
Issue: Converting TensorFlow dataset example

Comment:
Do you want to convert a dataset script to the tfds format ?
If so, we currently have a comversion script nlp/commands/convert.py but it is a conversion script that goes from tfds to nlp.
I think it shouldn't be too hard to do the changes in reverse (at some manual adjustments).
If you manage to make it work in reverse, feel free to open a PR to share it with the community :)
Issue: TypeError: Receiver() takes no arguments

Comment:
Which version of Apache Beam do you have (can you copy your full environment info here)?
Issue: TypeError: Receiver() takes no arguments

Comment:
apache-beam==2.23.0
nlp==0.4.0

For me this was resolved by running the same python script on Linux (or really WSL).
Issue: TypeError: Receiver() takes no arguments

Comment:
Do you manage to run a dummy beam pipeline with python on windows ? 
You can test a dummy pipeline with [this code](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py)

If you get the same error, it means that the issue comes from apache beam.
Otherwise we'll investigate what went wrong here
Issue: TypeError: Receiver() takes no arguments

Comment:
Still, same error, so I guess it is on apache beam then. 
Thanks for the investigation.
Issue: TypeError: Receiver() takes no arguments

Comment:
Thanks for trying
Let us know if you find clues of what caused this issue, or if you find a fix
Issue: Errors when I use 

Comment:
Looks like an issue with 3.0.2 transformers version. Works fine when I use "master" version of transformers.
Issue: Caching doesn't work for map (non-deterministic)

Comment:
Thanks for reporting !

To store the cache file, we compute a hash of the function given in `.map`, using our own hashing function.
The hash doesn't seem to stay the same over sessions for the tokenizer.
Apparently this is because of the regex at `tokenizer.pat` is not well supported by our hashing function.

I'm working on a fix
Issue: Caching doesn't work for map (non-deterministic)

Comment:
Hi. I believe the fix was for the nlp library. Is there a solution to handle compiled regex expressions in .map() with the caching. I want to run a simple regex pattern on a big dataset, but I am running into the issue of compiled expression not being cached. 

Instead of opening a new issue, I thought I would put my query here. Let me know if a new issue would be more suitable. Thanks
Issue: Caching doesn't work for map (non-deterministic)

Comment:
Hi @MaveriQ! This fix is also included in the `datasets` library. Can you provide a reproducer?
Issue: nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema

Comment:
In 0.4.0, the assertion in `concatenate_datasets ` is on the features, and not the schema.
Could you try to update `nlp` ?

Also, since 0.4.0, you can use `dset_wikipedia.cast_(dset_books.features)` to avoid the schema cast hack.
Issue: nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema

Comment:
I'm using the master branch. The assertion failure comes from the underlying `pa.concat_tables()`, which is in the pyarrow package. That method does check schemas.

Since `features.type` does not contain information about nullable vs non-nullable features, the `cast_()` method won't resolve the schema mismatch. There is information in a schema which is not stored in features.
Issue: nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema

Comment:
I'm doing a refactor of type inference in #363 . Both text fields should match after that
Issue: nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema

Comment:
It should be good now. I was able to run

```python
>>> from nlp import concatenate_datasets, load_dataset
>>>
>>> bookcorpus = load_dataset("bookcorpus", split="train")
>>> wiki = load_dataset("wikipedia", "20200501.en", split="train")
>>> wiki.remove_columns_("title")  # only keep the text
>>>
>>> assert bookcorpus.features.type == wiki.features.type
>>> bert_dataset = concatenate_datasets([bookcorpus, wiki])
```
Issue: issues with downloading datasets for wmt16 and wmt19

Comment:
I found `UNv1.0.en-ru.tar.gz` here: https://conferences.unite.un.org/uncorpus/en/downloadoverview, so it can be reconstructed with:
```
wget -c https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-ru.tar.gz.00
wget -c https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-ru.tar.gz.01
wget -c https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-ru.tar.gz.02
cat UNv1.0.en-ru.tar.gz.0* > UNv1.0.en-ru.tar.gz
```
it has other languages as well, in case https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/ is gone
Issue: issues with downloading datasets for wmt16 and wmt19

Comment:
Further, `nlp.load_dataset('wmt19', 'ru-en')` has only the `train` and `val` datasets. `test` is missing.

Fixed locally for summarization needs, by running:
```
pip install sacrebleu
sacrebleu -t wmt19 -l ru-en --echo src > test.source
sacrebleu -t wmt19 -l ru-en --echo ref > test.target
```
h/t @sshleifer
Issue: Bookcorpus data contains pretokenized text

Comment:
Yes indeed it looks like some `'` and spaces are missing (for example in `dont` or `didnt`).
Do you know if there exist some copies without this issue ?
How would you fix this issue on the current data exactly ? I can see that the data is raw text (not tokenized) so I'm not sure I understand how you would do it. Could you provide more details ?
Issue: Bookcorpus data contains pretokenized text

Comment:
I'm afraid that I don't know how to obtain the original BookCorpus data. I believe this version came from an anonymous Google Drive link posted in another issue.

Going through the raw text in this version, it's apparent that NLTK's TreebankWordTokenizer was applied on it (I gave some examples in my original post), followed by:
`' '.join(tokens)`
You can retrieve the tokenization by splitting on whitespace. You can then "detokenize" it with TreebankWordDetokenizer class of NLTK (though, as I suggested, use the fixed version in my repo). This will bring the text closer to its original form, but some steps of TreebankWordTokenizer are destructive, so it wouldn't be one-to-one. Something along the lines of the following should work:
```
treebank_detokenizer = nltk.tokenize.treebank.TreebankWordDetokenizer()
db = nlp.load_dataset('bookcorpus', split=nlp.Split.TRAIN)
db = db.map(lambda x: treebank_detokenizer.detokenize(x['text'].split()))
```

Regarding other issues beyond the above, I'm afraid that I can't help with that.
Issue: Bookcorpus data contains pretokenized text

Comment:
Ok I get it, that would be very cool indeed

What kinds of patterns the detokenizer can't retrieve ?
Issue: Bookcorpus data contains pretokenized text

Comment:
The TreebankTokenizer makes some assumptions about whitespace, parentheses, quotation marks, etc. For instance, while tokenizing the following text:
```
Dwayne "The Rock" Johnson
```
will result in:
```
Dwayne `` The Rock '' Johnson
```
where the left and right quotation marks are turned into distinct symbols. Upon reconstruction, we can attach the left part to its token on the right, and respectively for the right part. However, the following texts would be tokenized exactly the same:
```
Dwayne " The Rock " Johnson
Dwayne " The Rock" Johnson
Dwayne     " The Rock" Johnson
...
```
In the above examples, the detokenizer would correct these inputs into the canonical text
```
Dwayne "The Rock" Johnson
```
However, there are cases where there the solution cannot easily be inferred (at least without a true LM - this tokenizer is just a bunch of regexes). For instance, in cases where you have a fragment that contains the end of quote, but not its beginning, plus an accidental space:
```
... and it sounds fantastic, " he said.
```
In the above case, the tokenizer would assume that the quotes refer to the next token, and so upon detokenization it will result in the following mistake:
```
... and it sounds fantastic, "he said.
```

While these are all odd edge cases (the basic assumptions do make sense), in noisy data they can occur, which is why I mentioned that the detokenizer cannot restore the original perfectly.
Issue: Bookcorpus data contains pretokenized text

Comment:
To confirm, since this is preprocessed, this was not the exact version of the Book Corpus used to actually train the models described here (particularly Distilbert)?  https://huggingface.co/datasets/bookcorpus

Or does this preprocessing exactly match that of the papers?
Issue: Bookcorpus data contains pretokenized text

Comment:
I believe these are just artifacts of this particular source. It might be better to crawl it again, or use another preprocessed source, as found here: https://github.com/soskek/bookcorpus
Issue: Bookcorpus data contains pretokenized text

Comment:
Yes actually the BookCorpus on hugginface is based on [this](https://github.com/soskek/bookcorpus/issues/24#issuecomment-643933352). And I kind of regret naming it as "BookCorpus" instead of something like "BookCorpusLike".

But there is a good news ! @shawwn has replicated BookCorpus in his way, and also provided a link to download the plain text files. see [here](https://github.com/soskek/bookcorpus/issues/27). There is chance we can have a "OpenBookCorpus" !
Issue: Bugs : dataset.map() is frozen on ELI5

Comment:
This comes from an overflow in pyarrow's array.
It is stuck inside the loop that reduces the batch size to avoid the overflow.
I'll take a look
Issue: Bugs : dataset.map() is frozen on ELI5

Comment:
I created a PR to fix the issue.
It was due to an overflow check that handled badly an empty list.

You can try the changes by using 
```
!pip install git+https://github.com/huggingface/nlp.git@fix-bad-type-in-overflow-check
```

Also I noticed that the first 1000 examples have an empty list in the `title_urls` field. The feature type inference in `.map` will consider it `null` because of that, and it will crash when it encounter the next example with a `title_urls` that is not empty.

Therefore to fix that, what you can do for now is increase the writer batch size so that the feature inference will take into account at least one example with a non-empty `title_urls`:

```python
# default batch size is 1_000 and it's not enough for feature type inference because of empty lists
valid_dataset = valid_dataset.map(make_input_target, writer_batch_size=3_000) 
```

I was able to run the frozen cell with these changes.
Issue: Bugs : dataset.map() is frozen on ELI5

Comment:
@lhoestq mapping the function `make_input_target` was passed by your fixing.

However, there is another error in the final step of `valid_dataset.map(convert_to_features, batched=True)`

`ArrowInvalid: Could not convert Thepiratebay.vg with type str: converting to null type`
(The [same colab notebook above with new error message](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing#scrollTo=5sRrJ3_C8rLt))

Do you have some ideas? (I am really sorry I could not debug it by myself since I never used `pyarrow` before) 
Note that `train_dataset.map(convert_to_features, batched=True)` can be run successfully even though train_dataset is 27x bigger than `valid_dataset` so I believe the problem lies in some field of `valid_dataset` again .
Issue: Bugs : dataset.map() is frozen on ELI5

Comment:
I got this issue too and fixed it by specifying `writer_batch_size=3_000` in `.map`.
This is because Arrow didn't expect `Thepiratebay.vg` in `title_urls `, as all previous examples have empty lists in `title_urls `
Issue: Bugs : dataset.map() is frozen on ELI5

Comment:
I'm getting a hanging `dataset.map()` when running a gradio app with `gradio` for auto-reloading instead of `python`
Issue: Bugs : dataset.map() is frozen on ELI5

Comment:
Maybe this is an issue with gradio, could you open an issue on their repo ? `Dataset.map` simply uses `multiprocess.Pool` for multiprocessing

If you interrupt the program mayeb the stack trace would give some information of where it was hanging in the code (maybe a lock somewhere ?)
Issue: Export TFRecord to GCP bucket

Comment:
Nevermind, I restarted my python session and it worked fine...

---

I had an authentification error, and I authenticated from another terminal. After that, no more error but it was not working. Restarting the sessions makes it work :)
Issue: Overview.ipynb throws exceptions with nlp 0.4.0

Comment:
Thanks for reporting this issue

There was a bug where numpy arrays would get returned instead of tensorflow tensors.
This is fixed on master.

I tried to re-run the colab and encountered this error instead:

```
AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'to_tensor'
```

This is because the dataset returns a Tensor and not a RaggedTensor.
But I think we should always return a RaggedTensor unless the length of the sequence is fixed (it that case they can be stack into a Tensor).
Issue: Overview.ipynb throws exceptions with nlp 0.4.0

Comment:
Hi, I got another error (on Colab):

```python
# You can read a few attributes of the datasets before loading them (they are python dataclasses)
from dataclasses import asdict

for key, value in asdict(datasets[6]).items():
    print('👉 ' + key + ': ' + str(value))

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-b8ace6c227a2> in <module>()
      2 from dataclasses import asdict
      3 
----> 4 for key, value in asdict(datasets[6]).items():
      5     print('👉 ' + key + ': ' + str(value))

/usr/local/lib/python3.6/dist-packages/dataclasses.py in asdict(obj, dict_factory)
   1008     """
   1009     if not _is_dataclass_instance(obj):
-> 1010         raise TypeError("asdict() should be called on dataclass instances")
   1011     return _asdict_inner(obj, dict_factory)
   1012 

TypeError: asdict() should be called on dataclass instances
```
Issue: test_load_real_dataset when config has BUILDER_CONFIGS that matter

Comment:
The `data_dir` parameter has been removed. Now the error is `ValueError: Config name is missing`

As mentioned in #470 I think we can have one test with the first config of BUILDER_CONFIGS, and another test that runs all of the configs in BUILDER_CONFIGS
Issue: test_load_real_dataset when config has BUILDER_CONFIGS that matter

Comment:
This was fixed in #527 

Closing this one, but feel free to re-open if you have other questions
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
Hi ! Did you try to set the output format to pytorch ? (or tensorflow if you're using tensorflow)
It can be done with `dataset.set_format("torch", columns=columns)`  (or "tensorflow").

Note that for pytorch, string columns can't be converted to `torch.Tensor`, so you have to specify in `columns=` the list of columns you want to keep (`input_ids` for example)
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
Hello . Yes, I did set the output format as below for the two columns 

  `train_dataset.set_format('torch',columns=['Text','Label'])`
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
I think you're having this issue because you try to format strings as pytorch tensors, which is not possible.
Indeed by having "Text" in `columns=['Text','Label']`, you try to convert the text values to pytorch tensors.

Instead I recommend you to first tokenize your dataset using a tokenizer from transformers. For example

```python
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

train_dataset.map(lambda x: tokenizer(x["Text"]), batched=True)
train_dataset.set_format("torch", column=["input_ids"])
```

Another way to fix your issue would be to not set the format to pytorch, and leave the dataset as it is by default. In that case, the strings are returned normally when you get examples from your dataloader. It means that you would have to tokenize the examples in the training loop (or using a data collator) though.

Let me know if you have other questions
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-145-ca218223c9fc> in <module>()
----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
      2 val_f1 = f1_score_func(predictions, true_val)
      3 tqdm.write(f'Validation loss: {val_loss}')
      4 tqdm.write(f'F1 Score (Weighted): {val_f1}')

6 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in <genexpr>(.0)
    160 
    161     def __getitem__(self, index):
--> 162         return tuple(tensor[index] for tensor in self.tensors)
    163 
    164     def __len__(self):

TypeError: new(): invalid data type 'str'
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
> Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
> I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error
> 
> TypeError Traceback (most recent call last)
> in ()
> ----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
> 2 val_f1 = f1_score_func(predictions, true_val)
> 3 tqdm.write(f'Validation loss: {val_loss}')
> 4 tqdm.write(f'F1 Score (Weighted): {val_f1}')
> 
> 6 frames
> /usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in (.0)
> 160
> 161 def **getitem**(self, index):
> --> 162 return tuple(tensor[index] for tensor in self.tensors)
> 163
> 164 def **len**(self):
> 
> TypeError: new(): invalid data type 'str'

I got the same error and fix it .
you can check your input where there may be string contained.
such as
```
a = [1,2,3,4,'<unk>']
torch.tensor(a)
```
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
I didn't know tokenizers could return strings in the token ids. Which tokenizer are you using to get this @Doragd ?
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
> I didn't know tokenizers could return strings in the token ids. Which tokenizer are you using to get this @Doragd ?

i'm sorry that i met this issue in another place (not in huggingface repo).
Issue: invalid data type 'str' at _convert_outputs in arrow_dataset.py

Comment:
@akhilkapil do you have strings in your dataset ? When you set the dataset format to "pytorch" you should exclude columns with strings as pytorch can't make tensors out of strings
Issue: UnicodeDecodeError while loading PAN-X task of XTREME dataset

Comment:
Indeed. Solution 1 is the simplest.

This is actually a recurring problem.
I think we should scan all the datasets with regexpr to fix the use of `open()` without encodings.
And probably add a test in the CI to forbid using this in the future.
Issue: UnicodeDecodeError while loading PAN-X task of XTREME dataset

Comment:
I've created a simple function that seems to do the trick:

```python
def apply_encoding_on_file_open(filepath: str):
    """Apply UTF-8 encoding for all instances where a non-binary file is opened."""
    
    with open(filepath, 'r', encoding='utf-8') as input_file:
        regexp = re.compile(r"""
                            (?!.*\b(?:encoding|rb|wb|wb+|ab|ab+)\b)
                            (open)
                            \((.*)\)
                            """)
        input_text = input_file.read()
        match = regexp.search(input_text)
        
        if match:
            print('Found match!', match.group())
            # append utf-8 encoding to matching groups in-place
            output = regexp.sub(lambda m: m.group()[:-1]+', encoding="utf-8")', input_text)
            with open(filepath, 'w', encoding='utf-8') as output_file:
                output_file.write(output)
        else:
            print("No match found!")
```

The regexp does a negative lookahead to avoid matching on cases where the encoding is already specified or when binary files are involved.

From an implementation perspective:

* Would it make sense to include this function in `nlp-cli` so that we can run something like
```
nlp-cli fix_encoding path/to/folder
```
and the command recursively fixes all files in the target?
* What is the desired behaviour in the CI test? Here we could either have a simple script that we run as a `job` in the CI and raises an error if a missing encoding is detected. Alternatively we could incorporate this behaviour into the CLI and run that in the CI.

Please let me know what you prefer among the alternatives.
Issue: UnicodeDecodeError while loading PAN-X task of XTREME dataset

Comment:
I realised I was overthinking the problem, so decided to just run the regexp over the codebase and make the PR. In other words, we can ignore my comments about using the CLI 😸
Issue: Keep loading old file even I specify a new file in load_dataset

Comment:
This is the only fix I could come up with without touching the repo's code.
```python
from nlp.builder import FORCE_REDOWNLOAD
dataset = load_dataset('csv', data_file='./a.csv', download_mode=FORCE_REDOWNLOAD, version='0.0.1')
```
You'll have to change the version each time you want to load a different csv file.
If you're willing to add a ```print```, you can go to ```nlp.load``` and add ```print(builder_instance.cache_dir)``` right before the ```return ds``` in the ```load_dataset``` method. It'll print the cache folder, and you'll just have to erase it (and then you won't need the change here above).
Issue: Cannot unpickle saved .pt dataset with torch.save()/load()

Comment:
This seems to be fixed in a non-released version. 

Installing nlp from source
```
git clone https://github.com/huggingface/nlp
cd nlp
pip install .
```
solves the issue.
Issue: Issues: Adding a FAISS or Elastic Search index to a Dataset

Comment:
`DPRContextEncoder` and `DPRContextEncoderTokenizer` will be available in the next release of `transformers`.

Right now you can experiment with it by installing `transformers` from the master branch.
You can also check the docs of DPR [here](https://huggingface.co/transformers/master/model_doc/dpr.html).

Moreover all the indexing features will also be available in the next release of `nlp`.
Issue: Issues: Adding a FAISS or Elastic Search index to a Dataset

Comment:
@lhoestq  I tried installing transformer from the master branch. Python imports for DPR again didnt' work.  Anyways, Looking forward to trying it in the next release of nlp
Issue: New Datasets: IWSLT15+, ITTB

Comment:
Thanks Sam, we now have a very detailed tutorial and template on how to add a new dataset to the library. It typically take 1-2 hours to add one. Do you want to give it a try ?
The tutorial on writing a new dataset loading script is here: https://huggingface.co/nlp/add_dataset.html
And the part on how to share a new dataset is here: https://huggingface.co/nlp/share_dataset.html