SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Error loading wikihow dataset

Comment:
Good call out. I did try that and that's when it told me to download the
dataset. Don't believe I have tried it with local files. Will try first
thing in the morning and get back to you.

On Mon, Apr 19, 2021, 11:17 PM Albert Villanova del Moral <
***@***.***> wrote:

> Hi @odellus <https://github.com/odellus>, thanks for reporting.
>
> The wikihow dataset has 2 versions:
>
>    - all: Consisting of the concatenation of all paragraphs as the
>    articles and the bold lines as the reference summaries.
>    - sep: Consisting of each paragraph and its summary.
>
> Therefore, in order to load it, you have to specify which version you
> would like, for example:
>
> dataset = load_dataset('wikihow', 'all')
>
> Please, tell me if this solves your problem.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2239#issuecomment-823004146>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABDYI3HVRTBI2QT3BOG262DTJUL57ANCNFSM43GV5BZQ>
> .
>
Issue: Error loading wikihow dataset

Comment:
Hi @odellus, yes you are right.

Due to the server where the `wikihow` dataset is hosted, the dataset can't be downloaded automatically by `huggingface` and you have to download it manually as you did.

Nevertheless, you have to specify which dataset version you would like to load anyway:
```python
dataset = load_dataset('wikihow', 'all', data_dir='./wikihow')
```
or
```python
dataset = load_dataset('wikihow', 'sep', data_dir='./wikihow')
```
I find that the instructions given by `huggingface` are not clear enough: I am going to fix this.
Please tell me if this eventually works for you.
Issue: Update Dataset.dataset_size after transformed with map

Comment:
@albertvillanova I would like to take this up. It would be great if you could point me as to how the dataset size is calculated in HF. Thanks!
Issue: Keys yielded while generating dataset are not being checked

Comment:
Hi ! Indeed there's no verification on the uniqueness nor the types of the keys.
Do you already have some ideas of what you would like to implement and how ?
Issue: Keys yielded while generating dataset are not being checked

Comment:
Hey @lhoestq, thank you so much for the opportunity.
Although I haven't had much experience with the HF Datasets code, after a careful look at how the `ArrowWriter` functions, I think we can implement this as follows:

1.  First, we would have to update the `ArrowWriter.write()` function here:
https://github.com/huggingface/datasets/blob/fcd3c3c8e3b1d9a2f3686a496082e21f06591380/src/datasets/arrow_writer.py#L296
so that it accepts an additional argument `key` which would be appended along with the example here after hashing.

2.  Then, we would need to create a `Hasher` class which will take the key as its input and return a hash for it (We might need to use some hash salt which can be passed to the ArrowWriter.writer() with value equal to the `split_name` for differentiating between same keys of different splits)

    We can use the `hashlib.md5` function for hashing which will conert each key to its byte code before hashing (depending on the data type of the key) **Thus, the `key` type will be verified here**.

3.  Now, we would have to edit this
https://github.com/huggingface/datasets/blob/fcd3c3c8e3b1d9a2f3686a496082e21f06591380/src/datasets/arrow_writer.py#L257
 so that it iterates over each `(hash, example)` pair (sorted according to hash). We can then simply **check whether each hash is different from the previous hash** (since they will be sorted)

However, since I'm not very familiar with how the data is being written on disk in the form of a table, I might need some guidance for Step 3. 
Please let me know your thought on this. Thanks!
Issue: Keys yielded while generating dataset are not being checked

Comment:
Interesting !
We keep the dataset sorted in the order examples are generated by the builder (we expect the dataset builders to generate examples in deterministic order). Therefore I don't think we should shuffle the examples with the hashing. Let me know what you think.
Other that that, I really like the idea of checking for keys duplicates in `write_examples_on_file` :)

This looks like a great plan ! Feel free to open a PR and ping me if you have questions or if I can help
Issue: Keys yielded while generating dataset are not being checked

Comment:
@lhoestq I'm glad you liked the idea!
I think that since the keys will be unique and deterministic in the nature themselves, so even if we shuffle the examples according to the hash, a deterministic order would still be maintained (as the keys will always have the same hash, whenever the dataset is generated). 
And since, we are not dealing with time series data (which would require the data to be in original order), I don't think the order of examples would matter much, as long as the order is deterministic and constant for all users.

I think that this is also what was originally envisioned as mentioned in the documentation here:
https://github.com/huggingface/datasets/blob/6775661b19d2ec339784f3d84553a3996a1d86c3/src/datasets/builder.py#L973

Also, if we avoid this, we would need to keep track of all the hashed keys in some place and compare each individual key with all others. This can cause some major overhead as each dataset consists of tens of thousands of examples.
Let me know your thoughts in it! I would be opening a PR soon :)
Issue: Keys yielded while generating dataset are not being checked

Comment:
When users load their own data, they expect the order to stay the same. I think that shuffling the data can make things inconvenient.

> I think that this is also what was originally envisioned as mentioned in the documentation here:

This part was originally developed by tensorflow datasets, and tensorflow datasets indeed does the shuffling. However in this library this is probably not what we want in the general case. But if @albertvillanova and @thomwolf you have opinions on this please let us know.

> Also, if we avoid this, we would need to keep track of all the hashed keys in some place and compare each individual key with all others. This can cause some major overhead as each dataset consists of tens of thousands of examples.

Maybe we cam simply keep track of the hashes of of each batch being written ? The size of the batch when the data are save in arrow is 10 000 examples. This would only ensure that we don't have duplicates in each batch, but there might still be duplicates across batches. For 10 000 examples the hashes can just be stored as a python `set`.

Otherwise if we want full deduplication, we need an extra tool that allows to temporarily save and query hashes that may need to use disk space rather than memory.
Issue: Keys yielded while generating dataset are not being checked

Comment:
Yes I think we want to keep the original order by default and only shuffle when the user ask for it (for instance by calling `dataset.shuffle()`). That’s how I had it in mind originally.
Issue: Keys yielded while generating dataset are not being checked

Comment:
Hey @lhoestq, I just had a more in-depth look at the original TFDS code about why the keys and hash were used in the first place.

In my opinion, the only use that the `hash(key)` serves is that it allows us to shuffle the examples in a deterministic order (as each example will always yield the same key and thus, the same hash on every system) so that the same dataset is generated for each user, irrespective of the order the examples are yielded by the dataset builder on different user systems.

Otherwise, if we are not shuffling, then while yielding and writing the data, after getting the key and hashing it for an example, I can't quite see the use of the hash or the key. The hash will simply be generated for each example but not actually used anywhere?

@lhoestq @thomwolf It would be great if you could explain a bit more about the usage of keys. Thanks!
Issue: Keys yielded while generating dataset are not being checked

Comment:
In `datasets` the keys are currently ignored.
For shuffling we don't use the keys. Instead we shuffle an array of indices. Since both the original order of the dataset and the indices shuffling are deterministic, then `dataset.shuffle` is deterministic as well.
We can use it to:
1. detect duplicates
2. verify that the generation order is indeed deterministic
3. maybe more ?
Issue: Keys yielded while generating dataset are not being checked

Comment:
Thanks a lot @lhoestq. I think I understand what we need to do now. The keys can indeed be used for detecting duplicates in generated examples as well as ensuring the order.

> Maybe we cam simply keep track of the hashes of of each batch being written ? The size of the batch when the data are save in arrow is 10 000 examples. This would only ensure that we don't have duplicates in each batch,

I think that checking for duplicates in every batch independently would be sufficient as the probability of collisions using something like `MD5` is very low. I would be opening a draft PR soon. It would be great to have your guidance. Thanks!
Issue: `xnli` dataset creating a tuple key while yielding instead of `str` or `int`

Comment:
Hi ! Sure sounds good. Also if you find other datasets that use tuples instead of str/int, you can also fix them !
thanks :)
Issue: `xnli` dataset creating a tuple key while yielding instead of `str` or `int`

Comment:
@lhoestq I have sent a PR for fixing the issue. Would be great if you could have a look! Thanks!
Issue: Batched map fails when removing all columns

Comment:
I found the problem. I called `set_format` on some columns before. This makes it crash. Here is a complete example to reproduce:

```python
from datasets import load_dataset
sst = load_dataset("sst")
sst.set_format("torch", columns=["label"], output_all_columns=True)
ds = sst["train"]

# crashes
ds.map(
    lambda x: {"a": list(range(20))},
    remove_columns=ds.column_names,
    load_from_cache_file=False,
    num_proc=1,
    batched=True,
)
```
Issue: Batched map fails when removing all columns

Comment:
Thanks for reporting and for providing this code to reproduce the issue, this is really helpful !
Issue: Batched map fails when removing all columns

Comment:
I merged a fix, it should work on `master` now :)
We'll do a new release soon !
Issue: Duplicates in the LAMA dataset

Comment:
Hi,

currently the datasets API doesn't have a dedicated function to remove duplicate rows, but since the LAMA dataset is not too big (it fits in RAM), we can leverage pandas to help us remove duplicates:
```python
>>> from datasets import load_dataset, Dataset
>>> dataset = load_dataset('lama', split='train')
>>> dataset = Dataset.from_pandas(dataset.to_pandas().drop_duplicates(subset=...)) # specify a subset of the columns to consider in a list or use all of the columns if None
```

Note that the same can be achieved with the `Dataset.filter` method but this would requrie some extra work (filter function, speed?).
Issue: Duplicates in the LAMA dataset

Comment:
Oh, seems like my question wasn't specified well. I'm _not_ asking how to remove duplicates, but whether duplicates should be removed if I want to do the evaluation on the LAMA dataset as it was proposed in the original paper/repository? In other words, will I get the same result if evaluate on the de-duplicated dataset loaded from HF's `datasets` as the results I'd get if I use the original data format and data processing script in https://github.com/facebookresearch/LAMA?
Issue: Duplicates in the LAMA dataset

Comment:
So it looks like the person who added LAMA to the library chose to have one item per piece of evidence rather than one per relation - and in this case, there are duplicate pieces of evidence for the target relation

If I understand correctly, to reproduce reported results, you would have to aggregate predictions for the several pieces of evidence provided for each relation (each unique `uuid`), but the original authors will know better 

cc @fabiopetroni
Issue: load_metric error: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'

Comment:
Hi @nsaphra, thanks for reporting.

This issue was fixed in `datasets` version 1.3.0. Could you please update `datasets` and tell me if the problem persists?
```shell
pip install -U datasets
```
Issue: load_metric error: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'

Comment:
There might be a bug in the conda version of `datasets` 1.2.1 where the datasets/metric scripts are downloaded from `master` instead of the `1.2.1` repo.

You can try setting the env var `HF_SCRIPTS_VERSION="1.2.1"` as a workaround. Let me know if that helps.
Issue: load_metric error: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'

Comment:
I just faced the same issue. I was using 1.2.1 from conda and received the same AttributeError complaining about 'add_start_docstrings'. Uninstalling the conda installed datasets and then installing the latest datasets (version 1.5.0) using pip install solved the issue for me. I don't like mixing up conda and pip installs in the same environments but this will have to do for now, until 1.5.0 is made available through conda.
Issue: load_metric error: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'

Comment:
Yep, seems to have fixed things! The conda package could really do with an update. Thanks!
Issue: Can't reach "https://storage.googleapis.com/illuin/fquad/train.json.zip" when trying to load fquad dataset

Comment:
Hi ! Apparently the data are not available from this url anymore. We'll replace it with the new url when it's available
Issue: Can't reach "https://storage.googleapis.com/illuin/fquad/train.json.zip" when trying to load fquad dataset

Comment:
I saw this on their website when we request to download the dataset:
![image](https://user-images.githubusercontent.com/19718818/114879600-fa458680-9e1e-11eb-9e05-f0963d68ff0f.png)

Can we still request them link for the dataset and make a PR? @lhoestq @yjernite
Issue: Can't reach "https://storage.googleapis.com/illuin/fquad/train.json.zip" when trying to load fquad dataset

Comment:
I've contacted Martin (first author of the fquad paper) regarding a possible new url. Hopefully we can get one soon !
Issue: Can't reach "https://storage.googleapis.com/illuin/fquad/train.json.zip" when trying to load fquad dataset

Comment:
They now made a website to force people who want to use the dataset for commercial purposes to seek a commercial license from them ...
Issue: Getting checksum error when trying to load lc_quad dataset

Comment:
Hi,

I've already opened a PR with the fix. If you are in a hurry, just build the project from source and run:
```bash
datasets-cli test datasets/lc_quad --save_infos --all_configs --ignore_verifications
```
Issue: dataloading slow when using HUGE dataset

Comment:
Hi ! Yes this is an issue with `datasets<=1.5.0`
This issue has been fixed by #2122 , we'll do a new release soon :)
For now you can test it on the `master` branch.
Issue: dataloading slow when using HUGE dataset

Comment:
Hi, thank you for your answer. I did not realize that my issue stems from the same problem.
Issue: making labels consistent across the datasets

Comment:
Hi ! The ClassLabel feature type encodes the labels as integers.
The integer corresponds to the index of the label name in the `names` list of the ClassLabel.
Here that means that the labels are 'entailment' (0), 'neutral' (1), 'contradiction' (2).

You can get the label names back by using `a.features['label'].int2str(i)`.
Issue: Got pyarrow error when loading a dataset while adding special tokens into the tokenizer

Comment:
Hi,

the output of the tokenizers is treated specially in the lib to optimize the dataset size (see the code [here](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_writer.py#L138-L141)). It looks like that one of the values in a dictionary returned by the tokenizer is out of the assumed range.
Can you please provide a minimal reproducible example for more help?
Issue: Got pyarrow error when loading a dataset while adding special tokens into the tokenizer

Comment:
Hi @yana-xuyan, thanks for reporting.

As clearly @mariosasko explained, `datasets` performs some optimizations in order to reduce the size of the dataset cache files. And one of them is storing the field `special_tokens_mask` as `int8`, which means that this field can only contain integers between `-128` to `127`. As your message error states, one of the values of this field is `50259`, and therefore it cannot be stored as an `int8`.

Maybe we could implement a way to disable this optimization and allow using any integer value; although the size of the cache files would be much larger.
Issue: Got pyarrow error when loading a dataset while adding special tokens into the tokenizer

Comment:
I'm facing same issue @mariosasko @albertvillanova 

```
ArrowInvalid: Integer value 50260 not in range: -128 to 127
```

To reproduce:
```python
SPECIAL_TOKENS = ['<bos>','<eos>','<speaker1>','<speaker2>','<pad>']
ATTR_TO_SPECIAL_TOKEN = {
    'bos_token': '<bos>', 
    'eos_token': '<eos>', 
    'pad_token': '<pad>',
    'additional_special_tokens': ['<speaker1>', '<speaker2>']
    }

tokenizer = AutoTokenizer.from_pretrained("gpt2", use_fast=False)
num_added_tokens =tokenizer.add_special_tokens(ATTR_TO_SPECIAL_TOKEN)
vocab_size = len(self.tokenizer.encoder) + num_added_tokens
vocab =tokenizer.get_vocab()

pad_index = tokenizer.pad_token_id
eos_index = tokenizer.eos_token_id
bos_index = tokenizer.bos_token_id
speaker1_index = vocab["<speaker1>"]
speaker2_index = vocab["<speaker2>"]
```

```python
tokenizer.decode(['50260'])
'<speaker1>'
```
Issue: Got pyarrow error when loading a dataset while adding special tokens into the tokenizer

Comment:
@mariosasko 
I am hitting this bug in the Bert tokenizer too.  I see that @albertvillanova labeled this as a bug back in April.  Has there been a fix released yet?
What I did for now is to just disable the optimization in the HF library.  @yana-xuyan and @thomas-happify, is that what you did and did that work for you?
Issue: Got pyarrow error when loading a dataset while adding special tokens into the tokenizer

Comment:
Hi @gregg-ADP, 

This is still a bug.

As @albertvillanova has suggested, maybe it's indeed worth adding a variable to `config.py` to have a way to disable this behavior.

In the meantime, this forced optimization can be disabled by specifying `features` (of the returned examples) in the `map` call:
```python
from datasets import *
... # dataset init
ds.map(process_example, features=Features({"special_tokens_mask": Sequence(Value("int32")), ... rest of the features}) 
```

cc @lhoestq so he is also aware of this issue
Issue: Got pyarrow error when loading a dataset while adding special tokens into the tokenizer

Comment:
Thanks for the quick reply @mariosasko.  What I did was to changed the optimizer to use int32 instead of int8. 
What you're suggesting specifies the type for each feature explicitly without changing the HF code. This is definitely a better option.  However, we are hitting a new error later:
```
  File "/Users/ccccc/PycharmProjects/aaaa-ml/venv-source/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'pos'

```
Where 'pos' is the name of a new feature we added.  Do you agree that your way of fixing the optimizer issue will not fix our new issue?  If not, I will continue with this optimizer fix until we resolve our other issue.
Issue: Got pyarrow error when loading a dataset while adding special tokens into the tokenizer

Comment:
Hi @gwc4github,

the fix was merged a few minutes ago, and it doesn't require any changes on the user side (e.g. no need for specifying `features`). If you find time, feel free to install `datasets` from master with:
```
pip install git+https://github.com/huggingface/datasets.git
```
and let us know if it works for your use case!
Issue: _prepare_split will overwrite DatasetBuilder.info.features

Comment:
Hi ! This might be related to #2153 

You're right the ArrowWriter should be initialized with `features=self.info.features` ! Good catch
I'm opening a PR to fix this and also to figure out how it was not caught in the tests

EDIT: opened #2201
Issue: _prepare_split will overwrite DatasetBuilder.info.features

Comment:
> Hi ! This might be related to #2153
> 
> You're right the ArrowWriter should be initialized with `features=self.info.features` ! Good catch
> I'm opening a PR to fix this and also to figure out how it was not caught in the tests
> 
> EDIT: opened #2201

Glad to hear that! Thank you for your fix, I'm new to huggingface, it's a fantastic project 😁
Issue: `load_dataset` caches two arrow files?

Comment:
Hi ! Files that starts with `cache-*` are cached computation files, i.e. they are the cached results of map/filter/cast/etc. operations. For example if you used `map` on your dataset to transform it, then the resulting dataset is going to be stored and cached in a `cache-*` file. These files are used to avoid having to load the dataset in RAM, even after many transforms
Issue: `load_dataset` caches two arrow files?

Comment:
Thanks @lhoestq! Hmm.. that's strange because I specifically turned off auto caching, and saved mapped result, using `save_to_disk`, to another location. At this location, the following file is created:`355G	cache-ed205e500a7dc44c.arrow`

To my observation, both `load_dataset` and `map` creates `cache-*` files, and I wonder what the `cache-*` file from `load_dataset` is for (as I believe the same information is stored in `json-train.arrow`.
Issue: `load_dataset` caches two arrow files?

Comment:
This is a wrong report -- `cache-*` files are created only my `map`, not by `load_dataset`.
Issue: KeyError: '_indices_files' in `arrow_dataset.py`

Comment:
Thanks @samsontmr this should be fixed on master now

Feel free to reopen if you're still having issues
Issue: Filtering/mapping on one column is very slow

Comment:
Hi ! Yes we are working on making `filter` significantly faster. You can look at related PRs here: #2060 #2178 

I think you can expect to have the fast version of `filter` available next week.

We'll make it only select one column, and we'll also make the overall filtering operation way faster by avoiding many arrow<->python conversions especially during writing.

I'll let you know how it goes !
Issue: Filtering/mapping on one column is very slow

Comment:
@lhoestq Thanks for the response— it's great to hear that we'll be getting a much faster `filter` method soon. However, my use case does also involve using `map` over a single column in order to pre-compute roughly uniformly sized batches, and right now that is also very slow. Is there any plan to make `map` faster for single column operations?

If that's not a priority for the maintainers right now, I could try my hand at adding the feature, but I can't guarantee I would do a good job given my lack of familiarity with pyarrow.
Issue: Filtering/mapping on one column is very slow

Comment:
Currently the optimal setup for single-column computations is probably to do something like
```python
result = dataset.map(f, input_columns="my_col", remove_columns=dataset.column_names)
```
This has two advantages:
- input_columns="my_col" allows to only read the column "my_col"
- remove_columns=dataset.column_names makes `map` only keep the output of your function `f`, and it drops the other columns of the dataset instead of keeping them.

Let me know if it improves speed on your side.

You can also get more speed by using `batched=True` and setting `num_proc=` for multiprocessing