SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Possible cache miss in datasets

Comment:
Thanks for the fast reply, waiting for the fix :)

I tried to use `cache_file_names` and wasn't sure how, I tried to give it the following:
```
tokenized_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    num_proc=60,
    load_from_cache_file=True,
    cache_file_names={k: f'.cache/{str(k)}' for k in tokenized_datasets}
)
```

and got an error:
```
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/venv/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1491, in _map_single
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
  File "/usr/lib/python3.6/tempfile.py", line 690, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/lib/python3.6/tempfile.py", line 401, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: '_00000_of_00060.cache/tmpsvszxtop'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 48, in <module>
    cache_file_names={k: f'.cache/{str(k)}' for k in tokenized_datasets}
  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
    for k, dataset in self.items()
  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in map
    transformed_shards = [r.get() for r in results]
  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 644, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '_00000_of_00060.cache/tmpsvszxtop'
```
Issue: Possible cache miss in datasets

Comment:
The documentation says
```
cache_file_names (`Optional[Dict[str, str]]`, defaults to `None`): Provide the name of a cache file to use to store the
    results of the computation instead of the automatically generated cache file name.
    You have to provide one :obj:`cache_file_name` per dataset in the dataset dictionary.
```
What is expected is simply the name of a file, not a path. The file will be located in the cache directory of the `wikitext` dataset. You can try again with something like
```python
cache_file_names = {k: f'tokenized_and_grouped_{str(k)}' for k in tokenized_datasets}
```
Issue: Possible cache miss in datasets

Comment:
Managed to get `cache_file_names` working and caching works well with it
Had to make a small modification for it to work:
```
cache_file_names = {k: f'tokenized_and_grouped_{str(k)}.arrow' for k in tokenized_datasets}
```
Issue: Possible cache miss in datasets

Comment:
Another comment on `cache_file_names`, it doesn't save the produced cached files in the dataset's cache folder, it requires to give a path to an existing directory for it to work.
I can confirm that this is how it works in `datasets==1.1.3`
Issue: Possible cache miss in datasets

Comment:
Oh yes indeed ! Maybe we need to update the docstring to mention that it is a path
Issue: Possible cache miss in datasets

Comment:
I upgraded to the latest version and I encountered some strange behaviour, the script I posted in the OP doesn't trigger recalculation, however, if I add the following change it does trigger partial recalculation, I am not sure if its something wrong on my machine or a bug:
```
from datasets import load_dataset
from transformers import AutoTokenizer

datasets = load_dataset('wikitext', 'wikitext-103-raw-v1')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)

column_names = datasets["train"].column_names
text_column_name = "text" if "text" in column_names else column_names[0]
def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
# CHANGE
print('hello')
# CHANGE

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
...
```
I am using datasets in the `run_mlm.py` script in the transformers examples and I found that if I change the script without touching any of the preprocessing. it still triggers recalculation which is very weird

Edit: accidently clicked the close issue button
Issue: Possible cache miss in datasets

Comment:
This is because the `group_texts` line definition changes (it is defined 3 lines later than in the previous call). Currently if a function is moved elsewhere in a script we consider it to be different.

Not sure this is actually a good idea to keep this behavior though. We had this as a security in the early development of the lib but now the recursive hashing of objects is robust so we can probably remove that.
Moreover we're already ignoring the line definition for lambda functions.
Issue: Possible cache miss in datasets

Comment:
Sounds great, thank you for your quick responses and help! Looking forward for the next release.
Issue: Possible cache miss in datasets

Comment:
This error happened to me too, when I tried to supply my own fingerprint to `map()` via the `new_fingerprint` arg.

Edit: realized it was because my path was weird and had colons and brackets and slashes in it, since one of the variable values I included in the fingerprint was a dataset split like "train[:10%]". I fixed it with [this solution](https://stackoverflow.com/a/13593932/2287177) from StackOverflow to just remove those invalid characters from the fingerprint.
Issue: Possible cache miss in datasets

Comment:
Good catch @jxmorris12, maybe we should do additional checks on the valid characters for fingerprints ! Would you like to contribute this ?

I think this can be added here, when we set the fingerprint(s) that are passed `map`:

https://github.com/huggingface/datasets/blob/25bb7c9cbf519fbbf9abf3898083b529e7762705/src/datasets/fingerprint.py#L449-L454

maybe something like
```python
if kwargs.get(fingerprint_name) is None:
    ...
else:
    # In this case, it's the user who specified the fingerprint manually:
    # we need to make sure it's a valid hash
    validate_fingerprint(kwargs[fingerprint_name])
```

Otherwise I can open a PR later
Issue: SciFact dataset - minor changes

Comment:
Hi Dave,
You are more than welcome to open a PR to make these changes! 🤗
You will find the relevant information about opening a PR in the [contributing guide](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md) and in the [dataset addition guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Pinging also @lhoestq for the Google cloud matter.
Issue: SciFact dataset - minor changes

Comment:
> I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

Sure ! Also feel free to ping us for reviews or if we can help :)

> It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

What makes you think that ?
Afaik there's no scifact on our google storage
Issue: SciFact dataset - minor changes

Comment:


> > I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?
> 
> Sure ! Also feel free to ping us for reviews or if we can help :)
> 
OK! We're organizing a [shared task](https://sdproc.org/2021/sharedtasks.html#sciver) based on the dataset, and I made some updates and changed the download URL - so the current code points to a dead URL. I'll update appropriately once the task is finalized and make a PR.

> > It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?
> 
> What makes you think that ?
> Afaik there's no scifact on our google storage

You're right, I had the data cached on my machine somewhere.
Issue: Installation using conda

Comment:
Great! Did you guys have a timeframe in mind for the next release?

Thank you for all the great work in developing this library.
Issue: Installation using conda

Comment:
I think we can have `datasets` on conda by next week. Will see what I can do!
Issue: Installation using conda

Comment:
`datasets` has been added to the huggingface channel thanks to @LysandreJik :)
It depends on conda-forge though

```
conda install -c huggingface -c conda-forge datasets
```
Issue: IsADirectoryError when trying to download C4

Comment:
I haven't tested C4 on my side so there so there may be a few bugs in the code/adjustments to make.
Here it looks like in c4.py, line 190 one of the `files_to_download` is `'/'` which is invalid.
Valid files are paths to local files or URLs to remote files.
Issue: Error when downloading a large dataset on slow connection.

Comment:
Hi ! Is this an issue you have with `openwebtext` specifically or also with other datasets ?

It looks like the downloaded file is corrupted and can't be extracted using `tarfile`.
Could you try loading it again with 
```python
import datasets
datasets.load_dataset("openwebtext", download_mode="force_redownload")
```
Issue: Some datasets miss dataset_infos.json or dummy_data.zip

Comment:
Thanks for reporting.
We should indeed add all the missing dummy_data.zip and also the dataset_infos.json at least for lm1b, reclor and wikihow.

For c4 I haven't tested the script and I think we'll require some optimizations regarding beam datasets before processing it.
Issue: Some datasets miss dataset_infos.json or dummy_data.zip

Comment:
Closing since the dummy data generation is deprecated now (and the issue with missing metadata seems to be addressed).
Issue: Question: Shouldn't .info be a part of DatasetDict?

Comment:
We could do something. There is a part of `.info` which is split specific (cache files, split instructions) but maybe if could be made to work.
Issue: Question: Shouldn't .info be a part of DatasetDict?

Comment:
Yes this was kinda the idea I was going for. DatasetDict.info would be the shared info amongs the datasets (maybe even some info on how they differ).
Issue: Dataset Error: DaNE contains empty samples at the end

Comment:
One the PR is merged the fix will be available in the next release of `datasets`.

If you don't want to wait the next release you can still load the script from the master branch with

```python
load_dataset("dane", script_version="master")
```
Issue: `ArrowInvalid` occurs while running `Dataset.map()` function for DPRContext

Comment:
Looks like the mapping function returns a dictionary with a 768-dim array in the `embeddings` field. Since the map is batched, we actually expect the `embeddings` field to be an array of shape (batch_size, 768) to have one embedding per example in the batch.

To fix that can you try to remove one of the `[0]` ? In my opinion you only need one of them, not two.
Issue: Dataset "dane" missing

Comment:
Hi @KennethEnevoldsen ,
I think the issue might be that this dataset was added during the community sprint and has not been released yet. It will be available with the v2 of datasets.
For now, you should be able to load the datasets after installing the latest (master) version of datasets using pip:
pip install git+https://github.com/huggingface/datasets.git@master
Issue: Dataset "dane" missing

Comment:
The `dane` dataset was added recently, that's why it wasn't available yet. We did an intermediate release today just before the v2.0.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `dane` with

```python
from datasets import load_dataset

dataset = load_dataset("dane")
```
Issue: Can't import cc100 dataset

Comment:
cc100 was added recently, that's why it wasn't available yet.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `cc100` with

```python
from datasets import load_dataset

lang = "en"
dataset = load_dataset("cc100", lang=lang, split="train")
```
Issue: Add the 800GB Pile dataset?

Comment:
The pile dataset would be very nice.
Benchmarks show that pile trained models achieve better results than most of actually trained models
Issue: Add the 800GB Pile dataset?

Comment:
The pile can very easily be added and adapted using this [tfds implementation](https://github.com/EleutherAI/The-Pile/blob/master/the_pile/tfds_pile.py) from the repo. 

However, the question is whether you'd be ok with 800GB+ cached in your local disk, since the tfds implementation was designed to offload the storage to Google Cloud Storage.
Issue: Add the 800GB Pile dataset?

Comment:
With the dataset streaming feature (see #2375) it will be more convenient to play with such big datasets :)
I'm currently adding C4 (see #2511 ) but I can probably start working on this afterwards
Issue: Add the 800GB Pile dataset?

Comment:
Hi folks! Just wanted to follow up on this -- would be really nice to get the Pile on HF Datasets... unclear if it would be easy to also add partitions of the Pile subject to the original 22 datasets used, but that would be nice too!
Issue: Add the 800GB Pile dataset?

Comment:
Hi folks, thanks to some awesome work by @lhoestq and @albertvillanova you can now stream the Pile as follows:

```python
# Install master branch of `datasets`
pip install git+https://github.com/huggingface/datasets.git#egg=datasets[streaming]
pip install zstandard

from datasets import load_dataset

dset = load_dataset("json", data_files="https://the-eye.eu/public/AI/pile/train/00.jsonl.zst", streaming=True, split="train")
next(iter(dset))
# {'meta': {'pile_set_name': 'Pile-CC'},
# 'text': 'It is done, and submitted. You can play “Survival of the Tastiest” on Android, and on the web ... '}
```

Next step is to add the Pile as a "canonical" dataset that can be streamed without specifying the file names explicitly :)
Issue: Add the 800GB Pile dataset?

Comment:
> Hi folks! Just wanted to follow up on this -- would be really nice to get the Pile on HF Datasets... unclear if it would be easy to also add partitions of the Pile subject to the original 22 datasets used, but that would be nice too!

Hi @siddk thanks to a tip from @richarddwang it seems we can access some of the partitions that EleutherAI created for the Pile [here](https://the-eye.eu/public/AI/pile_preliminary_components/). What's missing are links to the preprocessed versions of pre-existing datasets like DeepMind Mathematics and OpenSubtitles, but worst case we do the processing ourselves and host these components on the Hub.

My current idea is that we could provide 23 configs: one for each of the 22 datasets and an `all` config that links to the train / dev / test splits that EleutherAI released [here](https://the-eye.eu/public/AI/pile/), e.g.

```python
from datasets import load_dataset

# Load a single component
youtube_subtitles = load_dataset("the_pile", "youtube_subtitles")
# Load the train / dev / test splits of the whole corpus
dset = load_dataset("the_pile", "all")
```

Ideally we'd like everything to be compatible with the streaming API and there's ongoing work by @albertvillanova to make this happen for the various compression algorithms.

cc @lhoestq
Issue: Add the 800GB Pile dataset?

Comment:
Ah I just saw that @lhoestq is already thinking about the specifying of one or more subsets in [this PR](https://github.com/huggingface/datasets/pull/2817#issuecomment-901874049) :)
Issue: dutch_social can't be loaded

Comment:
Hi @koenvandenberge and @alighofrani95!
The datasets you're experiencing issues with were most likely added recently to the `datasets` library, meaning they have not been released yet. They will be released with the v2 of the library.
Meanwhile, you can still load the datasets using one of the techniques described in this issue: #1641 
Let me know if this helps!
Issue: dutch_social can't be loaded

Comment:
I just did the release :)

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `dutch_social` with

```python
from datasets import load_dataset

dataset = load_dataset("dutch_social")
```
Issue: dutch_social can't be loaded

Comment:
@lhoestq could you also shed light on the Hindi Wikipedia Dataset for issue number #1673. Will this also be available in the new release that you committed recently?
Issue: dutch_social can't be loaded

Comment:
Okay. Could you comment on the #1673 thread? Actually @thomwolf had commented that if i use datasets library from source, it would allow me to download the Hindi Wikipedia Dataset but even the version 1.1.3 gave me the same issue. The details are there in the issue #1673 thread.
Issue: Unable to Download Hindi Wikipedia Dataset

Comment:
Currently this dataset is only available when the library is installed from source since it was added after the last release.

We pin the dataset version with the library version so that people can have a reproducible dataset and processing when pinning the library.

We'll see if we can provide access to newer datasets with a warning that they are newer than your library version, that would help in cases like yours.
Issue: Unable to Download Hindi Wikipedia Dataset

Comment:
So for now, should i try and install the library from source and then try out the same piece of code? Will it work then, considering both the versions will match then?
Issue: Unable to Download Hindi Wikipedia Dataset

Comment:
Hey, so i tried installing the library from source using the commands : **git clone https://github.com/huggingface/datasets**,  **cd datasets** and then **pip3 install -e .**. But i still am facing the same error that file is not found. Please advise.

The Datasets library version now is 1.1.3 by installing from source as compared to the earlier 1.0.3 that i had loaded using pip command but I am still getting same error

![Error](https://user-images.githubusercontent.com/30871963/103479005-69f3b080-4df0-11eb-83ae-58d7bb56a90e.png)
Issue: Unable to Download Hindi Wikipedia Dataset

Comment:
Looks like the wikipedia dump for hindi at the date of 05/05/2020 is not available anymore.
You can try to load a more recent version of wikipedia
```python
from datasets import load_dataset

d = load_dataset("wikipedia", language="hi", date="20210101", split="train", beam_runner="DirectRunner")
```
Issue: connection issue 

Comment:
Also, mayjor issue for me is the format issue, even if I go through changing the whole code to use load_from_disk, then if I do 

d = datasets.load_from_disk("imdb")
d = d["train"][:10] => the format of this is no more in datasets format
this is different from you call load_datasets("train[10]")

could you tell me how I can make the two datastes the same format @lhoestq
Issue: connection issue 

Comment:
> `
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/glue/glue.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6d6c60a20>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))`

Do you have an internet connection on the machine ? Is there a proxy that might block requests to aws ?

> I tried to do read the data, save it to a path and then set HF_HOME, which does not work and this is still not reading from the old set path, could you assist me how to save the datasets in a path, and let dataset library read from this path to avoid connection issue. thanks

HF_HOME is used to specify the directory for the cache files of this library.
You can use save_to_disk and load_from_disk without changing the HF_HOME:
```python
imdb = datasets.load_dataset("imdb")
imdb.save_to_disk("/idiap/temp/rkarimi/hf_datasets/imdb")
imdb = datasets.load_from_disk("/idiap/temp/rkarimi/hf_datasets/imdb")
```

> could you tell me how I can make the two datastes the same format

Indeed they returns different things:
- `load_dataset` returns a `Dataset` object if the split is specified, or a `DatasetDict` if no split is given. Therefore `load_datasets("imdb", split="train[10]")` returns a `Dataset` object containing 10 elements.
- doing `d["train"][:10]` on a DatasetDict "d" gets the train split `d["train"]` as a `Dataset` object and then gets the first 10 elements as a dictionary
Issue: wiki_dpr pre-processing performance

Comment:
Hi ! And thanks for the tips :) 

Indeed currently `wiki_dpr` takes some time to be processed.
Multiprocessing for dataset generation is definitely going to speed up things.

Regarding the index note that for the default configurations, the index is downloaded instead of being built, which avoid spending time on constructing the index. However in other cases it would be awesome to make the construction faster.

Any contribution that can help things faster are welcome. In particular in you have some code that can build a wiki_dpr IVF PQ index in a sharded GPU setup and would like to share it, we can add it to an `examples` folder. In particular since faiss is becoming the library of reference for dataset indexing for tasks like Open Domain Question Answering.
Issue: wiki_dpr pre-processing performance

Comment:
I'd be happy to contribute something when I get the time, probably adding multiprocessing and / or cython support to wiki_dpr. I've written cythonized apache beam code before as well.

For sharded index building, I used the FAISS example code for indexing 1 billion vectors as a start. I'm sure you're aware that the documentation isn't great, but the source code is fairly easy to follow.
Issue: Arrow file is too large when saving vector data

Comment:
Hi !
The arrow file size is due to the embeddings. Indeed if they're stored as float32 then the total size of the embeddings is

20 000 000 vectors * 768 dimensions * 4 bytes per dimension ~= 60GB

If you want to reduce the size you can consider using quantization for example, or maybe using dimension reduction techniques.
Issue: Arrow file is too large when saving vector data

Comment:
Thanks for your reply @lhoestq.
I want to save original embedding for these sentences for subsequent calculations. So does arrow have a way to save in a compressed format to reduce the size of the file?
Issue: NarrativeQA fails to load with `load_dataset`

Comment:
Hi @eric-mitchell,
I think the issue might be that this dataset was added during the community sprint and has not been released yet. It will be available with the v2 of `datasets`.
For now, you should be able to load the datasets after installing the latest (master) version of `datasets` using pip:
`pip install git+https://github.com/huggingface/datasets.git@master`
Issue: NarrativeQA fails to load with `load_dataset`

Comment:
Update: HuggingFace did an intermediate release yesterday just before the v2.0.

To load it you can just update `datasets`

`pip install --upgrade datasets`
Issue: HoVeR dataset fails to load

Comment:
Hover was added recently, that's why it wasn't available yet.

To load it you can just update `datasets`
```
pip install --upgrade datasets
```

and then you can load `hover` with

```python
from datasets import load_dataset

dataset = load_dataset("hover")
```