SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Indices incorrect with multiprocessing

Comment:
Still the case on master.
I guess we should have an offset in the multi-procs indeed (hopefully it's enough).

Also, side note is that we should add some logging before the "test" to say we are testing the function otherwise its confusing for the user to see two outputs I think. Proposal (see the "Testing the mapped function outputs:" lines):
```
>>> d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2)
Done writing 10 indices in 80 bytes .
Done writing 5 indices in 41 bytes .
Done writing 5 indices in 41 bytes .
Spawning 2 processes
Testing the mapped function outputs:
inds: [0, 1]
inds: [0, 1]
Testing finished, running the mapped function on the dataset:
#0:   0%|                                                                                                                                                                    | 0/1 [00:00<?, ?ba/s]
inds: [0, 1, 2, 3, 4]                                                                                                                                                                             inds: [0, 1, 2, 3, 4]                                                                                                                                                         | 0/1 [00:00<?, ?ba/s]
#0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1321.04ba/s]
#1: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1841.22ba/s]
Concatenating 2 shards from multiprocessing
Dataset(features: {'text': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None)}, num_rows: 10)
```
Issue: `Dataset`/`DatasetDict` has no attribute 'save_to_disk'

Comment:
`pip install git+https://github.com/huggingface/nlp.git` should have done the job.

Did you uninstall `nlp` before installing from github ?
Issue: `Dataset`/`DatasetDict` has no attribute 'save_to_disk'

Comment:
> Did you uninstall `nlp` before installing from github ?

I did not. I created a new environment and installed `nlp` directly from `github` and it worked!

Thanks.
Issue: The process cannot access the file because it is being used by another process (windows)

Comment:
Hi, which version of `nlp` are you using?

By the way we'll be releasing today a significant update fixing many issues (but also comprising a few breaking changes).
You can see more informations here #545 and try it by installing from source from the master branch.
Issue: The process cannot access the file because it is being used by another process (windows)

Comment:
Ok, it's probably fixed on master. Otherwise if you can give me a fully self-contained exemple to reproduce the error, I can try to investigate.
Issue: The process cannot access the file because it is being used by another process (windows)

Comment:
I get the same behavior, on Windows, when `map`ping a function to a loaded dataset. 
The error doesn't occur if I re-run the cell a second time though! 
I'm on version 1.0.1.
Issue: The process cannot access the file because it is being used by another process (windows)

Comment:
@saareliad I got the same issue that troubled me quite a while. Unfortunately, there are no good answers to this issue online, I tried it on Linux and that's absolutely fine. After hacking the source code, I solved this problem as follows.

In the source code file: arrow_dataset.py -> _map_single(...)

change
```python
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```
to
```python
tmp_file.close()
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```

Then it works without needing multiple times runs to avoid the permission error.
I know this solution is unusual since it changes the source code. Hopefully, the lib's contributors can have better solutions in the future.
Issue: The process cannot access the file because it is being used by another process (windows)

Comment:
@wangcongcong123 thanks  for sharing.
(BTW I also solved it locally on windows by putting the problematic line under try except and not using cache... On windows I just needed 1% of the dataset anyway)
Issue: nlp re-creates already-there caches when using a script, but not within a shell

Comment:
Couln't reproduce on my side :/ 
let me know if you manage to reproduce on another env (colab for example)
Issue: Some languages in wikipedia dataset are not loading

Comment:
Some wikipedia languages have already been processed by us and are hosted on our google storage. This is the case for "fr" and "en" for example.

For other smaller languages (in terms of bytes), they are directly downloaded and parsed from the wikipedia dump site.
Parsing can take some time for languages with hundreds of MB of xml.

Let me know if you encounter an error or if you feel that is is taking too long for you.
We could process those that really take too much time
Issue: Some languages in wikipedia dataset are not loading

Comment:
Thanks ! This will be very helpful.

About the date issue, I think it's possible to use another date with

```python
load_dataset("wikipedia", language="es", date="...", beam_runner="...")
```

However we've not processed wikipedia dumps for other dates than 20200501 (yet ?)

One more thing that is specific to 20200501.es: it was available once but the `mwparserfromhell` was not able to parse it for some reason, so we didn't manage to get a processed version of 20200501.es (see #321 )
Issue: Some languages in wikipedia dataset are not loading

Comment:
Hey @gaguilar ,

I just found the ["char2subword" paper](https://arxiv.org/pdf/2010.12730.pdf) and I'm really interested in trying it out on own vocabs/datasets like for historical texts (I've already [trained some lms](https://github.com/stefan-it/europeana-bert) on newspaper articles with OCR errors).

Do you plan to release the code for your paper or is it possible to get the implementation 🤔 Many thanks :hugs:
Issue: Some languages in wikipedia dataset are not loading

Comment:
Hi @stefan-it! Thanks for your interest in our work! We do plan to release the code, but we will make it available once the paper has been published at a conference. Sorry for the inconvenience!

Hi @lhoestq, do you have any insights for this issue by any chance? Thanks!
Issue: Some languages in wikipedia dataset are not loading

Comment:
This is an issue on the `mwparserfromhell` side. You could try to update `mwparserfromhell` and see if it fixes the issue. If it doesn't we'll have to create an issue on their repo for them to fix it.
But first let's see if the latest version of `mwparserfromhell` does the job.
Issue: Some languages in wikipedia dataset are not loading

Comment:
Hi ! The link https://dumps.wikimedia.org/idwiki/20210501/dumpstatus.json seems to be working fine for me.

Regarding the time outs, it must come either from an issue on the wikimedia host side, or from your internet connection.
Feel free to try again several times.
Issue: Some languages in wikipedia dataset are not loading

Comment:
Hi ! This looks related to this issue: https://github.com/huggingface/datasets/issues/1994
Basically the parser that is used (mwparserfromhell) has some issues for some pages in `es`.
We already reported some issues for `es` on their repo at https://github.com/earwig/mwparserfromhell/issues/247 but it looks like there are still a few issues. Might be a good idea to open a new issue on the mwparserfromhell repo
Issue: Some languages in wikipedia dataset are not loading

Comment:
The issue:
```
KeyError: "000nbsp [while running 'train/Clean content']"
```
reported in comments:
- https://github.com/huggingface/datasets/issues/577#issuecomment-701890059 (by @gaguilar)
- https://github.com/huggingface/datasets/issues/577#issuecomment-879513227 (by @mmiakashs)

was normally fixed in the `mwparserfromhell` library and will be accessible in their next release version `0.7`:
- https://github.com/earwig/mwparserfromhell/issues/288
Issue: Some languages in wikipedia dataset are not loading

Comment:
mwparserfromhell 0.7 has still not been released, but you might have luck with the dev version:
`pip install git+https://github.com/earwig/mwparserfromhell.git@0f89f44`
Issue: Couldn't reach certain URLs and for the ones that can be reached, code just blocks after downloading.

Comment:
Update:

The imdb download completed after a long time (about 45 mins). Ofcourse once download loading was instantaneous. Also, the loaded object was of type `arrow_dataset`. 

The urls for glue still doesn't work though.
Issue: Couldn't reach certain URLs and for the ones that can be reached, code just blocks after downloading.

Comment:
I am also seeing a similar error when running the following:

```
import nlp
dataset = load_dataset('cola')
```
Error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/load.py", line 509, in load_dataset
    module_path = prepare_module(path, download_config=download_config, dataset=True)
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/load.py", line 248, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 191, in cached_path
    output_path = get_from_cache(
  File "/home/js11133/.conda/envs/jiant/lib/python3.8/site-packages/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cola/cola.py
```
Issue: Couldn't reach certain URLs and for the ones that can be reached, code just blocks after downloading.

Comment:
@jeswan `"cola"` is not a valid dataset identifier (you can check the up-to-date list on https://huggingface.co/datasets) but you can find cola inside glue.
Issue: Couldn't reach certain URLs and for the ones that can be reached, code just blocks after downloading.

Comment:
Hi. Closing this one since #626 updated the glue urls.

> 1. Why is it still blocking? Is it still downloading?

After downloading it generates the arrow file by iterating through the examples.
The number of examples processed by second is shown during the processing (not sure why it was not the case for you)

> 2. I specified split as train, so why is the test folder being populated?

It downloads every split
Issue: `metric.compute` throws `ArrowInvalid` error

Comment:
Could you try to update to `datasets>=1.0.0` (we changed the name of the library) and try again ?
If is was related to the distributed setup settings it must be fixed.
If it was related to empty metric inputs it's going to be fixed in #654
Issue: `metric.compute` throws `ArrowInvalid` error

Comment:
Closing this one as it was fixed in #654 
Feel free to re-open if you have other questions
Issue: No module named 'nlp.logging'

Comment:
Thanks for reporting.

Apparently this is a versioning issue: the lib downloaded the `bleurt` script from the master branch where we did this change recently. We'll fix that in a new release this week or early next week. Cc @thomwolf 

Until that, I'd suggest you to download the right bleurt folder from github ([this one](https://github.com/huggingface/nlp/tree/0.4.0/metrics/bleurt)) and do

```python
from nlp import load_metric

bleurt = load_metric("path/to/bleurt/folder")
```

To download it you can either clone the repo or download the `bleurt.py` file and place it in a folder named `bleurt`
Issue: No module named 'nlp.logging'

Comment:
Actually we can fix this on our side, this script didn't had to be updated. I'll do it in a few minutes
Issue: Using custom DownloadConfig results in an error

Comment:
From my limited understanding, part of the issue seems related to the `prepare_module`  and `download_and_prepare` functions each handling the case where no config is passed. For example, `prepare_module` does mutate the object passed and forces the flags `extract_compressed_file` and `force_extract` to `True`.

See:
* https://github.com/huggingface/nlp/blob/5fb61e1012bda724a9b6b847307d90a1380abfa5/src/nlp/load.py#L227
* https://github.com/huggingface/nlp/blob/5fb61e1012bda724a9b6b847307d90a1380abfa5/src/nlp/builder.py#L388

Maybe a cleaner solution would be to always instantiate a default `DownloadConfig` object at the top-level, have it as non-optional for the lower-level functions and treat it as immutable.
Issue: Using custom DownloadConfig results in an error

Comment:
Thanks for the report, I'll take a look.

What is your specific use-case for providing a DownloadConfig object?
Issue: Using custom DownloadConfig results in an error

Comment:
Thanks. Our use case involves running a training job behind a corporate firewall with no access to any external resources (S3, GCP or other web resources).

I was thinking about a 2-steps process:
1) Download the resources / artifacts using some secure corporate channel, ie run `nlp.load_dataset()` without a specific `DownloadConfig`. After that, collect the files from the `$HF_HOME` folder
2) Copy the `$HF_HOME` folder in the firewalled environment. Run `nlp.load_dataset()` with a custom config `DownloadConfig(local_files_only=True)`

However this ends up a bit clunky in practice, even when solving the `DownloadConfig` issue above. For example, the `filename` hash computed in `get_from_cache()` differs in the `local_files_only=False` vs `local_files_only=True` case (local case defaults `etag` to `None`, which results in a different hash). So effectively step 2) above doesn't work because the hash computed differs from the hash in the cache folder. Some hacks / workaround are possible but this solution becomes very convoluted.
https://github.com/huggingface/nlp/blob/c214aa5a4430c1df1bcd0619fd94d6abdf9d2da7/src/nlp/utils/file_utils.py#L417

Would you recommend a different path?
Issue: Using custom DownloadConfig results in an error

Comment:
I see.

Probably the easiest way for you would be that we add simple serialization/deserialization methods to the Dataset and DatasetDict objects once the data files have been downloaded and all the dataset is processed.

What do you think @lhoestq ?
Issue: nlp downloads to its module path

Comment:
Indeed this is a known issue arising from the fact that we try to be compatible with cloupickle.

Does this also happen if you are installing in a virtual environment?
Issue: nlp downloads to its module path

Comment:
> Indeed this is a know issue with the fact that we try to be compatible with cloupickle.
> 
> Does this also happen if you are installing in a virtual environment?

Then it would work, because the package is in a writable path.
Issue: nlp downloads to its module path

Comment:
> If it's fine for you then this is the recommended way to solve this issue.

I don't want to use a virtual environment, because Nix is fully reproducible, and virtual environments are not. And I am the maintainer of the `transformers` in nixpkgs, so sooner or later I will have to package `nlp`, since it is becoming a dependency of `transformers` ;).
Issue: nlp downloads to its module path

Comment:
Ok interesting. We could have another check to see if it's possible to download and import the datasets script at another location than the module path. I think this would probably involve tweaking the python system path dynamically.

I don't know anything about Nix so if you want to give this a try your self we can guide you or you can give us more information on your general project and how this works.

Regarding `nlp` and `transformers`, we are not sure `nlp` will become a required dependency for `transformers`.  It will probably be used a lot in the examples but I think it probably won't be a required dependency for the main package since we try to keep it as light as possible in terms of deps.

Happy to help you make all these things work better for your use-case
Issue: nlp downloads to its module path

Comment:
@danieldk modules are now installed in a different location (by default in the cache directory of the lib, in `~/.cache/huggingface/modules`). You can also change that using the environment variable `HF_MODULES_PATH`

Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
We plan to do a release in the next coming days
Issue: nlp downloads to its module path

Comment:
> Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
> We plan to do a release in the next coming days

Thanks for making this change! I just packaged the latest commit on master and it works like a charm now! :partying_face:
Issue: Very slow data loading on large dataset

Comment:
When you load a text file for the first time with `nlp`, the file is converted into Apache Arrow format. Arrow allows to use memory-mapping, which means that you can load an arbitrary large dataset.

Note that as soon as the conversion has been done once, the next time you'll load the dataset it will be much faster.

However for a 1TB dataset, the conversion can indeed take time. You could try to load parts of it in parallel, and then use `nlp.concatenate_datasets` to get your full dataset.
Issue: Very slow data loading on large dataset

Comment:
Humm, we can give a look at these large scale datasets indeed.

Do you mind sharing a few stats on your dataset so I can try to test on a similar one?

In particular some orders of magnitudes for the number of files, number of lines per files, line lengths.
Issue: Very slow data loading on large dataset

Comment:
@lhoestq Yes, I understand that the first time requires more time. The concatenate_datasets seems to be a workaround, but I believe a multi-processing method should be integrated into load_dataset to make it easier and more efficient for users.

@thomwolf Sure, here are the statistics:
Number of lines: 4.2 Billion
Number of files: 6K
Number of tokens: 800 Billion
The number of lines is distributed equally across these 6k files.
The line length varies between 100 tokens to 40k tokens.
Issue: Very slow data loading on large dataset

Comment:
@agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new `text` processing script which is [here](https://github.com/huggingface/nlp/blob/07d92a82b7594498ff702f3cca55c074e2052257/datasets/text/text.py) somewhere on your drive and give it's local path instead of `text` to `load_dataset`. E.g. in your example:
```python
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```
Issue: Very slow data loading on large dataset

Comment:
I have already generated the dataset, but now I tried to reload it and it is still very slow.

I also have installed your commit and it is slow, even after the dataset was already generated.
`pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257`

It uses only a single thread.

Did I miss something ?
Issue: Very slow data loading on large dataset

Comment:
As mentioned in #548 , each time you call `load_dataset` with `data_files=`, they are hashed to get the cache directory name. Hashing can be too slow with 1TB of data. I feel like we should have a faster way of getting a hash that identifies the input data files
Issue: Very slow data loading on large dataset

Comment:
I believe this is really a very important feature, otherwise, we will still have the issue of too slow loading problems even if the data cache generation is fast.
Issue: Very slow data loading on large dataset

Comment:
Hmm ok then maybe it's the hashing step indeed.

Let's see if we can improve this as well.

(you will very likely have to regenerate your dataset if we change this part of the lib though since I expect modifications on this part of the lib to results in new hashes)
Issue: Very slow data loading on large dataset

Comment:
Also, @agemagician you have to follow the step I indicate in my previous message [here](https://github.com/huggingface/nlp/issues/546#issuecomment-684648927) to use the new text loading script.

Just doing `pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257` like you did won't use the new script (they are not inside the library but hosted on our hub).
Issue: Very slow data loading on large dataset

Comment:
No problem, I will regenerate it. This will make us see if we solved both issues and now both the data generation step, as well as the hashing step, is fast.
Issue: Very slow data loading on large dataset

Comment:
Ok so now the text files won't be hashed.

I also updated #548 to include this change.
Let us know if it helps @agemagician :)
Issue: Very slow data loading on large dataset

Comment:
Right now, for caching 18Gb data, it is taking 1 hour 10 minute. Is that proper expected time? @lhoestq @agemagician 
In this rate (assuming large file will caching at the same rate) caching full mC4 (27TB) requires a month (~26 days).
Issue: Very slow data loading on large dataset

Comment:
Hi ! Currently it is that slow because we haven't implemented parallelism for the dataset generation yet.
Though we will definitely work on this :)

For now I'd recommend loading the dataset shard by shard in parallel, and then concatenate them:
```python
# in one process, load first 100 files for english
shard1 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-000**.json.gz")
# in another process load next 100 files for english
shard2 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-001**.json.gz")

# finally
concatenate_datasets([shard1, shard2, ...])
Issue: Very slow data loading on large dataset

Comment:
Sorry to write on a closed issue but, has there been any progress on parallelizing the `load_dataset` function?
Issue: Very slow data loading on large dataset

Comment:
I'm literally crying waiting for the trainer to restart from checkpoint. It's getting stuck at `get_train_dataloader` and I think this is to do with the same issue... has there been any progress on this?
Issue: Very slow data loading on large dataset

Comment:
> I'm literally crying waiting for the trainer to restart from checkpoint. It's getting stuck at get_train_dataloader and I think this is to do with the same issue...

Once the dataset is cached once, it's not regenerated again. Your issue seems different
Issue: Very slow data loading on large dataset

Comment:
hmmm, yes. I'll come back with details on this, fairly easy to reproduce. Takes about 30 minutes to get from checkpoint loading to starting training...
Issue: [Dataset] `NonMatchingChecksumError` due to an update in the LinCE benchmark data

Comment:
Hi @gaguilar 

If you want to take care of this, it very simple, you just need to regenerate the `dataset_infos.json` file as indicated [in the doc](https://huggingface.co/nlp/share_dataset.html#adding-metadata) by [installing from source](https://huggingface.co/nlp/installation.html#installing-from-source) and running the following command from the root of the repo:
```bash
python nlp-cli test ./datasets/lince --save_infos --all_configs
```
And then you can open a pull-request with the updated json file.

Otherwise we'll do it sometime this week.