devopsmarc/my-issues-dataset huggingface dataset:
nlp re-creates already-there caches when using a script, but not within a shell 
 `nlp` keeps creating new caches for the same file when launching `filter` from a script, and behaves correctly from within the shell.

Example: try running

```
import nlp

hans_easy_data = nlp.load_dataset('hans', split="validation").filter(lambda x: x['label'] == 0)
hans_hard_data = nlp.load_dataset('hans', split="validation").filter(lambda x: x['label'] == 1)
```

twice. If launched from a `file.py` script, the cache will be re-created the second time. If launched as 3 shell/`ipython` commands, `nlp` will correctly re-use the cache.
As observed with @lhoestq. 
 Couln't reproduce on my side :/ 
let me know if you manage to reproduce on another env (colab for example)
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 Some wikipedia languages have already been processed by us and are hosted on our google storage. This is the case for "fr" and "en" for example.

For other smaller languages (in terms of bytes), they are directly downloaded and parsed from the wikipedia dump site.
Parsing can take some time for languages with hundreds of MB of xml.

Let me know if you encounter an error or if you feel that is is taking too long for you.
We could process those that really take too much time
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 Thanks ! This will be very helpful.

About the date issue, I think it's possible to use another date with

```python
load_dataset("wikipedia", language="es", date="...", beam_runner="...")
```

However we've not processed wikipedia dumps for other dates than 20200501 (yet ?)

One more thing that is specific to 20200501.es: it was available once but the `mwparserfromhell` was not able to parse it for some reason, so we didn't manage to get a processed version of 20200501.es (see #321 )
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 Hey @gaguilar ,

I just found the ["char2subword" paper](https://arxiv.org/pdf/2010.12730.pdf) and I'm really interested in trying it out on own vocabs/datasets like for historical texts (I've already [trained some lms](https://github.com/stefan-it/europeana-bert) on newspaper articles with OCR errors).

Do you plan to release the code for your paper or is it possible to get the implementation 🤔 Many thanks :hugs:
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 Hi @stefan-it! Thanks for your interest in our work! We do plan to release the code, but we will make it available once the paper has been published at a conference. Sorry for the inconvenience!

Hi @lhoestq, do you have any insights for this issue by any chance? Thanks!
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 This is an issue on the `mwparserfromhell` side. You could try to update `mwparserfromhell` and see if it fixes the issue. If it doesn't we'll have to create an issue on their repo for them to fix it.
But first let's see if the latest version of `mwparserfromhell` does the job.
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 Hi ! The link https://dumps.wikimedia.org/idwiki/20210501/dumpstatus.json seems to be working fine for me.

Regarding the time outs, it must come either from an issue on the wikimedia host side, or from your internet connection.
Feel free to try again several times.
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 Hi ! This looks related to this issue: https://github.com/huggingface/datasets/issues/1994
Basically the parser that is used (mwparserfromhell) has some issues for some pages in `es`.
We already reported some issues for `es` on their repo at https://github.com/earwig/mwparserfromhell/issues/247 but it looks like there are still a few issues. Might be a good idea to open a new issue on the mwparserfromhell repo
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 The issue:
```
KeyError: "000nbsp [while running 'train/Clean content']"
```
reported in comments:
- https://github.com/huggingface/datasets/issues/577#issuecomment-701890059 (by @gaguilar)
- https://github.com/huggingface/datasets/issues/577#issuecomment-879513227 (by @mmiakashs)

was normally fixed in the `mwparserfromhell` library and will be accessible in their next release version `0.7`:
- https://github.com/earwig/mwparserfromhell/issues/288
Some languages in wikipedia dataset are not loading 
 Hi,

I am working with the `wikipedia` dataset and I have a script that goes over 92 of the available languages in that dataset. So far I have detected that `ar`, `af`, `an` are not loading. Other languages like `fr` and `en` are working fine. Here's how I am loading them:

```
import nlp

langs = ['ar'. 'af', 'an']

for lang in langs:
    data = nlp.load_dataset('wikipedia', f'20200501.{lang}', beam_runner='DirectRunner', split='train') 
    print(lang, len(data))
```

Here's what I see for 'ar' (it gets stuck there):
```
Downloading and preparing dataset wikipedia/20200501.ar (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to /home/gaguilar/.cache/huggingface/datasets/wikipedia/20200501.ar/1.0.0/7be7f4324255faf70687be8692de57cf79197afdc33ff08d6a04ed602df32d50...
```

Note that those languages are indeed in the list of expected languages. Any suggestions on how to work around this? Thanks! 
 mwparserfromhell 0.7 has still not been released, but you might have luck with the dev version:
`pip install git+https://github.com/earwig/mwparserfromhell.git@0f89f44`
Couldn't reach certain URLs and for the ones that can be reached, code just blocks after downloading. 
 Hi,

I'm following the [quick tour](https://huggingface.co/nlp/quicktour.html) and tried to load the glue dataset:
```
>>> from nlp import load_dataset
>>> dataset = load_dataset('glue', 'mrpc', split='train')
```

However, this ran into a `ConnectionError` saying it could not reach the URL (just pasting the last few lines):
```

/net/vaosl01/opt/NFS/su0/miniconda3/envs/hf/lib/python3.7/site-packages/nlp/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only)
    354                 " to False."
    355             )
--> 356         raise ConnectionError("Couldn't reach {}".format(url))
    357 
    358     # From now on, connected is True.

ConnectionError: Couldn't reach https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2Fmrpc_dev_ids.tsv?alt=media&token=ec5c0836-31d5-48f4-b431-7480817f1adc
```

I tried glue with cola and sst2. I got the same error, just instead of mrpc in the URL, it was replaced with cola and sst2.

Since this was not working, I thought I'll try another dataset. So I tried downloading the imdb dataset:
```
ds = load_dataset('imdb', split='train')
```
This downloads the data, but it just blocks after that:
```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.56k/4.56k [00:00<00:00, 1.38MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.07k/2.07k [00:00<00:00, 1.15MB/s]
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown sizetotal: 207.28 MiB) to /net/vaosl01/opt/NFS/su0/huggingface/datasets/imdb/plain_text/1.0.0/76cdbd7249ea3548c928bbf304258dab44d09cd3638d9da8d42480d1d1be3743...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 84.1M/84.1M [00:07<00:00, 11.1MB/s]
```

I checked the folder `$HF_HOME/datasets/downloads/extracted/<id>/aclImdb`. This folder is constantly growing in size. When I navigated to the train folder within, there was no file. However, the test folder seemed to be populating. The last time I checked it was 327M. I thought the Imdb dataset was smaller than that. My questions are:
1. Why is it still blocking? Is it still downloading?
2. I specified split as train, so why is the test folder being populated?
3. I read somewhere that after downloading, `nlp` converts the text files into some sort of `arrow` files, which will also take a while. Is this also happening here?

Thanks.
 
 @jeswan `"cola"` is not a valid dataset identifier (you can check the up-to-date list on https://huggingface.co/datasets) but you can find cola inside glue.
`metric.compute` throws `ArrowInvalid` error 
 I get the following error with `rouge.compute`. It happens only with distributed training, and it occurs randomly I can't easily reproduce it. This is using `nlp==0.4.0`

```
  File "/home/beltagy/trainer.py", line 92, in validation_step
    rouge_scores = rouge.compute(predictions=generated_str, references=gold_str, rouge_types=['rouge2', 'rouge1', 'rougeL'])
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/metric.py", line 224, in compute
    self.finalize(timeout=timeout)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/metric.py", line 213, in finalize
    self.data = Dataset(**reader.read_files(node_files))
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 217, in read_files
    dataset_kwargs = self._read_files(files=files, info=self._info, original_instructions=original_instructions)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 162, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 276, in _get_dataset_from_filename
    f = pa.ipc.open_stream(mmap)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/pyarrow/ipc.py", line 173, in open_stream
    return RecordBatchStreamReader(source)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/pyarrow/ipc.py", line 64, in __init__
    self._open(source)
  File "pyarrow/ipc.pxi", line 469, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
``` 
 Could you try to update to `datasets>=1.0.0` (we changed the name of the library) and try again ?
If is was related to the distributed setup settings it must be fixed.
If it was related to empty metric inputs it's going to be fixed in #654
`metric.compute` throws `ArrowInvalid` error 
 I get the following error with `rouge.compute`. It happens only with distributed training, and it occurs randomly I can't easily reproduce it. This is using `nlp==0.4.0`

```
  File "/home/beltagy/trainer.py", line 92, in validation_step
    rouge_scores = rouge.compute(predictions=generated_str, references=gold_str, rouge_types=['rouge2', 'rouge1', 'rougeL'])
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/metric.py", line 224, in compute
    self.finalize(timeout=timeout)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/metric.py", line 213, in finalize
    self.data = Dataset(**reader.read_files(node_files))
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 217, in read_files
    dataset_kwargs = self._read_files(files=files, info=self._info, original_instructions=original_instructions)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 162, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/nlp/arrow_reader.py", line 276, in _get_dataset_from_filename
    f = pa.ipc.open_stream(mmap)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/pyarrow/ipc.py", line 173, in open_stream
    return RecordBatchStreamReader(source)
  File "/home/beltagy/miniconda3/envs/allennlp/lib/python3.7/site-packages/pyarrow/ipc.py", line 64, in __init__
    self._open(source)
  File "pyarrow/ipc.pxi", line 469, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
``` 
 Closing this one as it was fixed in #654 
Feel free to re-open if you have other questions
No module named 'nlp.logging' 
 Hi, I am using nlp version 0.4.0. Trying to use bleurt as an eval metric, however, the bleurt script imports nlp.logging which creates the following error. What am I missing?

```
>>> import nlp
2020-09-02 13:47:09.210310: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> bleurt = nlp.load_metric("bleurt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/load.py", line 443, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/load.py", line 61, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/metrics/bleurt/43448cf2959ea81d3ae0e71c5c8ee31dc15eed9932f197f5f50673cbcecff2b5/bleurt.py", line 20, in <module>
    from nlp.logging import get_logger
ModuleNotFoundError: No module named 'nlp.logging'
```

Just to show once again that I can't import the logging module:

```
>>> import nlp
2020-09-02 13:48:38.190621: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> nlp.__version__
'0.4.0'
>>> from nlp.logging import get_logger
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'nlp.logging'
``` 
 Thanks for reporting.

Apparently this is a versioning issue: the lib downloaded the `bleurt` script from the master branch where we did this change recently. We'll fix that in a new release this week or early next week. Cc @thomwolf 

Until that, I'd suggest you to download the right bleurt folder from github ([this one](https://github.com/huggingface/nlp/tree/0.4.0/metrics/bleurt)) and do

```python
from nlp import load_metric

bleurt = load_metric("path/to/bleurt/folder")
```

To download it you can either clone the repo or download the `bleurt.py` file and place it in a folder named `bleurt`
No module named 'nlp.logging' 
 Hi, I am using nlp version 0.4.0. Trying to use bleurt as an eval metric, however, the bleurt script imports nlp.logging which creates the following error. What am I missing?

```
>>> import nlp
2020-09-02 13:47:09.210310: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> bleurt = nlp.load_metric("bleurt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/load.py", line 443, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/load.py", line 61, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/melody/anaconda3/envs/transformers/lib/python3.6/site-packages/nlp/metrics/bleurt/43448cf2959ea81d3ae0e71c5c8ee31dc15eed9932f197f5f50673cbcecff2b5/bleurt.py", line 20, in <module>
    from nlp.logging import get_logger
ModuleNotFoundError: No module named 'nlp.logging'
```

Just to show once again that I can't import the logging module:

```
>>> import nlp
2020-09-02 13:48:38.190621: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> nlp.__version__
'0.4.0'
>>> from nlp.logging import get_logger
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'nlp.logging'
``` 
 Actually we can fix this on our side, this script didn't had to be updated. I'll do it in a few minutes
Using custom DownloadConfig results in an error 
 ## Version / Environment

Ubuntu 18.04
Python 3.6.8
nlp 0.4.0

## Description

Loading `imdb` dataset works fine when when I don't specify any `download_config` argument. When I create a custom `DownloadConfig` object and pass it to the `nlp.load_dataset` function, this results in an error.

## How to reproduce

### Example without DownloadConfig --> works

```python
import os

os.environ["HF_HOME"] = "/data/hf-test-without-dl-config-01/"

import logging
import nlp

logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    imdb = nlp.load_dataset(path="imdb")
```

### Example with DownloadConfig --> doesn't work

```python
import os

os.environ["HF_HOME"] = "/data/hf-test-with-dl-config-01/"

import logging
import nlp
from nlp.utils import DownloadConfig

logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    download_config = DownloadConfig()
    imdb = nlp.load_dataset(path="imdb", download_config=download_config)
```

Error traceback:

```
Traceback (most recent call last):
  File "/.../example_with_dl_config.py", line 13, in <module>
    imdb = nlp.load_dataset(path="imdb", download_config=download_config)
  File "/.../python3.6/python3.6/site-packages/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/.../python3.6/python3.6/site-packages/nlp/builder.py", line 463, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/.../python3.6/python3.6/site-packages/nlp/builder.py", line 518, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/.../python3.6/python3.6/site-packages/nlp/datasets/imdb/76cdbd7249ea3548c928bbf304258dab44d09cd3638d9da8d42480d1d1be3743/imdb.py", line 86, in _split_generators
    arch_path = dl_manager.download_and_extract(_DOWNLOAD_URL)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 220, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 158, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 108, in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum_dict(path)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/info_utils.py", line 79, in get_size_checksum_dict
    with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/data/hf-test-with-dl-config-01/datasets/extracted/b6802c5b61824b2c1f7dbf7cda6696b5f2e22214e18d171ce1ed3be90c931ce5'
```

 
 Thanks for the report, I'll take a look.

What is your specific use-case for providing a DownloadConfig object?
Using custom DownloadConfig results in an error 
 ## Version / Environment

Ubuntu 18.04
Python 3.6.8
nlp 0.4.0

## Description

Loading `imdb` dataset works fine when when I don't specify any `download_config` argument. When I create a custom `DownloadConfig` object and pass it to the `nlp.load_dataset` function, this results in an error.

## How to reproduce

### Example without DownloadConfig --> works

```python
import os

os.environ["HF_HOME"] = "/data/hf-test-without-dl-config-01/"

import logging
import nlp

logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    imdb = nlp.load_dataset(path="imdb")
```

### Example with DownloadConfig --> doesn't work

```python
import os

os.environ["HF_HOME"] = "/data/hf-test-with-dl-config-01/"

import logging
import nlp
from nlp.utils import DownloadConfig

logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    download_config = DownloadConfig()
    imdb = nlp.load_dataset(path="imdb", download_config=download_config)
```

Error traceback:

```
Traceback (most recent call last):
  File "/.../example_with_dl_config.py", line 13, in <module>
    imdb = nlp.load_dataset(path="imdb", download_config=download_config)
  File "/.../python3.6/python3.6/site-packages/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/.../python3.6/python3.6/site-packages/nlp/builder.py", line 463, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/.../python3.6/python3.6/site-packages/nlp/builder.py", line 518, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/.../python3.6/python3.6/site-packages/nlp/datasets/imdb/76cdbd7249ea3548c928bbf304258dab44d09cd3638d9da8d42480d1d1be3743/imdb.py", line 86, in _split_generators
    arch_path = dl_manager.download_and_extract(_DOWNLOAD_URL)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 220, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 158, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 108, in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum_dict(path)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/info_utils.py", line 79, in get_size_checksum_dict
    with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/data/hf-test-with-dl-config-01/datasets/extracted/b6802c5b61824b2c1f7dbf7cda6696b5f2e22214e18d171ce1ed3be90c931ce5'
```

 
 I see.

Probably the easiest way for you would be that we add simple serialization/deserialization methods to the Dataset and DatasetDict objects once the data files have been downloaded and all the dataset is processed.

What do you think @lhoestq ?
nlp downloads to its module path 
 I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`? 
 Indeed this is a known issue arising from the fact that we try to be compatible with cloupickle.

Does this also happen if you are installing in a virtual environment?
nlp downloads to its module path 
 I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`? 
 > Indeed this is a know issue with the fact that we try to be compatible with cloupickle.
> 
> Does this also happen if you are installing in a virtual environment?

Then it would work, because the package is in a writable path.
nlp downloads to its module path 
 I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`? 
 > If it's fine for you then this is the recommended way to solve this issue.

I don't want to use a virtual environment, because Nix is fully reproducible, and virtual environments are not. And I am the maintainer of the `transformers` in nixpkgs, so sooner or later I will have to package `nlp`, since it is becoming a dependency of `transformers` ;).
nlp downloads to its module path 
 I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`? 
 Ok interesting. We could have another check to see if it's possible to download and import the datasets script at another location than the module path. I think this would probably involve tweaking the python system path dynamically.

I don't know anything about Nix so if you want to give this a try your self we can guide you or you can give us more information on your general project and how this works.

Regarding `nlp` and `transformers`, we are not sure `nlp` will become a required dependency for `transformers`.  It will probably be used a lot in the examples but I think it probably won't be a required dependency for the main package since we try to keep it as light as possible in terms of deps.

Happy to help you make all these things work better for your use-case
nlp downloads to its module path 
 I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`? 
 @danieldk modules are now installed in a different location (by default in the cache directory of the lib, in `~/.cache/huggingface/modules`). You can also change that using the environment variable `HF_MODULES_PATH`

Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
We plan to do a release in the next coming days
nlp downloads to its module path 
 I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`? 
 > Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
> We plan to do a release in the next coming days

Thanks for making this change! I just packaged the latest commit on master and it works like a charm now! :partying_face:
Very slow data loading on large dataset 
 I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ? 
 When you load a text file for the first time with `nlp`, the file is converted into Apache Arrow format. Arrow allows to use memory-mapping, which means that you can load an arbitrary large dataset.

Note that as soon as the conversion has been done once, the next time you'll load the dataset it will be much faster.

However for a 1TB dataset, the conversion can indeed take time. You could try to load parts of it in parallel, and then use `nlp.concatenate_datasets` to get your full dataset.
Very slow data loading on large dataset 
 I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ? 
 Humm, we can give a look at these large scale datasets indeed.

Do you mind sharing a few stats on your dataset so I can try to test on a similar one?

In particular some orders of magnitudes for the number of files, number of lines per files, line lengths.
Very slow data loading on large dataset 
 I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ? 
 @lhoestq Yes, I understand that the first time requires more time. The concatenate_datasets seems to be a workaround, but I believe a multi-processing method should be integrated into load_dataset to make it easier and more efficient for users.

@thomwolf Sure, here are the statistics:
Number of lines: 4.2 Billion
Number of files: 6K
Number of tokens: 800 Billion
The number of lines is distributed equally across these 6k files.
The line length varies between 100 tokens to 40k tokens.
Very slow data loading on large dataset 
 I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ? 
 @agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new `text` processing script which is [here](https://github.com/huggingface/nlp/blob/07d92a82b7594498ff702f3cca55c074e2052257/datasets/text/text.py) somewhere on your drive and give it's local path instead of `text` to `load_dataset`. E.g. in your example:
```python
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```
Very slow data loading on large dataset 
 I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ? 
 I have already generated the dataset, but now I tried to reload it and it is still very slow.

I also have installed your commit and it is slow, even after the dataset was already generated.
`pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257`

It uses only a single thread.

Did I miss something ?