SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: not being able to get wikipedia es language

Comment:
@lhoestq  I really appreciate if you could help me providiing processed datasets, I do not really have access to enough resources to run the apache-beam and need to run the codes on these datasets. Only en/de/fr currently works, but I need all the languages more or less. thanks
Issue: not being able to get wikipedia es language

Comment:
Hi @dorost1234, I think I can help you a little. I’ve processed some Wikipedia datasets (Spanish inclusive) using the HF/datasets library during recent research.

@lhoestq Could you help me to upload these preprocessed datasets to Huggingface's repositories? To be more precise, I've built datasets from the following languages using the 20201201 dumps: Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish. Process these datasets have high costs that most of the community can't afford. I think these preprocessed datasets I have could be helpful for someone without access to high-resource machines to process Wikipedia's dumps like @dorost1234
Issue: not being able to get wikipedia es language

Comment:
Thank you so much @jonatasgrosman , I greatly appreciate your help with them. 
Yes, I unfortunately does not have access to a good resource and need it for my
research. I greatly appreciate @lhoestq  your help with uploading the processed datasets in huggingface datasets. This would be really helpful for some users like me with not access to high-memory GPU resources.

thank you both so much again.

On Sat, Mar 6, 2021 at 12:55 AM Jonatas Grosman <notifications@github.com>
wrote:

> Hi @dorost1234 <https://github.com/dorost1234>, I think I can help you a
> little. I’ve processed some Wikipedia datasets (Spanish inclusive) using
> the HF/datasets library during recent research.
>
> @lhoestq <https://github.com/lhoestq> Could you help me to upload these
> preprocessed datasets to Huggingface's repositories? To be more precise,
> I've built datasets from the following languages using the 20201201 dumps:
> Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish.
> Process these datasets have high costs that most of the community can't
> afford. I think these preprocessed datasets I have could be helpful for
> someone without access to high-resource machines to process Wikipedia's
> dumps like @dorost1234 <https://github.com/dorost1234>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1994#issuecomment-791798195>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMWMK5GFJFU3ACCJFUDTCFVNZANCNFSM4YUZIF4A>
> .
>
Issue: not being able to get wikipedia es language

Comment:
Hi @dorost1234, so sorry, but looking at my files here, I figure out that I've preprocessed files using the HF/datasets for all the languages previously listed by me (Portuguese, Russian, French, Japanese, Chinese, and Turkish) except the Spanish (on my tests I've used the [wikicorpus](https://www.cs.upc.edu/~nlp/wikicorpus/) instead).

Only with the Spanish Wikipedia's dump, I had the same `KeyError: '000nbsp'` problem already reported here https://github.com/huggingface/datasets/issues/577

So nowadays, even with access to a high resource machine, you couldn't be able to get Wikipedia's Spanish data using the HF/datasets :(
Issue: not being able to get wikipedia es language

Comment:
Thanks a lot for the information and help. This would be great to have
these datasets.
@lhoestq <https://github.com/lhoestq>  Do you know a way I could get
smaller amount of these data like 1 GBtype of each language to deal with
computatioanl requirements? thanks

On Sat, Mar 6, 2021 at 5:36 PM Jonatas Grosman <notifications@github.com>
wrote:

> Hi @dorost1234 <https://github.com/dorost1234>, so sorry, but looking at
> my files here, I figure out that I've preprocessed files using the
> HF/datasets for all the languages previously listed by me (Portuguese,
> Russian, French, Japanese, Chinese, and Turkish) except the Spanish (on my
> tests I've used the wikicorpus <https://www.cs.upc.edu/~nlp/wikicorpus/>
> instead).
>
> Only with the Spanish Wikipedia's dump, I had the same KeyError: '000nbsp'
> problem already reported here #577
> <https://github.com/huggingface/datasets/issues/577>
>
> So nowadays, even with access to a high resource machine, you couldn't be
> able to get Wikipedia's Spanish data using the HF/datasets :(
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1994#issuecomment-791985546>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMWMO7WOHWLOROPD6Q3TCJKXPANCNFSM4YUZIF4A>
> .
>
Issue: not being able to get wikipedia es language

Comment:
Hi ! As mentioned above the Spanish configuration have parsing issues from `mwparserfromhell`. I haven't tested with the latest `mwparserfromhell` >=0.6 though. Which version of `mwparserfromhell` are you using ?

> @lhoestq Could you help me to upload these preprocessed datasets to Huggingface's repositories? To be more precise, I've built datasets from the following languages using the 20201201 dumps: Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish. Process these datasets have high costs that most of the community can't afford. I think these preprocessed datasets I have could be helpful for someone without access to high-resource machines to process Wikipedia's dumps like @dorost1234

That would be awesome ! Feel free to ping me on slack so we can put the processed wikipedia files on google storage with the other ones we've already preprocessed.

> Do you know a way I could get smaller amount of these data like 1 GBtype of each language to deal with computatioanl requirements? thanks

I'd suggest to copy the [wikipedia.py](https://github.com/huggingface/datasets/blob/master/datasets/wikipedia/wikipedia.py) to a new script `custom_wikipedia.py` and modify it to only download and process only a subset of the raw data files.
You can for example replace [this line](https://github.com/huggingface/datasets/blob/64e59fc45ca2134218b3e42e83fddddbe840ff74/datasets/wikipedia/wikipedia.py#L446) by:
```python
            if total_bytes >= (1 << 30):  # stop if the total amount of data is >= 1GB
                break
            else:
                xml_urls.append(_base_url(lang) + fname)
```

Then you can load your custom wikipedia dataset with
```python
load_dataset("path/to/my/custom_wikipedia.py", f"{date}.{language}")
```
Issue: not being able to get wikipedia es language

Comment:
Hi @lhoestq!

> Hi ! As mentioned above the Spanish configuration have parsing issues from mwparserfromhell. I haven't tested with the latest mwparserfromhell >=0.6 though. Which version of mwparserfromhell are you using ?

I'm using the latest mwparserfromhell version (0.6)

> That would be awesome ! Feel free to ping me on slack so we can put the processed wikipedia files on google storage with the other ones we've already preprocessed.

I'll ping you there 👍
Issue: not being able to get wikipedia es language

Comment:
Thank you so much @jonatasgrosman  and @lhoestq  this would be a great help. I am really thankful to you both and to wonderful Huggingface dataset library allowing us to train models at scale.
Issue: How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 

Comment:
Hi ! That looks like a bug, can you provide some code so that we can reproduce ?
It's not supposed to update the original dataset
Issue: How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 

Comment:
Hi, I experimented with RAG. 

Actually, you can run the [use_own_knowldge_dataset.py](https://github.com/shamanez/transformers/blob/rag-end-to-end-retrieval/examples/research_projects/rag/use_own_knowledge_dataset.py#L80). In the 80 you can save the dataset object to the disk with save_to_disk. Then in order to compute the embeddings in this use **load_from_disk**. 

Then finally save it. You can see the original dataset object (CSV after splitting also will be changed)

One more thing- when I save the dataset object with **save_to_disk** it name the arrow file with cache.... rather than using dataset. arrow.  Can you add a variable that we can feed a name to save_to_disk function?
Issue: How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 

Comment:
@lhoestq   I also found that cache in tmp directory gets updated after transformations. This is really problematic when using datasets interactively. Let's say we use the shards function to a dataset loaded with csv, atm when we do transformations to shards and combine them it updates the original csv cache.
Issue: How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 

Comment:
I plan to update the save_to_disk method in #2025 so I can make sure the new save_to_disk doesn't corrupt your cache files.
But from your last message it looks like save_to_disk isn't the root cause right ?
Issue: How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 

Comment:
ok, one more thing. When we use save_to_disk there are two files other than .arrow. dataset_info.json and state.json. Sometimes most of the fields in the dataset_infor.json are null, especially when saving dataset objects. Anyways I think load_from_disk uses the arrow files mentioned in state.json right?
Issue: How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 

Comment:
Perfect.  For now, I am loading the dataset from CSV in my interactive process and will wait until you make the PR!
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Hi @hwijeen, you might want to look at issues #1796 and #1949. I think it could be something related to the I/O operations being performed.
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
I see that many people are experiencing the same issue. Is this problem considered an "official" bug that is worth a closer look? @lhoestq
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Yes this looks like a bug. On my side I haven't managed to reproduce it but @theo-m has. We'll investigate this !
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Thank you for the reply! I would be happy to follow the discussions related to the issue.
If you do not mind, could you also give a little more explanation on my p.s.2? I am having a hard time figuring out why the single processing `map` uses all of my cores.
@lhoestq @theo-m
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Regarding your ps2: It depends what function you pass to `map`.
For example, fast tokenizers from `transformers` in Rust tokenize texts and parallelize the tokenization over all the cores.
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
I am still experiencing this issue with datasets 1.9.0..
Has there been a further investigation? 
<img width="442" alt="image" src="https://user-images.githubusercontent.com/29157715/126143387-8b5ddca2-a896-4e18-abf7-4fbf62a48b41.png">
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Hi. Is there any update on this issue? I am desperately trying to decrease my times, and multiprocessing "should" be the solution, but it literally takes 5 times longer.
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Hi,

I’m running into the same issue and trying to come up with a simple benchmark. 

# environment info
I have a total of 80 CPUs.

- `datasets` version: 2.4.0
- Platform: Linux-4.18.0-305.65.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

# How to reproduce

```py
In [1]: from datasets import Dataset, set_caching_enabled                                                                                                       
In [2]: import numpy as np              
In [3]: set_caching_enabled(False)                                                                                                    
In [4]: d = Dataset.from_dict({'foo': np.random.randn(1000,256)})                                                                           
In [9]: d.set_format('np')
In [14]: def sort(array):                                                                                                                   
    ...:     np.sort(array)        
# multiprocessing disabled
In [19]: %%timeit                                                                                                                           
    ...: d.map(sort, input_columns='foo')                                                                                                   
78.8 ms ± 1.22 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)    
# multiprocessing enabled   
In [27]: %%timeit                                                                                                                           
    ...: d.map(sort, input_columns='foo',num_proc=10)               
858 ms ± 45.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)                                                                         
```
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Spawning multiple processes has an overhead. For small datasets the processing is likely to be faster than spawning the processes and passing the data to them.

Especially since your dataset is in memory: the data has to be copied to the subprocesses.
On the other hand, datasets loaded from disk are much faster to reload from a subprocess thanks to memory mapping.
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
Thanks for the clarifications! 

Indeed, when saving then loading the above dataset to disk, and increasing the number of rows to 10K or 100K, the performance gap narrows.

```py
# with 10000 rows
In [3]: %%timeit
   ...: d.map(sort, input_columns='foo')
578 ms ± 5.89 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [4]: %%timeit                                                                                                                                                                                                  
   ...: d.map(sort, input_columns='foo',num_proc=10)                                                                                                                                                              
1.06 s ± 47.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# with 100000 rows
In [6]: %%timeit
   ...: d.map(sort, input_columns='foo')
5.8 s ± 25.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [7]: %%timeit
   ...: d.map(sort, input_columns='foo',num_proc=10)
7.23 s ± 154 ms per loop (mean ± std. dev. of 7 runs, 1 loop each
```
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
any updates on this issue? 
I'm using `datasets=2.12.0`. Adding `num_proc` to the mapping function makes it at least 5x slower than using a single process.
Issue: `datasets.map` multi processing much slower than single processing 

Comment:
What kind of function are you passing to `map` ? How many CPUs do you have and what did you set for `num_proc` ?
Issue: OSError: Memory mapping file failed: Cannot allocate memory

Comment:
Do you think this is trying to bring the dataset into memory and if I can avoid it to save on memory so it only brings a batch into memory? @lhoestq  thank you
Issue: OSError: Memory mapping file failed: Cannot allocate memory

Comment:
It's not trying to bring the dataset into memory.

Actually, it's trying to memory map the dataset file, which is different. It allows to load large dataset files without filling up memory.

What dataset did you use to get this error ?
On what OS are you running ? What's your python and pyarrow version ?
Issue: OSError: Memory mapping file failed: Cannot allocate memory

Comment:
Dear @lhoestq 
thank you so much for coming back to me. Please find info below:
1) Dataset name: I used  wikipedia with config 20200501.en
2) I got these pyarrow in my environment:
pyarrow                   2.0.0                     <pip>
pyarrow                   3.0.0                     <pip>

3) python version 3.7.10
4) OS version 

lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster


Is there a way I could solve the memory issue and if I could run this model, I am using  GeForce GTX 108, 
thanks
Issue: OSError: Memory mapping file failed: Cannot allocate memory

Comment:
I noticed that the error happens when loading the validation dataset.
What value of `data_args.validation_split_percentage` did you use ?
Issue: OSError: Memory mapping file failed: Cannot allocate memory

Comment:
Dear @lhoestq 

thank you very much for the very sharp observation, indeed, this happens there, I use the default value of 5, I basically plan to subsample a part of the large  dataset and choose it as validation set. Do you think this is bringing the data into memory during subsampling? Is there a way I could avoid this?

Thank you very much for the great help.


On Mon, Mar 8, 2021 at 11:28 AM Quentin Lhoest ***@***.***>
wrote:

> I noticed that the error happens when loading the validation dataset.
> What value of data_args.validation_split_percentage did you use ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1990#issuecomment-792655644>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMS337ZUJ7HGGVVCCR3TCSREFANCNFSM4YTYAQ2A>
> .
>
Issue: OSError: Memory mapping file failed: Cannot allocate memory

Comment:
Methods like `dataset.shard`, `dataset.train_test_split`, `dataset.select` etc. don't bring the dataset in memory. 
The only time when samples are brought to memory is when you access elements via `dataset[0]`, `dataset[:10]`, `dataset["my_column_names"]`.

But it's possible that trying to use those methods to build your validation set doesn't fix the issue since, if I understand correctly, the error happens when when the dataset arrow file is opened (just before the 5% percentage is applied).

Did you try to reproduce this issue in a google colab ? This would be super helpful to investigate why this happened.

Also maybe you can try clearing your cache at `~/.cache/huggingface/datasets` and try again. If the arrow file was corrupted somehow, removing it and rebuilding may fix the issue.
Issue: Question/problem with dataset labels

Comment:
It seems that I get parsing errors for various fields in my data. For example now I get this:
```
  File "../../../models/tr-4.3.2/run_puppets.py", line 523, in <module>
    main()
  File "../../../models/tr-4.3.2/run_puppets.py", line 249, in main
    datasets = load_dataset("csv", data_files=data_files)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 572, in download_and_prepare
    self._download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 1028, in _prepare_split
    writer.write_table(table)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/arrow_writer.py", line 292, in write_table
    pa_table = pa_table.cast(self._schema)
  File "pyarrow/table.pxi", line 1311, in pyarrow.lib.Table.cast
  File "pyarrow/table.pxi", line 265, in pyarrow.lib.ChunkedArray.cast
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/pyarrow/compute.py", line 87, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 298, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 192, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Failed to parse string: https://www.netgalley.com/catalog/book/121872
```
Issue: Question/problem with dataset labels

Comment:
Not sure if this helps, this is how I load my files (as in the sample scripts on transformers):

```
    if data_args.train_file.endswith(".csv"):
        # Loading a dataset from local csv files
        datasets = load_dataset("csv", data_files=data_files)
```
Issue: Question/problem with dataset labels

Comment:
Since this worked out of the box in a few examples before, I wonder if it's some quoting issue or something else.
Issue: Question/problem with dataset labels

Comment:
Hi @ioana-blue,
Can you share a sample from your .csv? A dummy where you get this error will also help.

I tried this csv:
```csv
feature,label
1.2,not nurse
1.3,nurse
1.5,surgeon
```
and the following snippet:
```python
from datasets import load_dataset

d = load_dataset("csv",data_files=['test.csv'])

print(d)
print(d['train']['label'])
```
and this works perfectly fine for me:
```sh
DatasetDict({
    train: Dataset({
        features: ['feature', 'label'],
        num_rows: 3
    })
})
['not nurse', 'nurse', 'surgeon']
```
I'm sure your csv is more complicated than this one. But it is hard to tell where the issue might be without looking at a sample.
Issue: Question/problem with dataset labels

Comment:
I've had versions where it worked fain. For this dataset, I had all kind of parsing issues that I couldn't understand. What I ended up doing is strip all the columns that I didn't need and also make the label 0/1. 

I think one line that may have caused a problem was the csv version of this:

```crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz        Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job. ^M ('Rose', '', 'Blakey')  journalist      F       38      journalist      https://www.netgalley.com/catalog/book/121872   _ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job.    She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.```

The error I got in this case is this one: https://github.com/huggingface/datasets/issues/1989#issuecomment-790842771

Note, this line was part of a much larger file and until this line I guess it was working fine.
Issue: Question/problem with dataset labels

Comment:
Hi @ioana-blue,

What is the separator you're using for the csv? I see there are only two commas in the given line, but they don't seem like appropriate points. Also, is this a string part of one line, or an entire line? There should also be a label, right?