SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: OSCAR from Inria group

Comment:
PR is already open here : #348 
The only thing remaining is to compute the metadata of each subdataset (one per language + shuffled/unshuffled).
As soon as #863 is merged we can start computing them. This will take a bit of time though
Issue: Have Trouble importing `datasets`

Comment:
I'm sorry, this was a problem with my environment.
Now that I have identified the cause of environmental dependency, I would like to fix it and try it.
Excuse me for making a noise.
Issue: Unable to download cnn_dailymail dataset

Comment:
Same here! My kaggle notebook stopped working like yesterday. It's strange because I have fixed version of datasets==1.1.2
Issue: Unable to download cnn_dailymail dataset

Comment:
I couldn't reproduce unfortunately. I tried
```python
from datasets import load_dataset

load_dataset("cnn_dailymail", "3.0.0", download_mode="force_redownload")
```
and it worked fine on both my env (python 3.7.2) and colab (python 3.6.9)

Maybe there was an issue with the google drive download link of the dataset ?
Are you still having the issue ? If so could your give me more info about your python and requests version ?
Issue: Unable to download cnn_dailymail dataset

Comment:
No, It's working fine now. Very strange. Here are my python and request versions

requests         2.24.0
Python 3.8.2
Issue: Possible Bug: Small training/dataset file creates gigantic output

Comment:
The preprocessing tokenizes the input text. Tokenization outputs `input_ids`, `attention_mask`, `token_type_ids` and `special_tokens_mask`. All those are of length`max_seq_length` because of padding. Therefore for each sample it generate 4 *`max_seq_length` integers. Currently they're all saved as int64. This is why the tokenization takes so much space.

I'm sure we can optimize that though
What do you think @sgugger ?
Issue: Possible Bug: Small training/dataset file creates gigantic output

Comment:
First I think we should disable padding in the dataset processing and let the data collator do it.

Then I'm wondering if you need attention_mask and token_type_ids at this point ?

Finally we can also specify the output feature types at this line https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py#L280 to use more optimized integer precisions for the output. Maybe something like:
- input_ids: uint16 or uint32
- token_type_ids: uint8 or bool
- attention_mask: bool
- special_tokens_mask: bool

Also IMO these changes are all on the `transformers` side. Maybe we should discuss on the `transformers` repo
Issue: Possible Bug: Small training/dataset file creates gigantic output

Comment:
> First I think we should disable padding in the dataset processing and let the data collator do it.

No, you can't do that on TPUs as dynamic shapes will result in a very slow training. The script can however be tweaked to use the `PaddingDataCollator` with a fixed max length instead of dynamic batching.

For the other optimizations, they can be done by changing the script directly for each user's use case. Not sure we can find something that is general enough to be in transformers or the examples script.
Issue: Possible Bug: Small training/dataset file creates gigantic output

Comment:
Oh yes right..
Do you think that a lazy map feature on the `datasets` side could help to avoid storing padded tokenized texts then ?
Issue: Possible Bug: Small training/dataset file creates gigantic output

Comment:
I think I can do the tweak mentioned above with the data collator as short fix (but fully focused on v4 right now so that will be for later this week, beginning of next week :-) ).
If it doesn't hurt performance to tokenize on the fly, that would clearly be the long-term solution however!
Issue: Possible Bug: Small training/dataset file creates gigantic output

Comment:
> Hey guys,
> 
> I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.
> 
> I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?
> 
> I've used the following CMD:
> `python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

It's actually because of the parameter 'preprocessing_num_worker' when using TPU. 
I am also planning to have my model trained on the google TPU with a 11gb text corpus. With x8 cores enabled, each TPU core has its own dataset.  When not using distributed training, the preprocessed file is about 77gb. On the opposite, if enable xla, the file produced will easily consume all my free space(more than 220gb, I think it will be, in the end, around 600gb ). 
So I think that's maybe where the problem came from. 

Is there any possibility that all of the cores share the same preprocess dataset?

@sgugger @RammMaschine
Issue: Possible Bug: Small training/dataset file creates gigantic output

Comment:
Hi @NebelAI, we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.
Issue: wmt16 does not download 

Comment:
Hi,I also posted it to the forum, but this is a bug, perhaps it needs to be reported here? thanks
Issue: wmt16 does not download 

Comment:
It looks like the official OPUS server for WMT16 doesn't provide the data files anymore (503 error).
I searched a bit and couldn't find a mirror except maybe http://nlp.ffzg.hr/resources/corpora/setimes/ (the data are a cleaned version of the original ones though)
Should we consider replacing the old urls with these ones even though it's not the exact same data ?
Issue: wmt16 does not download 

Comment:
The data storage is down at the moment. Sorry. Hopefully, it will come back soon. Apologies for the inconvenience ...
Issue: wmt16 does not download 

Comment:
Dear great huggingface team, this is not working yet, I really appreciate some temporary fix on this, I need this for my project and this is time sensitive and I will be grateful for your help on this.
Issue: wmt16 does not download 

Comment:
We have reached out to the OPUS team which is currently working on making the data available again. Cc @jorgtied
Issue: wmt16 does not download 

Comment:
Hi, this is still down, I would be really grateful if you could ping them one more time. thank you so much.
Issue: wmt16 does not download 

Comment:
Hi
I am trying with multiple setting of wmt datasets and all failed so far, I need to have at least one dataset working for testing somecodes, and this is really time sensitive, I greatly appreciate letting me know of one translation datasets currently working. thanks
Issue: wmt16 does not download 

Comment:
It is still down, unfortunately. I'm sorry for that. It should come up again later today or tomorrow at the latest if no additional complications will happen.
Issue: concatenate_datasets support axis=0 or 1 ？

Comment:
Unfortunately `concatenate_datasets` only supports concatenating the rows, while what you want to achieve is concatenate the columns.
Currently to add more columns to a dataset, one must use `map`.
What you can do is somehting like this:
```python
# suppose you have datasets d1, d2, d3
def add_columns(example, index):
    example.update(d2[index])
    example.update(d3[index])
    return example

full_dataset = d1.map(add_columns, with_indices=True)
```
Issue: concatenate_datasets support axis=0 or 1 ？

Comment:
That's not really difficult to add, though, no?
I think it can be done without copy.
Maybe let's add it to the roadmap?
Issue: concatenate_datasets support axis=0 or 1 ？

Comment:
Actually it's doable but requires to update the `Dataset._data_files` schema to support this.
I'm re-opening this since we may want to add this in the future
Issue: concatenate_datasets support axis=0 or 1 ？

Comment:
Hi @lhoestq, I would love to help and add this feature if still needed. My plan is to add an axis variable in the `concatenate_datasets` function in `arrow_dataset.py` and when that is set to 1 concatenate columns instead of rows.
Issue: concatenate_datasets support axis=0 or 1 ？

Comment:
Hi ! I would love to see this feature implemented as well :) Thank you for proposing your help !

Here is a few things about the current implementation:
- A dataset object is a wrapper of one `pyarrow.Table` that contains the data
- Pyarrow offers an API that allows to transform Table objects. For example there are functions like `concat_tables`, `Table.rename_columns`, `Table.add_column` etc.

Therefore adding columns from another dataset is possible thanks to the pyarrow API and in particular `Table.add_column` :) 

However this breaks some features we have regarding pickle. A dataset object can be pickled and unpickled without loading all the data in memory. It is useful for multiprocessing for example. Pickling a dataset object is possible thanks to the `Dataset._data_files` which defines the list of arrow files that will be used to form the final Table (basically all the data from each files are concatenated on axis 0).

Therefore to be able to add columns to a Dataset and still be able to work with it in a multiprocessing setup, we need to extend this last aspect to be able to reconstruct a Table object from multiple arrow files that are combined in both axis 0 and 1. Currently this reconstruction mechanism only supports axis 0.

I'm sure we can figure something out that enables users to add columns from another dataset while keeping the multiprocessing support.
Issue: concatenate_datasets support axis=0 or 1 ？

Comment:
@lhoestq, we have two Pull Requests to implement:
- Dataset.add_item: #1870
- Dataset.add_column: #2145
which add a single row or column, repectively.

The request here is to implement the concatenation of *multiple* rows/columns. Am I right?

We should agree on the API:
- `concatenate_datasets` with `axis`?
- other Dataset method name?
Issue: concatenate_datasets support axis=0 or 1 ？

Comment:
For the API, I like `concatenate_datasets` with `axis` personally :)
From a list of `Dataset` objects, it would concatenate them to a new `Dataset` object backed by a `ConcatenationTable`, that is the concatenation of the tables of each input dataset. The concatenation is either on axis=0 (append rows) or on axis=1 (append columns).

Regarding what we need to implement:
The axis=0 is already supported and is the current behavior of `concatenate_datasets`.
Also `add_item` is not needed to implement axis=1 (though it's an awesome addition to this library).

To implement axis=1, we either need `add_column` or a `ConcatenationTable` constructor to concatenate tables horizontally.
I have a preference for using a `ConcatenationTable` constructor because this way we can end up with a `ConcatenationTable` with only 1 additional block per table, while `add_column` would add 1 block per new column.

Maybe we can simply have an equivalent of `ConcatenationTable.from_tables` but for axis=1 ?
`axis` could also be an argument of `ConcatenationTable.from_tables`
Issue: Load amazon dataset

Comment:
Thanks for reporting !
We plan to show information about the different configs of the datasets on the website, with the corresponding `load_dataset` calls.

Also I think the bullet points formatting has been fixed
Issue: Error when concatenate_datasets

Comment:
As you can see in the error the test checks if `indices_mappings_in_memory` is True or not, which is different from the test you do in your script. In a dataset, both the data and the indices mapping can be either on disk or in memory.

The indices mapping correspond to a mapping on top of the data table that is used to re-order/select a sample of the original data table. For example if you do `dataset.train_test_split`, then the resulting train and test datasets will have both an indices mapping to tell which examples are in train and which ones in test.

Before saving your datasets on disk, you should call `dataset.flatten_indices()` to remove the indices mapping. It should fix your issue. Under the hood it will create a new data table using the indices mapping. The new data table is going to be a subset of the old one (for example taking only the test set examples), and since the indices mapping will be gone you'll be able to concatenate your datasets.
Issue: Error when concatenate_datasets

Comment:
> As you can see in the error the test checks if `indices_mappings_in_memory` is True or not, which is different from the test you do in your script. In a dataset, both the data and the indices mapping can be either on disk or in memory.
> 
> The indices mapping correspond to a mapping on top of the data table that is used to re-order/select a sample of the original data table. For example if you do `dataset.train_test_split`, then the resulting train and test datasets will have both an indices mapping to tell which examples are in train and which ones in test.
> 
> Before saving your datasets on disk, you should call `dataset.flatten_indices()` to remove the indices mapping. It should fix your issue. Under the hood it will create a new data table using the indices mapping. The new data table is going to be a subset of the old one (for example taking only the test set examples), and since the indices mapping will be gone you'll be able to concatenate your datasets.

`dataset.flatten_indices()` solved my problem, thanks so much!
Issue: Error when concatenate_datasets

Comment:
@lhoestq we can add a mention of `dataset.flatten_indices()` in the error message (no rush, just put it on your TODO list or I can do it when I come at it)
Issue: multiprocessing in dataset map "can only test a child process"

Comment:
It looks like an issue with wandb/tqdm here.
We're using the `multiprocess` library instead of the `multiprocessing` builtin python package to support various types of mapping functions. Maybe there's some sort of incompatibility.

Could you make a minimal script to reproduce or a google colab ?
Issue: multiprocessing in dataset map "can only test a child process"

Comment:
It looks like this warning : 
"Truncation was not explicitly activated but max_length is provided a specific value, "
is not handled well by wandb.

The error occurs when calling the tokenizer.
Maybe you can try to specify `truncation=True` when calling the tokenizer to remove the warning ?
Otherwise I don't know why wandb would fail on a warning. Maybe one of its logging handlers have some issues with the logging of tokenizers. Maybe @n1t0 knows more about this ?
Issue: multiprocessing in dataset map "can only test a child process"

Comment:
Isn't it more the pytorch warning on the use of non-writable memory for tensor that trigger this here @lhoestq? (since it seems to be a warning triggered in `torch.tensor()`
Issue: multiprocessing in dataset map "can only test a child process"

Comment:
Yep this time this is a warning from pytorch that causes wandb to not work properly.
Could this by a wandb issue ?
Issue: multiprocessing in dataset map "can only test a child process"

Comment:
Hi @timothyjlaurent @gaceladri 
If you're running `transformers` from `master` you can try setting the env var `WAND_DISABLE=true` (from https://github.com/huggingface/transformers/pull/9896) and try again ?
This issue might be related to https://github.com/huggingface/transformers/issues/9623
Issue: multiprocessing in dataset map "can only test a child process"

Comment:
I have commented the lines that cause my code break. I'm now seeing my reports on Wandb and my code does not break. I am training now, so I will check probably in 6 hours. I suppose that setting wandb disable will work as well.
Issue: Add HoVer multi-hop fact verification dataset

Comment:
Hi @yjernite  I'm new but wanted to contribute. Has anyone already taken this problem and do you think it is suitable for newbies?
Issue: Add HoVer multi-hop fact verification dataset

Comment:
Hi @tenjjin! This dataset is still up for grabs! Here's the link with the guide to add it. You should play around with the library first (download and look at a few datasets), then follow the steps here:

https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md
Issue: use_custom_baseline still produces errors for bertscore

Comment:
Thanks for reporting ! That's a bug indeed
If you want to contribute, feel free to fix this issue and open a PR :)
Issue: use_custom_baseline still produces errors for bertscore

Comment:
This error is because of a mismatch between `datasets` and `bert_score`. With `datasets=1.1.2` and `bert_score>=0.3.6` it works ok. So `pip install -U bert_score` should fix the problem.
Issue: use_custom_baseline still produces errors for bertscore

Comment:
Hello everyone,

I think the  problem is not solved: 

```
from datasets import load_metric
metric=load_metric('bertscore')
metric.compute(
    predictions=predictions,
    references=references,
    lang='fr',
    rescale_with_baseline=True
)
TypeError: get_hash() missing 2 required positional arguments: 'use_custom_baseline' and 'use_fast_tokenizer'
```
This code is produced using `Python 3.6.9 datasets==1.1.2 and bert_score==0.3.10`
Issue: use_custom_baseline still produces errors for bertscore

Comment:
Hi ! This has been fixed by https://github.com/huggingface/datasets/pull/2770, we'll do a new release soon to make the fix available :)

In the meantime please use an older version of `bert_score`
Issue: How to enable `.map()` pre-processing pipelines to support multi-node parallelism?

Comment:
Right now multiprocessing only runs on single node.

However it's probably possible to extend it to support multi nodes. Indeed we're using the `multiprocess` library from the `pathos` project to do multiprocessing in `datasets`, and `pathos` is made to support parallelism on several nodes. More info about pathos [on the pathos repo](https://github.com/uqfoundation/pathos).

If you're familiar with pathos or if you want to give it a try, it could be a nice addition to the library :)
Issue: How to enable `.map()` pre-processing pipelines to support multi-node parallelism?

Comment:
Curious to hear if anything on that side changed or if you suggestions to do it changed @lhoestq :)

For our use-case, we are entering the regime where trading a few more instances to save a few days would be nice :)
Issue: How to enable `.map()` pre-processing pipelines to support multi-node parallelism?

Comment:
Currently for multi-node setups we're mostly going towards a nice integration with Dask. But I wouldn't exclude exploring `pathos` more at one point
Issue: Can not reuse datasets already downloaded

Comment:
It seems the process needs '/datasets.huggingface.co/datasets/datasets/wikipedia/wikipedia.py'
Where and how to assign this ```wikipedia.py``` after I manually download it ?
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
Which version of pyarrow do you have ? Could you try to update pyarrow and try again ?
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
Thanks for the fast response. I have the latest version '2.0.0' (I tried to update)
I am working with Python 3.8.5
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
I think that the issue is similar to this one:https://issues.apache.org/jira/browse/ARROW-9612
The problem is in arrow when the column data contains long strings.
Any ideas on how to bypass this?
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
We should expose the [`block_size` argument](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions) of Apache Arrow csv `ReadOptions` in the [script](https://github.com/huggingface/datasets/blob/master/datasets/csv/csv.py).


In the meantime you can specify yourself the `ReadOptions` config like this:
```python
import pyarrow.csv as pac   # PyArrow is installed with `datasets`

read_options = pac.ReadOptions(block_size=1e9)  # try to find the right value for your use-case
dataset = load_dataset('csv', data_files=files, read_options=read_options)
```
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
This did help to load the data. But the problem now is that I get:
ArrowInvalid: CSV parse error: Expected 5 columns, got 187

It seems that this change the parsing so I changed the table to tab-separated and tried to load it directly from pyarrow
But I got a similar error, again it loaded fine in pandas so I am not sure what to do.
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
Got almost the same error loading a ~5GB TSV file, first got the same error as OP, then tried giving it my own ReadOptions and also got the same CSV parse error.
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
> We should expose the [`block_size` argument](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions) of Apache Arrow csv `ReadOptions` in the [script](https://github.com/huggingface/datasets/blob/master/datasets/csv/csv.py).
> 
> In the meantime you can specify yourself the `ReadOptions` config like this:
> 
> ```python
> import pyarrow.csv as pac   # PyArrow is installed with `datasets`
> 
> read_options = pac.ReadOptions(block_size=1e9)  # try to find the right value for your use-case
> dataset = load_dataset('csv', data_files=files, read_options=read_options)
> ```

This did not work for me, I got
`TypeError: __init__() got an unexpected keyword argument 'read_options'`
Issue: load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas

Comment:
Hi ! Yes because of issues with PyArrow's CSV reader we switched to using the Pandas CSV reader. In particular the `read_options` argument is not supported anymore, but you can pass any parameter of Pandas' `read_csv` function (see the list here in [Pandas documentation](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html))
Issue: Wikipedia postprocessing

Comment:
Hi @bminixhofer ! Parsing WikiMedia is notoriously difficult: this processing used [mwparserfromhell](https://github.com/earwig/mwparserfromhell) which is pretty good but not perfect.

As an alternative, you can also use the Wiki40b dataset which was pre-processed using an un-released Google internal tool
Issue: [GEM] add WikiLingua cross-lingual abstractive summarization dataset

Comment:
Hey @yjernite. This is a very interesting dataset. Would love to work on adding it but I see that the link to the data is to a gdrive folder. Can I just confirm wether dlmanager can handle gdrive urls or would this have to be a manual dl?