devopsmarc/my-issues-dataset huggingface dataset:
imdb dataset cannot be loaded  
 Hi
I am trying to load the imdb train dataset

`dataset = datasets.load_dataset("imdb", split="train")`

getting following errors, thanks for your help 
```
Traceback (most recent call last):        
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=32660064, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='test', num_bytes=26476338, num_examples=20316, dataset_name='imdb')}, {'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
>>> dataset = datasets.load_dataset("imdb", split="train")

```
 
 Hi ! I just tried in 1.8.0 and it worked fine. Can you try again ? Maybe the dataset host had some issues that are fixed now
terminate called after throwing an instance of 'google::protobuf::FatalException' 
 Hi
I am using the dataset "iwslt2017-en-nl", and after downloading it I am getting this error when trying to evaluate it on T5-base with seq2seq_trainer.py in the huggingface repo could you assist me please? thanks 


100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 63/63 [02:47<00:00,  2.18s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0): 
run_t5_base_eval.sh: line 19:  5795 Aborted  
 Loading the iwslt2017-en-nl config of iwslt2017 works fine on my side. 
Maybe you can open an issue on transformers as well ? And also add more details about your environment (OS, python version, version of transformers and datasets etc.)
terminate called after throwing an instance of 'google::protobuf::FatalException' 
 Hi
I am using the dataset "iwslt2017-en-nl", and after downloading it I am getting this error when trying to evaluate it on T5-base with seq2seq_trainer.py in the huggingface repo could you assist me please? thanks 


100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 63/63 [02:47<00:00,  2.18s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0): 
run_t5_base_eval.sh: line 19:  5795 Aborted  
 closing now, figured out this is because the max length of decoder was set smaller than the input_dimensions. thanks
[Feature Request] Add optional parameter in text loading script to preserve linebreaks 
 I'm working on a project about rhyming verse using phonetic poetry and song lyrics, and line breaks are a vital part of the data. 

I recently switched over to use the datasets library when my various corpora grew larger than my computer's memory. And so far, it is SO great. 

But the first time I processed all of my data into a dataset, I hadn't realized the text loader script was processing the source files line-by-line and stripping off the newlines. 

Once I caught the issue, I made my own data loader by modifying one line in the default text loader (changing `batch = batch.splitlines()` to `batch = batch.splitlines(True)` inside `_generate_tables`). And so I'm all set as far as my project is concerned.

But if my use case is more general, it seems like it'd be pretty trivial to add a kwarg to the default text loader called keeplinebreaks or something, which would default to False and get passed to `splitlines()`.  
 Hi ! Thanks for your message.
Indeed it's a free feature we can add and that can be useful.
If you want to contribute, feel free to open a PR to add it to the text dataset script :)
OSCAR from Inria group 
 ## Adding a Dataset
- **Name:** *OSCAR* (Open Super-large Crawled ALMAnaCH coRpus), multilingual parsing of Common Crawl (separate crawls for many different languages), [here](https://oscar-corpus.com/).
- **Description:** *OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture.*
- **Paper:** *[here](https://hal.inria.fr/hal-02148693)*
- **Data:** *[here](https://oscar-corpus.com/)*
- **Motivation:** *useful for unsupervised tasks in separate languages. In an ideal world, your team would be able to obtain the unshuffled version, that could be used to train GPT-2-like models (the shuffled version, I suppose, could be used for translation).*

I am aware that you do offer the "colossal" Common Crawl dataset already, but this has the advantage to be available in many subcorpora for different languages.
 
 PR is already open here : #348 
The only thing remaining is to compute the metadata of each subdataset (one per language + shuffled/unshuffled).
As soon as #863 is merged we can start computing them. This will take a bit of time though
Have Trouble importing `datasets` 
 I'm failing to import transformers (v4.0.0-dev), and tracing the cause seems to be failing to import datasets.

I cloned the newest version of datasets (master branch), and do `pip install -e .`.

Then, `import datasets` causes the error below.

```
~/workspace/Clone/datasets/src/datasets/utils/file_utils.py in <module>
    116 sys.path.append(str(HF_MODULES_CACHE))
    117 
--> 118 os.makedirs(HF_MODULES_CACHE, exist_ok=True)
    119 if not os.path.exists(os.path.join(HF_MODULES_CACHE, "__init__.py")):
    120     with open(os.path.join(HF_MODULES_CACHE, "__init__.py"), "w"):

~/.pyenv/versions/anaconda3-2020.07/lib/python3.8/os.py in makedirs(name, mode, exist_ok)
    221             return
    222     try:
--> 223         mkdir(name, mode)
    224     except OSError:
    225         # Cannot rely on checking for EEXIST, since the operating system 

FileNotFoundError: [Errno 2] No such file or directory: '<MY_HOME_DIRECTORY>/.cache/huggingface/modules'
```

The error occurs in `os.makedirs` in `file_utils.py`, even though `exist_ok = True` option is set.
(I use Python 3.8, so `exist_ok` is expected to work.)

I've checked some environment variables, and they are set as below.

```
*** NameError: name 'HF_MODULES_CACHE' is not defined
*** NameError: name 'hf_cache_home' is not defined
*** NameError: name 'XDG_CACHE_HOME' is not defined
```

Should I set some environment variables before using this library?
And, do you have any idea why "No such file or directory" occurs even though the `exist_ok = True` option is set?

Thank you in advance. 
 I'm sorry, this was a problem with my environment.
Now that I have identified the cause of environmental dependency, I would like to fix it and try it.
Excuse me for making a noise.
Possible Bug: Small training/dataset file creates gigantic output 
 Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

 
 The preprocessing tokenizes the input text. Tokenization outputs `input_ids`, `attention_mask`, `token_type_ids` and `special_tokens_mask`. All those are of length`max_seq_length` because of padding. Therefore for each sample it generate 4 *`max_seq_length` integers. Currently they're all saved as int64. This is why the tokenization takes so much space.

I'm sure we can optimize that though
What do you think @sgugger ?
Possible Bug: Small training/dataset file creates gigantic output 
 Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

 
 First I think we should disable padding in the dataset processing and let the data collator do it.

Then I'm wondering if you need attention_mask and token_type_ids at this point ?

Finally we can also specify the output feature types at this line https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py#L280 to use more optimized integer precisions for the output. Maybe something like:
- input_ids: uint16 or uint32
- token_type_ids: uint8 or bool
- attention_mask: bool
- special_tokens_mask: bool

Also IMO these changes are all on the `transformers` side. Maybe we should discuss on the `transformers` repo
Possible Bug: Small training/dataset file creates gigantic output 
 Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

 
 > First I think we should disable padding in the dataset processing and let the data collator do it.

No, you can't do that on TPUs as dynamic shapes will result in a very slow training. The script can however be tweaked to use the `PaddingDataCollator` with a fixed max length instead of dynamic batching.

For the other optimizations, they can be done by changing the script directly for each user's use case. Not sure we can find something that is general enough to be in transformers or the examples script.
Possible Bug: Small training/dataset file creates gigantic output 
 Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

 
 Oh yes right..
Do you think that a lazy map feature on the `datasets` side could help to avoid storing padded tokenized texts then ?
Possible Bug: Small training/dataset file creates gigantic output 
 Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

 
 I think I can do the tweak mentioned above with the data collator as short fix (but fully focused on v4 right now so that will be for later this week, beginning of next week :-) ).
If it doesn't hurt performance to tokenize on the fly, that would clearly be the long-term solution however!
Possible Bug: Small training/dataset file creates gigantic output 
 Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

 
 > Hey guys,
> 
> I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.
> 
> I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?
> 
> I've used the following CMD:
> `python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

It's actually because of the parameter 'preprocessing_num_worker' when using TPU. 
I am also planning to have my model trained on the google TPU with a 11gb text corpus. With x8 cores enabled, each TPU core has its own dataset.  When not using distributed training, the preprocessed file is about 77gb. On the opposite, if enable xla, the file produced will easily consume all my free space(more than 220gb, I think it will be, in the end, around 600gb ). 
So I think that's maybe where the problem came from. 

Is there any possibility that all of the cores share the same preprocess dataset?

@sgugger @RammMaschine
Possible Bug: Small training/dataset file creates gigantic output 
 Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

 
 Hi @NebelAI, we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.
concatenate_datasets support axis=0 or 1 ？ 
 I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
 
 Unfortunately `concatenate_datasets` only supports concatenating the rows, while what you want to achieve is concatenate the columns.
Currently to add more columns to a dataset, one must use `map`.
What you can do is somehting like this:
```python
# suppose you have datasets d1, d2, d3
def add_columns(example, index):
    example.update(d2[index])
    example.update(d3[index])
    return example

full_dataset = d1.map(add_columns, with_indices=True)
```
concatenate_datasets support axis=0 or 1 ？ 
 I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
 
 That's not really difficult to add, though, no?
I think it can be done without copy.
Maybe let's add it to the roadmap?
concatenate_datasets support axis=0 or 1 ？ 
 I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
 
 Actually it's doable but requires to update the `Dataset._data_files` schema to support this.
I'm re-opening this since we may want to add this in the future
concatenate_datasets support axis=0 or 1 ？ 
 I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
 
 Hi @lhoestq, I would love to help and add this feature if still needed. My plan is to add an axis variable in the `concatenate_datasets` function in `arrow_dataset.py` and when that is set to 1 concatenate columns instead of rows.
concatenate_datasets support axis=0 or 1 ？ 
 I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
 
 Hi ! I would love to see this feature implemented as well :) Thank you for proposing your help !

Here is a few things about the current implementation:
- A dataset object is a wrapper of one `pyarrow.Table` that contains the data
- Pyarrow offers an API that allows to transform Table objects. For example there are functions like `concat_tables`, `Table.rename_columns`, `Table.add_column` etc.

Therefore adding columns from another dataset is possible thanks to the pyarrow API and in particular `Table.add_column` :) 

However this breaks some features we have regarding pickle. A dataset object can be pickled and unpickled without loading all the data in memory. It is useful for multiprocessing for example. Pickling a dataset object is possible thanks to the `Dataset._data_files` which defines the list of arrow files that will be used to form the final Table (basically all the data from each files are concatenated on axis 0).

Therefore to be able to add columns to a Dataset and still be able to work with it in a multiprocessing setup, we need to extend this last aspect to be able to reconstruct a Table object from multiple arrow files that are combined in both axis 0 and 1. Currently this reconstruction mechanism only supports axis 0.

I'm sure we can figure something out that enables users to add columns from another dataset while keeping the multiprocessing support.
concatenate_datasets support axis=0 or 1 ？ 
 I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
 
 @lhoestq, we have two Pull Requests to implement:
- Dataset.add_item: #1870
- Dataset.add_column: #2145
which add a single row or column, repectively.

The request here is to implement the concatenation of *multiple* rows/columns. Am I right?

We should agree on the API:
- `concatenate_datasets` with `axis`?
- other Dataset method name?
concatenate_datasets support axis=0 or 1 ？ 
 I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
 
 For the API, I like `concatenate_datasets` with `axis` personally :)
From a list of `Dataset` objects, it would concatenate them to a new `Dataset` object backed by a `ConcatenationTable`, that is the concatenation of the tables of each input dataset. The concatenation is either on axis=0 (append rows) or on axis=1 (append columns).

Regarding what we need to implement:
The axis=0 is already supported and is the current behavior of `concatenate_datasets`.
Also `add_item` is not needed to implement axis=1 (though it's an awesome addition to this library).

To implement axis=1, we either need `add_column` or a `ConcatenationTable` constructor to concatenate tables horizontally.
I have a preference for using a `ConcatenationTable` constructor because this way we can end up with a `ConcatenationTable` with only 1 additional block per table, while `add_column` would add 1 block per new column.

Maybe we can simply have an equivalent of `ConcatenationTable.from_tables` but for axis=1 ?
`axis` could also be an argument of `ConcatenationTable.from_tables`
Load amazon dataset 
 Hi,
I was going through amazon_us_reviews dataset and found that example API usage given on website is different from the API usage while loading dataset. 

Eg. what API usage is on the [website](https://huggingface.co/datasets/amazon_us_reviews) 
```
from datasets import load_dataset
dataset = load_dataset("amazon_us_reviews")
```
How it is when I tried (the error generated does point me to the right direction though)
```
from datasets import load_dataset
dataset = load_dataset("amazon_us_reviews", 'Books_v1_00')
``` 
Also, there is some issue with formatting as it's not showing bullet list in description with new line. Can I work on it? 
 Thanks for reporting !
We plan to show information about the different configs of the datasets on the website, with the corresponding `load_dataset` calls.

Also I think the bullet points formatting has been fixed
Error when concatenate_datasets 
 Hello, when I concatenate two dataset loading  from disk, I encountered a problem:
```
test_dataset = load_from_disk('data/test_dataset')
trn_dataset = load_from_disk('data/train_dataset')

train_dataset = concatenate_datasets([trn_dataset, test_dataset])
```
And it reported ValueError blow:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-38-74fa525512ca> in <module>
----> 1 train_dataset = concatenate_datasets([trn_dataset, test_dataset])

/opt/miniconda3/lib/python3.7/site-packages/datasets/arrow_dataset.py in concatenate_datasets(dsets, info, split)
   2547                 "However datasets' indices {} come from memory and datasets' indices {} come from disk.".format(
   2548                     [i for i in range(len(dsets)) if indices_mappings_in_memory[i]],
-> 2549                     [i for i in range(len(dsets)) if not indices_mappings_in_memory[i]],
   2550                 )
   2551             )

ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```

But it's curious both of my datasets loading from disk, so I check the source code in `arrow_dataset.py` about the Error:
```
trn_dataset._data_files
# output
[{'filename': 'data/train_dataset/csv-train.arrow', 'skip': 0, 'take': 593264}]

test_dataset._data_files
# output
[{'filename': 'data/test_dataset/csv-test.arrow', 'skip': 0, 'take': 424383}]

print([not dset._data_files for dset in [trn_dataset, test_dataset]])
# [False, False]

# And I tested the code the same as arrow_dataset, but nothing happened
dsets = [trn_dataset, test_dataset]
dsets_in_memory = [not dset._data_files for dset in dsets]
if any(dset_in_memory != dsets_in_memory[0] for dset_in_memory in dsets_in_memory):
    raise ValueError(
        "Datasets should ALL come from memory, or should ALL come from disk.\n"
        "However datasets {} come from memory and datasets {} come from disk.".format(
            [i for i in range(len(dsets)) if dsets_in_memory[i]],
            [i for i in range(len(dsets)) if not dsets_in_memory[i]],
        )
    )
```

Any suggestions would be greatly appreciated! 
Thanks! 
 @lhoestq we can add a mention of `dataset.flatten_indices()` in the error message (no rush, just put it on your TODO list or I can do it when I come at it)
Add HoVer multi-hop fact verification dataset 
 ## Adding a Dataset
- **Name:** HoVer
- **Description:** https://twitter.com/YichenJiang9/status/1326954363806429186 contains 20K claim verification examples
- **Paper:** https://arxiv.org/abs/2011.03088
- **Data:** https://hover-nlp.github.io/
- **Motivation:** There are still few multi-hop information extraction benchmarks (HotpotQA, which dataset wase based off, notwithstanding)

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).
 
 Hi @yjernite  I'm new but wanted to contribute. Has anyone already taken this problem and do you think it is suitable for newbies?
Add HoVer multi-hop fact verification dataset 
 ## Adding a Dataset
- **Name:** HoVer
- **Description:** https://twitter.com/YichenJiang9/status/1326954363806429186 contains 20K claim verification examples
- **Paper:** https://arxiv.org/abs/2011.03088
- **Data:** https://hover-nlp.github.io/
- **Motivation:** There are still few multi-hop information extraction benchmarks (HotpotQA, which dataset wase based off, notwithstanding)

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).
 
 Hi @tenjjin! This dataset is still up for grabs! Here's the link with the guide to add it. You should play around with the library first (download and look at a few datasets), then follow the steps here:

https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md
use_custom_baseline still produces errors for bertscore 
 `metric = load_metric('bertscore')`
`a1 = "random sentences"`
`b1 = "random sentences"`
`metric.compute(predictions = [a1], references = [b1], lang = 'en')`

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/stephen_chan/.cache/huggingface/modules/datasets_modules/metrics/bertscore/361e597a01a41d6cf95d94bbfb01dea16261687abc0c6c74cc9930f80488f363/bertscore.py", line 108, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() missing 1 required positional argument: 'use_custom_baseline'`

Adding 'use_custom_baseline = False' as an argument produces this error

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
TypeError: _compute() got an unexpected keyword argument 'use_custom_baseline'`

This is on Ubuntu 18.04, Python 3.6.9, datasets version 1.1.2 
 Thanks for reporting ! That's a bug indeed
If you want to contribute, feel free to fix this issue and open a PR :)
use_custom_baseline still produces errors for bertscore 
 `metric = load_metric('bertscore')`
`a1 = "random sentences"`
`b1 = "random sentences"`
`metric.compute(predictions = [a1], references = [b1], lang = 'en')`

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/stephen_chan/.cache/huggingface/modules/datasets_modules/metrics/bertscore/361e597a01a41d6cf95d94bbfb01dea16261687abc0c6c74cc9930f80488f363/bertscore.py", line 108, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() missing 1 required positional argument: 'use_custom_baseline'`

Adding 'use_custom_baseline = False' as an argument produces this error

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
TypeError: _compute() got an unexpected keyword argument 'use_custom_baseline'`

This is on Ubuntu 18.04, Python 3.6.9, datasets version 1.1.2 
 This error is because of a mismatch between `datasets` and `bert_score`. With `datasets=1.1.2` and `bert_score>=0.3.6` it works ok. So `pip install -U bert_score` should fix the problem.
use_custom_baseline still produces errors for bertscore 
 `metric = load_metric('bertscore')`
`a1 = "random sentences"`
`b1 = "random sentences"`
`metric.compute(predictions = [a1], references = [b1], lang = 'en')`

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/stephen_chan/.cache/huggingface/modules/datasets_modules/metrics/bertscore/361e597a01a41d6cf95d94bbfb01dea16261687abc0c6c74cc9930f80488f363/bertscore.py", line 108, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() missing 1 required positional argument: 'use_custom_baseline'`

Adding 'use_custom_baseline = False' as an argument produces this error

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
TypeError: _compute() got an unexpected keyword argument 'use_custom_baseline'`

This is on Ubuntu 18.04, Python 3.6.9, datasets version 1.1.2 
 Hello everyone,

I think the  problem is not solved: 

```
from datasets import load_metric
metric=load_metric('bertscore')
metric.compute(
    predictions=predictions,
    references=references,
    lang='fr',
    rescale_with_baseline=True
)
TypeError: get_hash() missing 2 required positional arguments: 'use_custom_baseline' and 'use_fast_tokenizer'
```
This code is produced using `Python 3.6.9 datasets==1.1.2 and bert_score==0.3.10`
use_custom_baseline still produces errors for bertscore 
 `metric = load_metric('bertscore')`
`a1 = "random sentences"`
`b1 = "random sentences"`
`metric.compute(predictions = [a1], references = [b1], lang = 'en')`

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/stephen_chan/.cache/huggingface/modules/datasets_modules/metrics/bertscore/361e597a01a41d6cf95d94bbfb01dea16261687abc0c6c74cc9930f80488f363/bertscore.py", line 108, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() missing 1 required positional argument: 'use_custom_baseline'`

Adding 'use_custom_baseline = False' as an argument produces this error

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
TypeError: _compute() got an unexpected keyword argument 'use_custom_baseline'`

This is on Ubuntu 18.04, Python 3.6.9, datasets version 1.1.2 
 Hi ! This has been fixed by https://github.com/huggingface/datasets/pull/2770, we'll do a new release soon to make the fix available :)

In the meantime please use an older version of `bert_score`
How to enable `.map()` pre-processing pipelines to support multi-node parallelism? 
 Hi,

Currently, multiprocessing can be enabled for the `.map()` stages on a single node. However, in the case of multi-node training, (since more than one node would be available) I'm wondering if it's possible to extend the parallel processing among nodes, instead of only 1 node running the `.map()` while the other node is waiting for it to finish?

Thanks! 
 Right now multiprocessing only runs on single node.

However it's probably possible to extend it to support multi nodes. Indeed we're using the `multiprocess` library from the `pathos` project to do multiprocessing in `datasets`, and `pathos` is made to support parallelism on several nodes. More info about pathos [on the pathos repo](https://github.com/uqfoundation/pathos).

If you're familiar with pathos or if you want to give it a try, it could be a nice addition to the library :)
How to enable `.map()` pre-processing pipelines to support multi-node parallelism? 
 Hi,

Currently, multiprocessing can be enabled for the `.map()` stages on a single node. However, in the case of multi-node training, (since more than one node would be available) I'm wondering if it's possible to extend the parallel processing among nodes, instead of only 1 node running the `.map()` while the other node is waiting for it to finish?

Thanks! 
 Curious to hear if anything on that side changed or if you suggestions to do it changed @lhoestq :)

For our use-case, we are entering the regime where trading a few more instances to save a few days would be nice :)
How to enable `.map()` pre-processing pipelines to support multi-node parallelism? 
 Hi,

Currently, multiprocessing can be enabled for the `.map()` stages on a single node. However, in the case of multi-node training, (since more than one node would be available) I'm wondering if it's possible to extend the parallel processing among nodes, instead of only 1 node running the `.map()` while the other node is waiting for it to finish?

Thanks! 
 Currently for multi-node setups we're mostly going towards a nice integration with Dask. But I wouldn't exclude exploring `pathos` more at one point
load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas 
 Hi All
I am trying to load a custom dataset  and I am trying to load a single file to make sure the file is loading correctly:
dataset = load_dataset('csv', data_files=files)
When I run it I get:

Downloading and preparing dataset csv/default-35575a1051604c88 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) tocache/huggingface/datasets/csv/default-35575a1051604c88/0.0.0/49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4...

I am getting this error:
6a4ac4/csv.py in _generate_tables(self, files)
     78     def _generate_tables(self, files):
     79         for i, file in enumerate(files):
---> 80             pa_table = pac.read_csv(
     81                 file,
     82                 read_options=self.config.pa_read_options,

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/_csv.pyx in pyarrow._csv.read_csv()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

**ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)**



The size of the file is 3.5 GB. When I try smaller files I do not have an issue. When I load it with 'text' parser I can see all data but it is not what I need.
There is no issue reading the file with pandas. any idea what could be the issue?
When I am running a different CSV I do not get  this line:
 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size)

Any ideas?
 
 Which version of pyarrow do you have ? Could you try to update pyarrow and try again ?
load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas 
 Hi All
I am trying to load a custom dataset  and I am trying to load a single file to make sure the file is loading correctly:
dataset = load_dataset('csv', data_files=files)
When I run it I get:

Downloading and preparing dataset csv/default-35575a1051604c88 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) tocache/huggingface/datasets/csv/default-35575a1051604c88/0.0.0/49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4...

I am getting this error:
6a4ac4/csv.py in _generate_tables(self, files)
     78     def _generate_tables(self, files):
     79         for i, file in enumerate(files):
---> 80             pa_table = pac.read_csv(
     81                 file,
     82                 read_options=self.config.pa_read_options,

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/_csv.pyx in pyarrow._csv.read_csv()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

**ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)**



The size of the file is 3.5 GB. When I try smaller files I do not have an issue. When I load it with 'text' parser I can see all data but it is not what I need.
There is no issue reading the file with pandas. any idea what could be the issue?
When I am running a different CSV I do not get  this line:
 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size)

Any ideas?
 
 Thanks for the fast response. I have the latest version '2.0.0' (I tried to update)
I am working with Python 3.8.5
load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas 
 Hi All
I am trying to load a custom dataset  and I am trying to load a single file to make sure the file is loading correctly:
dataset = load_dataset('csv', data_files=files)
When I run it I get:

Downloading and preparing dataset csv/default-35575a1051604c88 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) tocache/huggingface/datasets/csv/default-35575a1051604c88/0.0.0/49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4...

I am getting this error:
6a4ac4/csv.py in _generate_tables(self, files)
     78     def _generate_tables(self, files):
     79         for i, file in enumerate(files):
---> 80             pa_table = pac.read_csv(
     81                 file,
     82                 read_options=self.config.pa_read_options,

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/_csv.pyx in pyarrow._csv.read_csv()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

**ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)**



The size of the file is 3.5 GB. When I try smaller files I do not have an issue. When I load it with 'text' parser I can see all data but it is not what I need.
There is no issue reading the file with pandas. any idea what could be the issue?
When I am running a different CSV I do not get  this line:
 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size)

Any ideas?
 
 I think that the issue is similar to this one:https://issues.apache.org/jira/browse/ARROW-9612
The problem is in arrow when the column data contains long strings.
Any ideas on how to bypass this?
load_dataset with 'csv' is not working. while the same file is loading with 'text' mode or with pandas 
 Hi All
I am trying to load a custom dataset  and I am trying to load a single file to make sure the file is loading correctly:
dataset = load_dataset('csv', data_files=files)
When I run it I get:

Downloading and preparing dataset csv/default-35575a1051604c88 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) tocache/huggingface/datasets/csv/default-35575a1051604c88/0.0.0/49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4...

I am getting this error:
6a4ac4/csv.py in _generate_tables(self, files)
     78     def _generate_tables(self, files):
     79         for i, file in enumerate(files):
---> 80             pa_table = pac.read_csv(
     81                 file,
     82                 read_options=self.config.pa_read_options,

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/_csv.pyx in pyarrow._csv.read_csv()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

**ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)**



The size of the file is 3.5 GB. When I try smaller files I do not have an issue. When I load it with 'text' parser I can see all data but it is not what I need.
There is no issue reading the file with pandas. any idea what could be the issue?
When I am running a different CSV I do not get  this line:
 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size)

Any ideas?
 
 We should expose the [`block_size` argument](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions) of Apache Arrow csv `ReadOptions` in the [script](https://github.com/huggingface/datasets/blob/master/datasets/csv/csv.py).


In the meantime you can specify yourself the `ReadOptions` config like this:
```python
import pyarrow.csv as pac   # PyArrow is installed with `datasets`

read_options = pac.ReadOptions(block_size=1e9)  # try to find the right value for your use-case
dataset = load_dataset('csv', data_files=files, read_options=read_options)
```