Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task
[This colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) implements a token classification input pipeline extending the logic from [this hugging example](https://huggingface.co/transformers/custom_datasets.html#tok-ner).

The pipeline works fine with most instance in different languages, but unfortunately, [the Japanese Kana ligature (a form of abbreviation? I don't know Japanese well)](https://en.wikipedia.org/wiki/Kana_ligature) break the alignment of `return_offsets_mapping`:
![image](https://user-images.githubusercontent.com/50871412/122904371-db192700-d382-11eb-8917-1775db76db69.png)

Without the try catch block, it riase `ValueError: NumPy boolean array indexing assignment cannot assign 88 input values to the 87 output values where the mask is true`, example shown here [(another colab notebook)](https://colab.research.google.com/drive/1MmOqf3ppzzdKKyMWkn0bJy6DqzOO0SSm?usp=sharing)

It is clear that the normalizer is the process that break the alignment, as it is observed that `tokenizer._tokenizer.normalizer.normalize_str('ヿ')` return 'コト'.

One workaround is to include `tokenizer._tokenizer.normalizer.normalize_str` before the tokenizer preprocessing pipeline, which is also provided in the [first colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) with the name `udposTestDatasetWorkaround`.

I guess similar logics should be included inside the tokenizer and the offsets_mapping generation process such that user don't need to include them in their code. But I don't understand the code of tokenizer well that I think I am not able to do this.

p.s.
**I am using my own dataset building script in the provided example, but the script should be equivalent to the changes made by this [update](https://github.com/huggingface/datasets/pull/2466)**
`get_dataset `is just a simple wrapping for `load_dataset`
and the `tokenizer` is just `XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-large")`
> Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**?

Oh, I am sorry
I would reopen the post on huggingface/transformers


Logging cannot be set to NOTSET similar to transformers
## Describe the bug
In the transformers library you can set the verbosity level to logging.NOTSET to work around the usage of tqdm and IPywidgets, however in Datasets this is no longer possible. This is because transformers set the verbosity level of tqdm with [this](https://github.com/huggingface/transformers/blob/b53bc55ba9bb10d5ee279eab51a2f0acc5af2a6b/src/transformers/file_utils.py#L1449)
`disable=bool(logging.get_verbosity() == logging.NOTSET)`
and datasets accomplishes this like [so](https://github.com/huggingface/datasets/blob/83554e410e1ab8c6f705cfbb2df7953638ad3ac1/src/datasets/utils/file_utils.py#L493)
`not_verbose = bool(logger.getEffectiveLevel() > WARNING)`
## Steps to reproduce the bug
```python
import datasets
import logging
datasets.logging.get_verbosity = lambda : logging.NOTSET
datasets.load_dataset("patrickvonplaten/librispeech_asr_dummy")
```

## Expected results
The code should download and load the dataset as normal without displaying progress bars

## Actual results
```ImportError                               Traceback (most recent call last)
<ipython-input-4-aec65c0509c6> in <module>
----> 1 datasets.load_dataset("patrickvonplaten/librispeech_asr_dummy")

~/venv/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, **config_kwargs)
    713         dataset=True,
    714         return_resolved_file_path=True,
--> 715         use_auth_token=use_auth_token,
    716     )
    717     # Set the base path for downloads as the parent of the script location

~/venv/lib/python3.7/site-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
    350                     file_path = hf_bucket_url(path, filename=name, dataset=False)
    351                 try:
--> 352                     local_path = cached_path(file_path, download_config=download_config)
    353                 except FileNotFoundError:
    354                     raise FileNotFoundError(

~/venv/lib/python3.7/site-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    289             use_etag=download_config.use_etag,
    290             max_retries=download_config.max_retries,
--> 291             use_auth_token=download_config.use_auth_token,
    292         )
    293     elif os.path.exists(url_or_filename):

~/venv/lib/python3.7/site-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    668                     headers=headers,
    669                     cookies=cookies,
--> 670                     max_retries=max_retries,
    671                 )
    672

~/venv/lib/python3.7/site-packages/datasets/utils/file_utils.py in http_get(url, temp_file, proxies, resume_size, headers, cookies, timeout, max_retries)
    493         initial=resume_size,
    494         desc="Downloading",
--> 495         disable=not_verbose,
    496     )
    497     for chunk in response.iter_content(chunk_size=1024):

~/venv/lib/python3.7/site-packages/tqdm/notebook.py in __init__(self, *args, **kwargs)
    217         total = self.total * unit_scale if self.total else self.total
    218         self.container = self.status_printer(
--> 219             self.fp, total, self.desc, self.ncols)
    220         self.sp = self.display
    221

~/venv/lib/python3.7/site-packages/tqdm/notebook.py in status_printer(_, total, desc, ncols)
     95         if IProgress is None:  # #187 #451 #558 #872
     96             raise ImportError(
---> 97                 "IProgress not found. Please update jupyter and ipywidgets."
     98                 " See https://ipywidgets.readthedocs.io/en/stable"
     99                 "/user_install.html")

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.95-42.163.amzn2.x86_64-x86_64-with-debian-10.8
- Python version: 3.7.10
- PyArrow version: 3.0.0
I am running this code on Deepnote and which important to this issue **does not** support IPywidgets

Hi @joshzwiebel, thanks for reporting. We are going to align with `transformers`.


Documentation Mistakes in Dataset: emotion
As per documentation,
Dataset: emotion
Homepage: https://github.com/dair-ai/emotion_dataset

Dataset: https://github.com/huggingface/datasets/blob/master/datasets/emotion/emotion.py

Permalink: https://huggingface.co/datasets/viewer/?dataset=emotion

Emotion is a dataset of English Twitter messages with eight basic emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. For more detailed information please refer to the paper.

But when we view the data, there are only 6 emotions, anger, fear, joy, sadness, surprise, and trust.
Hi,

this issue has been already reported in the dataset repo (https://github.com/dair-ai/emotion_dataset/issues/2), so this is a bug on their side.


datasets.map pickle issue resulting in invalid mapping function
I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples

Hi ! `map` calls `__getstate__` using `dill` to hash your map function. This is used by the caching mechanism to recover previously computed results. That's why you don't see any `__setstate__` call.

Why do you change an attribute of your tokenizer when `__getstate__` is called ?


datasets.map pickle issue resulting in invalid mapping function
I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples

@lhoestq because if I try to pickle my custom tokenizer (it contains a pure python pretokenization step in an otherwise rust backed tokenizer) I get

> Exception: Error while attempting to pickle Tokenizer: Custom PreTokenizer cannot be serialized

So I remove the Custom PreTokenizer in `__getstate__` and then restore it in `__setstate__` (since it doesn't contain any state). This is what my `__getstate__`  / `__setstate__` looks like:

    def __getstate__(self):
        """
        Removes pre_tokenizer since it cannot be pickled
        """
        logger.debug("Copy state dict")
        out = self.__dict__.copy()
        logger.debug("Detaching pre_tokenizer")
        out['_tokenizer'].pre_tokenizer = tokenizers.pre_tokenizers.Sequence([])
        return out

    def __setstate__(self, d):
        """
        Reinstates pre_tokenizer
        """
        logger.debug("Reattaching pre_tokenizer")
        self.__dict__ = d
        self.backend_tokenizer.pre_tokenizer = self._pre_tokenizer()

If this is the case can you think of another way of avoiding my issue?


datasets.map pickle issue resulting in invalid mapping function
I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples

Actually, maybe I need to deep copy `self.__dict__`? That way `self` isn't modified. That was my intention and I thought it was working - I'll double-check after the weekend.


datasets.map pickle issue resulting in invalid mapping function
I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples

Doing a deep copy results in the warning:

> 06/20/2021 16:02:15 - WARNING - datasets.fingerprint -   Parameter 'function'=<function tokenize_function at 0x7f1e95f05d40> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.

```
def __getstate__(self):
    """
    Removes pre_tokenizer since it cannot be pickled
    """
    logger.debug("Copy state dict")
    out = copy.deepcopy(self.__dict__)
    logger.debug("Detaching pre_tokenizer")
    out['_tokenizer'].pre_tokenizer = tokenizers.pre_tokenizers.Sequence([])
    return out
```


datasets.map pickle issue resulting in invalid mapping function
I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples

Looks like there is still an object that is not pickable in your `tokenize_function` function.

You can test if an object can be pickled and hashed by using
```python
from datasets.fingerprint import Hasher

Hasher.hash(my_object)
```

Under the hood it pickles the object to compute its hash, so it calls `__getstate__` when applicable.


datasets.map pickle issue resulting in invalid mapping function
I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples

I figured it out, the problem is deep copy itself uses pickle (unless you implement `__deepcopy__`). So when I changed `__getstate__` it started throwing an error.

I'm sure there's a better way of doing this, but in order to return the `__dict__` without the non-pikelable pre-tokeniser and without modifying self I removed the pre-tokenizers, did a deep copy and then re-generated it.

It does work -  although I noticed Hasher doesn't call `__hash__` if the object being hashed implements it which I feel it should? If it did I could return a hash of the tokenizers.json file instead.

```
    def __getstate__(self):
        """
        Removes pre_tokenizer since it cannot be pickled
        """
        logger.debug("Copy state dict")
        self.backend_tokenizer.pre_tokenizer = tokenizers.pre_tokenizers.Sequence([])
        out = copy.deepcopy(self.__dict__)  #self.__dict__.copy()
        self.backend_tokenizer.pre_tokenizer = self._pre_tokenizer()

        return out
```


datasets.map pickle issue resulting in invalid mapping function
I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples

I'm glad you figured something out :)

Regarding hashing: we're not using hashing for the same purpose as the python `__hash__` purpose (which is in general for dictionary lookups). For example it is allowed for python hashing to not return the same hash across sessions, while our hashing must return the same hashes across sessions for the caching to work properly.


Can datasets remove duplicated rows?

Can datasets remove duplicated rows?
**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..

**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...

**Additional context**
no
Hi ! For now this is probably the best option.
We might add a feature like this in the feature as well.

Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases


Can datasets remove duplicated rows?

Can datasets remove duplicated rows?
**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..

**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...

**Additional context**
no
Yes, I'd like to work on this feature once I'm done with #2500, but first I have to do some research, and see if the implementation wouldn't be too complex.

In the meantime, maybe [this lib](https://github.com/TomScheffers/pyarrow_ops) can help. However, note that this lib operates directly on pyarrow tables and relies only on `hash` to find duplicates (e.g. `-1` and `-2` have the same hash in Python 3, so this lib will treat them as duplicates), which doesn't make much sense.


Can datasets remove duplicated rows?

Can datasets remove duplicated rows?
**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..

**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...

**Additional context**
no
> Hi ! For now this is probably the best option.
> We might add a feature like this in the feature as well.
>
> Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
> Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases

Great if this is can be done. Thanks!!

Not sure if you are asking me. In any case I don't know of any unfortunately :( in practice if data is really large we normally do it with spark (only for info. I understand this is not useful in developing this library..)


Corelation should be Correlation

Corelation should be Correlation
https://github.com/huggingface/datasets/blob/0e87e1d053220e8ecddfa679bcd89a4c7bc5af62/metrics/matthews_correlation/matthews_correlation.py#L66
Hi @colbym-MM, thanks for reporting. We are fixing it.


seqeval metric does not work with a recent version of sklearn: classification_report() got an unexpected keyword argument 'output_dict'
## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
seqeval = load_metric("seqeval")
seqeval.compute(predictions=[['A']], references=[['A']])
```

## Expected results
The function computes a dict with metrics

## Actual results
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-39-69a57f5cf06f> in <module>
      1 from datasets import load_dataset, load_metric
      2 seqeval = load_metric("seqeval")
----> 3 seqeval.compute(predictions=[['A']], references=[['A']])

~/p3/lib/python3.7/site-packages/datasets/metric.py in compute(self, *args, **kwargs)
    396             references = self.data["references"]
    397             with temp_seed(self.seed):
--> 398                 output = self._compute(predictions=predictions, references=references, **kwargs)
    399
    400             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/seqeval/81eda1ff004361d4fa48754a446ec69bb7aa9cf4d14c7215f407d1475941c5ff/seqeval.py in _compute(self, predictions, references, suffix)
     95
     96     def _compute(self, predictions, references, suffix=False):
---> 97         report = classification_report(y_true=references, y_pred=predictions, suffix=suffix, output_dict=True)
     98         report.pop("macro avg")
     99         report.pop("weighted avg")

TypeError: classification_report() got an unexpected keyword argument 'output_dict'
```

## Environment info
sklearn=0.24
datasets=1.1.3

Sorry, I was using an old version of sequeval


Add C4

Add C4
## Adding a Dataset
- **Name:** *C4*
- **Description:** *https://github.com/allenai/allennlp/discussions/5056*
- **Paper:** *https://arxiv.org/abs/1910.10683*
- **Data:** *https://huggingface.co/datasets/allenai/c4*
- **Motivation:** *Used a lot for pretraining*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Should fix https://github.com/huggingface/datasets/issues/1710
Update on this: I'm computing the checksums of the data files. It will be available soon


Add C4

Add C4
## Adding a Dataset
- **Name:** *C4*
- **Description:** *https://github.com/allenai/allennlp/discussions/5056*
- **Paper:** *https://arxiv.org/abs/1910.10683*
- **Data:** *https://huggingface.co/datasets/allenai/c4*
- **Motivation:** *Used a lot for pretraining*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Should fix https://github.com/huggingface/datasets/issues/1710
Added in #2575 :)


Load Image Classification Dataset from Local
**Is your feature request related to a problem? Please describe.**
Yes - we would like to load an image classification dataset with datasets without having to write a custom data loader.

**Describe the solution you'd like**

Given a folder structure with images of each class in each folder, the ability to load these folders into a HuggingFace dataset like "cifar10".

**Describe alternatives you've considered**

Implement ViT training outside of the HuggingFace Trainer and without datasets (we did this but prefer to stay on the main path)

Write custom data loader logic

**Additional context**

We're training ViT on custom dataset

Hi ! Is this folder structure a standard, a bit like imagenet ?
In this case maybe we can consider having a dataset loader for cifar-like, imagenet-like, squad-like, conll-like etc. datasets ?
```python
from datasets import load_dataset

my_custom_cifar = load_dataset("cifar_like", data_dir="path/to/data/dir")
```

Let me know what you think


Load Image Classification Dataset from Local
**Is your feature request related to a problem? Please describe.**
Yes - we would like to load an image classification dataset with datasets without having to write a custom data loader.

**Describe the solution you'd like**

Given a folder structure with images of each class in each folder, the ability to load these folders into a HuggingFace dataset like "cifar10".

**Describe alternatives you've considered**

Implement ViT training outside of the HuggingFace Trainer and without datasets (we did this but prefer to stay on the main path)

Write custom data loader logic

**Additional context**

We're training ViT on custom dataset

Yep that would be sweet - closing for now as we found a workaround.


Load Image Classification Dataset from Local
**Is your feature request related to a problem? Please describe.**
Yes - we would like to load an image classification dataset with datasets without having to write a custom data loader.

**Describe the solution you'd like**

Given a folder structure with images of each class in each folder, the ability to load these folders into a HuggingFace dataset like "cifar10".

**Describe alternatives you've considered**

Implement ViT training outside of the HuggingFace Trainer and without datasets (we did this but prefer to stay on the main path)

Write custom data loader logic

**Additional context**

We're training ViT on custom dataset

@lhoestq I think we'll want a generic `image-folder` dataset (same as 'imagenet-like'). This is like `torchvision.datasets.ImageFolder`, and is something vision folks are used to seeing.


Load Image Classification Dataset from Local
**Is your feature request related to a problem? Please describe.**
Yes - we would like to load an image classification dataset with datasets without having to write a custom data loader.

**Describe the solution you'd like**

Given a folder structure with images of each class in each folder, the ability to load these folders into a HuggingFace dataset like "cifar10".

**Describe alternatives you've considered**

Implement ViT training outside of the HuggingFace Trainer and without datasets (we did this but prefer to stay on the main path)

Write custom data loader logic

**Additional context**

We're training ViT on custom dataset

Opening this back up, since I'm planning on tackling this. Already posted a quick version of it on my account on the hub.

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
```


SubjQA wrong boolean values in entries

SubjQA wrong boolean values in entries
## Describe the bug
SubjQA seems to have a boolean that's consistently wrong.

It defines:
- question_subj_level: The subjectiviy level of the question (on a 1 to 5 scale with 1 being the most subjective).
- is_ques_subjective: A boolean subjectivity label derived from question_subj_level (i.e., scores below 4 are considered as subjective)

However, `is_ques_subjective` seems to have wrong values in the entire dataset.

For instance, in the example in the dataset card, we have:
- "question_subj_level": 2
- "is_ques_subjective": false

However, according to the description, the question should be subjective since the `question_subj_level` is below 4

Hi @arnaudstiegler, thanks for reporting. I'm investigating it.


SubjQA wrong boolean values in entries

SubjQA wrong boolean values in entries
## Describe the bug
SubjQA seems to have a boolean that's consistently wrong.

It defines:
- question_subj_level: The subjectiviy level of the question (on a 1 to 5 scale with 1 being the most subjective).
- is_ques_subjective: A boolean subjectivity label derived from question_subj_level (i.e., scores below 4 are considered as subjective)

However, `is_ques_subjective` seems to have wrong values in the entire dataset.

For instance, in the example in the dataset card, we have:
- "question_subj_level": 2
- "is_ques_subjective": false

However, according to the description, the question should be subjective since the `question_subj_level` is below 4

@arnaudstiegler I have just checked that these mismatches are already present in the original dataset: https://github.com/megagonlabs/SubjQA

We are going to contact the dataset owners to report this.


SubjQA wrong boolean values in entries

SubjQA wrong boolean values in entries
## Describe the bug
SubjQA seems to have a boolean that's consistently wrong.

It defines:
- question_subj_level: The subjectiviy level of the question (on a 1 to 5 scale with 1 being the most subjective).
- is_ques_subjective: A boolean subjectivity label derived from question_subj_level (i.e., scores below 4 are considered as subjective)

However, `is_ques_subjective` seems to have wrong values in the entire dataset.

For instance, in the example in the dataset card, we have:
- "question_subj_level": 2
- "is_ques_subjective": false

However, according to the description, the question should be subjective since the `question_subj_level` is below 4

I have:
- opened an issue in their repo: https://github.com/megagonlabs/SubjQA/issues/3
- written an email to all the paper authors


SubjQA wrong boolean values in entries

SubjQA wrong boolean values in entries
## Describe the bug
SubjQA seems to have a boolean that's consistently wrong.

It defines:
- question_subj_level: The subjectiviy level of the question (on a 1 to 5 scale with 1 being the most subjective).
- is_ques_subjective: A boolean subjectivity label derived from question_subj_level (i.e., scores below 4 are considered as subjective)

However, `is_ques_subjective` seems to have wrong values in the entire dataset.

For instance, in the example in the dataset card, we have:
- "question_subj_level": 2
- "is_ques_subjective": false

However, according to the description, the question should be subjective since the `question_subj_level` is below 4

Please [see my response](https://github.com/megagonlabs/SubjQA/issues/3#issuecomment-905160010). There will be a fix in a couple of days.


Python Programming Puzzles

 Python Programming Puzzles
## Adding a Dataset
- **Name:** Python Programming Puzzles
- **Description:** Programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis
- **Paper:** https://arxiv.org/pdf/2106.05784.pdf
- **Data:** https://github.com/microsoft/PythonProgrammingPuzzles ([Scrolling through the data](https://github.com/microsoft/PythonProgrammingPuzzles/blob/main/problems/README.md))
- **Motivation:** Spans a large range of difficulty, problems, and domains. A useful resource for evaluation as we don't have a clear understanding of the abilities and skills of extremely large LMs.

Note: it's a growing dataset (contributions are welcome), so we'll need careful versioning for this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

👀 @TalSchuster


Python Programming Puzzles

 Python Programming Puzzles
## Adding a Dataset
- **Name:** Python Programming Puzzles
- **Description:** Programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis
- **Paper:** https://arxiv.org/pdf/2106.05784.pdf
- **Data:** https://github.com/microsoft/PythonProgrammingPuzzles ([Scrolling through the data](https://github.com/microsoft/PythonProgrammingPuzzles/blob/main/problems/README.md))
- **Motivation:** Spans a large range of difficulty, problems, and domains. A useful resource for evaluation as we don't have a clear understanding of the abilities and skills of extremely large LMs.

Note: it's a growing dataset (contributions are welcome), so we'll need careful versioning for this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Thanks @VictorSanh!
There's also a [notebook](https://aka.ms/python_puzzles) and [demo](https://aka.ms/python_puzzles_study) available now to try out some of the puzzles


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

That’s interesting thanks, let’s see what we can do. Can you detail your last sentence? I’m not sure I understand it well.


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

Hi ! I just re-ran a quick benchmark and using `to_numpy()` seems to be faster now:

```python
import pyarrow as pa  # I used pyarrow 3.0.0
import numpy as np

n, max_length = 1_000, 512
low, high, size = 0, 2 << 16, (n, max_length)

table = pa.Table.from_pydict({
    "input_ids": np.random.default_rng(42).integers(low=low, high=high, size=size).tolist()
})

%%timeit
_ = table.to_pandas()["input_ids"].to_numpy()
# 1.44 ms ± 80.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_pandas().to_numpy()
# 461 µs ± 14.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_numpy()
# 317 µs ± 5.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Currently the conversion from arrow to numpy is done in the NumpyArrowExtractor here:

https://github.com/huggingface/datasets/blob/d6d0ede9486ffad7944642ca9a326e058b676788/src/datasets/formatting/formatting.py#L143-L166

Let's update the NumpyArrowExtractor to call `to_numpy` directly and see how our github benchmarks evolve ?__


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

Sounds like a plan @lhoestq  If you create a PR I'll pick it up and try it out right away!


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

@lhoestq I can also prepare the PR, just lmk.


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

I’m not exactly sure how to read the graph but it seems that to_categorical take a lot of time here. Could you share more informations on the features/stats of your datasets so we could maybe design a synthetic datasets that looks more similar for debugging testing?


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

I created https://github.com/huggingface/datasets/pull/2505 if you want to play with it @vblagoje


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

> I’m not exactly sure how to read the graph but it seems that to_categorical take a lot of time here. Could you share more informations on the features/stats of your datasets so we could maybe design a synthetic datasets that looks more similar for debugging testing?

@thomwolf starting from the top, each rectangle represents the cumulative amount of it takes to execute the method call. Therefore, format_batch in torch_formatter.py takes ~20 sec, and the largest portion of that call is taken by to_pandas call and the smaller portion (grey rectangle) by the other method invocation(s) in format_batch (series_to_numpy etc).

Features of the dataset are BERT pre-training model input columns i.e:
```
f = Features({
 "input_ids": Sequence(feature=Value(dtype="int32")),
 "attention_mask": Sequence(feature=Value(dtype="int8")),
 "token_type_ids": Sequence(feature=Value(dtype="int8")),
 "labels": Sequence(feature=Value(dtype="int32")),
 "next_sentence_label": Value(dtype="int8")
})
```

I'll work with @lhoestq till we get to the bottom of this one.


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

@lhoestq the proposed branch is faster, but overall training speedup is a few percentage points. I couldn't figure out how to include the GitHub branch into setup.py, so I couldn't start NVidia optimized Docker-based pre-training run.  But on bare metal, there is a slight improvement. I'll do some more performance traces.


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

Hi @vblagoje, to install Datasets from @lhoestq PR reference #2505, you can use:
```shell
pip install git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head#egg=datasets
```


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

Hey @albertvillanova yes thank you, I am aware, I can easily pull it from a terminal command line but then I can't automate docker image builds as dependencies are picked up from setup.py and for some reason setup.py doesn't accept this string format.


Improve torch formatting performance

Improve torch formatting performance
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.

@vblagoje in that case, you can add this to your `setup.py`:
```python
    install_requires=[
        "datasets @ git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head",
```