Failure to hash function when using .map() 
Opened by venzen (closed)

### Describe the bug

_Parameter 'function'=<function process at 0x7f1ec4388af0> of the transform datasets.arrow_dataset.Dataset.\_map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed._

This issue with `.map()` happens for me consistently, as also described in closed issue #4506

Dataset indices can be individually serialized using dill and pickle without any errors.  I'm using tiktoken to encode in the function passed to map(). Similarly, indices can be individually encoded without error.

### Steps to reproduce the bug

```py
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

Should encode simple text objects.

### Environment info


Python versions tried: both 3.8 and 3.10.10
`PYTHONUTF8=1` as env variable

Datasets tried: 
- stas/openwebtext-10k
- rotten_tomatoes
- local text file

OS: Ubuntu Linux 20.04

Package versions:
- torch 1.13.1
- dill 0.3.4 (if using 0.3.6 - same issue)
- datasets 2.9.0
- tiktoken 0.2.0



Comments:
Hi ! `enc` is not hashable:
```python
import tiktoken
from datasets.fingerprint import Hasher

enc = tiktoken.get_encoding("gpt2")
Hasher.hash(enc)
# raises TypeError: cannot pickle 'builtins.CoreBPE' object
```
It happens because it's not picklable, and because of that it's not possible to cache the result of `map`, hence the warning message.

You can find more details about caching here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument.
Or disable caching using
```python
import datasets
datasets.disable_caching()
```

@lhoestq Thank you for the explanation and advice. Will relay all of this to the repo where this (non)issue arose. 

Great job with huggingface! 

We made tiktoken tokenizers hashable in #5552, which is included in today's release `datasets==2.10.0`

Just a heads up that when I'm trying to use TikToken along with the a given Dataset `.map()` method, I am still met with the following error :

```
  File "/opt/conda/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/conda/lib/python3.8/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle 'builtins.CoreBPE' object
```

My current environment is running datasets v2.10.0.

cc @mariosasko 

@lhoestq @edhenry I am also seeing this, do you have any suggested solution?

With which `datasets` version ? Can you try to udpate ?

@lhoestq @edhenry I am on datasets version `'2.12.0'. I see the same `TypeError: cannot pickle 'builtins.CoreBPE' object` that others are seeing.

I am able to reproduce this on datasets 2.14.2. The `datasets.disable_caching()` doesn't work around it.

@lhoestq - you might want to reopen this issue. Because of this issue folks won't be able run Karpathy's NanoGPT :(.

update： temporarily solved the problem by setting
```
--preprocess_num_workers  1
```

-------------
I  have met the same problem, here is my env:
```
datasets                      2.14.4
transformers                  4.31.0
tiktoken                      0.4.0
torch                         1.13.1
```

@mengban I cannot reproduce the issue even with these versions installed. It would help if you could provide info about your system and the `pip list` output.

@mariosasko Please take a look at this
```python
from typing import Any
from datasets import Dataset
import tiktoken

dataset = Dataset.from_list([{"n": str(i)} for i in range(20)])
enc = tiktoken.get_encoding("gpt2")


class A:
    tokenizer = enc #tiktoken.get_encoding("gpt2")

    def __call__(self, example) -> Any:
        ids = self.tokenizer.encode(example["n"])
        example["len"] = len(ids)
        return example

a = A()

def process(example):
    ids = a.tokenizer.encode(example["n"])
    example["len"] = len(ids)
    return example

# success
tokenized = dataset.map(process, desc="tiktoken", num_proc=2)

# raise TypeError: cannot pickle 'builtins.CoreBPE' object
tokenized = dataset.map(a, desc="tiktoken", num_proc=2)
```

pip list
```
datasets                      2.14.4
tiktoken                      0.4.0
```

Thanks @maxwellzh! Our `Hasher` works with this snippet, but the problem is running multiprocessing with a non-serializable `tiktoken.Encoding` object.

Inserting the following code before the `map` should fix this:
```python
import copyreg

def pickle_Encoding(enc):
  return (functools.partial(tiktoken.core.Encoding, enc.name, pat_str=enc._pat_str, mergeable_ranks=enc._mergeable_ranks, special_tokens=enc._special_tokens), ())

copyreg.pickle(tiktoken.core.Encoding, pickle_Encoding)
```

But the best fix would be implementing `__reduce__` for `tiktoken.Encoding` or `tiktoken.CoreBPE`. If I find time, I'll try to fix this in the `tiktoken` repo.

I think the right way to fix this would be to have new tokenizer instance for each process. This applies to many other tokenizers that don't support multi-process or have bugs. To do this, first define tokenizer factory class like this:

```
    class TikTokenFactory:
        def __init__(self):
            self._enc = None
            self.eot_token = None

        def encode_ordinary(self, text):
            if self._enc is None:
                self._enc = tiktoken.get_encoding("gpt2")
                self.eot_token = self._enc.eot_token
            return self._enc.encode_ordinary(text)
```

Now use this in `.map()` like this:

```
    # tokenize the dataset
    tokenized = dataset.map(
        partial(process, TikTokenFactory()),
        remove_columns=['text'],
        desc="tokenizing the splits",
        num_proc=max(1, cpu_count()//2),
    )
```

A full working example is here: https://github.com/sytelus/nanoGPT/blob/refactor/nanogpt_common/hf_data_prepare.py