GitHub issue #5985: Cannot reuse tokenizer object for dataset map
State: closed | Created: 2023-06-23T14:45:31Z

### Describe the bug

Related to https://github.com/huggingface/transformers/issues/24441. Not sure if this is a tokenizer issue or caching issue, so filing in both.

Passing the tokenizer to the dataset map function causes the tokenizer to be fingerprinted weirdly. After calling the tokenizer with arguments like padding and truncation the tokenizer object changes interanally, even though the hash remains the same.

But dumps is able to detect that internal change which causes the tokenizer object's fingerprint to change.



### Steps to reproduce the bug

```python
from transformers import AutoTokenizer
from datasets.utils.py_utils import dumps # Huggingface datasets

t = AutoTokenizer.from_pretrained('bert-base-uncased')
t.save_pretrained("tok1")
th1 = hash(dumps(t))
text = "This is an example text"
ttext = t(text, max_length=512, padding="max_length", truncation=True)
t.save_pretrained("tok2")
th2 = hash(dumps(t))

assert th1 == th2 # Assertion Error
```

But if you use just the hash of the object without dumps, the hashes don't change

```python
from transformers import AutoTokenizer
from datasets.utils.py_utils import dumps # Huggingface datasets

t = AutoTokenizer.from_pretrained('bert-base-uncased')
th1 = hash(t)  # Just hash no dumps
text = "This is an example text"
ttext = t(text, max_length=512, padding="max_length", truncation=True)
th2 = hash(t) # Just hash no dumps

assert th1 == th2 # This is OK
```

This causes situations such as the following

1. Create a text file like this `yes "This is an example text" | head -n 10000 > lines.txt`

```python
from transformers import AutoTokenizer
import datasets


class TokenizeMapper(object):
    """Mapper for tokenizer.

    This is needed because the caching mechanism of HuggingFace does not work on
    lambdas. Each time a new lambda will be created by a new process which will
    lead to a different hash.
    This way we can have a universal mapper object in init and reuse it with the same
    hash for each process.
    """

    def __init__(self, tokenizer):
        """Initialize the tokenizer."""
        self.tokenizer = tokenizer

    def __call__(self, examples, **kwargs):
        """Run the mapper."""
        texts = examples["text"]
        tt = self.tokenizer(texts, max_length=256, padding="max_length", truncation=True)
        batch_outputs = {
            "input_ids": tt.input_ids,
            "attention_mask": tt.attention_mask,
        }
        return batch_outputs


t = AutoTokenizer.from_pretrained('bert-base-uncased')
mapper = TokenizeMapper(t)

ds = datasets.load_dataset("text", data_files="lines.txt")

mds1 = ds.map(
    mapper,
    batched=False,
    remove_columns=["text"],
).with_format("torch")

mds2 = ds.map(
    mapper,
    batched=False,
    remove_columns=["text"],
).with_format("torch")
```

The second call to map should reuse the cached processed dataset from mds1, but it instead it redoes the tokenization because of the behavior of dumps.

### Expected behavior

We should be able to initialize a tokenizer. And reusing it should let us reuse the same map computation for the same dataset.

The second call to map should reuse the cached processed dataset from mds1, but it instead it redoes the tokenization because of the behavior of dumps.

### Environment info

- `datasets` version: 2.13.0
- Platform: Linux-6.1.31_1-x86_64-with-glibc2.36
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.2