BertWordPieceTokenizer cannot be pickled
Author: Sriharsha-hatwar
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
**Bert**

Language I am using the model on (English, Chinese ...):
**English**

The problem arises when using:
* [X] my own modified scripts: 


The tasks I am working on is:
* [X] my own task or dataset: 

## To reproduce



``` python
import torch
import tokenizers
import pandas as pd
from torch.utils import data

class config:
    MAX_LEN = 128
    TRAIN_BATCH_SIZE = 64
    VALID_BATCH_SIZE = 16
    EPOCHS = 5
    BERT_PATH = "../input/bert-base-uncased/"
    MODEL_PATH = "model.bin"
    TRAINING_FILE = "../input/tweet-sentiment-extraction/train_folds.csv"
    TOKENIZER = tokenizers.BertWordPieceTokenizer(
        f"{BERT_PATH}/vocab.txt", 
        lowercase=True
    )

def process_data(tweet, selected_text, sentiment, tokenizer, max_len):
    len_st = len(selected_text)
    idx0 = -1
    idx1 = -1
    for ind in (i for i, e in enumerate(tweet) if e == selected_text[0]):
        if tweet[ind: ind+len_st] == selected_text:
            idx0 = ind
            idx1 = ind + len_st - 1
            break

    char_targets = [0] * len(tweet)
    if idx0 != -1 and idx1 != -1 :
        for ct in range(idx0, idx1 + 1):
            char_targets[ct] = 1
    
    tok_tweet = tokenizer.encode(tweet)
    input_ids_orig = tok_tweet.ids[1:-1]
    tweet_offsets = tok_tweet.offsets[1:-1]
    
    target_idx = []
    for j, (offset1, offset2) in enumerate(tweet_offsets):
        if sum(char_targets[offset1: offset2]) > 0:
            target_idx.append(j)
    
    targets_start = target_idx[0]
    targets_end = target_idx[-1]

    sentiment_id = {
        'positive': 3893,
        'negative': 4997,
        'neutral': 8699
    }
    
    input_ids = [101] + [sentiment_id[sentiment]] + [102] + input_ids_orig + [102]
    token_type_ids = [0, 0, 0] + [1] * (len(input_ids_orig) + 1)
    mask = [1] * len(token_type_ids)
    tweet_offsets = [(0, 0)] * 3 + tweet_offsets + [(0, 0)]
    targets_start += 3
    targets_end += 3

    padding_length = max_len - len(input_ids)
    if padding_length > 0:
        input_ids = input_ids + ([0] * padding_length)
        mask = mask + ([0] * padding_length)
        token_type_ids = token_type_ids + ([0] * padding_length)
        tweet_offsets = tweet_offsets + ([(0, 0)] * padding_length)
    
    return {
        'ids': input_ids,
        'mask': mask,
        'token_type_ids': token_type_ids,
        'targets_start': targets_start,
        'targets_end': targets_end,
        'orig_tweet': tweet,
        'orig_selected': selected_text,
        'sentiment': sentiment,
        'offsets': tweet_offsets
    }

class TweetDataset(data.Dataset):
    def __init__(self, tweet, sentiment, selected_text):
        self.tweet = tweet
        self.sentiment = sentiment
        self.selected_text = selected_text
        self.tokenizer = config.TOKENIZER
        self.max_len = config.MAX_LEN
    
    def __len__(self):
        return len(self.tweet)

    def __getitem__(self, item):
        data = process_data(
            self.tweet[item], 
            self.selected_text[item], 
            self.sentiment[item],
            self.tokenizer,
            self.max_len
        )

        return {
            'ids': torch.tensor(data["ids"], dtype=torch.long),
            'mask': torch.tensor(data["mask"], dtype=torch.long),
            'token_type_ids': torch.tensor(data["token_type_ids"], dtype=torch.long),
            'targets_start': torch.tensor(data["targets_start"], dtype=torch.long),
            'targets_end': torch.tensor(data["targets_end"], dtype=torch.long),
            'orig_tweet': data["orig_tweet"],
            'orig_selected': data["orig_selected"],
            'sentiment': data["sentiment"],
            'offsets': torch.tensor(data["offsets"], dtype=torch.long)
        }

dfx = pd.read_csv(config.TRAINING_FILE)
fold = 4
df_train = dfx[dfx.kfold != fold].reset_index(drop=True)
df_valid = dfx[dfx.kfold == fold].reset_index(drop=True)

train_dataset = TweetDataset(
        tweet=dfx.text.values,
        sentiment=dfx.sentiment.values,
        selected_text=dfx.selected_text.values
    )

train_data_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=config.TRAIN_BATCH_SIZE,
        num_workers=1
    )

if __name__ =='__main__':
    a = enumerate(train_data_loader)
```


## Expected behavior

The enumerate should return the iterable.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
Output of `transformers-cli env`

transformers version: 2.9.1
Platform: Windows-10-10.0.18362-SP0
Python version: 3.8.2
PyTorch version (GPU?): 1.5.0 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

Comments:
If needed I can even provide the dataset (Did not want to clutter): 
the error stacktrace : 
```
Traceback (most recent call last):
  File "error.py", line 129, in <module>
    a = enumerate(train_data_loader)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
    w.start()
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\context.py", line 224, in _Popen
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\context.py", line 326, in _Popen
  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\spawn.py", line 116, in spawn_main
    return Popen(process_obj)
exitcode = _main(fd, parent_sentinel)  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__

  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\spawn.py", line 126, in _main
    reduction.dump(process_obj, to_child)
self = reduction.pickle.load(from_parent)  File "C:\Users\admin\miniconda3\envs\machine_learning\lib\multiprocessing\reduction.py", line 60, in dump

EOFError: Ran out of input
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Tokenizer' object
```
But when run in kaggle notebook this works perfectly well. (The same script having same tokenizer and transformers version)
@julien-c @sshleifer any help here?

~Yes, this is fixed by PR #4389 , so you could `pip install -e .` off of that branch.~

Hi @sshleifer I did these steps : 
1. git fetch origin pull/4389/head:temp_fix
2. git checkout temp_fix
3. pip install -e  .
still the above fix doesn't seem to work. 

By looking into the PR , I am guessing that it is fixed for `MarianTokenizer ` and not for `BertWordPieceTokenizer` that I am using in the above script.
 

Tested with another environment with python = 3.7.7, same issue is observed.

Just wanted to mention that providing `num_workers = 0` bypasses the problem. So it only fails when multiprocessing is involved. This issue is not only in `BertWordPieceTokenizer`, It also fails with `ByteLevelBPETokenizer` .

And this probably should be moved to the tokenizer repo @sshleifer to confirm.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.