'BertTokenizer' object has no attribute 'tokens_trie'
Author: Momentum-TN
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
- Used machine: google colab
- `transformers` version: 4.11.3
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
- [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I have a trained model on custom tags (for invoices)
1.  I passed a text block to the model using `model.predict([block])`
2.  The error raised

```
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_utils.py", line 233, in convert_examples_with_multiprocessing
    for example in example_group
  File "/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_utils.py", line 233, in <listcomp>
    for example in example_group
  File "/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_utils.py", line 280, in convert_example_to_feature
    word_tokens = tokenizer.tokenize(word)
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils.py", line 476, in tokenize
    tokens = self.tokens_trie.split(text)
AttributeError: 'BertTokenizer' object has no attribute 'tokens_trie'
"""


The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)

/tmp/ipykernel_1136/2044312605.py in <module>
      1 extraction_info=[]
      2 for txt in text_blocks.get_texts():
----> 3     prediction, model_output = model_bert.predict([txt])
      4     extraction_info.append(prediction)
      5     print(prediction, end='\n---\n')

12 frames

/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_model.py in predict(self, to_predict, split_on_space)
   1458 
   1459             eval_dataset = self.load_and_cache_examples(
-> 1460                 None, to_predict=predict_examples
   1461             )
   1462             eval_sampler = SequentialSampler(eval_dataset)

/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_model.py in load_and_cache_examples(self, data, evaluate, no_cache, to_predict)
   1738                         chunksize=args.multiprocessing_chunksize,
   1739                         mode=mode,
-> 1740                         use_multiprocessing_for_evaluation=args.use_multiprocessing_for_evaluation,
   1741                     )
   1742 

/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_utils.py in convert_examples_to_features(examples, label_list, max_seq_length, tokenizer, cls_token_at_end, cls_token, cls_token_segment_id, sep_token, sep_token_extra, pad_on_left, pad_token, pad_token_segment_id, pad_token_label_id, sequence_a_segment_id, mask_padding_with_zero, process_count, chunksize, silent, use_multiprocessing, mode, use_multiprocessing_for_evaluation)
    471                     ),
    472                     total=len(examples),
--> 473                     disable=silent,
    474                 )
    475             )

/usr/local/lib/python3.7/dist-packages/tqdm/notebook.py in __iter__(self)
    255     def __iter__(self):
    256         try:
--> 257             for obj in super(tqdm_notebook, self).__iter__():
    258                 # return super(tqdm...) will not catch exception
    259                 yield obj

/usr/local/lib/python3.7/dist-packages/tqdm/std.py in __iter__(self)
   1178 
   1179         try:
-> 1180             for obj in iterable:
   1181                 yield obj
   1182                 # Update and possibly print the progressbar.

/usr/lib/python3.7/multiprocessing/pool.py in <genexpr>(.0)
    323                     result._set_length
    324                 ))
--> 325             return (item for chunk in result for item in chunk)
    326 
    327     def imap_unordered(self, func, iterable, chunksize=1):

/usr/lib/python3.7/multiprocessing/pool.py in next(self, timeout)
    746         if success:
    747             return value
--> 748         raise value
    749 
    750     __next__ = next                    # XXX

/usr/lib/python3.7/multiprocessing/pool.py in worker()
    119         job, i, func, args, kwds = task
    120         try:
--> 121             result = (True, func(*args, **kwds))
    122         except Exception as e:
    123             if wrap_exception and func is not _helper_reraises_exception:

/usr/lib/python3.7/multiprocessing/pool.py in mapstar()
     42 
     43 def mapstar(args):
---> 44     return list(map(*args))
     45 
     46 def starmapstar(args):

/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_utils.py in convert_examples_with_multiprocessing()
    231             mask_padding_with_zero,
    232         )
--> 233         for example in example_group
    234     ]
    235 

/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_utils.py in <listcomp>()
    231             mask_padding_with_zero,
    232         )
--> 233         for example in example_group
    234     ]
    235 

/usr/local/lib/python3.7/dist-packages/simpletransformers/ner/ner_utils.py in convert_example_to_feature()
    278         for i, (word, label) in enumerate(zip(example.words, example.labels)):
    279             if example.tokenized_word_ids is None:
--> 280                 word_tokens = tokenizer.tokenize(word)
    281             else:
    282                 word_tokens = example.tokenized_word_ids[i]

/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils.py in tokenize()
    474 
    475         no_split_token = set(self.unique_no_split_tokens)
--> 476         tokens = self.tokens_trie.split(text)
    477         # ["This is something", "<special_token_1>", "  else"]
    478         for i, token in enumerate(tokens):

AttributeError: 'BertTokenizer' object has no attribute 'tokens_trie'

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The predict method should return me a model_output and a list of tagged words:
`[[{'Philip': 'NAME'}, {'Norris,': 'NAME'}, {'Inc.': 'O'}, {'Research': 'O'}, {'Counter': 'O'}, {'P.': 'O'}, {'0.': 'O'}, {'Box': 'O'}, {'3D': 'O'}, {'Richsond': 'NAME'}, {'6,': 'DATE'}, {'Virginia': 'LOCATION'}, {'Att:': 'O'}, {'Mr.': 'O'}, {'Frank': 'NAME'}, {'Resnik': 'NAME'}]]`
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hello! Could you post a full reproducible code sample so that we may take a look? Thank you, cc @Narsil 

Hi, `tokens_trie` should be initiated in the `__init__` of every slow tokenizer.

Did you copy the tokenizer in some way ? It could be when the tokenizer get moved to the other thread maybe ?

I have the same issue with the `DistilBertTokenizer` when version was bumped to >=4.11.0

<Result AttributeError("'DistilBertTokenizer' object has no attribute 'tokens_trie'")>

@gbmarc1 do you have more complete script so we can reproduce ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.