Pretrainedtokenizerfast Segmentation fault
Author: Facerman-cloud
Pull request: no
Labels: name: bug

### System Info

**Problem:**
`Fatal Python error: Segmentation fault

Thread 0x00007fd9c172e700 (most recent call first):
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/threading.py", line 316 in wait
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/threading.py", line 574 in wait
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/threading.py", line 930 in _bootstrap

Current thread 0x00007fdb742c1740 (most recent call first):
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/tokenization/middle/TokenizerMiddleMapping.py", line 68 in __init__
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/tokenization/middle/TokenizerMiddleMapping.py", line 48 in from_hf_tokenizer
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/tokenization/byte_trie.py", line 54 in from_tokenizer
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/token_grammar_recognizer.py", line 29 in __init__
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/token_grammar_recognizer.py", line 102 in __init__
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/llm/llama.py", line 64 in generate_hf_llm
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/llm/hf_model.py", line 129 in generate_hf
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/llm/hf_model.py", line 97 in generate
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/planner/llm_planner.py", line 557 in replan
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/planner/llm_planner.py", line 638 in get_next_action
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/evaluation/decentralized_evaluation_runner.py", line 128 in get_low_level_actions
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/evaluation/evaluation_runner.py", line 618 in run_instruction
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/examples/planner_demo.py", line 347 in run_planner
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/examples/planner_demo.py", line 188 in run_eval
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/hydra/core/utils.py", line 186 in run_job
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 119 in run
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458 in <lambda>
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220 in run_and_report
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457 in _run_app
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394 in _run_hydra
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/site-packages/hydra/main.py", line 94 in decorated_main
  File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/examples/planner_demo.py", line 455 in <module>
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/runpy.py", line 87 in _run_code
  File "/home/taiic/anaconda3/envs/habitat-llm/lib/python3.9/runpy.py", line 197 in _run_module_as_main`
**device**：
the device I am using is Ubuntu 22.04 and the driver I am using is Nvidia-driver-550. The graphics cards used simultaneously are two 48GB NVIDIA GeForce RTX 4090 D.
**Version**
 The versions of transformer and tokenizer are tokenizers-0.20.3 transformer-4.46.2



### Who can help?

@ArthurZucker  and @itazap 
I really hope to receive help from all the teachers here.

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**Code related links**
[](https://github.com/facebookresearch/partnr-planner)

### Expected behavior

The error mentioned above was encountered while running QuickStart in the above link. While solving the problem, I found that it was related to the HuggingFaceFastTokenizer compiled using Rust. I have been thinking for a long time but have not been able to solve it. I sincerely hope to receive help from all the teachers here. I would greatly appreciate it if this issue could be resolved.

Comments:
Hey! Sorry it is not apparent from the stacktrace that this issue is related to `tokenizers`:
```python
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/tokenization/middle/TokenizerMiddleMapping.py", line 68 in init
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/tokenization/middle/TokenizerMiddleMapping.py", line 48 in from_hf_tokenizer
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/tokenization/byte_trie.py", line 54 in from_tokenizer
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/token_grammar_recognizer.py", line 29 in init
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/third_party/transformers-CFG/transformers_cfg/token_grammar_recognizer.py", line 102 in init
```
shows mostly third party using `tokenizers` maybe? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.