Repository: vllm-project/vllm

Issue:
Bug in LLaMA fast tokenizer
In my environment, using the LLaMA fast tokenizer raises an error about protobuf:
```
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/convert_slow_tokenizer.py", line 445, in __init__
    from .utils import sentencepiece_model_pb2 as model_pb2
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/utils/sentencepiece_model_pb2.py", line 91, in <module>
    _descriptor.EnumValueDescriptor(
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 796, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
```

While downgrading the protobuf version removed the error, it slowed down the initialization time by ~8x.
* Initialization with fast tokenizer & protobuf==3.20.3 
```
real    4m18.476s
user    3m52.706s
sys     0m27.644s
```
* Initialization with slow tokenizer
```
real    0m27.620s
user    0m8.011s
sys     0m19.237s
```



Discussion:
@merrymercy has confirmed that this also happens in his environment. Maybe we can use the slow tokenizer until the bug gets fixed.
+1 Met the same error on my side.

Resolving patch:
diff --git a/cacheflow/sampling_params.py b/cacheflow/sampling_params.py
--- a/cacheflow/sampling_params.py
+++ b/cacheflow/sampling_params.py
@@ -129,7 +129,7 @@ def __repr__(self) -> str:
                 f"frequency_penalty={self.frequency_penalty}, "
                 f"temperature={self.temperature}, "
                 f"top_p={self.top_p}, "
-                f"top_k={self.top_k},"
+                f"top_k={self.top_k}, "
                 f"use_beam_search={self.use_beam_search}, "
                 f"stop={self.stop}, "
                 f"ignore_eos={self.ignore_eos}, "
diff --git a/cacheflow/server/tokenizer_utils.py b/cacheflow/server/tokenizer_utils.py
--- a/cacheflow/server/tokenizer_utils.py
+++ b/cacheflow/server/tokenizer_utils.py
@@ -7,11 +7,7 @@
 
 logger = init_logger(__name__)
 
-_MODEL_TYPES_WITH_SLOW_TOKENIZER = [
-    # LLaMA fast tokenizer has a bug related to protobuf.
-    # See https://github.com/WoosukKwon/cacheflow/issues/80#issue-1698550554
-    "llama",
-]
+_MODEL_TYPES_WITH_SLOW_TOKENIZER = []
 
 
 def get_tokenizer(
@@ -20,7 +16,15 @@ def get_tokenizer(
     **kwargs,
 ) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:
     config = AutoConfig.from_pretrained(model_name)
-    if config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:
+    if config.model_type == "llama" and getattr(kwargs, "use_fast", True):
+        # LLaMA fast tokenizer causes protobuf errors in some environments.
+        # However, we found that the below LLaMA fast tokenizer works well in
+        # most environments.
+        model_name = "hf-internal-testing/llama-tokenizer"
+        logger.info(
+            f"Using the LLaMA fast tokenizer in '{model_name}' to avoid "
+            "potential protobuf errors.")
+    elif config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:
         if getattr(kwargs, "use_fast", False) == True:
             raise ValueError(
                 f"Cannot use the fast tokenizer for {config.model_type} due to "


Test patch: