Repository: vllm-project/vllm

Issue:
[BUG] vLLM Baichuan model is not working in latest vLLM release
A simple test from official script
```bash
python benchmarks/benchmark_latency.py --trust-remote-code \
 --model baichuan-inc/Baichuan-13B-Base \
 --input-len 256 --output-len 256 \
 --batch-size 32 \
 --tensor-parallel 2
```

Shows error info: ```AttributeError: 'BaichuanTokenizer' object has no attribute 'sp_model'```

This is due to the newest transformers==4.34.0 is not compatiable with currrent vllm Baichuan model. Yet this version required for Mistral model, hence introduces a breaking change in vLLM. @WoosukKwon @zhuohan123 


Discussion:
encountered the same issue, waiting for a solution~

You need to Modify  `tokenization_baichuan.py` ,`__init__ `code in `BaichuanTokenizer `.
```python
  def __init__(
        self,
        vocab_file,
        unk_token="<unk>",
        bos_token="
",
        eos_token="
",
        pad_token=None,
        sp_model_kwargs: Optional[Dict[str, Any]] = None,
        add_bos_token=True,
        add_eos_token=False,
        clean_up_tokenization_spaces=False,
        **kwargs,
    ):
        self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
        bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token
        eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token
        unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token
        pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token
        self.vocab_file = vocab_file
        self.add_bos_token = add_bos_token
        self.add_eos_token = add_eos_token
        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
        self.sp_model.Load(vocab_file)
        super().__init__(
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            pad_token=pad_token,
            add_bos_token=add_bos_token,
            add_eos_token=add_eos_token,
            sp_model_kwargs=self.sp_model_kwargs,
            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
            **kwargs,
        )
``` 
It's moving the `super()._init__ ` to the end.
@BWZhang-NLP It works, thank you!
If the transformers version is lower than 4.34, an error "ImportError: cannot import name 'MistralConfig' from 'transformers'" will be reported. No matter what model I run, it will be reported.
but when the version is 4.34, error "AttributeError: 'BaichuanTokenizer' object has no attribute 'sp_model'" will be reported only when running the Baichuan model.

This result is unsolvable unless transformers 4.34 adapts to Baichuan, or Baichuan adapts to 4.34, or adapt by yourself.
> 您需要修改 `tokenization_baichuan.py`，`__init__ `编码在`BaichuanTokenizer `.
> 
> ```python
>   def __init__(
>         self,
>         vocab_file,
>         unk_token="<unk>",
>         bos_token="
",
>         eos_token="
",
>         pad_token=None,
>         sp_model_kwargs: Optional[Dict[str, Any]] = None,
>         add_bos_token=True,
>         add_eos_token=False,
>         clean_up_tokenization_spaces=False,
>         **kwargs,
>     ):
>         self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
>         bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token
>         eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token
>         unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token
>         pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token
>         self.vocab_file = vocab_file
>         self.add_bos_token = add_bos_token
>         self.add_eos_token = add_eos_token
>         self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
>         self.sp_model.Load(vocab_file)
>         super().__init__(
>             bos_token=bos_token,
>             eos_token=eos_token,
>             unk_token=unk_token,
>             pad_token=pad_token,
>             add_bos_token=add_bos_token,
>             add_eos_token=add_eos_token,
>             sp_model_kwargs=self.sp_model_kwargs,
>             clean_up_tokenization_spaces=clean_up_tokenization_spaces,
>             **kwargs,
>         )
> ```
> 
> 它正在把事情拖到`super()._init__ `最后。

Try it.
same problem
@BWZhang-NLP Excuse me, I also encountered AttributeError when deploying Baihuan2 using this framework: 'BaihuanTokenizer' object has no attribute 'sp_ Model 'issue, but I found that tokenization_ Super () in baichuan. py__ Init__ is in the back. Is there any solution？
> @BWZhang-NLP Excuse me, I also encountered AttributeError when deploying Baihuan2 using this framework: 'BaihuanTokenizer' object has no attribute 'sp_ Model 'issue, but I found that tokenization_ Super () in baichuan. py__ Init__ is in the back. Is there any solution？

no other idea
> BWZhang

Thank you for your reply. I have resolved the issue through your method. Thank you again.
Thank you! this help me solve  the "sp_model "problem
> You need to Modify `tokenization_baichuan.py` ,`__init__ `code in `BaichuanTokenizer `.
> 
> ```python
>   def __init__(
>         self,
>         vocab_file,
>         unk_token="<unk>",
>         bos_token="
",
>         eos_token="
",
>         pad_token=None,
>         sp_model_kwargs: Optional[Dict[str, Any]] = None,
>         add_bos_token=True,
>         add_eos_token=False,
>         clean_up_tokenization_spaces=False,
>         **kwargs,
>     ):
>         self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
>         bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token
>         eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token
>         unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token
>         pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token
>         self.vocab_file = vocab_file
>         self.add_bos_token = add_bos_token
>         self.add_eos_token = add_eos_token
>         self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
>         self.sp_model.Load(vocab_file)
>         super().__init__(
>             bos_token=bos_token,
>             eos_token=eos_token,
>             unk_token=unk_token,
>             pad_token=pad_token,
>             add_bos_token=add_bos_token,
>             add_eos_token=add_eos_token,
>             sp_model_kwargs=self.sp_model_kwargs,
>             clean_up_tokenization_spaces=clean_up_tokenization_spaces,
>             **kwargs,
>         )
> ```
> 
> It's moving the `super()._init__ ` to the end.

I try to moving the "super()._init_" to the end of the function of _init_. Then I check this file again to be sure to do this.

However, When I use chatglm-6b in below code.
'''
from vllm import LLM, SamplingParams
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="./baichuan-7b",trust_remote_code=True)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
'''
I still met the same problem. AttributeError: 'BaiChuanTokenizer' object has no attribute 'sp_model'.
I check the tokenization_baichuan.py again. I found the super()._init_ appear the initial position not the end.
Do you do that's why? and have any idea to solve this problem?
Thank you! 
> > You need to Modify `tokenization_baichuan.py` ,`__init__ `code in `BaichuanTokenizer `.
> > ```python
> >   def __init__(
> >         self,
> >         vocab_file,
> >         unk_token="<unk>",
> >         bos_token="
",
> >         eos_token="
",
> >         pad_token=None,
> >         sp_model_kwargs: Optional[Dict[str, Any]] = None,
> >         add_bos_token=True,
> >         add_eos_token=False,
> >         clean_up_tokenization_spaces=False,
> >         **kwargs,
> >     ):
> >         self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
> >         bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token
> >         eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token
> >         unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token
> >         pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token
> >         self.vocab_file = vocab_file
> >         self.add_bos_token = add_bos_token
> >         self.add_eos_token = add_eos_token
> >         self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
> >         self.sp_model.Load(vocab_file)
> >         super().__init__(
> >             bos_token=bos_token,
> >             eos_token=eos_token,
> >             unk_token=unk_token,
> >             pad_token=pad_token,
> >             add_bos_token=add_bos_token,
> >             add_eos_token=add_eos_token,
> >             sp_model_kwargs=self.sp_model_kwargs,
> >             clean_up_tokenization_spaces=clean_up_tokenization_spaces,
> >             **kwargs,
> >         )
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > It's moving the `super()._init__ ` to the end.
> 
> I try to moving the "super()._init_" to the end of the function of _init_. Then I check this file again to be sure to do this.
> 
> However, When I use chatglm-6b in below code. ''' from vllm import LLM, SamplingParams prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95) llm = LLM(model="./baichuan-7b",trust_remote_code=True) outputs = llm.generate(prompts, sampling_params) for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ''' I still met the same problem. AttributeError: 'BaiChuanTokenizer' object has no attribute 'sp_model'. I check the tokenization_baichuan.py again. I found the super()._init_ appear the initial position not the end. Do you do that's why? and have any idea to solve this problem? Thank you!

I alse met this question!
You need to modify `tokenization_baichuan.py` in the Baichuan2 model dir you downloaded, not in the .cache dir of huggingface.
@Linzecong 
The same error cycles i have met.

Resolving patch:
diff --git a/vllm/transformers_utils/tokenizer.py b/vllm/transformers_utils/tokenizer.py
--- a/vllm/transformers_utils/tokenizer.py
+++ b/vllm/transformers_utils/tokenizer.py
@@ -4,6 +4,7 @@
                           PreTrainedTokenizerFast)
 
 from vllm.logger import init_logger
+from vllm.transformers_utils.tokenizers import *
 
 logger = init_logger(__name__)
 
@@ -61,6 +62,18 @@ def get_tokenizer(
             raise RuntimeError(err_msg) from e
         else:
             raise e
+    except AttributeError as e:
+        if "BaichuanTokenizer" in str(e):
+            # This is for the error "'BaichuanTokenizer' object has no
+            # attribute 'sp_model'".
+            tokenizer = BaichuanTokenizer.from_pretrained(
+                tokenizer_name,
+                *args,
+                trust_remote_code=trust_remote_code,
+                tokenizer_revision=tokenizer_revision,
+                **kwargs)
+        else:
+            raise e
 
     if not isinstance(tokenizer, PreTrainedTokenizerFast):
         logger.warning(
diff --git a/vllm/transformers_utils/tokenizers/__init__.py b/vllm/transformers_utils/tokenizers/__init__.py
new file mode 100644
--- /dev/null
+++ b/vllm/transformers_utils/tokenizers/__init__.py
@@ -0,0 +1,5 @@
+from vllm.transformers_utils.tokenizers.baichuan import BaichuanTokenizer
+
+__all__ = [
+    "BaichuanTokenizer",
+]
diff --git a/vllm/transformers_utils/tokenizers/baichuan.py b/vllm/transformers_utils/tokenizers/baichuan.py
new file mode 100644
--- /dev/null
+++ b/vllm/transformers_utils/tokenizers/baichuan.py
@@ -0,0 +1,263 @@
+# yapf: disable
+# Adapted from
+# https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat/blob/8f6e343d545c503b91429582231d1d354dac2740/tokenization_baichuan.py
+# This includes a fix suggested in
+# https://github.com/vllm-project/vllm/issues/1403#issuecomment-1767503058
+# Copyright (c) 2023, Baichuan Intelligent Technology. All rights reserved.
+
+import os
+from shutil import copyfile
+from typing import Any, Dict, List, Optional, Tuple
+
+import sentencepiece as spm
+from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+VOCAB_FILES_NAMES = {"vocab_file": "tokenizer.model"}
+
+PRETRAINED_VOCAB_FILES_MAP = {
+    "vocab_file": {},
+    "tokenizer_file": {},
+}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {}
+
+
+class BaichuanTokenizer(PreTrainedTokenizer):
+    """
+    Construct a Baichuan tokenizer. Based on byte-level Byte-Pair-Encoding.
+
+    Args:
+        vocab_file (`str`):
+            Path to the vocabulary file.
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["input_ids", "attention_mask"]
+
+    def __init__(
+        self,
+        vocab_file,
+        unk_token="<unk>",
+        bos_token="
",
+        eos_token="
",
+        pad_token=None,
+        sp_model_kwargs: Optional[Dict[str, Any]] = None,
+        add_bos_token=True,
+        add_eos_token=False,
+        clean_up_tokenization_spaces=False,
+        **kwargs,
+    ):
+        self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
+        bos_token = (
+            AddedToken(bos_token, lstrip=False, rstrip=False)
+            if isinstance(bos_token, str)
+            else bos_token
+        )
+        eos_token = (
+            AddedToken(eos_token, lstrip=False, rstrip=False)
+            if isinstance(eos_token, str)
+            else eos_token
+        )
+        unk_token = (
+            AddedToken(unk_token, lstrip=False, rstrip=False)
+            if isinstance(unk_token, str)
+            else unk_token
+        )
+        pad_token = (
+            AddedToken(pad_token, lstrip=False, rstrip=False)
+            if isinstance(pad_token, str)
+            else pad_token
+        )
+        self.vocab_file = vocab_file
+        self.add_bos_token = add_bos_token
+        self.add_eos_token = add_eos_token
+        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
+        self.sp_model.Load(vocab_file)
+        super().__init__(
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            pad_token=pad_token,
+            add_bos_token=add_bos_token,
+            add_eos_token=add_eos_token,
+            sp_model_kwargs=self.sp_model_kwargs,
+            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+            **kwargs,
+        )
+
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        state["sp_model"] = None
+        return state
+
+    def __setstate__(self, d):
+        self.__dict__ = d
+        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
+        self.sp_model.Load(self.vocab_file)
+
+    @property
+    def vocab_size(self):
+        """Returns vocab size"""
+        return self.sp_model.get_piece_size()
+
+    def get_vocab(self):
+        """Returns vocab as a dict"""
+        vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text):
+        """Returns a tokenized string."""
+        return self.sp_model.encode(text, out_type=str)
+
+    def _convert_token_to_id(self, token):
+        """Converts a token (str) in an id using the vocab."""
+        return self.sp_model.piece_to_id(token)
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        token = self.sp_model.IdToPiece(index)
+        return token
+
+    def convert_tokens_to_string(self, tokens):
+        """Converts a sequence of tokens (string) in a single string."""
+        current_sub_tokens = []
+        out_string = ""
+        prev_is_special = False
+        for i, token in enumerate(tokens):
+            # make sure that special tokens are not decoded using sentencepiece model
+            if token in self.all_special_tokens:
+                if not prev_is_special and i != 0:
+                    out_string += " "
+                out_string += self.sp_model.decode(current_sub_tokens) + token
+                prev_is_special = True
+                current_sub_tokens = []
+            else:
+                current_sub_tokens.append(token)
+                prev_is_special = False
+        out_string += self.sp_model.decode(current_sub_tokens)
+        return out_string
+
+    def save_vocabulary(
+        self, save_directory, filename_prefix: Optional[str] = None
+    ) -> Tuple[str]:
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (`str`):
+                The directory in which to save the vocabulary.
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        if not os.path.isdir(save_directory):
+            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
+            return
+        out_vocab_file = os.path.join(
+            save_directory,
+            (filename_prefix + "-" if filename_prefix else "")
+            + VOCAB_FILES_NAMES["vocab_file"],
+        )
+
+        if os.path.abspath(self.vocab_file) != os.path.abspath(
+            out_vocab_file
+        ) and os.path.isfile(self.vocab_file):
+            copyfile(self.vocab_file, out_vocab_file)
+        elif not os.path.isfile(self.vocab_file):
+            with open(out_vocab_file, "wb") as fi:
+                content_spiece_model = self.sp_model.serialized_model_proto()
+                fi.write(content_spiece_model)
+
+        return (out_vocab_file,)
+
+    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
+        bos_token_id = [self.bos_token_id] if self.add_bos_token else []
+        eos_token_id = [self.eos_token_id] if self.add_eos_token else []
+
+        output = bos_token_id + token_ids_0 + eos_token_id
+
+        if token_ids_1 is not None:
+            output = output + bos_token_id + token_ids_1 + eos_token_id
+
+        return output
+
+    def get_special_tokens_mask(
+        self,
+        token_ids_0: List[int],
+        token_ids_1: Optional[List[int]] = None,
+        already_has_special_tokens: bool = False,
+    ) -> List[int]:
+        """
+        Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer `prepare_for_model` method.
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
+                Whether or not the token list is already formatted with special tokens for the model.
+
+        Returns:
+            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0,
+                token_ids_1=token_ids_1,
+                already_has_special_tokens=True,
+            )
+
+        bos_token_id = [1] if self.add_bos_token else []
+        eos_token_id = [1] if self.add_eos_token else []
+
+        if token_ids_1 is None:
+            return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id
+        return (
+            bos_token_id
+            + ([0] * len(token_ids_0))
+            + eos_token_id
+            + bos_token_id
+            + ([0] * len(token_ids_1))
+            + eos_token_id
+        )
+
+    def create_token_type_ids_from_sequences(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT
+        sequence pair mask has the following format:
+
+        ```
+        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
+        | first sequence    | second sequence |
+        ```
+
+        if token_ids_1 is None, only returns the first portion of the mask (0s).
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of ids.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
+        """
+        bos_token_id = [self.bos_token_id] if self.add_bos_token else []
+        eos_token_id = [self.eos_token_id] if self.add_eos_token else []
+
+        output = [0] * len(bos_token_id + token_ids_0 + eos_token_id)
+
+        if token_ids_1 is not None:
+            output += [1] * len(bos_token_id + token_ids_1 + eos_token_id)
+
+        return output


Test patch: