remove files

CHANGELOG.md

@@ -1,3 +0,0 @@
-## v1.0
-
-- Add accelerated ChatGLM-6B model (from: https://huggingface.co/THUDM/chatglm-6b)

Dockerfile

@@ -1,11 +0,0 @@
-FROM nvcr.io/nvidia/pytorch:23.02-py3
-
-WORKDIR /workdir
-
-COPY requirements.txt /workdir/
-
-# since installing icetk will install protobuf 3.18.3, and we need protobuf==3.20.3
-RUN pip install -r requirements.txt && \
-    pip install protobuf==3.20.3 
-
-

README.md

@@ -1,120 +0,0 @@
----
-license: creativeml-openrail-m
-language:
-- en
-tags:
-- LLM
-- tensorRT
-- ChatGLM
----
-## Model Card for lyraChatGLM
-
-lyraChatGLM is currently the **fastest ChatGLM-6B** available. To the best of our knowledge, it is the **first accelerated version of ChatGLM-6B**.
-
-The inference speed of lyraChatGLM has achieved **10x** acceleration upon the ealry original version. We are still working hard to further improve the performance.
-
-Among its main features are:
-
-- weights: original ChatGLM-6B weights released by THUDM.
-- device: lyraChatGLM is mainly based on TensorRT compiled for SM=80 (A100, for example).
-- batch_size: compiled with dynamic batch size, max batch_size = 8
-
-## Speed
-
-### test environment
-
-- device: Nvidia A100 40G
-- batch size: 8
-
-**Since early chatGLM version didn't suport batch inference, `original` in below table was measured on batch_size=1**
-
-
-**According to [this discussion](https://huggingface.co/TMElyralab/lyraChatGLM/discussions/6), this bug has been fixed and the speed on batch_size=8 reachs up to 137 tokens/s. We will evaluate and update the latest performance.**
-
-|version|speed|
-|:-:|:-:|
-|original|30 tokens/s|
-|lyraChatGLM|310 tokens/s|
-
-
-## Model Sources
-
-- **Repository:** https://huggingface.co/THUDM/chatglm-6b
-
-## Try Demo in 2 fast steps
-
-``` bash
-#step 1
-git clone https://huggingface.co/TMElyralab/lyraChatGLM
-cd lyraChatGLM
-
-#step 2
-docker run --gpus=1 --rm --net=host -v ${PWD}:/workdir yibolu96/lyra-chatglm-env:0.0.1 python3 /workdir/demo.py
-```
-
-## Uses
-
-```python
-from transformers import AutoTokenizer
-from lyraChatGLM import GLM6B, FasterChatGLM
-import os
-
-current_workdir = os.path.dirname(__file__)
-
-MAX_OUT_LEN = 100
-chatglm6b_dir = os.path.join(current_workdir, "models")
-tokenizer = AutoTokenizer.from_pretrained(chatglm6b_dir, trust_remote_code=True)
-input_str = ["为什么我们需要对深度学习模型加速？", ]
-inputs = tokenizer(input_str, return_tensors="pt", padding=True)
-input_ids = inputs.input_ids.to('cuda:0')
-
-plan_path = os.path.join(current_workdir, "models/glm6b-bs8.ftm")
-
-# kernel for chat model.
-kernel = GLM6B(plan_path=plan_path,
-               batch_size=1,
-               num_beams=1,
-               use_cache=True,
-               num_heads=32,
-               emb_size_per_heads=128,
-               decoder_layers=28,
-               vocab_size=150528,
-               max_seq_len=MAX_OUT_LEN)
-
-chat = FasterChatGLM(model_dir=chatglm6b_dir, kernel=kernel).half().cuda()
-
-# generate
-sample_output = chat.generate(inputs=input_ids, max_length=MAX_OUT_LEN)
-# de-tokenize model output to text
-res = tokenizer.decode(sample_output[0], skip_special_tokens=True)
-print(res)
-```
-## Demo output
-
-### input
-为什么我们需要对深度学习模型加速? 。
-
-### output
-为什么我们需要对深度学习模型加速? 深度学习模型的训练需要大量计算资源,特别是在训练模型时,需要大量的内存、GPU(图形处理器)和其他计算资源。因此,训练深度学习模型需要一定的时间,并且如果模型不能快速训练,则可能会导致训练进度缓慢或无法训练。
-
-以下是一些原因我们需要对深度学习模型加速:
-
-1. 训练深度神经网络需要大量的计算资源,特别是在训练深度神经网络时,需要更多的计算资源,因此需要更快的训练速度。
-
-### TODO：
-
-We plan to implement a FasterTransformer version to publish a much faster release. Stay tuned!
-
-## Citation
-``` bibtex
-@Misc{lyraChatGLM2023,
-  author =       {Kangjian Wu, Zhengtao Wang, Yibo Lu, Bin Wu},
-  title =        {lyraChatGLM: Accelerating ChatGLM by 10x+},
-  howpublished = {\url{https://huggingface.co/TMElyralab/lyraChatGLM}},
-  year =         {2023}
-}
-```
-
-## Report bug
-- start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraChatGLM/discussions
-- report bug with a `[bug]` mark in the title.

demo.py

@@ -1,35 +0,0 @@
-# coding=utf-8
-
-from transformers import AutoTokenizer
-from lyraChatGLM import GLM6B, FasterChatGLM
-import os
-
-current_workdir = os.path.dirname(__file__)
-
-MAX_OUT_LEN = 100
-chatglm6b_dir = os.path.join(current_workdir, "models")
-tokenizer = AutoTokenizer.from_pretrained(chatglm6b_dir, trust_remote_code=True)
-input_str = ["为什么我们需要对深度学习模型加速？", ]
-inputs = tokenizer(input_str, return_tensors="pt", padding=True)
-input_ids = inputs.input_ids.to('cuda:0')
-
-plan_path = os.path.join(current_workdir, "models/glm6b-bs8.ftm")
-
-# kernel for chat model.
-kernel = GLM6B(plan_path=plan_path,
-               batch_size=1,
-               num_beams=1,
-               use_cache=True,
-               num_heads=32,
-               emb_size_per_heads=128,
-               decoder_layers=28,
-               vocab_size=150528,
-               max_seq_len=MAX_OUT_LEN)
-
-chat = FasterChatGLM(model_dir=chatglm6b_dir, kernel=kernel).half().cuda()
-
-# generate
-sample_output = chat.generate(inputs=input_ids, max_length=MAX_OUT_LEN)
-# de-tokenize model output to text
-res = tokenizer.decode(sample_output[0], skip_special_tokens=True)
-print(res)

lyraChatGLM/__init__.py

@@ -1,10 +0,0 @@
-import os
-import ctypes
-
-current_workdir = os.path.dirname(__file__)
-ctypes.cdll.LoadLibrary(os.path.join(current_workdir, "libnvinfer_plugin.so"))
-os.environ["TORCH_USE_RTLD_GLOBAL"]="YES"
-
-import torch
-from .glm import GLM6B
-from .model import FasterChatGLM

lyraChatGLM/model.py

@@ -1,131 +0,0 @@
-import torch
-from transformers.modeling_outputs import CausalLMOutputWithPast
-from transformers.modeling_utils import PreTrainedModel
-from transformers import AutoConfig
-from typing import Dict, List, Tuple, Union, Optional
-
-
-class FasterChatGLM(PreTrainedModel):
-    def __init__(self, model_dir, kernel, *inputs, **kwargs):
-        config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
-        config.n_head = config.num_attention_heads
-        config.n_embd = config.hidden_size
-        config.n_layer = config.num_layers
-        super().__init__(config, *inputs, **kwargs)
-        self.kernel = kernel
-        self.fake_reg = torch.nn.Linear(2, 2)
-        self.position_encoding_2d = True
-
-    def forward(self, input_ids, position_ids, attention_mask, past_key_values, *args, **kwargs):
-        inputs_values = [input_ids, position_ids, attention_mask]
-        if past_key_values is not None:
-            inputs_values = inputs_values + past_key_values
-
-        computed = self.kernel.infer(inputs_values)
-        logits = computed[0]
-        if len(computed) == 1:
-            present_key_values = None
-        else:
-            present_key_values = computed[1:]
-
-        return CausalLMOutputWithPast(logits=logits, past_key_values=present_key_values)
-
-    def get_masks_and_position_ids(self, seq, mask_position, context_length, device, gmask=False):
-        attention_mask = torch.ones((1, context_length, context_length), device=device)
-        attention_mask.tril_()
-        attention_mask[..., :context_length - 1] = 1
-        attention_mask.unsqueeze_(1)
-        attention_mask = (attention_mask < 0.5).bool()
-
-        if self.position_encoding_2d:
-            seq_length = seq.index(150004)
-            position_ids = torch.arange(context_length, dtype=torch.long, device=device)
-            if not gmask:
-                position_ids[seq_length:] = mask_position
-            block_position_ids = torch.cat((
-                torch.zeros(seq_length, dtype=torch.long, device=device),
-                torch.arange(context_length - seq_length, dtype=torch.long, device=device) + 1
-            ))
-            position_ids = torch.stack((position_ids, block_position_ids), dim=0)
-        else:
-            position_ids = torch.arange(context_length, dtype=torch.long, device=device)
-            if not gmask:
-                position_ids[context_length - 1:] = mask_position
-
-        position_ids = position_ids.unsqueeze(0)
-
-        return attention_mask, position_ids
-
-    def prepare_one_sample(self, input_id, mask_token, past, past_key_values, use_gmask):
-
-        seq = input_id.tolist()
-        mask_position = seq.index(mask_token)
-
-        if mask_token not in seq:
-            raise ValueError("You have to add either [MASK] or [gMASK] in your input")
-
-        # only last token for input_ids if past is not None
-        if past is not None or past_key_values is not None:
-            context_length = seq.index(150004)
-            last_token = input_id[-1].unsqueeze(-1).unsqueeze(0)  # 2 dim
-            proc_input_id = last_token
-            if self.position_encoding_2d:
-                position_ids = torch.tensor([[[mask_position], [len(seq) - context_length]]], dtype=torch.long,
-                                            device=input_id.device)
-            else:
-                position_ids = torch.tensor([[mask_position]], dtype=torch.long, device=input_id.device)
-
-            attention_mask = torch.zeros(1, 1, 1, 1, device=input_id.device)
-        else:
-            proc_input_id = input_id.unsqueeze(0)
-            attention_mask, position_ids = self.get_masks_and_position_ids(
-                seq=seq,
-                mask_position=mask_position,
-                context_length=len(seq),
-                device=input_id.device,
-                gmask=use_gmask
-            )
-
-        return (proc_input_id.to(torch.int32), position_ids.to(torch.int32),
-                attention_mask.to(torch.bool))
-
-    def prepare_inputs_for_generation(
-            self,
-            input_ids: torch.LongTensor,
-            past: Optional[torch.Tensor] = None,
-            past_key_values: Optional[torch.Tensor] = None,
-            attention_mask: Optional[torch.Tensor] = None,
-            use_cache: bool = None,
-            **kwargs
-    ) -> dict:
-
-        MASK, gMASK = 150000, 150001
-        mask_token = MASK if MASK in input_ids else gMASK
-        use_gmask = False if MASK in input_ids else gMASK
-
-        batch_input_ids, batch_position_ids, batch_attention_mask = [], [], []
-        for input_id in input_ids:
-            proc_input_id, position_id, attention_mask = self.prepare_one_sample(
-                input_id, mask_token, past, past_key_values, use_gmask)
-            batch_input_ids.append(proc_input_id)
-            batch_position_ids.append(position_id)
-            batch_attention_mask.append(attention_mask)
-
-        batch_input_ids = torch.vstack(batch_input_ids)
-        batch_position_ids = torch.vstack(batch_position_ids)
-        batch_attention_mask = torch.vstack(batch_attention_mask)
-
-        if past is None:
-            past = past_key_values
-
-        if past is not None or past_key_values is not None:
-            self.kernel.set_context_mode(False)
-        else:
-            self.kernel.set_context_mode(self.config.use_cache)
-
-        return {
-            "input_ids": batch_input_ids,
-            "past_key_values": past_key_values,
-            "position_ids": batch_position_ids,
-            "attention_mask": batch_attention_mask
-        }

models/config.json

@@ -1,25 +0,0 @@
-{
-  "_name_or_path": "THUDM/chatglm-6b",
-  "architectures": [
-    "ChatGLMModel"
-  ],
-  "auto_map": {
-    "AutoConfig": "configuration_chatglm.ChatGLMConfig",
-    "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
-    "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration"
-  },
-  "bos_token_id": 150004,
-  "eos_token_id": 150005,
-  "hidden_size": 4096,
-  "inner_hidden_size": 16384,
-  "layernorm_epsilon": 1e-05,
-  "max_sequence_length": 2048,
-  "model_type": "chatglm",
-  "num_attention_heads": 32,
-  "num_layers": 28,
-  "position_encoding_2d": true,
-  "torch_dtype": "float16",
-  "transformers_version": "4.23.1",
-  "use_cache": true,
-  "vocab_size": 150528
-}

models/configuration_chatglm.py

@@ -1,92 +0,0 @@
-""" ChatGLM model configuration """
-
-from transformers.configuration_utils import PretrainedConfig
-from transformers.utils import logging
-
-logger = logging.get_logger(__name__)
-
-
-class ChatGLMConfig(PretrainedConfig):
-    r"""
-    This is the configuration class to store the configuration of a [`~ChatGLMModel`].
-    It is used to instantiate an ChatGLM model according to the specified arguments, defining the model
-    architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
-    the ChatGLM-6B [THUDM/ChatGLM-6B](https://huggingface.co/THUDM/chatglm-6b) architecture.
-
-    Configuration objects inherit from  [`PretrainedConfig`] and can be used
-    to control the model outputs. Read the documentation from  [`PretrainedConfig`]
-    for more information.
-
-
-    Args:
-        vocab_size (`int`, *optional*, defaults to 150528):
-            Vocabulary size of the ChatGLM-6B model. Defines the number of different tokens that can be represented by the
-            `inputs_ids` passed when calling [`~ChatGLMModel`] or
-            [`~TFChatGLMModel`].
-        hidden_size (`int`, *optional*, defaults to 4096):
-            Dimension of the encoder layers and the pooler layer.
-        num_hidden_layers (`int`, *optional*, defaults to 28):
-            Number of hidden layers in the Transformer encoder.
-        num_attention_heads (`int`, *optional*, defaults to 32):
-            Number of attention heads for each attention layer in the Transformer encoder.
-        inner_hidden_size (`int`, *optional*, defaults to 16384):
-            Dimension of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
-        max_sequence_length (`int`, *optional*, defaults to 512):
-            The maximum sequence length that this model might ever be used with.
-            Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
-        layernorm_epsilon (`float`, *optional*, defaults to 1e-5):
-            The epsilon used by the layer normalization layers.
-        use_cache (`bool`, *optional*, defaults to `True`):
-            Whether the model should return the last key/values attentions (not used by all models).
-        Example:
-
-    ```python
-    >>> from configuration_chatglm import ChatGLMConfig
-    >>> from modeling_chatglm import ChatGLMModel
-
-    >>> # Initializing a ChatGLM-6B THUDM/ChatGLM-6B style configuration
-    >>> configuration = ChatGLMConfig()
-
-    >>> # Initializing a model from the THUDM/ChatGLM-6B style configuration
-    >>> model = ChatGLMModel(configuration)
-
-    >>> # Accessing the model configuration
-    >>> configuration = model.config
-    ```
-"""
-    model_type = "chatglm"
-
-    def __init__(
-            self,
-            vocab_size=150528,
-            hidden_size=4096,
-            num_layers=28,
-            num_attention_heads=32,
-            layernorm_epsilon=1e-5,
-            use_cache=False,
-            bos_token_id=150004,
-            eos_token_id=150005,
-            pad_token_id=0,
-            max_sequence_length=2048,
-            inner_hidden_size=16384,
-            position_encoding_2d=True,
-            **kwargs
-    ):
-        self.num_layers = num_layers
-        self.vocab_size = vocab_size
-        self.hidden_size = hidden_size
-        self.num_attention_heads = num_attention_heads
-        self.max_sequence_length = max_sequence_length
-        self.layernorm_epsilon = layernorm_epsilon
-        self.inner_hidden_size = inner_hidden_size
-        self.use_cache = use_cache
-        self.bos_token_id = bos_token_id
-        self.eos_token_id = eos_token_id
-        self.pad_token_id = pad_token_id
-        self.position_encoding_2d = position_encoding_2d
-        super().__init__(
-            pad_token_id=pad_token_id,
-            bos_token_id=bos_token_id,
-            eos_token_id=eos_token_id,
-            **kwargs
-        )

models/ice_text.model

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:99871e0c85db81ad7af1028854fd091cd5778c8414ae9d94bbbc10d02c831c21
-size 2699926

models/tokenization_chatglm.py

@@ -1,346 +0,0 @@
-"""Tokenization classes for ChatGLM."""
-import sys
-import unicodedata
-from typing import List, Optional, Union
-from functools import lru_cache
-import os
-import collections
-import re
-
-from transformers.tokenization_utils import PreTrainedTokenizer
-from icetk.text_tokenizer import TextTokenizer
-from icetk.utils import auto_create
-import icetk.sentencepiece_model_pb2 as sp_model
-from transformers.utils import logging
-
-logger = logging.get_logger(__name__)
-
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "THUDM/chatglm-6b": 2048,
-}
-
-
-class SPTokenizer:
-    def __init__(
-        self,
-        vocab_file,
-        max_blank_length=80,
-        byte_fallback=True,
-    ):
-        assert vocab_file is not None
-        self.vocab_file = vocab_file
-        self.special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "<unused_0>", "<sop>", "<eop>", "<ENC>", "<dBLOCK>"]
-        self.max_blank_length = max_blank_length
-        self.byte_fallback = byte_fallback
-        self.text_tokenizer = self._build_text_tokenizer(encode_special_tokens=False)
-        self.special_text_tokenizer = self._build_text_tokenizer(encode_special_tokens=True)
-
-    @staticmethod
-    def _configure_tokenizer(
-        text_tokenizer: TextTokenizer,
-        special_tokens: List[str],
-        max_blank_length: int,
-        byte_fallback: bool,
-        encode_special_tokens=False,
-    ):
-        # special token
-        special_token_type = 4 if encode_special_tokens else 3  # 3 - CONTROL, 4 - USER_DEFINE
-        for token in special_tokens:
-            text_tokenizer.proto.pieces.append(
-                sp_model.ModelProto.SentencePiece(piece=token, score=0.0, type=special_token_type)
-            )
-        # whitespaces
-        for token in [SPTokenizer.get_tab_token()] + [
-            SPTokenizer.get_blank_token(i) for i in range(2, max_blank_length + 1)
-        ]:
-            text_tokenizer.proto.pieces.append(sp_model.ModelProto.SentencePiece(piece=token, score=0.0, type=4))
-        # byte fallback
-        if byte_fallback:
-            text_tokenizer.proto.trainer_spec.byte_fallback = True
-            for i in range(256):
-                text_tokenizer.proto.pieces.append(
-                    sp_model.ModelProto.SentencePiece(piece="<0x{:02X}>".format(i), score=0.0, type=6)
-                )
-        text_tokenizer.refresh()
-
-    def _build_text_tokenizer(self, encode_special_tokens=False):
-        tokenizer = TextTokenizer(self.vocab_file)
-        self._configure_tokenizer(
-            tokenizer, self.special_tokens, self.max_blank_length, self.byte_fallback, encode_special_tokens
-        )
-        return tokenizer
-
-    def _get_text_tokenizer(self, encode_special_tokens=False):
-        if encode_special_tokens:
-            return self.special_text_tokenizer
-        else:
-            return self.text_tokenizer
-
-    @staticmethod
-    def get_blank_token(length: int):
-        assert length >= 2
-        return f"<|blank_{length}|>"
-
-    @staticmethod
-    def get_tab_token():
-        return f"<|tab|>"
-
-    @property
-    def num_image_tokens(self):
-        return 20000
-
-    @property
-    def num_text_tokens(self):
-        return self.text_tokenizer.num_tokens
-
-    @property
-    def num_tokens(self):
-        return self.num_image_tokens + self.num_text_tokens
-
-    @staticmethod
-    def _encode_whitespaces(text: str, max_len: int = 80):
-        text = text.replace("\t", SPTokenizer.get_tab_token())
-        for i in range(max_len, 1, -1):
-            text = text.replace(" " * i, SPTokenizer.get_blank_token(i))
-        return text
-
-    def _preprocess(self, text: str, linebreak=True, whitespaces=True):
-        if linebreak:
-            text = text.replace("\n", "<n>")
-        if whitespaces:
-            text = self._encode_whitespaces(text, max_len=self.max_blank_length)
-        return text
-
-    def encode(
-        self, text: str, linebreak=True, whitespaces=True, special_tokens=False, add_dummy_prefix=True
-    ) -> List[int]:
-        """
-        @param text: Text to encode.
-        @param linebreak: Whether to encode newline (\n) in text.
-        @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding.
-        @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text.
-        @param add_dummy_prefix: Whether to add dummy blank space in the beginning.
-        """
-        text = self._preprocess(text, linebreak, whitespaces)
-        if not add_dummy_prefix:
-            text = "<n>" + text
-        tmp = self._get_text_tokenizer(encode_special_tokens=special_tokens).encode(text)
-        tokens = [x + self.num_image_tokens for x in tmp]
-        return tokens if add_dummy_prefix else tokens[2:]
-
-    def decode(self, text_ids: List[int], special_tokens=False) -> str:
-        ids = [int(_id) - self.num_image_tokens for _id in text_ids]
-        ids = [_id for _id in ids if _id >= 0]
-        text = self._get_text_tokenizer(encode_special_tokens=special_tokens).decode(ids)
-        text = text.replace("<n>", "\n")
-        text = text.replace(SPTokenizer.get_tab_token(), "\t")
-        for i in range(2, self.max_blank_length + 1):
-            text = text.replace(self.get_blank_token(i), " " * i)
-        return text
-
-    def tokenize(
-        self, text: str, linebreak=True, whitespaces=True, special_tokens=False, add_dummy_prefix=True
-    ) -> List[str]:
-        """
-        @param text: Text to encode.
-        @param linebreak: Whether to encode newline (\n) in text.
-        @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding.
-        @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text.
-        @param add_dummy_prefix: Whether to add dummy blank space in the beginning.
-        """
-        text = self._preprocess(text, linebreak, whitespaces)
-        if not add_dummy_prefix:
-            text = "<n>" + text
-        tokens = self._get_text_tokenizer(encode_special_tokens=special_tokens).tokenize(text)
-        return tokens if add_dummy_prefix else tokens[2:]
-
-    def __getitem__(self, x: Union[int, str]):
-        if isinstance(x, int):
-            if x < self.num_image_tokens:
-                return "<image_{}>".format(x)
-            else:
-                return self.text_tokenizer.convert_id_to_token(x - self.num_image_tokens)
-        elif isinstance(x, str):
-            if x.startswith("<image_") and x.endswith(">") and x[7:-1].isdigit():
-                return int(x[7:-1])
-            else:
-                return self.text_tokenizer.convert_token_to_id(x) + self.num_image_tokens
-        else:
-            raise ValueError("The key should be str or int.")
-
-
-class ChatGLMTokenizer(PreTrainedTokenizer):
-    """
-    Construct a ChatGLM tokenizer. Based on byte-level Byte-Pair-Encoding.
-
-    Args:
-        vocab_file (`str`):
-            Path to the vocabulary file.
-    """
-
-    vocab_files_names = {"vocab_file": "ice_text.model"}
-    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
-    model_input_names = ["input_ids"]
-
-    def __init__(
-            self,
-            vocab_file,
-            do_lower_case=False,
-            remove_space=False,
-            bos_token='sop',
-            eos_token='eos',
-            eop_token='eop',
-            mask_token='[MASK]',
-            gmask_token='[gMASK]',
-            padding_side="left",
-            **kwargs
-    ) -> None:
-        super().__init__(
-            do_lower_case=do_lower_case,
-            remove_space=remove_space,
-            padding_side=padding_side,
-            **kwargs
-        )
-
-        self.do_lower_case = do_lower_case
-        self.remove_space = remove_space
-        self.vocab_file = vocab_file
-
-        self.bos_token = bos_token
-        self.eos_token = eos_token
-        self.eop_token = eop_token
-        self.mask_token = mask_token
-        self.gMASK_token = gmask_token
-
-        self.sp_tokenizer = SPTokenizer(vocab_file)
-
-        """ Initialisation """
-
-    @property
-    def eop_token_id(self) -> Optional[int]:
-        """
-        `Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been
-        set.
-        """
-        if self.eop_token is None:
-            return None
-        return self.convert_tokens_to_ids(self.eop_token)
-
-    @property
-    def vocab_size(self):
-        """ Returns vocab size """
-        return self.sp_tokenizer.num_tokens
-
-    def get_vocab(self):
-        """ Returns vocab as a dict """
-        vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
-        vocab.update(self.added_tokens_encoder)
-        return vocab
-
-    def preprocess_text(self, inputs):
-        if self.remove_space:
-            outputs = " ".join(inputs.strip().split())
-        else:
-            outputs = inputs
-
-        if self.do_lower_case:
-            outputs = outputs.lower()
-
-        return outputs
-
-    def _tokenize(self, text, **kwargs):
-        """ Returns a tokenized string. """
-        text = self.preprocess_text(text)
-
-        seq = self.sp_tokenizer.tokenize(text)
-
-        return seq
-
-    def decode(
-            self,
-            token_ids: Union[List[int], List[List[int]]],
-            skip_special_tokens: bool = False,
-            clean_up_tokenization_spaces: bool = True,
-            spaces_between_special_tokens: bool = True,
-            **kwargs
-    ) -> str:
-        if isinstance(token_ids[0], list):
-            tokens = []
-            for single_token_ids in token_ids:
-                if self.pad_token_id in single_token_ids:  # remove pad
-                    single_token_ids = list(filter((self.pad_token_id).__ne__, single_token_ids))
-                tokens.append(self.sp_tokenizer.decode(single_token_ids))
-            return (tokens)
-        else:
-            if self.pad_token_id in token_ids:  # remove pad
-                token_ids = list(filter((self.pad_token_id).__ne__, token_ids))
-            return self.sp_tokenizer.decode(token_ids)
-
-    def _convert_token_to_id(self, token):
-        """ Converts a token (str) in an id using the vocab. """
-        return self.sp_tokenizer[token]
-
-    def _convert_id_to_token(self, index):
-        """Converts an index (integer) in a token (str) using the vocab."""
-        return self.sp_tokenizer[index]
-
-    def save_vocabulary(self, save_directory, filename_prefix=None):
-        """
-        Save the vocabulary and special tokens file to a directory.
-
-        Args:
-            save_directory (`str`):
-                The directory in which to save the vocabulary.
-            filename_prefix (`str`, *optional*):
-                An optional prefix to add to the named of the saved files.
-
-        Returns:
-            `Tuple(str)`: Paths to the files saved.
-        """
-        if os.path.isdir(save_directory):
-            vocab_file = os.path.join(
-                save_directory, VOCAB_FILES_NAMES["vocab_file"]
-            )
-        else:
-            vocab_file = save_directory
-
-        with open(self.vocab_file, 'rb') as fin:
-            proto_str = fin.read()
-
-        with open(vocab_file, "wb") as writer:
-            writer.write(proto_str)
-
-        return (vocab_file,)
-
-    def build_inputs_with_special_tokens(
-            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
-    ) -> List[int]:
-        """
-        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
-        adding special tokens. A BERT sequence has the following format:
-
-        - single sequence: `[CLS] X [SEP]`
-        - pair of sequences: `[CLS] A [SEP] B [SEP]`
-
-        Args:
-            token_ids_0 (`List[int]`):
-                List of IDs to which the special tokens will be added.
-            token_ids_1 (`List[int]`, *optional*):
-                Optional second list of IDs for sequence pairs.
-
-        Returns:
-            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
-        """
-        if token_ids_1 is not None:
-            token_ids_0 += token_ids_1
-        mask_ids = self.sp_tokenizer[self.mask_token]
-        gmask_ids = self.sp_tokenizer[self.gMASK_token]
-        if mask_ids not in token_ids_0 and gmask_ids not in token_ids_0:
-            token_ids_0 += [gmask_ids]
-
-        if token_ids_0[-1] != mask_ids and token_ids_0[-1] != gmask_ids:
-            token_ids_0 += [self.sp_tokenizer[self.eos_token]]
-
-        token_ids_0 += [self.sp_tokenizer[self.bos_token]]
-
-        return token_ids_0

models/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "name_or_path": "THUDM/chatglm-6b",
-  "bos_token": "<sop>",
-  "eop_token": "<eop>",
-  "eos_token": "</s>",
-  "gmask_token": "[gMASK]",
-  "mask_token": "[MASK]",
-  "pad_token": "<pad>",
-  "unk_token": "<unk>",
-  "remove_space": false,
-  "do_lower_case": false,
-  "tokenizer_class": "ChatGLMTokenizer",
-  "auto_map": {
-    "AutoTokenizer": [
-      "tokenization_chatglm.ChatGLMTokenizer",
-      null
-      ]
-  }
-}

requirements.txt

@@ -1,4 +0,0 @@
-icetk
-torch
-transformers
-