zRzRzRzRzRzRzR JeremyHibiki commited on
Commit
f6e0743
1 Parent(s): aae8bd7

Update tokenization_chatglm.py (#72)

Browse files

- Update tokenization_chatglm.py (bc26b66c0862517e6f7c28337206ba5acf1fd206)


Co-authored-by: Jeremy JIANG <JeremyHibiki@users.noreply.huggingface.co>

Files changed (1) hide show
  1. tokenization_chatglm.py +1 -2
tokenization_chatglm.py CHANGED
@@ -3,10 +3,9 @@ import base64
3
  import os
4
  import json
5
  import tiktoken
6
- from torch import TensorType
7
  from typing import List, Optional, Union, Dict, Any
8
  from transformers import PreTrainedTokenizer
9
- from transformers.utils import logging, PaddingStrategy
10
  from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
11
 
12
 
 
3
  import os
4
  import json
5
  import tiktoken
 
6
  from typing import List, Optional, Union, Dict, Any
7
  from transformers import PreTrainedTokenizer
8
+ from transformers.utils import logging, PaddingStrategy, TensorType
9
  from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
10
 
11