Repository: PaddlePaddle/PaddleNLP
Pull request: Fix albert tokenizer
<!-- Demo: https://github.com/PaddlePaddle/PaddleNLP/pull/26 -->
### PR types
<!-- One of [ New features | Bug fixes | Function optimization | Performance optimization | Breaking changes | Others ] -->
Bug fixes
### PR changes
<!-- One of [ Models | APIs | Docs | Others ] -->
Models
### Description
<!-- Describe what this PR does -->
Fix albert tokenizer

Problem statement:
AlbertTokenizer Pad的id是否正确？
欢迎您反馈PaddleNLP使用问题，非常感谢您对PaddleNLP的贡献！
在留下您的问题时，辛苦您同步提供如下信息：
- 版本、环境信息
1）PaddleNLP和PaddlePaddle版本：请提供您的PaddleNLP和PaddlePaddle版本号，例如PaddleNLP 2.0.4，PaddlePaddle2.1.1
PaddleNLP 2.1.1 PaddlePaddle-GPU 2.2.0
2）系统环境：请您描述系统类型，例如Linux/Windows/MacOS/，python版本
Linux Python 3.8.11
- 复现信息：如为报错，请给出复现环境、复现步骤

在使用albert_tokenizer时：设置如下代码
```python
    encode_inputs = tokenizer(
        text=raw_inputs,
        max_seq_len=128,
        pad_to_max_seq_len=True,
    )
```

得到的部分输出如下：
```
'input_ids': [101, 1184, 697, 1921, 2769, 2970, 1168, 8287, 9632, 8157, 5314, 2769, 2802, 4510, 6413, 1285, 5277, 4635, 7032, 1305, 749, 8024, 6656, 2769, 6432, 5543, 6444, 7583, 8024, 3221, 4696, 4638, 1408, 8024, 679, 3221, 7745, 2094, 1416, 511, 102, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100]
```

可以看到Pad的值为100，而albert doc中有如下描述：
```
"pad_token_id": 0,
```

请问Pad的id应该是0还是100？

Hints from issue discussion:
我在调整batch_size的大小时，发现不同的batch_size得到了截然不同的结果
> 我在调整batch_size的大小时，发现不同的batch_size得到了截然不同的结果

可以给出具体的代码吗？
我刚刚试了一下pad_token_id=0
测试代码
```python
import paddle
from paddlenlp.transformers import AlbertTokenizer
from paddlenlp.transformers import AlbertModel

model = AlbertModel.from_pretrained('albert-chinese-small')
tokenizer = AlbertTokenizer.from_pretrained('albert-chinese-small')

with open('test_file', 'r') as fin:
    raw_inputs = [line.strip() for line in fin.readlines()]

encode_inputs = tokenizer(
    text=raw_inputs,
    max_seq_len=128,
    pad_to_max_seq_len=True,
)

print(encode_inputs)
```

测试样例
```
今天天气真好
paddle好用的不得了
```

输出
```
[{'input_ids': [101, 791, 1921, 1921, 3698, 4696, 1962, 102, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}, {'input_ids': [101, 10257, 8168, 8268, 1962, 4500, 4638, 679, 2533, 749, 102, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}]
```

查阅词表发现，[PAD]对应的id应该是0才对，这里的100是否是有错误的？
> 查阅词表发现，[PAD]对应的id应该是0才对，这里的100是否是有错误的？

这里确实有问题，感谢指出～
好像只是PAD识别成了UNK，我以为词表全对应混了，使用albert-chinese-tiny训练相似度模型的时候，训练loss下降非常缓慢，最后验证集表现非常差，看起来就像是没有初始化参数，用自己的数据重新训了一个模型一样。楼主有遇到这样的问题么

Commit message: Fix albert tokenizer

Patch to review:
diff --git a/paddlenlp/transformers/albert/tokenizer.py b/paddlenlp/transformers/albert/tokenizer.py
index 7a854b7c0821..5d63a1e6f166 100644
--- a/paddlenlp/transformers/albert/tokenizer.py
+++ b/paddlenlp/transformers/albert/tokenizer.py
@@ -134,46 +134,148 @@ class AlbertTokenizer(PretrainedTokenizer):
 
     pretrained_init_configuration = {
         "albert-base-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-large-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xlarge-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xxlarge-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-base-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-large-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xlarge-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xxlarge-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-tiny": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-small": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-base": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-large": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-xlarge": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-xxlarge": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
     }
 
@@ -183,13 +285,6 @@ def __init__(self,
                  do_lower_case=True,
                  remove_space=True,
                  keep_accents=False,
-                 bos_token="[CLS]",
-                 eos_token="[SEP]",
-                 unk_token="<unk>",
-                 sep_token="[SEP]",
-                 pad_token="<pad>",
-                 cls_token="[CLS]",
-                 mask_token="[MASK]",
                  **kwargs):
         self.do_lower_case = do_lower_case
         self.remove_space = remove_space
@@ -199,12 +294,14 @@ def __init__(self,
 
         if vocab_file is not None:
             self.tokenizer = AlbertChineseTokenizer(
-                vocab_file=vocab_file,
-                do_lower_case=False, )
+                vocab_file=vocab_file, do_lower_case=do_lower_case, **kwargs)
         elif sentencepiece_model_file is not None:
             self.tokenizer = AlbertEnglishTokenizer(
                 sentencepiece_model_file=sentencepiece_model_file,
-                do_lower_case=True, )
+                do_lower_case=do_lower_case,
+                remove_space=remove_space,
+                keep_accents=keep_accents,
+                **kwargs)
         else:
             raise ValueError(
                 "You should only specify either one(not both) of 'vocal_file'"
@@ -443,28 +540,100 @@ class AlbertEnglishTokenizer(PretrainedTokenizer):
 
     pretrained_init_configuration = {
         "albert-base-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-large-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xlarge-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xxlarge-v1": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-base-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-large-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xlarge-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-xxlarge-v2": {
-            "do_lower_case": True
+            "do_lower_case": True,
+            "remove_space": True,
+            "keep_accents": False,
+            "bos_token": "[CLS]",
+            "eos_token": "[SEP]",
+            "unk_token": "<unk>",
+            "sep_token": "[SEP]",
+            "pad_token": "<pad>",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
     }
 
@@ -473,13 +642,6 @@ def __init__(self,
                  do_lower_case=True,
                  remove_space=True,
                  keep_accents=False,
-                 bos_token="[CLS]",
-                 eos_token="[SEP]",
-                 unk_token="<unk>",
-                 sep_token="[SEP]",
-                 pad_token="<pad>",
-                 cls_token="[CLS]",
-                 mask_token="[MASK]",
                  **kwargs):
 
         self.do_lower_case = do_lower_case
@@ -663,32 +825,55 @@ class AlbertChineseTokenizer(BertTokenizer):
     }
     pretrained_init_configuration = {
         "albert-chinese-tiny": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-small": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-base": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-large": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-xlarge": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-xxlarge": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
     }
 
-    def __init__(self,
-                 vocab_file,
-                 do_lower_case=False,
-                 unk_token="[UNK]",
-                 sep_token="[SEP]",
-                 pad_token="[PAD]",
-                 cls_token="[CLS]",
-                 mask_token="[MASK]"):
+    def __init__(self, vocab_file, do_lower_case=False, **kwargs):
         super(AlbertChineseTokenizer, self).__init__(
-            vocab_file, do_lower_case=do_lower_case)
+            vocab_file, do_lower_case=do_lower_case, **kwargs)


Code review comments:
diff_hunk: @@ -663,32 +825,55 @@ class AlbertChineseTokenizer(BertTokenizer):
     }
     pretrained_init_configuration = {
         "albert-chinese-tiny": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-small": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-base": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-large": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-xlarge": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
         "albert-chinese-xxlarge": {
-            "do_lower_case": False
+            "do_lower_case": False,
+            "unk_token": "[UNK]",
+            "sep_token": "[SEP]",
+            "pad_token": "[PAD]",
+            "cls_token": "[CLS]",
+            "mask_token": "[MASK]",
         },
     }
 
-    def __init__(self,
-                 vocab_file,
-                 do_lower_case=False,
-                 unk_token="[UNK]",
-                 sep_token="[SEP]",
-                 pad_token="[PAD]",
-                 cls_token="[CLS]",
-                 mask_token="[MASK]"):
+    def __init__(self, vocab_file, do_lower_case=False, **kwargs):
path: paddlenlp/transformers/albert/tokenizer.py
text: @user1:
还要确认下这里使用`**kwargs`是否合适及加载既有保存的config是否有问题，咱们的基类会将init的参数作为config进行保存

@author:
这里使用 **kwargs** 可以使得用 **from_pretrained** 方法的时候可以传入自定义的config参数；
经过自测，可以加载既有保存的config。