DeBERTaRefCheckTokenizer / added_tokens.json
AIcademic's picture
Add DeBERTa tokenizer with SEC_* tokens
4989982 verified
{
"[MASK]": 128000,
"[SEC_ABSTRACT]": 128001,
"[SEC_ACKNOWLEDGMENTS]": 128005,
"[SEC_ACKNOW]": 128002,
"[SEC_BACKGROUND]": 128006,
"[SEC_CATEGORIES_AND_SUBJECT_DESCRIPTORS]": 128007,
"[SEC_CONCLUSIONS]": 128008,
"[SEC_CONCL]": 128003,
"[SEC_DISCUSSIONS]": 128009,
"[SEC_EVALUATION]": 128010,
"[SEC_GENERAL_TERMS]": 128011,
"[SEC_INTRODUCTION]": 128012,
"[SEC_INTRO]": 128004,
"[SEC_KEYWORDS]": 128013,
"[SEC_METHOD]": 128014,
"[SEC_REFERENCES]": 128015,
"[SEC_RELATED_WORK]": 128016
}