HerBERT tokenizer
HerBERT tokenizer is a character level byte-pair encoding with
vocabulary size of 50k tokens. The tokenizer was trained on Wolne Lektury and a publicly available subset of
National Corpus of Polish with fastBPE library.
Tokenizer utilize XLMTokenizer
implementation from transformers.
Tokenizer usage
Herbert tokenizer should be used together with HerBERT model:
from transformers import XLMTokenizer, RobertaModel
tokenizer = XLMTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")
encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors='pt')
outputs = model(encoded_input)
License
CC BY-SA 4.0
Citation
If you use this tokenizer, please cite the following paper:
@inproceedings{rybak-etal-2020-klej,
title = "{KLEJ}: Comprehensive Benchmark for {P}olish Language Understanding",
author = "Rybak, Piotr and
Mroczkowski, Robert and
Tracz, Janusz and
Gawlik, Ireneusz",
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
month = jul,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.acl-main.111",
doi = "10.18653/v1/2020.acl-main.111",
pages = "1191--1201",
}
Authors
Tokenizer was created by Allegro Machine Learning Research team.
You can contact us at: klejbenchmark@allegro.pl
- Downloads last month
- 124