YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
from transformers import RobertaTokenizerFast tokenizer = RobertaTokenizerFast.from_pretrained('Andrija/RobertaFastBPE', bos_token="<s>", eos_token="</s>")
encoded = tokenizer('Stručnjaci te bolnice, predvođeni dr Alisom Lim')
{'input_ids': [0, 47541, 34632, 603, 24817, 16, 27540, 6768, 2350, 2803, 3991, 2733, 81, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
tokenizer.decode(encoded['input_ids'])
<s>Stručnjaci te bolnice, predvođeni dr Alisom Lim</s>
- Downloads last month
- 0
Unable to determine this model's library. Check the
docs
.