wangchanberta-base-att-spm-uncased / sentencepiece.bpe.vocab

Commit History

revert to ▁ as previous fix did not work
7d568ae

charipol commited on

change ▁ to ▁▁ token to avoid additional token problem with tokenized
51ee40b

charipol commited on

Change space token from <th_roberta_space_token> to <_>
04ff256

lalital commited on

Add vocab file and tokenizer_config.json
247d65f

lalital commited on