xgen-7b-8k-base / tokenization_xgen.py

Commit History

fix tokenizer save_pretrained method
927e4b7
verified

katuni4ka commited on

Update _decode method to accept integer element and convert it to sequence (#30)
582db72
verified

rooa ramkrithiks commited on

Update tokenization_xgen.py
a473726

tianxie-sf commited on

fix issue in get vocab (#18)
1a0f468

tianxie-sf commited on

_convert_id_to_token (#12)
3987e09

tianxie-sf commited on

add pad token and default eos token (#6)
e9a901f

rooa tianxie-sf commited on

.
d2cc612

rooa commited on

.
b9f4adf

rooa commited on

.
04d6513

rooa commited on