Santiagot1105's picture
add tokenizer
c91d774
raw
history blame contribute delete
No virus
480 Bytes
{"l": 0, "ã": 1, "¿": 2, "毵": 3, "m": 4, "j": 5, "z": 6, "g": 7, "ü": 8, "f": 9, "b": 10, "ə": 11, "ñ": 12, "á": 13, "o": 14, "°": 15, "–": 16, "d": 17, "k": 18, "¡": 19, "ū": 20, "é": 21, "u": 22, "ë": 23, "q": 24, "ú": 25, "í": 26, "i": 27, "w": 28, "t": 29, "v": 30, "h": 31, "=": 33, "ō": 34, "’": 35, "n": 36, "e": 37, "x": 38, "'": 39, "p": 40, "ö": 41, "ó": 42, "ş": 43, "s": 44, "r": 45, "a": 46, "c": 47, "y": 48, "|": 32, "[UNK]": 49, "[PAD]": 50}