countrysideid's picture
add tokenizer
07d0414
raw
history blame
No virus
268 Bytes
{"f": 0, "j": 1, "d": 2, "x": 3, "b": 4, "k": 5, "r": 6, "i": 7, "l": 9, "y": 10, "c": 11, "u": 12, "e": 13, "'": 14, "w": 15, "z": 16, "a": 17, "o": 18, "s": 19, "m": 20, "v": 21, "p": 22, "n": 23, "g": 24, "t": 25, "q": 26, "h": 27, "|": 8, "[UNK]": 28, "[PAD]": 29}