Elaben's picture
add tokenizer
31c0090
{"j": 0, "h": 1, "t": 2, "k": 3, "p": 4, "'": 5, "r": 6, "v": 7, "f": 8, "b": 9, "x": 10, "g": 11, "c": 12, "y": 13, "d": 14, "a": 15, "u": 16, "w": 17, "q": 18, "s": 19, "m": 20, "e": 22, "n": 23, "i": 24, "z": 25, "o": 26, "l": 27, "|": 21, "[UNK]": 28, "[PAD]": 29}