RWKV

/

v5-Eagle-7B-HF

like 68

class RWKVWorldTokenizer(PreTrainedTokenizer):

....

            self.encoder[idx] = x

        self.decoder = {}
        for k, v in self.encoder.items():
            self.decoder[v] = int(k)

I think encoder should convert strings into ids, and decoder should convert ids back to strings.