codellama

/

CodeLlama-7b-hf

like 297

What are these tokens in the tokenizer.

If I run this line

tokenizer.convert_ids_to_tokens(tokenizer.encode('int add(int a, int b) {\n    return a + b;\n}'))

It gives me

['<s>', '▁int', '▁add', '(', 'int', '▁a', ',', '▁int', '▁b', ')', '▁{', '<0x0A>', '▁▁▁', '▁return', '▁a', '▁+', '▁b', ';', '<0x0A>', '}']

Looks like <0x0A> is used as the newline. Initially, I thought these tokens are special tokens for hex-decimal values.

Is this supposed to be correct? What the other tokens mean?