AttributeError: 'Qwen2TokenizerFast' object has no attribute 'eod_id'
I'm trying to finetune Qwen-1.5-0.5B with axolotl and i'm getting the following tokenizer error:
AttributeError: 'Qwen2TokenizerFast' object has no attribute 'eod_id'
Hello, and thank you for bringing this issue to our attention. As a temporary solution, we recommend editing the config.json file manually and setting both the unk_token_id and pad_token_id values to 151643. This should bypass the current problem until an official fix is implemented in axolotl.
Please be aware that manual modifications to configuration files can sometimes lead to unintended consequences if not done correctly. Therefore, always keep a backup of your original file before making any changes. We anticipate that a permanent fix will be included in an upcoming axolotl release soon.