WizardLMTeam

/

WizardCoder-15B-V1.0

like 736

Hello, I'm going to fine-tune on WizardCoder-15B-V1.0 , do I need this code:

if "starcoder" in model_args.model_name_or_path:
        tokenizer.add_special_tokens(
            {
                "eos_token": DEFAULT_EOS_TOKEN,
                "bos_token": DEFAULT_BOS_TOKEN,
                "unk_token": DEFAULT_UNK_TOKEN,
                "pad_token": DEFAULT_PAD_TOKEN,
            }
        )