IDEA-CCNL

/

Ziya-LLaMA-13B-v1

like 273

We added 7000 chinese token to vocab.txt，which leads to mismatch error with the original apply_delta.py script from fastchat. Here is our modified script : https://github.com/IDEA-CCNL/Fengshenbang-LM/blob/main/fengshen/utils/apply_delta.py

The main difference is (line 140), the weight of embedding and lm_head doesn't need to convert.

        # param.data += delta.state_dict()[name]
        if "embed_tokens" in name or "lm_head.weight" in name or "self_attn.rotary_emb.inv_freq" in name:
            continue
        else:
            param.data += base.state_dict()[name]