Are the new update of weights having error?

#17
by wyxwangmed - opened

I test the newest model weight. But there is something wrong. The loss downed very quick. I want to ask that is there someone has the same issue as me.

@wyxwangmed Would you mind elaborating a bit more on how you tested this?

@wyxwangmed Would you mind elaborating a bit more on how you tested this?

@MeisterDeLaV https://github.com/InternLM/xtuner/blob/main/xtuner/model/modules/dispatch/llama.py I used xtuner to fine-tuned Yi-34b-200k, and xtuner will use this code to use flash_attn, I comment this section of code to close the flash_attn, the loss is right. So I think is not the reason of weights. But I don't know why I need to comment this section of code.

wyxwangmed changed discussion status to closed

Sign up or log in to comment