Hugging Face's logo

THUDM
/
chatglm3-6b

Transformers
PyTorch
Safetensors
Chinese
English
chatglm
glm
thudm
custom_code
Inference Endpoints
Model card Files Community
59

fix rmsnorm init weight bug.

#59
by Shan1990 - opened
base: refs/heads/main
←
from: refs/pr/59
Discussion Files changed
+1
-1
Shan1990

Using torch.ones to init rmsnorm weight. And torch.empty gets random weight tensor, which maybe out of float value limits.

fix rmsnorm init weight bug.9d3d7be5
Shan1990

@chielo pls review this pr, thx.

chielo

@Shan1990 Well. I don't belong to the organization, nor do I have the permission to merge.
But this PR looks good and friendly to initialize chatglm3 for training, etc.
It will be better to have this fix.

DuanKW

@zRzRzRzRzRzRzR pls review this pr, thx.

zRzRzRzRzRzRzR
 Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University org

check now

zRzRzRzRzRzRzR changed pull request status to merged

Sign up or log in to comment