Is `rms_norm_eps` 1e-5 or 1e-6
#9
by
danielhanchen
- opened
Oh hey Qwen team!
For Qwen 2.5 32B, "rms_norm_eps" is 1e-05, but Qwen 2.5 32B Instruct is 1e-06.
The new QwQ model has rms_norm_eps 1e-05. Are there supposed to be differences?