Set dropout in config.json to be 0 ?
#7
by
hails
- opened
As far as I'm aware, and according to the GPT-NeoX-20b arXiv paper, the model wasn't trained with dropout. Is there a reason there is dropout in this config on both attention and hiddens? (is this because dropout is recommended for finetuning or something?)
stellaathena
changed pull request status to
merged