AGIEval and others
#19
by
migtissera
- opened
Hey @teknium ,
When you run AGIEval using Eluther, I'm guessing you run the Nous version as shown here? https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/agieval/README.md
What's the number of shots? Is it one-shot, 5-shot or more?
Thanks!
Also, do we use acc, or acc_norm, when calculating the average?
I use this branch that added agieval 7 or 8 months ago
https://github.com/dmahan93/lm-evaluation-harness/tree/add-agieval
0 shot
acc norm