README.md

metadata

license: apache-2.0

Evaluation

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
- medmcqa	Yaml	none	0	acc	0.5408	±	0.0077
		none	0	acc_norm	0.5408	±	0.0077
- medqa_4options	Yaml	none	0	acc	0.5711	±	0.0139
		none	0	acc_norm	0.5711	±	0.0139
- anatomy (mmlu)	0	none	0	acc	0.6815	±	0.0402
- clinical_knowledge (mmlu)	0	none	0	acc	0.7434	±	0.0269
- college_biology (mmlu)	0	none	0	acc	0.8056	±	0.0331
- college_medicine (mmlu)	0	none	0	acc	0.6647	±	0.0360
- medical_genetics (mmlu)	0	none	0	acc	0.7300	±	0.0446
- professional_medicine (mmlu)	0	none	0	acc	0.7353	±	0.0268
stem	N/A	none	0	acc_norm	0.5478	±	0.0067
		none	0	acc	0.5909	±	0.0058
- pubmedqa	1	none	0	acc	0.7620	±	0.0191

Groups	Version	Filter	n-shot	Metric	Value		Stderr
stem	N/A	none	0	acc_norm	0.5478	±	0.0067
		none	0	acc	0.5909	±	0.0058