--- datasets: - xzuyn/chatdoctor-200k-stripped - Technoculture/riddle_sense - axiong/pmc_llama_instructions - Open-Orca/SlimOrca-Dedup language: - en tags: - medical --- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/63486df1f8f01fcc4b23e97d/nMuS3Qnb5m0dENIixWv0q.png) [Technoculture/MT7Bi-alpha](https://huggingface.co/Technoculture/MT7Bi-alpha) adapter merged with its Base Model (Meditron 7B) # Evaluations ## Open LLM Leaderboard | Model | ARC |HellaSwag|TruthfulQA|Winogrande|GSM8K| |---------------------------------------------------|----:|--------:|---------:|---------:|----:| |[MT7Bi-sft (epoch 4)](https://huggingface.co/Technoculture/MT7Bi-sft)|54.1| 75.11| 43.08| 72.14|15.54| |[MT7Bi-sft (epoch 1)](https://huggingface.co/Technoculture/MT7Bi)|50.94| 73.24| 43.04| 72.06|22.52| ### Model Evaluation Benchmark | | | | | | | | | | | -------- | ------ |----- |----- |----- |----- |----- |----- |------ | |Category | MT7Bi | meditron-70b | llama-2-70b | med42-70b* | meditron-7b | llama-2-7b | PMC-llama-7b | |Health | | 81.8 | 69.1 | 83.6 | 27.3 | 16.4 | 3.6 | |Nutrition | | 77.9 | 68.8 | 62.5 | 31.1 | 12.5 | 6.3 | |Psychology| | 47.4 | 36.8 | 52.6 | 21.1 | 10.5 | 0.0 | |Science | | 77.8 | 44.4 | 33.3 | 33.3 | 11.1 | 0.0 | |Avg | | 71.2 | 54.8 | 58.0 | 28.3 | 12.6 | 2.5 | | | | | | | | | | | | | | | | | | | --- | ------ | ------ |----- |----- |----- |----- | |Dataset| MT7Bi | meditron-70b | llama-2-70b | med42-70b* | clinical-camel-70b* | |MMLU-Medical | 46.9 | 77.6 | 77.9 | 74.5 | 65.7 | |PubMedQA | 65.2 | 81.6 | 80.0 | 61.2 | 67.0 | |MedMCQA | 42.7 | 66.0 | 62.6 | 59.2 | 46.7 | |MedQA | | 64.4 | 61.5 | 59.1 | 50.8 | |MedQA-4-Option| 44.3 | 70.2 | 63.8 | 63.9 | 56.8 | |Avg | | 72.0 | 69.2 | 63.6 | 57.4 | | | | | | | | | | | | | | | | | | --- | ------ |----- |----- |----- |----- |------ | |Dataset | meditron-7b | llama-2-7b | pmc-llama-7b | Zephyr-7B-beta* | Mistral-7B-instruct* | MT7Bi | |MMLU-Medical | 54.2 | 53.7 | 56.4 | 63.3 | 60.0 | 46.9 | |PubMedQA | 74.4 | 61.8 | 59.2 | 46.0 | 17.8 | 65.2 | |MedMCQA | 59.2 | 54.4 | 57.6 | 43.0 | 40.2 | 42.7 | |MedQA | 47.9 | 44.0 | 42.4 | 42.8 | 32.4 | | |MedQA-4-Option| 52.0 | 49.6 | 49.2 | 48.5 | 41.1 | 44.3 | |Avg | 57.5 | 52.7 | 53.0 | 48.7 | 38.3 | | | | | | | | | | | Model Name | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K | | ------------------ | -------- | --------- | ---- | ---------- | ---------- | -------- | | Orca-2-7b | **78.4** | 76.1 | 53.7 | **52.4** | **74.2** | **47.2** | | LLAMA-2-7b | 43.2 | **77.1** | 44.4 | 38.7 | 69.5 | 16 | | MT7Bi-sft | 54.1 | 75.11 | - | 43.08 | 72.14 | 15.54 | ### ARC: 54.1% | Task |Version| Metric | Value | |Stderr| |-------------|------:|--------------------|-------------|---|------| |arc_challenge| 1|acc,none | 0.51| | | | | |acc_stderr,none | 0.01| | | | | |acc_norm,none | 0.54| | | | | |acc_norm_stderr,none| 0.01| | | | | |alias |arc_challenge| | | ### HellaSwag: 75.11% | Task |Version| Metric | Value | |Stderr| |---------|------:|--------------------|---------|---|------| |hellaswag| 1|acc,none | 0.57| | | | | |acc_stderr,none | 0| | | | | |acc_norm,none | 0.75| | | | | |acc_norm_stderr,none| 0| | | | | |alias |hellaswag| | | ### TruthfulQA: 43.08% | Task |Version| Metric | Value | |Stderr| |--------------|-------|-----------------------|-----------------|---|------| |truthfulqa |N/A |bleu_max,none | 18.31| | | | | |bleu_max_stderr,none | 0.46| | | | | |bleu_acc,none | 0.39| | | | | |bleu_acc_stderr,none | 0| | | | | |bleu_diff,none | -1.63| | | | | |bleu_diff_stderr,none | 0.39| | | | | |rouge1_max,none | 41.99| | | | | |rouge1_max_stderr,none | 0.71| | | | | |rouge1_acc,none | 0.39| | | | | |rouge1_acc_stderr,none | 0| | | | | |rouge1_diff,none | -2.88| | | | | |rouge1_diff_stderr,none| 0.66| | | | | |rouge2_max,none | 27.42| | | | | |rouge2_max_stderr,none | 0.80| | | | | |rouge2_acc,none | 0.32| | | | | |rouge2_acc_stderr,none | 0| | | | | |rouge2_diff,none | -3.11| | | | | |rouge2_diff_stderr,none| 0.78| | | | | |rougeL_max,none | 38.81| | | | | |rougeL_max_stderr,none | 0.71| | | | | |rougeL_acc,none | 0.38| | | | | |rougeL_acc_stderr,none | 0| | | | | |rougeL_diff,none | -3.01| | | | | |rougeL_diff_stderr,none| 0.66| | | | | |acc,none | 0.33| | | | | |acc_stderr,none | 0.05| | | | | |alias |truthfulqa | | | |truthfulqa_gen| 3|bleu_max,none | 18.31| | | | | |bleu_max_stderr,none | 0.68| | | | | |bleu_acc,none | 0.39| | | | | |bleu_acc_stderr,none | 0.02| | | | | |bleu_diff,none | -1.63| | | | | |bleu_diff_stderr,none | 0.62| | | | | |rouge1_max,none | 41.99| | | | | |rouge1_max_stderr,none | 0.84| | | | | |rouge1_acc,none | 0.39| | | | | |rouge1_acc_stderr,none | 0.02| | | | | |rouge1_diff,none | -2.88| | | | | |rouge1_diff_stderr,none| 0.81| | | | | |rouge2_max,none | 27.42| | | | | |rouge2_max_stderr,none | 0.89| | | | | |rouge2_acc,none | 0.32| | | | | |rouge2_acc_stderr,none | 0.02| | | | | |rouge2_diff,none | -3.11| | | | | |rouge2_diff_stderr,none| 0.88| | | | | |rougeL_max,none | 38.81| | | | | |rougeL_max_stderr,none | 0.84| | | | | |rougeL_acc,none | 0.38| | | | | |rougeL_acc_stderr,none | 0.02| | | | | |rougeL_diff,none | -3.01| | | | | |rougeL_diff_stderr,none| 0.82| | | | | |alias | - truthfulqa_gen| | | |truthfulqa_mc1| 2|acc,none | 0.28| | | | | |acc_stderr,none | 0.02| | | | | |alias | - truthfulqa_mc1| | | |truthfulqa_mc2| 2|acc,none | 0.43| | | | | |acc_stderr,none | 0.01| | | | | |alias | - truthfulqa_mc2| | | ### Winogrande: 72.14% | Task |Version| Metric | Value | |Stderr| |----------|------:|---------------|----------|---|------| |winogrande| 1|acc,none | 0.72| | | | | |acc_stderr,none| 0.01| | | | | |alias |winogrande| | | ### GSM8K: 15.54% |Task |Version| Metric |Value| |Stderr| |-----|------:|-----------------------------|-----|---|------| |gsm8k| 2|exact_match,get-answer | 0.16| | | | | |exact_match_stderr,get-answer| 0.01| | | | | |alias |gsm8k| | | Elapsed time: 04:06:36