Update README.md
Browse files
README.md
CHANGED
|
@@ -22,6 +22,4 @@ Prompt template:
|
|
| 22 |
GGML quants available [here](https://huggingface.co/TheBloke/Marx-3b-GGML).</br>
|
| 23 |
GPTQ quants available [here](https://huggingface.co/TheBloke/Marx-3b-GPTQ).
|
| 24 |
|
| 25 |
-
Note: Don't expect this model to be good, I was just starting out to finetune. So don't roast me please!
|
| 26 |
-
|
| 27 |
-
Benchmarks:<table><tr><th>arc_challenge</th><td><table><tr><th>acc</th><td>0.38993174061433444</td></tr><tr><th>acc_stderr</th><td>0.014252959848892884</td></tr><tr><th>acc_norm</th><td>0.4308873720136519</td></tr><tr><th>acc_norm_stderr</th><td>0.014471133392642475</td></tr></table></td></tr><tr><th>hellaswag</th><td><table><tr><th>acc</th><td>0.5513841864170484</td></tr><tr><th>acc_stderr</th><td>0.004963362085275556</td></tr><tr><th>acc_norm</th><td>0.7257518422624976</td></tr><tr><th>acc_norm_stderr</th><td>0.00445222854104355</td></tr></table></td></tr><tr><th>hendrycksTest-abstract_algebra</th><td><table><tr><th>acc</th><td>0.23</td></tr><tr><th>acc_stderr</th><td>0.04229525846816506</td></tr><tr><th>acc_norm</th><td>0.23</td></tr><tr><th>acc_norm_stderr</th><td>0.04229525846816506</td></tr></table></td></tr><tr><th>hendrycksTest-anatomy</th><td><table><tr><th>acc</th><td>0.2962962962962963</td></tr><tr><th>acc_stderr</th><td>0.03944624162501116</td></tr><tr><th>acc_norm</th><td>0.2962962962962963</td></tr><tr><th>acc_norm_stderr</th><td>0.03944624162501116</td></tr></table></td></tr><tr><th>hendrycksTest-astronomy</th><td><table><tr><th>acc</th><td>0.32894736842105265</td></tr><tr><th>acc_stderr</th><td>0.03823428969926603</td></tr><tr><th>acc_norm</th><td>0.32894736842105265</td></tr><tr><th>acc_norm_stderr</th><td>0.03823428969926603</td></tr></table></td></tr><tr><th>hendrycksTest-business_ethics</th><td><table><tr><th>acc</th><td>0.3</td></tr><tr><th>acc_stderr</th><td>0.046056618647183814</td></tr><tr><th>acc_norm</th><td>0.3</td></tr><tr><th>acc_norm_stderr</th><td>0.046056618647183814</td></tr></table></td></tr><tr><th>hendrycksTest-clinical_knowledge</th><td><table><tr><th>acc</th><td>0.2641509433962264</td></tr><tr><th>acc_stderr</th><td>0.027134291628741713</td></tr><tr><th>acc_norm</th><td>0.2641509433962264</td></tr><tr><th>acc_norm_stderr</th><td>0.027134291628741713</td></tr></table></td></tr><tr><th>hendrycksTest-college_biology</th><td><table><tr><th>acc</th><td>0.2569444444444444</td></tr><tr><th>acc_stderr</th><td>0.03653946969442099</td></tr><tr><th>acc_norm</th><td>0.2569444444444444</td></tr><tr><th>acc_norm_stderr</th><td>0.03653946969442099</td></tr></table></td></tr><tr><th>hendrycksTest-college_chemistry</th><td><table><tr><th>acc</th><td>0.22</td></tr><tr><th>acc_stderr</th><td>0.041633319989322695</td></tr><tr><th>acc_norm</th><td>0.22</td></tr><tr><th>acc_norm_stderr</th><td>0.041633319989322695</td></tr></table></td></tr><tr><th>hendrycksTest-college_computer_science</th><td><table><tr><th>acc</th><td>0.26</td></tr><tr><th>acc_stderr</th><td>0.0440844002276808</td></tr><tr><th>acc_norm</th><td>0.26</td></tr><tr><th>acc_norm_stderr</th><td>0.0440844002276808</td></tr></table></td></tr><tr><th>hendrycksTest-college_mathematics</th><td><table><tr><th>acc</th><td>0.31</td></tr><tr><th>acc_stderr</th><td>0.04648231987117316</td></tr><tr><th>acc_norm</th><td>0.31</td></tr><tr><th>acc_norm_stderr</th><td>0.04648231987117316</td></tr></table></td></tr><tr><th>hendrycksTest-college_medicine</th><td><table><tr><th>acc</th><td>0.23121387283236994</td></tr><tr><th>acc_stderr</th><td>0.032147373020294696</td></tr><tr><th>acc_norm</th><td>0.23121387283236994</td></tr><tr><th>acc_norm_stderr</th><td>0.032147373020294696</td></tr></table></td></tr><tr><th>hendrycksTest-college_physics</th><td><table><tr><th>acc</th><td>0.27450980392156865</td></tr><tr><th>acc_stderr</th><td>0.04440521906179327</td></tr><tr><th>acc_norm</th><td>0.27450980392156865</td></tr><tr><th>acc_norm_stderr</th><td>0.04440521906179327</td></tr></table></td></tr><tr><th>hendrycksTest-computer_security</th><td><table><tr><th>acc</th><td>0.36</td></tr><tr><th>acc_stderr</th><td>0.048241815132442176</td></tr><tr><th>acc_norm</th><td>0.36</td></tr><tr><th>acc_norm_stderr</th><td>0.048241815132442176</td></tr></table></td></tr><tr><th>hendrycksTest-conceptual_physics</th><td><table><tr><th>acc</th><td>0.2765957446808511</td></tr><tr><th>acc_stderr</th><td>0.029241883869628837</td></tr><tr><th>acc_norm</th><td>0.2765957446808511</td></tr><tr><th>acc_norm_stderr</th><td>0.029241883869628837</td></tr></table></td></tr><tr><th>hendrycksTest-econometrics</th><td><table><tr><th>acc</th><td>0.2631578947368421</td></tr><tr><th>acc_stderr</th><td>0.04142439719489363</td></tr><tr><th>acc_norm</th><td>0.2631578947368421</td></tr><tr><th>acc_norm_stderr</th><td>0.04142439719489363</td></tr></table></td></tr><tr><th>hendrycksTest-electrical_engineering</th><td><table><tr><th>acc</th><td>0.20689655172413793</td></tr><tr><th>acc_stderr</th><td>0.03375672449560554</td></tr><tr><th>acc_norm</th><td>0.20689655172413793</td></tr><tr><th>acc_norm_stderr</th><td>0.03375672449560554</td></tr></table></td></tr><tr><th>hendrycksTest-elementary_mathematics</th><td><table><tr><th>acc</th><td>0.2698412698412698</td></tr><tr><th>acc_stderr</th><td>0.022860838309232072</td></tr><tr><th>acc_norm</th><td>0.2698412698412698</td></tr><tr><th>acc_norm_stderr</th><td>0.022860838309232072</td></tr></table></td></tr><tr><th>hendrycksTest-formal_logic</th><td><table><tr><th>acc</th><td>0.2619047619047619</td></tr><tr><th>acc_stderr</th><td>0.039325376803928704</td></tr><tr><th>acc_norm</th><td>0.2619047619047619</td></tr><tr><th>acc_norm_stderr</th><td>0.039325376803928704</td></tr></table></td></tr><tr><th>hendrycksTest-global_facts</th><td><table><tr><th>acc</th><td>0.35</td></tr><tr><th>acc_stderr</th><td>0.047937248544110196</td></tr><tr><th>acc_norm</th><td>0.35</td></tr><tr><th>acc_norm_stderr</th><td>0.047937248544110196</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_biology</th><td><table><tr><th>acc</th><td>0.24193548387096775</td></tr><tr><th>acc_stderr</th><td>0.0243625996930311</td></tr><tr><th>acc_norm</th><td>0.24193548387096775</td></tr><tr><th>acc_norm_stderr</th><td>0.0243625996930311</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_chemistry</th><td><table><tr><th>acc</th><td>0.28078817733990147</td></tr><tr><th>acc_stderr</th><td>0.0316185633535861</td></tr><tr><th>acc_norm</th><td>0.28078817733990147</td></tr><tr><th>acc_norm_stderr</th><td>0.0316185633535861</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_computer_science</th><td><table><tr><th>acc</th><td>0.33</td></tr><tr><th>acc_stderr</th><td>0.04725815626252605</td></tr><tr><th>acc_norm</th><td>0.33</td></tr><tr><th>acc_norm_stderr</th><td>0.04725815626252605</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_european_history</th><td><table><tr><th>acc</th><td>0.296969696969697</td></tr><tr><th>acc_stderr</th><td>0.03567969772268048</td></tr><tr><th>acc_norm</th><td>0.296969696969697</td></tr><tr><th>acc_norm_stderr</th><td>0.03567969772268048</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_geography</th><td><table><tr><th>acc</th><td>0.2878787878787879</td></tr><tr><th>acc_stderr</th><td>0.03225883512300993</td></tr><tr><th>acc_norm</th><td>0.2878787878787879</td></tr><tr><th>acc_norm_stderr</th><td>0.03225883512300993</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_government_and_politics</th><td><table><tr><th>acc</th><td>0.2538860103626943</td></tr><tr><th>acc_stderr</th><td>0.0314102478056532</td></tr><tr><th>acc_norm</th><td>0.2538860103626943</td></tr><tr><th>acc_norm_stderr</th><td>0.0314102478056532</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_macroeconomics</th><td><table><tr><th>acc</th><td>0.2743589743589744</td></tr><tr><th>acc_stderr</th><td>0.022622765767493207</td></tr><tr><th>acc_norm</th><td>0.2743589743589744</td></tr><tr><th>acc_norm_stderr</th><td>0.022622765767493207</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_mathematics</th><td><table><tr><th>acc</th><td>0.26296296296296295</td></tr><tr><th>acc_stderr</th><td>0.026842057873833706</td></tr><tr><th>acc_norm</th><td>0.26296296296296295</td></tr><tr><th>acc_norm_stderr</th><td>0.026842057873833706</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_microeconomics</th><td><table><tr><th>acc</th><td>0.2647058823529412</td></tr><tr><th>acc_stderr</th><td>0.028657491285071977</td></tr><tr><th>acc_norm</th><td>0.2647058823529412</td></tr><tr><th>acc_norm_stderr</th><td>0.028657491285071977</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_physics</th><td><table><tr><th>acc</th><td>0.304635761589404</td></tr><tr><th>acc_stderr</th><td>0.03757949922943343</td></tr><tr><th>acc_norm</th><td>0.304635761589404</td></tr><tr><th>acc_norm_stderr</th><td>0.03757949922943343</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_psychology</th><td><table><tr><th>acc</th><td>0.28623853211009176</td></tr><tr><th>acc_stderr</th><td>0.019379436628919968</td></tr><tr><th>acc_norm</th><td>0.28623853211009176</td></tr><tr><th>acc_norm_stderr</th><td>0.019379436628919968</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_statistics</th><td><table><tr><th>acc</th><td>0.25462962962962965</td></tr><tr><th>acc_stderr</th><td>0.02971127586000535</td></tr><tr><th>acc_norm</th><td>0.25462962962962965</td></tr><tr><th>acc_norm_stderr</th><td>0.02971127586000535</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_us_history</th><td><table><tr><th>acc</th><td>0.23039215686274508</td></tr><tr><th>acc_stderr</th><td>0.029554292605695053</td></tr><tr><th>acc_norm</th><td>0.23039215686274508</td></tr><tr><th>acc_norm_stderr</th><td>0.029554292605695053</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_world_history</th><td><table><tr><th>acc</th><td>0.2869198312236287</td></tr><tr><th>acc_stderr</th><td>0.029443773022594693</td></tr><tr><th>acc_norm</th><td>0.2869198312236287</td></tr><tr><th>acc_norm_stderr</th><td>0.029443773022594693</td></tr></table></td></tr><tr><th>hendrycksTest-human_aging</th><td><table><tr><th>acc</th><td>0.3811659192825112</td></tr><tr><th>acc_stderr</th><td>0.03259625118416827</td></tr><tr><th>acc_norm</th><td>0.3811659192825112</td></tr><tr><th>acc_norm_stderr</th><td>0.03259625118416827</td></tr></table></td></tr><tr><th>hendrycksTest-human_sexuality</th><td><table><tr><th>acc</th><td>0.1984732824427481</td></tr><tr><th>acc_stderr</th><td>0.03498149385462472</td></tr><tr><th>acc_norm</th><td>0.1984732824427481</td></tr><tr><th>acc_norm_stderr</th><td>0.03498149385462472</td></tr></table></td></tr><tr><th>hendrycksTest-international_law</th><td><table><tr><th>acc</th><td>0.3884297520661157</td></tr><tr><th>acc_stderr</th><td>0.04449270350068382</td></tr><tr><th>acc_norm</th><td>0.3884297520661157</td></tr><tr><th>acc_norm_stderr</th><td>0.04449270350068382</td></tr></table></td></tr><tr><th>hendrycksTest-jurisprudence</th><td><table><tr><th>acc</th><td>0.23148148148148148</td></tr><tr><th>acc_stderr</th><td>0.04077494709252627</td></tr><tr><th>acc_norm</th><td>0.23148148148148148</td></tr><tr><th>acc_norm_stderr</th><td>0.04077494709252627</td></tr></table></td></tr><tr><th>hendrycksTest-logical_fallacies</th><td><table><tr><th>acc</th><td>0.2331288343558282</td></tr><tr><th>acc_stderr</th><td>0.03322015795776741</td></tr><tr><th>acc_norm</th><td>0.2331288343558282</td></tr><tr><th>acc_norm_stderr</th><td>0.03322015795776741</td></tr></table></td></tr><tr><th>hendrycksTest-machine_learning</th><td><table><tr><th>acc</th><td>0.21428571428571427</td></tr><tr><th>acc_stderr</th><td>0.03894641120044792</td></tr><tr><th>acc_norm</th><td>0.21428571428571427</td></tr><tr><th>acc_norm_stderr</th><td>0.03894641120044792</td></tr></table></td></tr><tr><th>hendrycksTest-management</th><td><table><tr><th>acc</th><td>0.3300970873786408</td></tr><tr><th>acc_stderr</th><td>0.04656147110012352</td></tr><tr><th>acc_norm</th><td>0.3300970873786408</td></tr><tr><th>acc_norm_stderr</th><td>0.04656147110012352</td></tr></table></td></tr><tr><th>hendrycksTest-marketing</th><td><table><tr><th>acc</th><td>0.2905982905982906</td></tr><tr><th>acc_stderr</th><td>0.029745048572674078</td></tr><tr><th>acc_norm</th><td>0.2905982905982906</td></tr><tr><th>acc_norm_stderr</th><td>0.029745048572674078</td></tr></table></td></tr><tr><th>hendrycksTest-medical_genetics</th><td><table><tr><th>acc</th><td>0.29</td></tr><tr><th>acc_stderr</th><td>0.04560480215720684</td></tr><tr><th>acc_norm</th><td>0.29</td></tr><tr><th>acc_norm_stderr</th><td>0.04560480215720684</td></tr></table></td></tr><tr><th>hendrycksTest-miscellaneous</th><td><table><tr><th>acc</th><td>0.31545338441890164</td></tr><tr><th>acc_stderr</th><td>0.016617501738763394</td></tr><tr><th>acc_norm</th><td>0.31545338441890164</td></tr><tr><th>acc_norm_stderr</th><td>0.016617501738763394</td></tr></table></td></tr><tr><th>hendrycksTest-moral_disputes</th><td><table><tr><th>acc</th><td>0.2861271676300578</td></tr><tr><th>acc_stderr</th><td>0.02433214677913413</td></tr><tr><th>acc_norm</th><td>0.2861271676300578</td></tr><tr><th>acc_norm_stderr</th><td>0.02433214677913413</td></tr></table></td></tr><tr><th>hendrycksTest-moral_scenarios</th><td><table><tr><th>acc</th><td>0.2122905027932961</td></tr><tr><th>acc_stderr</th><td>0.01367664468583173</td></tr><tr><th>acc_norm</th><td>0.2122905027932961</td></tr><tr><th>acc_norm_stderr</th><td>0.01367664468583173</td></tr></table></td></tr><tr><th>hendrycksTest-nutrition</th><td><table><tr><th>acc</th><td>0.2875816993464052</td></tr><tr><th>acc_stderr</th><td>0.02591780611714716</td></tr><tr><th>acc_norm</th><td>0.2875816993464052</td></tr><tr><th>acc_norm_stderr</th><td>0.02591780611714716</td></tr></table></td></tr><tr><th>hendrycksTest-philosophy</th><td><table><tr><th>acc</th><td>0.2765273311897106</td></tr><tr><th>acc_stderr</th><td>0.02540383297817961</td></tr><tr><th>acc_norm</th><td>0.2765273311897106</td></tr><tr><th>acc_norm_stderr</th><td>0.02540383297817961</td></tr></table></td></tr><tr><th>hendrycksTest-prehistory</th><td><table><tr><th>acc</th><td>0.3117283950617284</td></tr><tr><th>acc_stderr</th><td>0.025773111169630446</td></tr><tr><th>acc_norm</th><td>0.3117283950617284</td></tr><tr><th>acc_norm_stderr</th><td>0.025773111169630446</td></tr></table></td></tr><tr><th>hendrycksTest-professional_accounting</th><td><table><tr><th>acc</th><td>0.26595744680851063</td></tr><tr><th>acc_stderr</th><td>0.026358065698880592</td></tr><tr><th>acc_norm</th><td>0.26595744680851063</td></tr><tr><th>acc_norm_stderr</th><td>0.026358065698880592</td></tr></table></td></tr><tr><th>hendrycksTest-professional_law</th><td><table><tr><th>acc</th><td>0.25684485006518903</td></tr><tr><th>acc_stderr</th><td>0.011158455853098832</td></tr><tr><th>acc_norm</th><td>0.25684485006518903</td></tr><tr><th>acc_norm_stderr</th><td>0.011158455853098832</td></tr></table></td></tr><tr><th>hendrycksTest-professional_medicine</th><td><table><tr><th>acc</th><td>0.1801470588235294</td></tr><tr><th>acc_stderr</th><td>0.023345163616544855</td></tr><tr><th>acc_norm</th><td>0.1801470588235294</td></tr><tr><th>acc_norm_stderr</th><td>0.023345163616544855</td></tr></table></td></tr><tr><th>hendrycksTest-professional_psychology</th><td><table><tr><th>acc</th><td>0.27941176470588236</td></tr><tr><th>acc_stderr</th><td>0.018152871051538802</td></tr><tr><th>acc_norm</th><td>0.27941176470588236</td></tr><tr><th>acc_norm_stderr</th><td>0.018152871051538802</td></tr></table></td></tr><tr><th>hendrycksTest-public_relations</th><td><table><tr><th>acc</th><td>0.3090909090909091</td></tr><tr><th>acc_stderr</th><td>0.044262946482000985</td></tr><tr><th>acc_norm</th><td>0.3090909090909091</td></tr><tr><th>acc_norm_stderr</th><td>0.044262946482000985</td></tr></table></td></tr><tr><th>hendrycksTest-security_studies</th><td><table><tr><th>acc</th><td>0.32653061224489793</td></tr><tr><th>acc_stderr</th><td>0.030021056238440313</td></tr><tr><th>acc_norm</th><td>0.32653061224489793</td></tr><tr><th>acc_norm_stderr</th><td>0.030021056238440313</td></tr></table></td></tr><tr><th>hendrycksTest-sociology</th><td><table><tr><th>acc</th><td>0.25870646766169153</td></tr><tr><th>acc_stderr</th><td>0.030965903123573026</td></tr><tr><th>acc_norm</th><td>0.25870646766169153</td></tr><tr><th>acc_norm_stderr</th><td>0.030965903123573026</td></tr></table></td></tr><tr><th>hendrycksTest-us_foreign_policy</th><td><table><tr><th>acc</th><td>0.32</td></tr><tr><th>acc_stderr</th><td>0.04688261722621504</td></tr><tr><th>acc_norm</th><td>0.32</td></tr><tr><th>acc_norm_stderr</th><td>0.04688261722621504</td></tr></table></td></tr><tr><th>hendrycksTest-virology</th><td><table><tr><th>acc</th><td>0.30120481927710846</td></tr><tr><th>acc_stderr</th><td>0.0357160923005348</td></tr><tr><th>acc_norm</th><td>0.30120481927710846</td></tr><tr><th>acc_norm_stderr</th><td>0.0357160923005348</td></tr></table></td></tr><tr><th>hendrycksTest-world_religions</th><td><table><tr><th>acc</th><td>0.32748538011695905</td></tr><tr><th>acc_stderr</th><td>0.035993357714560276</td></tr><tr><th>acc_norm</th><td>0.32748538011695905</td></tr><tr><th>acc_norm_stderr</th><td>0.035993357714560276</td></tr></table></td></tr><tr><th>truthfulqa_mc</th><td><table><tr><th>mc1</th><td>0.2423500611995104</td></tr><tr><th>mc1_stderr</th><td>0.01500067437357034</td></tr><tr><th>mc2</th><td>0.3859757929597962</td></tr><tr><th>mc2_stderr</th><td>0.013898628036488968</td></tr></table></td></tr></table>
|
|
|
|
| 22 |
GGML quants available [here](https://huggingface.co/TheBloke/Marx-3b-GGML).</br>
|
| 23 |
GPTQ quants available [here](https://huggingface.co/TheBloke/Marx-3b-GPTQ).
|
| 24 |
|
| 25 |
+
Note: Don't expect this model to be good, I was just starting out to finetune. So don't roast me please!
|
|
|
|
|
|