Text Generation
Transformers
Safetensors
English
llama
smol_llama
llama2
Inference Endpoints
text-generation-inference
pszemraj's picture
Upload 81m_tied.md
096e543

hf-causal-experimental (pretrained=BEE-spoke-data/smol_llama-81M-tied,trust_remote_code=True,dtype=float), limit: None, provide_description: False, num_fewshot: 0, batch_size: 64

Task Version Metric Value Stderr
arc_easy 0 acc 0.4162 ± 0.0101
acc_norm 0.3885 ± 0.0100
boolq 1 acc 0.5832 ± 0.0086
lambada_openai 0 ppl 79.4522 ± 3.1355
acc 0.2523 ± 0.0061
openbookqa 0 acc 0.1540 ± 0.0162
acc_norm 0.2780 ± 0.0201
piqa 0 acc 0.6050 ± 0.0114
acc_norm 0.5898 ± 0.0115
winogrande 0 acc 0.5272 ± 0.0140

hf-causal-experimental (pretrained=BEE-spoke-data/smol_llama-81M-tied,trust_remote_code=True,dtype=float), limit: None, provide_description: False, num_fewshot: 25, batch_size: 64

Task Version Metric Value Stderr
arc_challenge 0 acc 0.1672 ± 0.0109
acc_norm 0.2218 ± 0.0121

hf-causal-experimental (pretrained=BEE-spoke-data/smol_llama-81M-tied,trust_remote_code=True,dtype=float), limit: None, provide_description: False, num_fewshot: 10, batch_size: 64

Task Version Metric Value Stderr
hellaswag 0 acc 0.2769 ± 0.0045
acc_norm 0.2923 ± 0.0045

hf-causal-experimental (pretrained=BEE-spoke-data/smol_llama-81M-tied,trust_remote_code=True,dtype=float), limit: None, provide_description: False, num_fewshot: 0, batch_size: 64

Task Version Metric Value Stderr
truthfulqa_mc 1 mc1 0.2424 ± 0.0150
mc2 0.4353 ± 0.0152

hf-causal-experimental (pretrained=BEE-spoke-data/smol_llama-81M-tied,trust_remote_code=True,dtype=float), limit: None, provide_description: False, num_fewshot: 5, batch_size: 64

Task Version Metric Value Stderr
hendrycksTest-abstract_algebra 1 acc 0.2200 ± 0.0416
acc_norm 0.2200 ± 0.0416
hendrycksTest-anatomy 1 acc 0.2741 ± 0.0385
acc_norm 0.2741 ± 0.0385
hendrycksTest-astronomy 1 acc 0.1776 ± 0.0311
acc_norm 0.1776 ± 0.0311
hendrycksTest-business_ethics 1 acc 0.2100 ± 0.0409
acc_norm 0.2100 ± 0.0409
hendrycksTest-clinical_knowledge 1 acc 0.2264 ± 0.0258
acc_norm 0.2264 ± 0.0258
hendrycksTest-college_biology 1 acc 0.2361 ± 0.0355
acc_norm 0.2361 ± 0.0355
hendrycksTest-college_chemistry 1 acc 0.1900 ± 0.0394
acc_norm 0.1900 ± 0.0394
hendrycksTest-college_computer_science 1 acc 0.2100 ± 0.0409
acc_norm 0.2100 ± 0.0409
hendrycksTest-college_mathematics 1 acc 0.1800 ± 0.0386
acc_norm 0.1800 ± 0.0386
hendrycksTest-college_medicine 1 acc 0.2023 ± 0.0306
acc_norm 0.2023 ± 0.0306
hendrycksTest-college_physics 1 acc 0.2157 ± 0.0409
acc_norm 0.2157 ± 0.0409
hendrycksTest-computer_security 1 acc 0.2400 ± 0.0429
acc_norm 0.2400 ± 0.0429
hendrycksTest-conceptual_physics 1 acc 0.2596 ± 0.0287
acc_norm 0.2596 ± 0.0287
hendrycksTest-econometrics 1 acc 0.2544 ± 0.0410
acc_norm 0.2544 ± 0.0410
hendrycksTest-electrical_engineering 1 acc 0.2207 ± 0.0346
acc_norm 0.2207 ± 0.0346
hendrycksTest-elementary_mathematics 1 acc 0.2169 ± 0.0212
acc_norm 0.2169 ± 0.0212
hendrycksTest-formal_logic 1 acc 0.1587 ± 0.0327
acc_norm 0.1587 ± 0.0327
hendrycksTest-global_facts 1 acc 0.1900 ± 0.0394
acc_norm 0.1900 ± 0.0394
hendrycksTest-high_school_biology 1 acc 0.3000 ± 0.0261
acc_norm 0.3000 ± 0.0261
hendrycksTest-high_school_chemistry 1 acc 0.2808 ± 0.0316
acc_norm 0.2808 ± 0.0316
hendrycksTest-high_school_computer_science 1 acc 0.2800 ± 0.0451
acc_norm 0.2800 ± 0.0451
hendrycksTest-high_school_european_history 1 acc 0.2424 ± 0.0335
acc_norm 0.2424 ± 0.0335
hendrycksTest-high_school_geography 1 acc 0.2576 ± 0.0312
acc_norm 0.2576 ± 0.0312
hendrycksTest-high_school_government_and_politics 1 acc 0.2228 ± 0.0300
acc_norm 0.2228 ± 0.0300
hendrycksTest-high_school_macroeconomics 1 acc 0.2231 ± 0.0211
acc_norm 0.2231 ± 0.0211
hendrycksTest-high_school_mathematics 1 acc 0.2370 ± 0.0259
acc_norm 0.2370 ± 0.0259
hendrycksTest-high_school_microeconomics 1 acc 0.2227 ± 0.0270
acc_norm 0.2227 ± 0.0270
hendrycksTest-high_school_physics 1 acc 0.2053 ± 0.0330
acc_norm 0.2053 ± 0.0330
hendrycksTest-high_school_psychology 1 acc 0.2110 ± 0.0175
acc_norm 0.2110 ± 0.0175
hendrycksTest-high_school_statistics 1 acc 0.4120 ± 0.0336
acc_norm 0.4120 ± 0.0336
hendrycksTest-high_school_us_history 1 acc 0.2990 ± 0.0321
acc_norm 0.2990 ± 0.0321
hendrycksTest-high_school_world_history 1 acc 0.2658 ± 0.0288
acc_norm 0.2658 ± 0.0288
hendrycksTest-human_aging 1 acc 0.2287 ± 0.0282
acc_norm 0.2287 ± 0.0282
hendrycksTest-human_sexuality 1 acc 0.2595 ± 0.0384
acc_norm 0.2595 ± 0.0384
hendrycksTest-international_law 1 acc 0.2975 ± 0.0417
acc_norm 0.2975 ± 0.0417
hendrycksTest-jurisprudence 1 acc 0.2315 ± 0.0408
acc_norm 0.2315 ± 0.0408
hendrycksTest-logical_fallacies 1 acc 0.2822 ± 0.0354
acc_norm 0.2822 ± 0.0354
hendrycksTest-machine_learning 1 acc 0.2321 ± 0.0401
acc_norm 0.2321 ± 0.0401
hendrycksTest-management 1 acc 0.1748 ± 0.0376
acc_norm 0.1748 ± 0.0376
hendrycksTest-marketing 1 acc 0.2308 ± 0.0276
acc_norm 0.2308 ± 0.0276
hendrycksTest-medical_genetics 1 acc 0.3000 ± 0.0461
acc_norm 0.3000 ± 0.0461
hendrycksTest-miscellaneous 1 acc 0.2375 ± 0.0152
acc_norm 0.2375 ± 0.0152
hendrycksTest-moral_disputes 1 acc 0.2486 ± 0.0233
acc_norm 0.2486 ± 0.0233
hendrycksTest-moral_scenarios 1 acc 0.2425 ± 0.0143
acc_norm 0.2425 ± 0.0143
hendrycksTest-nutrition 1 acc 0.2288 ± 0.0241
acc_norm 0.2288 ± 0.0241
hendrycksTest-philosophy 1 acc 0.2090 ± 0.0231
acc_norm 0.2090 ± 0.0231
hendrycksTest-prehistory 1 acc 0.2377 ± 0.0237
acc_norm 0.2377 ± 0.0237
hendrycksTest-professional_accounting 1 acc 0.2234 ± 0.0248
acc_norm 0.2234 ± 0.0248
hendrycksTest-professional_law 1 acc 0.2471 ± 0.0110
acc_norm 0.2471 ± 0.0110
hendrycksTest-professional_medicine 1 acc 0.4081 ± 0.0299
acc_norm 0.4081 ± 0.0299
hendrycksTest-professional_psychology 1 acc 0.2565 ± 0.0177
acc_norm 0.2565 ± 0.0177
hendrycksTest-public_relations 1 acc 0.2182 ± 0.0396
acc_norm 0.2182 ± 0.0396
hendrycksTest-security_studies 1 acc 0.2408 ± 0.0274
acc_norm 0.2408 ± 0.0274
hendrycksTest-sociology 1 acc 0.2338 ± 0.0299
acc_norm 0.2338 ± 0.0299
hendrycksTest-us_foreign_policy 1 acc 0.2500 ± 0.0435
acc_norm 0.2500 ± 0.0435
hendrycksTest-virology 1 acc 0.2892 ± 0.0353
acc_norm 0.2892 ± 0.0353
hendrycksTest-world_religions 1 acc 0.2105 ± 0.0313
acc_norm 0.2105 ± 0.0313