150M model with OLMo-3 style arch (with 2:1 SWA with a 512 token window), pretrained on ~150B tokens of Dolma 3. Compute sponsored by lium.io, thank you <3!
| Stage | Model | Available Checkpoints | Data |
|---|---|---|---|
| Pretrained | allura-org/Rambley-150M-RealBase (you are here!) | allura-forge/Rambley-150M-Base-Checkpoints | 150B tokens of Dolma 3 |
| Midtrained | allura-org/Rambley-150M-Base | allura-forge/Rambley-150M-Midtrain-Checkpoints | +25B tokens of the OLMo 3 midtraining mix |
Motivation
I swear to god, if I see one more of these random Huggingface posts made by a 5 year old using Claude to pretrain a 150M with some homebrewed datamix, I'm going to lose it. Why not make my own? :) (it was also a fun learning experience so oh well)
Ablations
This cost way too much so I didn't have time for many ablations, but the main thing I tried was XSA attention but I found it to not converge as quickly so we just use regular attention
Benchmarks
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.2039 | ± | 0.0118 |
| none | 0 | acc_norm | ↑ | 0.2440 | ± | 0.0126 | ||
| arc_easy | 1 | none | 0 | acc | ↑ | 0.4945 | ± | 0.0103 |
| none | 0 | acc_norm | ↑ | 0.4474 | ± | 0.0102 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2933 | ± | 0.0045 |
| none | 0 | acc_norm | ↑ | 0.3229 | ± | 0.0047 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.6300 | ± | 0.0113 |
| none | 0 | acc_norm | ↑ | 0.6257 | ± | 0.0113 | ||
| blimp | 2 | none | 0 | acc | ↑ | 0.7261 | ± | 0.0015 |
While not SOTA by any means, for not having mixed in any benchmaxxing/domain-specific data in the actual pretraining mix, it's not too bad
Code / Logs
- Pretraining/midtraining codebase: https://code.allura.moe/fizz/OLMo-core
- Wandb pretraining logs
- Wandb midtraining logs
Other random benchmarks
BananaMind Base Bench 1.1
Overall Elo: 1006
Accuracy: 183/350 (52.29%)
Weighted accuracy: 48.76%
language_completion: Elo 1331 | 47/50 (94.00%) | weighted 93.31%
commonsense: Elo 1043 | 33/50 (66.00%) | weighted 62.15%
world_knowledge: Elo 1031 | 31/50 (62.00%) | weighted 60.50%
context_tracking: Elo 859 | 16/50 (32.00%) | weighted 31.53%
quantitative: Elo 822 | 11/50 (22.00%) | weighted 21.77%
logical_reasoning: Elo 1043 | 23/50 (46.00%) | weighted 42.32%
code_completion: Elo 1045 | 22/50 (44.00%) | weighted 43.31%
Loaded 1000 ArithMark 3.0 examples from /root/arith/arithmark-3.jsonl
ArithMark 3.0 dataset SHA-256: bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435
Using device: cuda
====================================================================
Loading allura-forge/Rambley-150M-RealBase...
====================================================================
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 178/178 [00:00<00:00, 5401.16it/s]
149,935,616 parameters (torch.bfloat16)
arithmark-3: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:02<00:00, 15.98it/s]
arithmark-3: raw 37.20% (372/1000) normalized 37.20% (372/1000)
speed: 499.2 examples/s (tokenize 0.44s, evaluate 2.00s)
Category N Raw Normalized
----------------------------------------------------------------------------------------------
elementary_school_math_continuation::addition::grades_1_2::easy 128 25.00% 25.00%
elementary_school_math_continuation::comparison::grades_2_3::medium 44 22.73% 22.73%
elementary_school_math_continuation::comparison_difference::grades_2_3::medium 48 25.00% 25.00%
elementary_school_math_continuation::data::grades_2_3::easy 43 20.93% 20.93%
elementary_school_math_continuation::division::grades_3_4::medium 54 33.33% 33.33%
elementary_school_math_continuation::fractions_counting::grades_3_4::medium 50 22.00% 22.00%
elementary_school_math_continuation::geometry_area::grades_4_5::medium 52 73.08% 73.08%
elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium 45 60.00% 60.00%
elementary_school_math_continuation::measurement::grades_2_3::easy 76 30.26% 30.26%
elementary_school_math_continuation::money::grades_3_4::medium 64 28.12% 28.12%
elementary_school_math_continuation::multiplication::grades_3_4::medium 74 63.51% 63.51%
elementary_school_math_continuation::patterns::grades_3_4::medium 53 32.08% 32.08%
elementary_school_math_continuation::subtraction::grades_1_2::easy 117 23.08% 23.08%
elementary_school_math_continuation::time::grades_2_3::easy 55 100.00% 100.00%
elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium 46 23.91% 23.91%
elementary_school_math_continuation::two_step_addition::grades_2_3::medium 19 31.58% 31.58%
elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium 32 34.38% 34.38%
====================================================================
allura-forge/Rambley-150M-RealBase (149,935,616 params) RESULTS
====================================================================
Raw continuation accuracy 37.20%
Length-normalized accuracy 37.20%
Primary (acc_norm) 37.20%
====================================================================
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| blimp | 2 | none | 0 | acc | ↑ | 0.7261 | ± | 0.0015 |
| - blimp_adjunct_island | 1 | none | 0 | acc | ↑ | 0.6230 | ± | 0.0153 |
| - blimp_anaphor_gender_agreement | 1 | none | 0 | acc | ↑ | 0.9490 | ± | 0.0070 |
| - blimp_anaphor_number_agreement | 1 | none | 0 | acc | ↑ | 0.9770 | ± | 0.0047 |
| - blimp_animate_subject_passive | 1 | none | 0 | acc | ↑ | 0.7290 | ± | 0.0141 |
| - blimp_animate_subject_trans | 1 | none | 0 | acc | ↑ | 0.9030 | ± | 0.0094 |
| - blimp_causative | 1 | none | 0 | acc | ↑ | 0.5990 | ± | 0.0155 |
| - blimp_complex_NP_island | 1 | none | 0 | acc | ↑ | 0.4610 | ± | 0.0158 |
| - blimp_coordinate_structure_constraint_complex_left_branch | 1 | none | 0 | acc | ↑ | 0.3310 | ± | 0.0149 |
| - blimp_coordinate_structure_constraint_object_extraction | 1 | none | 0 | acc | ↑ | 0.8150 | ± | 0.0123 |
| - blimp_determiner_noun_agreement_1 | 1 | none | 0 | acc | ↑ | 0.9510 | ± | 0.0068 |
| - blimp_determiner_noun_agreement_2 | 1 | none | 0 | acc | ↑ | 0.9390 | ± | 0.0076 |
| - blimp_determiner_noun_agreement_irregular_1 | 1 | none | 0 | acc | ↑ | 0.8640 | ± | 0.0108 |
| - blimp_determiner_noun_agreement_irregular_2 | 1 | none | 0 | acc | ↑ | 0.9160 | ± | 0.0088 |
| - blimp_determiner_noun_agreement_with_adj_2 | 1 | none | 0 | acc | ↑ | 0.9140 | ± | 0.0089 |
| - blimp_determiner_noun_agreement_with_adj_irregular_1 | 1 | none | 0 | acc | ↑ | 0.8560 | ± | 0.0111 |
| - blimp_determiner_noun_agreement_with_adj_irregular_2 | 1 | none | 0 | acc | ↑ | 0.8970 | ± | 0.0096 |
| - blimp_determiner_noun_agreement_with_adjective_1 | 1 | none | 0 | acc | ↑ | 0.9340 | ± | 0.0079 |
| - blimp_distractor_agreement_relational_noun | 1 | none | 0 | acc | ↑ | 0.6510 | ± | 0.0151 |
| - blimp_distractor_agreement_relative_clause | 1 | none | 0 | acc | ↑ | 0.6760 | ± | 0.0148 |
| - blimp_drop_argument | 1 | none | 0 | acc | ↑ | 0.7570 | ± | 0.0136 |
| - blimp_ellipsis_n_bar_1 | 1 | none | 0 | acc | ↑ | 0.8340 | ± | 0.0118 |
| - blimp_ellipsis_n_bar_2 | 1 | none | 0 | acc | ↑ | 0.8310 | ± | 0.0119 |
| - blimp_existential_there_object_raising | 1 | none | 0 | acc | ↑ | 0.8660 | ± | 0.0108 |
| - blimp_existential_there_quantifiers_1 | 1 | none | 0 | acc | ↑ | 0.8810 | ± | 0.0102 |
| - blimp_existential_there_quantifiers_2 | 1 | none | 0 | acc | ↑ | 0.1700 | ± | 0.0119 |
| - blimp_existential_there_subject_raising | 1 | none | 0 | acc | ↑ | 0.8480 | ± | 0.0114 |
| - blimp_expletive_it_object_raising | 1 | none | 0 | acc | ↑ | 0.7590 | ± | 0.0135 |
| - blimp_inchoative | 1 | none | 0 | acc | ↑ | 0.5880 | ± | 0.0156 |
| - blimp_intransitive | 1 | none | 0 | acc | ↑ | 0.7280 | ± | 0.0141 |
| - blimp_irregular_past_participle_adjectives | 1 | none | 0 | acc | ↑ | 0.9940 | ± | 0.0024 |
| - blimp_irregular_past_participle_verbs | 1 | none | 0 | acc | ↑ | 0.9350 | ± | 0.0078 |
| - blimp_irregular_plural_subject_verb_agreement_1 | 1 | none | 0 | acc | ↑ | 0.8490 | ± | 0.0113 |
| - blimp_irregular_plural_subject_verb_agreement_2 | 1 | none | 0 | acc | ↑ | 0.8190 | ± | 0.0122 |
| - blimp_left_branch_island_echo_question | 1 | none | 0 | acc | ↑ | 0.4820 | ± | 0.0158 |
| - blimp_left_branch_island_simple_question | 1 | none | 0 | acc | ↑ | 0.2790 | ± | 0.0142 |
| - blimp_matrix_question_npi_licensor_present | 1 | none | 0 | acc | ↑ | 0.2440 | ± | 0.0136 |
| - blimp_npi_present_1 | 1 | none | 0 | acc | ↑ | 0.6020 | ± | 0.0155 |
| - blimp_npi_present_2 | 1 | none | 0 | acc | ↑ | 0.5220 | ± | 0.0158 |
| - blimp_only_npi_licensor_present | 1 | none | 0 | acc | ↑ | 0.1050 | ± | 0.0097 |
| - blimp_only_npi_scope | 1 | none | 0 | acc | ↑ | 0.4300 | ± | 0.0157 |
| - blimp_passive_1 | 1 | none | 0 | acc | ↑ | 0.8810 | ± | 0.0102 |
| - blimp_passive_2 | 1 | none | 0 | acc | ↑ | 0.8650 | ± | 0.0108 |
| - blimp_principle_A_c_command | 1 | none | 0 | acc | ↑ | 0.6740 | ± | 0.0148 |
| - blimp_principle_A_case_1 | 1 | none | 0 | acc | ↑ | 1.0000 | ± | 0.0000 |
| - blimp_principle_A_case_2 | 1 | none | 0 | acc | ↑ | 0.9150 | ± | 0.0088 |
| - blimp_principle_A_domain_1 | 1 | none | 0 | acc | ↑ | 0.9800 | ± | 0.0044 |
| - blimp_principle_A_domain_2 | 1 | none | 0 | acc | ↑ | 0.6630 | ± | 0.0150 |
| - blimp_principle_A_domain_3 | 1 | none | 0 | acc | ↑ | 0.5440 | ± | 0.0158 |
| - blimp_principle_A_reconstruction | 1 | none | 0 | acc | ↑ | 0.3390 | ± | 0.0150 |
| - blimp_regular_plural_subject_verb_agreement_1 | 1 | none | 0 | acc | ↑ | 0.9040 | ± | 0.0093 |
| - blimp_regular_plural_subject_verb_agreement_2 | 1 | none | 0 | acc | ↑ | 0.7770 | ± | 0.0132 |
| - blimp_sentential_negation_npi_licensor_present | 1 | none | 0 | acc | ↑ | 0.9950 | ± | 0.0022 |
| - blimp_sentential_negation_npi_scope | 1 | none | 0 | acc | ↑ | 0.5410 | ± | 0.0158 |
| - blimp_sentential_subject_island | 1 | none | 0 | acc | ↑ | 0.3560 | ± | 0.0151 |
| - blimp_superlative_quantifiers_1 | 1 | none | 0 | acc | ↑ | 0.5850 | ± | 0.0156 |
| - blimp_superlative_quantifiers_2 | 1 | none | 0 | acc | ↑ | 0.9540 | ± | 0.0066 |
| - blimp_tough_vs_raising_1 | 1 | none | 0 | acc | ↑ | 0.5800 | ± | 0.0156 |
| - blimp_tough_vs_raising_2 | 1 | none | 0 | acc | ↑ | 0.7760 | ± | 0.0132 |
| - blimp_transitive | 1 | none | 0 | acc | ↑ | 0.8200 | ± | 0.0122 |
| - blimp_wh_island | 1 | none | 0 | acc | ↑ | 0.8030 | ± | 0.0126 |
| - blimp_wh_questions_object_gap | 1 | none | 0 | acc | ↑ | 0.7590 | ± | 0.0135 |
| - blimp_wh_questions_subject_gap | 1 | none | 0 | acc | ↑ | 0.9180 | ± | 0.0087 |
| - blimp_wh_questions_subject_gap_long_distance | 1 | none | 0 | acc | ↑ | 0.8340 | ± | 0.0118 |
| - blimp_wh_vs_that_no_gap | 1 | none | 0 | acc | ↑ | 0.9630 | ± | 0.0060 |
| - blimp_wh_vs_that_no_gap_long_distance | 1 | none | 0 | acc | ↑ | 0.9630 | ± | 0.0060 |
| - blimp_wh_vs_that_with_gap | 1 | none | 0 | acc | ↑ | 0.5100 | ± | 0.0158 |
| - blimp_wh_vs_that_with_gap_long_distance | 1 | none | 0 | acc | ↑ | 0.2390 | ± | 0.0135 |
- Downloads last month
- 117
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support