SlayerLab/fabryka-english-base-250m-e01

Fabryka English Base 250M — E01

An early experimental English base model trained from scratch. It has 249,944,832 parameters and completed 50,000,000 prediction targets in 191 optimizer updates. This checkpoint is substantially undertrained: sampled continuations are repetitive and often incoherent. It is not an instruction-tuned assistant or a completed research-program result.

Benchmarks

Benchmark	Metric	Result	Coverage
ARC Easy	acc_norm,none	29.42%	2376/2376
ARC Challenge	acc_norm,none	23.21%	1172/1172
PIQA	acc_norm,none	51.47%	1838/1838
HellaSwag	acc_norm,none	24.97%	10042/10042
ArithMark 3	Token-length-normalized accuracy	31.30%	313/1,000 correct
ArithMark 2	Raw continuation accuracy	25.36%	634/2,500 correct
INT Index	Open SLM-style aggregate	2.77	All five inputs
BananaMind Base Bench 1.1	Overall Elo	842	350/350
BananaMind Base Bench 1.1	Accuracy	30.29%	106/350 correct
BananaMind Base Bench 1.1	Weighted accuracy	28.82%	350/350

Evaluated on the published E01 weights. The complete category breakdown, pinned revisions, evaluation settings, and reproduction command follow below.

Additional native CORE evaluation (experimental)

The complete 22-task native Foundry evaluation produced CORE v2 = -0.0434304233. All 91,037 item outcomes and 204,534 continuation prompt records were verified against pinned source identities, and task scores and the aggregate were recomputed. This signed, baseline-centered aggregate can be negative. It is not INT, a percentage accuracy, or directly comparable to the zero-shot acc_norm table above.

Settings: FP32, TF32 disabled, 2,048-token context, native BOS/context-answer tokenization, and the task-specific few-shot settings recorded in the report. 14,600 continuations required prompt truncation. The evaluation is complete, but its protocol claim remains provisional pending interpretation of the recorded prompt effects. This score does not establish contamination-free training or broad model quality.

Full task scores, protocol hashes, lineage and verification summary. The original item ledger and prompt audit are retained in the research workspace; their SHA256 hashes are included in the summary.

Standard zero-shot benchmarks

Full lm-eval 0.4.13 evaluation of the released E01 revision, FP32 on one RTX 3090, batch size 8, 2,048-token context, no BOS and no chat template. ARC uses the test splits; PIQA and HellaSwag use validation. acc_norm divides continuation log likelihood by answer character length. All examples were evaluated, and recorded per-choice scores, predictions, complete sample coverage, aggregate accuracies and standard errors were verified.

Benchmark	Raw accuracy	Normalized accuracy	Normalized standard error
ARC Easy	28.75%	29.42%	0.94 percentage points
ARC Challenge	18.17%	23.21%	1.23 percentage points
PIQA	53.86%	51.47%	1.17 percentage points
HellaSwag	25.90%	24.97%	0.43 percentage points

Machine-readable scores, settings, dataset revisions and verification.

To reproduce the standard scores, use Python 3.12 in a fresh environment, download evaluation/reproduce_standard.py, and run:

pip install 'torch==2.11.0' 'transformers==5.3.0' 'tokenizers==0.22.2' 'lm_eval[hf]==0.4.13'
python reproduce_standard.py --device cuda:0 --output e01-standard-results

The script loads the pinned E01 revision with trust_remote_code=True, validates the canonical task hashes, and pins all dataset revisions. It writes complete results and per-question records. --prepare-only validates setup and writes the command without inference. Its generated task definitions and scoring arguments were checked against the completed run; a second full run was not needed for this packaging check. CPU execution is available with --device cpu; hardware/runtime changes can affect floating-point results.

BananaMind Base Bench 1.1

Overall Elo: 842 · Accuracy: 106/350 (30.29%) · Weighted accuracy: 28.82%.

Full 350-example run using the unchanged official runner. FP32 on CPU, batch size 2, 2 threads, 2,048-token context, no BOS, no chat template, and mean conditional token log-probability per continuation. No examples were omitted and no contexts were truncated. Random-choice accuracy for this four-choice benchmark is 25%; the current model remains weak overall. The Elo is this benchmark's fixed-item rating, not a cross-benchmark intelligence index.

Category	Correct	Accuracy	Elo
Language Completion	30/50	60.00%	993
Commonsense	15/50	30.00%	772
World Knowledge	15/50	30.00%	791
Context Tracking	11/50	22.00%	740
Quantitative	16/50	32.00%	925
Logical Reasoning	14/50	28.00%	939
Code Completion	5/50	10.00%	729

Machine-readable aggregate results and verification. Model revision: 1e9760c8a1cc0a41f2461d49b5f12d7e4148a2d5. Dataset/runner revision: d4aade51312889e8580963e1ce960c6eaef1a450. Dataset SHA-256: 2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21. Measured runtime: torch 2.14.0+cpu, transformers 5.3.0, tokenizers 0.22.2.

To reproduce the protocol, obtain benchmark.py from the pinned dataset revision, accept the dataset's evaluation-only access terms, and run:

python benchmark.py \
  --model SlayerLab/fabryka-english-base-250m-e01 \
  --model-revision 1e9760c8a1cc0a41f2461d49b5f12d7e4148a2d5 \
  --dataset-revision d4aade51312889e8580963e1ce960c6eaef1a450 \
  --device cpu --dtype float32 --threads 2 --batch-size 2 \
  --out-dir runs/fabryka-e01-bananamind-1.1

Results are self-reported from the official runner, with complete record coverage, choice/weight arithmetic, and Elo aggregation checked. This is not an independent replication. Dataset questions and answers are not redistributed here.

ArithMark and INT

ArithMark 3: 313/1,000 = 31.30%, using the official runner's mean-token-log-likelihood continuation scoring. CUDA FP32, batch 16 questions (64 continuations), no BOS, 1,024-token evaluation limit, zero truncated examples. Full per-choice predictions, token normalization and category/overall aggregation were verified.

ArithMark 2: 634/2,500 = 25.36%, using the official runner's raw continuation log likelihood. CUDA BF16, batch 16 questions (64 continuations), no BOS, 1,024-token evaluation limit, zero truncated examples. Official full-run completion, source/runner identities and aggregate arithmetic were checked; this runner does not save per-item predictions, so no independent item-level rescore is claimed.

Runtime: torch 2.11.0+cu130, transformers 5.3.0, one RTX 3090. Pinned dataset/runner revisions:

• ArithMark 3: AxiomicLabs/Arithmark-3.0@6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746
• ArithMark 2: AxiomicLabs/ArithMark-2.0@5bd986c7305dcccec88432d630aae1ac9280a71f

INT = 2.7679707593, calculated from unrounded standard normalized accuracy scores and ArithMark 3. Following the Open SLM formula, with inputs in percentages:

N(s,c) = 100 * (s-c) / (100-c)
INT = [N(HellaSwag,25) + N((ARC_Easy+ARC_Challenge)/2,25)
       + N(PIQA,50) + 0.65*N(ArithMark_3,25)] / 3.65

ArithMark 3 verification, ArithMark 2 results, INT inputs and calculation.

The separate seven-task continuous diagnostic is complete; its scoring differs from standard lm-eval and is not used for INT. Instruct Bench is a distinct suite and has not been evaluated for E01; Base Bench Elo cannot be compared to Instruct Bench Elo.

Near-duplicate grouping and the final held-out/benchmark exclusion union were not complete before E01 training. Its scores cannot be presented as an uncontaminated final evaluation. Generation probes show repetition, weak semantic coherence, and failure to follow the prompt. No useful capability, commercial suitability, safety, or state-of-the-art claim is made.

Training

36 layers, hidden width 768, 6 query heads, 2 KV heads, head dimension 128, FFN width 3040, ReLU², parameter-free QK normalization, tied embeddings, 32,768-token vocabulary, and 2,048-token context. Hidden matrices use Muon; embeddings and remaining parameters use AdamW. Learning rate 0.001 with 5M-target warmup; effective batch 262,144 prediction targets, BF16 compute with FP32 master weights, training/data seed 0. Training used one RTX 3090. Final minibatch training loss was 5.39894075; this is not held-out performance.

Training input was a biased 100,000-document ClimbMix cohort: the first 5,000 retained documents from each of 20 audited shards. The source mirror was pinned at gvlassis/ClimbMix@15905979e830dc1a9dc91ce2587a9c2b8fa62424. This sample does not reproduce full-release mixture weights. The tokenizer was reused from the pinned Plain ClimbMix 268M reference, with two chat-special IDs renamed to EOS and PAD; ordinary token IDs were preserved.

Load and generate

Install the versions in requirements.txt. This repository includes the custom implementation required for ReLU² and QK normalization. Inspect model.py and recipe.py before enabling trust_remote_code.

from transformers import AutoTokenizer, AutoModelForCausalLM

# Use the published repository, or the downloaded package directory locally.
model_id = "SlayerLab/fabryka-english-base-250m-e01"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
inputs = tokenizer("The purpose of science is to", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The packaged tokenizer adds BOS once by default and does not append EOS to generation prompts. add_special_tokens=False preserves the original ordinary-text tokenization. Weights are unchanged from the verified FP32 training export. lineage.json records the training checkpoint and recipe, data, tokenizer, and code identities. packaging.json records packaging changes.

The packaged loader includes a compatibility hook for Transformers 5.3 remote-code loading: it preserves the missing output-head key until the loader ties it to the input embeddings. This does not change trained weights or forward computation.

BananaMind Instruct Bench 1.1 — E01 base

The complete official 300-task evaluation scored 163 overall Elo, 1/300 passed (0.33%), and 0.35% weighted score. This is the original base checkpoint.

Category	Passed	Elo
General	0/120	129
Multi-turn	0/75	318
System prompts	1/60	516
Context recall	0/30	537
Code	0/15	667

The experimental SFT branch scored 296 Elo, 5/300 overall and 0/15 code. Base uses the official runner's Alpaca instruction fallback because it has no chat template; SFT uses its native chat template. This comparison includes prompt-format differences and does not isolate the effect of weight updates. Neither result establishes useful instruction following or coding ability. Category Elo includes a rating prior; nonzero Elo does not imply any passed tasks.

Protocol: unmodified official runner, dataset BananaMind/BananaMind-Instruct-Bench-1.1 at revision 40494cb4a9224bfd78722968efd2bff440e08186, RTX 3090, BF16, greedy generation, seed 42, repetition penalty 1.1, KV cache enabled, model context 2048 and official per-item generation limits. All 300 ordered item identities, prompts, scoring metadata, pass judgments and aggregates were verified against the pinned dataset and runner. Code grading checks syntax and patterns; it does not execute generated programs.

Access was obtained after pretraining. No pretraining-set decontamination against this Instruct benchmark is established by this evaluation. Gated task contents and model transcripts are not redistributed. Instruct Elo is separate from Base Bench Elo and INT.

Aggregate Instruct results and verification