Meridian-MRM/Zedev-134m-Fudas

Zedev-Fudas-134M

Base causal language model with 134.47M parameters.

---

Architecture

Field	Value
Parameters	134.47M
Layers	26
Hidden size	640
Attention	GQA (10 Query / 2 KV heads, head_dim 64)
Intermediate size	1536 SwiGLU
Norm	RMSNorm (eps 1e-5) + per-head QK-Norm
RoPE theta	10000.0
Context length	1024
Vocab size	50304 (50261 used, padded to /128)
Tokenizer	GPT-2 BPE
Embeddings	Tied (lm_head == embed_tokens)
HF Layout	Qwen3ForCausalLM

Training & Corpus

Parameter	Value
Training Time	22 hours, 36 minutes
Trained on Context Length	1024
Tokens seen	20.788B~

Pretraining Data

Source	Tokens	License
Wikipedia (20231101.en)	4.585B	CC BY-SA 4.0
Gutenberg	4.500B	Public Domain
DCLM-baseline s1	4.384B	CC-BY-4.0
DCLM-baseline s2	4.261B	CC-BY-4.0
OpenHermes	0.385B	Apache 2.0
Total	18.115B

Benchmarks

Evaluated with lm-evaluation-harness (0-shot, float16, batch size 16).

Benchmark	Metric	Zedev	Fudas	Improvement
ARC-Challenge	acc_norm	25.51%	26.28%	🟢 +0.77%
	acc	23.12%	22.70%	🔴 -0.42%
ARC-Easy	acc	45.62%	46.38%	🟢 +0.76%
	acc_norm	41.67%	41.58%	🔴 -0.09%
BoolQ	acc	45.84%	52.35%	🟢 +6.51%
HellaSwag	acc_norm	31.17%	31.39%	🟢 +0.22%
	acc	28.72%	28.91%	🟢 +0.19%
OpenBookQA	acc_norm	29.00%	30.00%	🟢 +1.00%
	acc	17.40%	17.20%	🔴 -0.20%
PIQA	acc	61.43%	61.21%	🔴 -0.22%
	acc_norm	60.94%	61.92%	🟢 +0.98%
WinoGrande	acc	49.64%	48.22%	🔴 -1.42%
BLiMP	acc	80.32%	81.69%	🟢 +1.37%
WikiText-2	word_perplexity (↓)	29.67	29.28	🟢 -0.39
	byte_perplexity (↓)	1.89	1.88	🟢 -0.01
	bits_per_byte (↓)	0.9146	0.9111	🟢 -0.0035

Special Tokens

Token	ID
<|endoftext|>	50256 (EOS/BOS/PAD)
<|im_start|>	50257
<|im_end|>	50258
<think>	50259
</think>	50260

Usage

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Meridian-MRM/Zedev-Fudas-134m")
model = AutoModelForCausalLM.from_pretrained(
    "Meridian-MRM/Zedev-Fudas-134m", torch_dtype=torch.float16
).cuda().eval()

prompt = "The old lighthouse keeper walked to the edge of the cliff and"
ids = tok(prompt, return_tensors="pt").input_ids.cuda()

with torch.no_grad():
    out = model.generate(
        ids, max_new_tokens=128, do_sample=True,
        temperature=0.7, top_k=40, top_p=0.95,
        repetition_penalty=1.15, pad_token_id=50256,
    )
print(tok.decode(out[0], skip_special_tokens=True))

llama.cpp f16.gguf

./llama-cli -m zedev-fudas-134m-f16.gguf \
    -p "The old lighthouse keeper walked to the edge of the cliff and" \
    -n 128 --temp 0.7 --top-k 40 --top-p 0.95 --repeat-penalty 1.15

Limitations

• English only.

License

Apache 2.0.

Credits

• JAX / Flax / Optax (Google)
• Wikipedia (CC BY-SA 4.0)
• Project Gutenberg (Public Domain)
• DCLM Dataset (CC-BY-4.0)
• OpenHermes (Apache 2.0)
• GPT-2 BPE (OpenAI)
• Hugging Face, llama.cpp
• Steve Carell

Citation

@misc{zedevfudas134m,
  title  = {Zedev-Fudas-134M},
  author = {Meridian-MRM},
  howpublished = {\url{https://huggingface.co/Meridian-MRM/zedev-fudas-134m}}
}