Gast-Mini

Base causal language model with 7.07M parameters, trained from scratch.

---

Architecture

Field	Value
Parameters	7.07M
Layers	16
Hidden size	192
Attention	GQA (3 Query / 1 KV head, head_dim 64)
Intermediate size	512 SwiGLU
Norm	RMSNorm (eps 1e-6) + per-head QK-Norm
RoPE theta	10000.0
Context length	512
Vocab size	4000
Tokenizer	Custom BPE (byte-level)
Embeddings	Tied (lm_head == embed_tokens)
HF Layout	Qwen3ForCausalLM

Training & Corpus

Parameter	Value
Optimizer	AdamW (beta1=0.9, beta2=0.95, wd=0.1)
LR	2e-3 (flat, 500-step warmup)
Batch	196,608 tokens/step
Precision	fp16
Tokens seen	~9B

Pretraining Data

Source	Tokens	License
FineWeb-Edu (sample/10BT)	~8.85B	ODC-By 1.0
WikiText-103-raw-v1	~0.17B	CC BY-SA 4.0
Total	~9B

Samples

Generated with llama.cpp, temperature=0.85, top_p=0.92, top_k=50, repeat_penalty=1.15.

Prompt: Artificial intelligence is

Artificial intelligence is the most efficient, and it is difficult to obtain data from human beings. It has no clearly defined or explicit identity, but only a firm assumption of a common law. The Digital Literature Society, Digital Literature Society, ABA, and Reference Library Society have been compiled to address the data collection needs of the digital literacy system. They provide advanced technical assistance in a variety of applications, including on-line communication with computers, and are now referred to as Dig

Prompt: Photosynthesis is the process by which

Photosynthesis is the process by which DNA can transfer a protein to a cell with the same function of its function. The most common function of a cell is to transfer it from one cell to another, and then to use it for a given cell or genome. The key step in creating a healthy organism is to create a bridge between living organisms that can survive. This process involves the formation of new cells in which the cells are made up of new cells. The cell contains only two cells, each one being formed into two sections, with an additional two cells having a different cell shape

Prompt: The theory of relativity states that

The theory of relativity states that the stars are all uniform in shape. For example, the star can be measured by a gravitational force, which is a small amount of mass that can be measured in terms of grams per kilogram. Similarly, the star is used to calculate its mass: This theory states that the grams of the stars are equal to 1,000,000,000 and their masses are equal to 2,500,000,000 and the masses are equal to 12,000,000.

Usage

llama.cpp

./llama-cli -m gast-7m-f16.gguf -p "The history of the Roman Empire" -n 128 --temp 0.85 --top-k 50 --top-p 0.92 --repeat-penalty 1.15

llama-cpp-python

from llama_cpp import Llama

llm = Llama(model_path="gast-7m-f16.gguf", n_ctx=512)

out = llm(
    "The history of the Roman Empire",
    max_tokens=120,
    temperature=0.85,
    top_p=0.92,
    top_k=50,
    repeat_penalty=1.15,
)
print(out["choices"][0]["text"])

Limitations

• English only.
• Base model, not instruction tuned, no chat template.
• Trained on ~9B tokens; not factual, barely coherent beyond a few words.

License

Apache 2.0.

Credits

• FineWeb-Edu - Hugging Face (ODC-By 1.0)
• WikiText-103 - Salesforce (CC BY-SA 4.0)
• llama.cpp - GGUF tooling
• Hugging Face

Citation

@misc{gastmini,
  title  = {Gast-Mini},
  author = {Meridian-MRM},
  howpublished = {https://huggingface.co/Meridian-MRM/Gast-Mini}
}