Flare-50M

Flare-50M is a compact 49M-parameter causal language model distributed in GGUF format. It is intended for research, experimentation, local inference, and studying the capabilities and limitations of very small language models.

The included checkpoint uses Q8_0 quantization and is approximately 53 MB.

Model details

Property	Value
Architecture	Qwen3-style causal LM
Parameters	49M (approximately 50M)
Layers	12
Hidden size	512
Attention heads	8
Key/value heads	2
Intermediate size	1,344
Vocabulary size	32,000
Maximum context	1,024 tokens
Quantization	GGUF Q8_0
Checkpoint	50m-q8_0.gguf

Intended use

Flare-50M may be useful for:

• experiments with highly constrained language models;
• local CPU inference;
• language-model training and evaluation research;
• testing compression, quantization, and fine-tuning pipelines;
• basic English completion and syntax experiments.

The model is not intended to be a general-purpose assistant or a reliable source of factual information.

Training data

Flare-50M was trained using data from:

• Project Axiom 1.0 — Opus 4.7, Kimi 2.6, GLM 5.2, DeepSeek 4, Mythos 5, Fable 5, and Qwen 3.7

Project Axiom is an English synthetic reasoning and instruction corpus containing chain-of-thought traces, multi-turn conversations, mathematical reasoning, code, tool trajectories, and other structured problem-solving examples. The dataset repository is published under the MIT license.

The exact subset, number of training tokens, preprocessing procedure, and number of epochs used for Flare-50M have not been documented in this model card. The dataset contains synthetic model-generated material, so its outputs may include incorrect reasoning, factual errors, stylistic artifacts, and biases inherited from the teacher models.

Usage

Install a recent version of llama-cpp-python. Older releases may not recognize the Qwen3 GGUF architecture.

pip install "llama-cpp-python>=0.3.34"

from llama_cpp import Llama

model = Llama(
    model_path="50m-q8_0.gguf",
    n_ctx=1024,
    verbose=False,
)

result = model(
    "The future of small language models is",
    max_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repeat_penalty=1.1,
)

print(result["choices"][0]["text"])

Do not set n_ctx above 1,024 unless you are intentionally testing unsupported context extrapolation.

Evaluation

The following results were measured on a fixed random sample of 1,000 examples per task. WikiText-2 used approximately 100,000 tokens. Candidate-based tasks were evaluated using conditional continuation log-likelihood instead of asking the model to generate an answer letter.

Benchmark	Metric	Flare-50M	Random baseline
WikiText-2	Perplexity ↓	340.38	—
LAMBADA OpenAI	Last-word accuracy ↑	7.5%	—
BLiMP	Accuracy ↑	61.8%	50%
HellaSwag	Accuracy ↑	25.8%	25%
PIQA	Accuracy ↑	53.3%	50%
ARC-Easy	Accuracy ↑	29.9%	approximately 25%

Additional length-normalized scores:

Benchmark	Normalized accuracy
BLiMP	59.0%
HellaSwag	25.8%
PIQA	52.8%
ARC-Easy	30.5%

Evaluation notes

• These results were produced with a custom llama-cpp-python evaluation script and may not be directly comparable to results from the EleutherAI LM Evaluation Harness or public leaderboards.
• Perplexity is tokenizer-dependent and should only be compared under the same tokenization and evaluation procedure.
• BLiMP is the clearest positive result and suggests that the model learned some English syntactic structure.
• HellaSwag is at approximately random performance. PIQA and ARC-Easy show only small improvements over their respective chance levels.
• A direct zero-shot MMLU answer-letter test was not considered valid: the model selected A for every item, demonstrating strong answer-token bias rather than measurable multiple-choice knowledge.

Limitations

Flare-50M is an extremely small model by modern standards. Users should expect:

• limited factual knowledge;
• weak multi-step reasoning and instruction following;
• poor performance on long or complex prompts;
• repetition, contradictions, and incoherent generations;
• a hard context limit of 1,024 tokens;
• biases and inaccuracies inherited from its training data.

Model outputs should not be used for medical, legal, financial, safety-critical, or other high-stakes decisions.

Reproducibility

Evaluation configuration:

seed: 1234
quantization: Q8_0
context length: 1024
examples per task: 1000
WikiText-2 evaluated tokens: 99903
scoring: conditional continuation log-likelihood

Dataset splits:

• WikiText-2 Raw: test
• LAMBADA OpenAI: test
• BLiMP: benchmark train splits (the dataset's evaluation examples)
• HellaSwag: validation
• PIQA: validation
• ARC-Easy: test