OrisTeam/Aster-0.5B

MultiBLiMP PL: 98.04% · Blind-6: 1st / 6 (avg. rank 2.433) · vs GoLLeM-v4: 60.17% (derived from 6-way rankings) · vs Bielik-1.5B: 28.25% (blind 1v1)

Aster-0.5B — About

Aster-0.5B is a 520M-parameter Polish decoder-only causal language model, pretrained from scratch as part of the Aster model family. Its primary purpose is to provide a foundation for continued pretraining, domain adaptation and task-specific fine-tuning, rather than a ready-to-use chatbot.

Stage 1 is complete: 16 billion processed training tokens and 244,141 optimizer updates. The release contains the final Stage 1 base weights, with BF16 as the default and original-precision FP32 available separately.

A base model, not a finished assistant. Aster-0.5B has not been instruction-tuned. It can lose coherence, repeat itself or invent facts after even one sentence. Fine-tuning, downstream evaluation and safeguards are necessary before practical use.

Architecture & Training

Property	Specification
Parameters	520,276,608 (~520M); tied input/output embeddings
Architecture	Decoder-only Transformer; 22 layers, hidden size 1,152
Attention	Grouped-query attention (GQA): 18 Q heads / 2 KV heads, head dimension 64
Transformer block	Pre-RMSNorm, RoPE, SwiGLU, bias-free linear layers
FFN dimension	5,504
Tokenizer	aster-v1-05, 32,000 tokens
Sequence length	2,048 trained, 4,096 configured positions
Training	AdamW, weight decay 0.1; peak LR 2e-4, final LR 2e-5
Schedule	Warmup, stable phase, cosine decay over last 20% of token budget
Effective batch	65,536 tokens/update
Training precision	FP32 weights and optimizer state; BF16 autocast
Completed training	244,141 updates / ~16B processed tokens

The 4,096-position setting is an architectural limit, not a validated 4,096-token quality claim. Stage 1 trained on 2,048-token sequences; use that as the conservative context budget including output.

Tokenizer

aster-v1-05 is the release designation for the same 32K tokenizer and token-ID mapping used during Stage 1 training. No vocabulary retraining or remapping was performed for this release. Verified special IDs are BOS 1, EOS 2 and PAD 3; the tokenizer metadata specifies a length of 2048.

For raw base-model completion, use add_special_tokens=False to reproduce the prefix format used in the release tests. The model does not use a chat template.

Training Data

Stage 1 used Polish-language text from the SlayerLab GoLLeM Polish corpus. The training preparation included normalization, Polish-language verification, deduplication, preliminary cleaning and filtering using Horyzont 2.0 procedures. These filters were applied to improve data quality; they are not a guarantee that every document is reliable or free of harmful material.

Property	Stage 1
Main source	SlayerLab/gollem-corpus-16b-pl
Language	Polish
Training sampling	60% web-derived / 40% curated
Processed exposure	~16 billion tokens, counted using the training tokenizer

The 60/40 figures describe the sampling configuration, not necessarily the corpus's physical proportions. The processed-token count does not represent unique text. Source datasets may have separate licensing and provenance conditions.

Evaluation

These benchmarks evaluate base-model behavior. Grammatical preference, six-way relative rankings and a separately judged A/B test measure different properties and must not be interpreted as interchangeable accuracy measures.

MultiBLiMP PL — Grammatical Preference

MultiBLiMP evaluates preference for the grammatical member of each of 3,272 Polish minimal sentence pairs. The higher summed token log-likelihood determines the prediction; no chat template is used.

Aster Stage 1 progression

Aster checkpoint	Training tokens	Correct / total	Accuracy	PL rank score	Mean margin
update_00095000.pt	6.23B	3,155 / 3,272	96.42%	+0.9285	+6.0860
update_00167500.pt	10.98B	3,161 / 3,272	96.61%	+0.9322	+6.2121
update_00244141.pt — Stage 1 Final	16.00B	3,208 / 3,272	98.04%	+0.9609	+6.8420

Reference results — Polish MultiBLiMP

Model	Parameters	Accuracy	Reported by
Poziomka-Baza-2026-09-05	4.1B	99.11%	Self-reported
GoLLeM-v4-250M-PL	248M	98.47%	Fabryka AI
GoLLeM-v6-250M	252.8M	98.14%	Fabryka AI
Aster-0.5B — Stage 1 Final	520M	98.04%	OrisTeam
Koliber-v1.0-Base	126M	96.39%	Self-reported
Koliber-v1.1-Base-Preview	126M	96.33%	Fabryka AI

These reference results originate from separately reported evaluations and were not all reproduced under one shared environment. MultiBLiMP measures grammatical preference on minimal pairs, not reliable long-form writing, factual accuracy, or downstream usefulness.

Blind-6 — Polish Base-Model Generation

The main generation benchmark compares six models on the same 600 Polish text prefixes, organized into 12 categories of 50 prompts. Each model supplied one completion per prompt (3,600 generations).

Generation and judging protocol

Setting	Value
Compared models	Aster-0.5B Stage 1 Final, GoLLeM-v4-Final, Koliber-1.2, APT3-0.5B, APT3-1B, Qra-1B
Prompt format	Raw text continuation, without chat templates or system prompts
Decoding	Sampling; temperature 0.8; top-p 0.95; top-k disabled; repetition penalty 1.0
Maximum output	160 new tokens per prompt
Generation seed	20261021 (per-prompt deterministic sampling)
Anonymous presentation	A–F labels independently randomized for each prompt; seed 20261022
Judge	GPT-6 (High reasoning), as identified in the evaluation session
Judging schedule	8 batches × 75 prompts, 600 judgments total
Output	Complete ranking of all six continuations, with ties allowed and an all_bad flag

The judge assessed relevance to the prefix, accuracy where applicable, reasoning coherence, Polish fluency, and unnecessary repetition without fixed criterion weights. It was instructed not to prefer model identity, longer outputs, or assistant-like formatting. The private model-to-label key was kept separate from the blinded responses until the judgments were recorded.

Judge instruction template (English) — a concise, publication-ready representation of the judging instructions; original evaluation artifacts are the authority for their exact wording:

Blind-6 — Independent Evaluation

Evaluate six anonymous Polish-language text continuations (A–F) for each
prompt. Judge relevance and task completion, factual correctness where
applicable, logical consistency, Polish fluency, coherence and repetition.
Apply criteria appropriate to the prompt rather than fixed weights.

Rank all six completions from best to worst, allowing ties for comparable
outputs. Do not favor length, assistant-style formatting or any slot label.
Mark all_bad=true only if none of the six continuations is acceptable.

Return valid structured JSON judgments with the original prompt ID,
complete A–F ranking and all_bad flag. Do not inspect the private
model-identity key, infer model names or use prior benchmark outcomes.

Overall ranking — 600 prompts

Model	1st	2nd	3rd	4th	5th	6th	Avg. rank ↓
Aster-0.5B — Stage 1 Final	224	131	98	78	46	23	2.433
GoLLeM-v4-Final	122	155	133	83	65	42	2.900
APT3-1B	101	107.5	129	114.5	76	72	3.288
Qra-1B	90	91	96	106	120	97	3.610
APT3-0.5B	41	71.5	96	128.5	132	131	4.053
Koliber-1.2	22	44	48	90	161	235	4.715

Per-category results — average rank (lower is better)

Category (50 prompts each)	Aster	GoLLeM	APT3-1B	Qra-1B	APT3-0.5B	Koliber	all_bad
Facts & Knowledge	2.080	3.240	2.680	3.560	4.440	5.000	12
Science & Technology	2.060	3.040	3.400	3.480	4.040	4.980	19
Logic & Arithmetic	3.420	3.220	3.410	3.140	3.470	4.340	37
Causality & Everyday Reasoning	2.100	3.280	3.020	4.380	3.460	4.760	6
Polish Language	2.160	2.900	4.370	3.420	3.910	4.240	42
Narrative & Coherence	2.220	2.940	2.840	4.200	3.760	5.040	2
Society & Decisions	2.400	2.660	3.360	3.660	3.960	4.960	1
Abstraction & Discourse	2.440	2.940	2.920	3.340	4.380	4.980	0
Text Editing & Transformation	2.880	2.860	4.100	3.060	4.060	4.040	31
Long Context & Coherence	3.020	3.020	3.000	3.600	3.780	4.580	19
Textual Instructions	2.460	2.320	3.240	3.280	4.560	5.140	2
Polish Culture & Context	1.960	2.380	3.120	4.200	4.820	4.520	3
Overall	2.433	2.900	3.288	3.610	4.053	4.715	174

Aster achieved the lowest average rank, 224 first-place finishes (37.3%), and the best category-level average rank in 8 of 12 categories. GoLLeM led two categories; APT3-1B and Qra-1B led one each. The mean rank includes fractional rank positions where responses tied.

Category leader	Number of categories led
Aster-0.5B	8 / 12
GoLLeM-v4-Final	2 / 12
APT3-1B	1 / 12
Qra-1B	1 / 12

Aster vs GoLLeM, derived from the 6-way rankings: 361 wins / 239 losses / 0 ties for Aster (60.17%). This is not a separately judged 1v1 experiment.

Relative preference is not the same as usable output. The judge marked 174/600 prompts (29.0%) as all_bad, meaning none of the six completions was acceptable on those prompts. All 600 prompts nevertheless remain included in the published ranking. Models in this comparison, including Aster, can produce nonsensical or fabricated content, lose the topic, or become incoherent even after a single sentence. The differences are visible in this evaluation, but they do not imply consistently reliable generation from any of the base models.

Blind 1v1 — Aster-0.5B vs Bielik-1.5B Base

A separate anonymous A/B judgment compared the same 600 Polish prefixes, using the final Aster Stage 1 checkpoint and Bielik-1.5B-v3 Base. Responses were independently anonymized and judged pairwise in A/B format, rather than inferred from Blind-6 rank positions. Bielik is a larger (~1.6B-parameter) model initialized from Qwen2.5-1.5B and further trained on approximately 292B tokens, according to its published model card. The Aster checkpoint was pretrained from scratch for 16B processed tokens, so this is not a compute- or scale-matched comparison.

A/B judge instruction template: for each prompt, select A, B, or TIE by comparing relevance, Polish fluency, factual/logical consistency and coherence. The judge was instructed not to reward verbosity or chat formatting, to preserve prompt IDs, and not to access the hidden identity key. These are reported LLM-judge preferences, not verified ground-truth labels.

Outcome	Count	Share of 600
Bielik-1.5B wins	429	71.50%
Aster-0.5B wins	168	28.00%
Ties	3	0.50%
Total	600	100%

Aster score: 28.25% when a tie is worth half a point ((168 + 0.5 × 3) / 600). Bielik score: 71.75%. These scores are not the same as the share of decisive comparisons won.

Breakdown by category

Category (50 prompts each)	Bielik wins	Aster wins	Ties
Facts & Knowledge	34	16	0
Science & Technology	38	12	0
Logic & Arithmetic	41	7	2
Causality & Everyday Reasoning	31	19	0
Polish Language	36	13	1
Narrative & Coherence	36	14	0
Society & Decisions	34	16	0
Abstraction & Discourse	36	14	0
Text Editing & Transformation	39	11	0
Long Context & Coherence	38	12	0
Textual Instructions	34	16	0
Polish Culture & Context	32	18	0
Total	429	168	3

Bielik led in all 12 categories, particularly logic and arithmetic, text editing, science, and longer-context continuation. This comparison documents a clear remaining gap to a stronger, larger base model; it is included alongside the more favorable Blind-6 result to avoid selective reporting.

Training Dynamics

Stage 1 completed 244,141 optimizer updates and processed approximately 16B training tokens. The figure plots measured training and validation cross-entropy loss against processed tokens. The full-range panel preserves the initial rapid decline; the detailed panel uses a narrower loss axis for readability.

Metric	Last recorded value
Training loss (last recorded batch)	2.4867
Validation loss	2.9452
Validation perplexity	19.02
Training updates	244,141
Processed tokens	~16.00B
Last validation update	244,000

The displayed training trend is EMA-smoothed over 151 recorded measurements; validation points are displayed as logged. The last training-loss value is an individual recorded batch, not the final EMA value. The last validation evaluation preceded the final checkpoint by 141 updates. Differences between training and validation streams should not be treated as a direct overfitting estimate without accounting for their respective data distributions.

Model Code & Inference

The repository contains the Transformers-compatible modeling_aster.py and configuration_aster.py. The default model.safetensors contains BF16 weights; the fp32/ folder contains an independently loadable FP32 model. Both use the same tokenizer and architecture.

The reference loader was tested with PyTorch 2.8.0, Transformers 5.20.2 and an NVIDIA RTX 5090. Other combinations may need additional verification. Custom model loading requires trust_remote_code=True; review the source files before enabling it.

Install and authenticate

pip install -U torch transformers safetensors huggingface_hub
hf auth login  # Required while the model repository is private

Default BF16 · text completion with KV cache

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "OrisTeam/Aster-0.5B"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prefix = "Historia Warszawy jest związana z"
inputs = tokenizer(prefix, return_tensors="pt", add_special_tokens=False).to(device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

continuation = output[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(continuation, skip_special_tokens=True))

Optional original-precision FP32

For a self-contained path to the optional FP32 weights, download the repository and load its fp32 subfolder:

import torch
from pathlib import Path
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_dir = Path(snapshot_download("OrisTeam/Aster-0.5B"))
model_dir = repo_dir / "fp32"
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
    trust_remote_code=True,
).eval()

print(model.config.model_type, next(model.parameters()).dtype)

Use KV caching for autoregressive generation. use_cache=True reuses previously computed keys and values, avoiding full-prefix recomputation for every next token. Aster uses 2 KV heads per layer for its 18 query heads and employs PyTorch SDPA with a DynamicCache implementation. The measured benefit depends on context length, hardware and batch size. The experimental StaticCache/torch.compile variant was not selected for this release.

At the time of release testing, Transformers emitted a future compatibility warning about cache_position in custom models. The tested loader and cached generation work in the validated environment; future library updates may require changes to modeling_aster.py.

Generation Examples

These are unaltered excerpts from the final BF16 release using temperature=0.8, top_p=0.95, max_new_tokens=160 and use_cache=True; sampling seeds are shown. Prefixes and continuations are separated to make the model's raw behavior clear. Excerpts are shorter than full output and have not been fact-checked.

On-topic start: science

Prefix: Współczesne badania nad energią odnawialną pokazują, że
Seed: 20261024

 jest to technologia przyszłości, ponieważ jej wykorzystanie może przynieść znaczne korzyści środowiskowe i ekonomiczne w porównaniu do tradycyjnych źródeł energii.
Jak działa energia odnawialna?
Energia odnawialna to energia, która nie pochodzi z paliw kopalnych, takich jak węgiel, ropa naftowa czy gaz.

The opening stays on topic, but the full sample repeats a heading and oversimplifies factual details.

Topic drift: Warsaw

Prefix: Warszawa jest stolicą Polski i od wieków
Seed: 20261023

 była ważnym ośrodkiem handlowym. W mieście rozwijało się rzemiosło, rolnictwo, architektura oraz sztuka.
Port w Gdańsku
W Gdańsku znajduje się port wojenny, gdzie nabrzeża są w pełni rozwinięte.

A plausible first sentence is followed by an abrupt change of subject.

Short-context failure: narrative

Prefix: Kiedy Maria znalazła stary list w szufladzie, początkowo
Seed: 20261026

 był to zwykły list. Później jednak w jego środku znajdowały się...
Składniki na 2 porcje: 50 dag ziemniaków, 1 średni jabłko, 1 szklanka mleka, 1 szklanka jogurtu naturalnego, 1 łyżeczka soku z cytryny, 1 łyżeczka żelatyny

The model switches from a story to a recipe almost immediately.

Repetition: longer continuation

Prefix: W latach dziewięćdziesiątych polskie miasta przechodziły intensywne zmiany gospodarcze i społeczne. Wpływało to na codzienne życie mieszkańców, ponieważ
Seed: 20261029

 warunki pracy i życia ulegały znacznej poprawie. W tym czasie miasta stopniowo stawały się coraz bardziej samowystarczalne i zależne od współpracy z innymi miastami i regionami.
W miastach powstają nowe miejsca pracy
W miastach powstają nowe miejsca pracy, ponieważ miasto jest siedzibą wielu firm i instytucji.

The model retains the broad topic but becomes formulaic and repetitive.

Fluent-looking text is not proof of correctness. Aster may invent historical claims, lose the topic, repeat fragments or generate nonsensical content after a single sentence. These samples illustrate model behavior, not production readiness.

Intended Use & Limitations

Intended: continued pretraining (CPT), domain adaptation, supervised fine-tuning, experiments with Polish-language generation and research on language modeling.

Not intended without substantial additional work: conversational assistance, reliable factual answering, autonomous decision-making, or direct deployment in user-facing/high-stakes systems. The model has not been instruction-tuned or safety-aligned as a chat assistant.

Fine-tuning and downstream validation are required for practical use. Strong relative benchmark results do not remove the risk of hallucinations, bias, incoherence or unsafe output. Fine-tuning alone does not guarantee safety or factual correctness.

License & Credits

Model weights: CC BY-SA 4.0. Source corpus licensing and original text rights are separate from the model-weight license; users are responsible for checking applicable upstream restrictions.

Data source: SlayerLab GoLLeM Polish corpus, prepared by SlayerLab / Arkadiusz Słota and contributors.

Development: Aleksander Ogrodzki / OrisTeam.

@misc{ogrodzki2026aster05b,
  author = {Ogrodzki, Aleksander},
  title = {Aster-0.5B: Polish Base Language Model, Stage 1 Final},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/OrisTeam/Aster-0.5B}
}