LexisLM-500m

LexisLM-500m is a Greek base language model with 502,193,664 parameters (0.502B), trained from random initialization on Greek text.

The full pretraining run completed 335,733 steps, approximately one pass over 22 billion tokens. This repository contains the best evaluated pretrained checkpoint at step 332,000, corresponding to 21.76 billion token positions (8 × 2 × 4,096 × 332,000), rather than the final training step. The model is pretrained-only, not instruction-tuned. It was exported from best.pt with 500/export_hf.py in bfloat16 format.

Architecture and tokenizer

The model follows a LLaMA-style decoder-only architecture with tied input/output embeddings, RMSNorm, rotary position embeddings, and SwiGLU feed-forward blocks.

Property	Value
Parameters (tied)	502,193,664
Layers	16
Hidden size	1,536
Attention heads	16
Head dimension	96
Feed-forward hidden size	4,096
Context length	4,096 tokens
Model vocabulary	32,004
RoPE theta	10,000
RMSNorm epsilon	0.00001
Dropout	0.0

The dedicated Greek byte-level BPE tokenizer has 32,000 base entries, including <unk>, <bos>, <eos>, and <pad>. It uses NFC normalization and preserves case and accents. Its training sample comprises 20 million word occurrences: 20% Greek Wikipedia and 80% FineWeb2-Greek. These percentages describe the tokenizer training sample, not the full pretraining corpus. Four additional model vocabulary entries are reserved for the append-only chat-token extension; their presence does not make this an instruction-tuned model.

Pretraining data and setup

The frozen training corpus contains 22,002,557,667 tokens from 34,079,145 documents, drawn primarily from Greek Wikipedia and the Greek subset of FineWeb2. Documents are separated by <eos> (ID 2). Training uses a seeded shuffle of disjoint 4,096-token blocks in a single pass, with masked padding at the end rather than repeated text.

Pretraining uses AdamW (betas 0.9 and 0.95), a peak learning rate of 0.0003 with 2,000 warmup steps and cosine decay to 0.00003, weight decay 0.1, and gradient clipping at 1.0. Each optimizer step uses 65,536 token positions (micro-batch 8 × accumulation 2 × context 4,096). Training uses bfloat16 on a single CUDA device with torch.compile.

Evaluation

All results below describe the base checkpoint at step 332,000 released in this repository.

Held-out language modeling

Metric	Value
Validation loss	2.426
Validation perplexity	11.31
Bits per byte	0.466
Bits per character	0.826

These metrics use the pretraining validation distribution (Greek Wikipedia and FineWeb2-Greek). Training-loop evaluation scores 256 windows of 4,096 tokens, using 32 batches of eight and a dedicated generator reset to seed 2026. The paper's two model sizes use different sampled windows, so their validation metrics compare the same distribution and token budget, not identical text.

Greek benchmarks

Scores are percentages, rounded to the precision reported in the paper.

Greek benchmark	Shots	Questions	Base checkpoint
ARC-Easy, raw accuracy	25	2,376	44.4%
ARC-Easy, normalized accuracy	25	2,376	41.9%
ARC-Challenge, normalized accuracy	25	1,168	25.3%
MMLU, accuracy	5	14,042	25.3%
TruthfulQA, MC1	0	817	24.7%
TruthfulQA, MC2	0	817	42.1%

These are full-set answer-likelihood evaluations on ILSP's Greek-translated ARC, MMLU, and TruthfulQA benchmarks. ARC normalized accuracy uses character-length normalization; MMLU accuracy is aggregated over questions. Demonstration selection uses seed 1337. All requested few-shot examples were used, and no questions were truncated. The base scores come from greek_500m_22bn_ctx4k/fewshot-evaluation-complete.json.

ARC-Easy accuracy is above chance, while ARC-Challenge and MMLU remain near the nominal 25% chance baseline. No uncertainty estimates or variation across training seeds are reported. Translation artifacts may affect results, and these scores are not directly comparable to English benchmark evaluations.

Usage

This is a base text-completion model. Pretraining documents were separated by <eos>, so prepend <eos> to a prompt to start a fresh document. The tokenizer does not add it automatically.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ComputationalLinguisticsLab/LexisLM-500m"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
inputs = tokenizer("<eos>Η Ελλάδα είναι μια χώρα", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(out[0], skip_special_tokens=True))