LexisLM-500m

LexisLM-500m is a Greek base language model with 502,193,664 parameters (0.502B), trained from random initialization on Greek text.

The full pretraining was performed on approximately one pass over 22 billion tokens. The model is pretrained-only, not instruction-tuned in bfloat16 format.

Architecture and tokenizer

The model follows a LLaMA-style decoder-only architecture with tied input/output embeddings, RMSNorm, rotary position embeddings, and SwiGLU feed-forward blocks.

Property	Value
Parameters (tied)	502,193,664
Layers	16
Hidden size	1,536
Attention heads	16
Head dimension	96
Feed-forward hidden size	4,096
Context length	4,096 tokens
Model vocabulary	32,004
RoPE theta	10,000
RMSNorm epsilon	0.00001
Dropout	0.0

The dedicated Greek byte-level BPE tokenizer has 32,000 base entries, including <unk>, <bos>, <eos>, and <pad>. It uses NFC normalization and preserves case and accents. Its training sample comprises 20 million word occurrences: 20% Greek Wikipedia and 80% FineWeb2-Greek. These percentages describe the tokenizer training sample, not the full pretraining corpus. Four additional model vocabulary entries are reserved for the append-only chat-token extension; their presence does not make this an instruction-tuned model.

Pretraining data and setup

Pretraining uses AdamW (betas 0.9 and 0.95), a peak learning rate of 0.0003 with 2,000 warmup steps and cosine decay to 0.00003, weight decay 0.1, and gradient clipping at 1.0.

Evaluation

Held-out language modeling

Metric	Value
Validation loss	2.426
Validation perplexity	11.31
Bits per byte	0.466
Bits per character	0.826

These metrics use the pretraining validation distribution (Greek Wikipedia and FineWeb2-Greek). Training-loop evaluation scores 256 windows of 4,096 tokens, using 32 batches of eight and a dedicated generator reset to seed 2026. The paper's two model sizes use different sampled windows, so their validation metrics compare the same distribution and token budget, not identical text.

Greek benchmarks

Scores are percentages, rounded to the precision reported in the paper.

Greek benchmark	Shots	Questions	Base checkpoint
ARC-Easy, raw accuracy	25	2,376	44.4%
ARC-Easy, normalized accuracy	25	2,376	41.9%
ARC-Challenge, normalized accuracy	25	1,168	25.3%
MMLU, accuracy	5	14,042	25.3%
TruthfulQA, MC1	0	817	24.7%
TruthfulQA, MC2	0	817	42.1%

These are full-set answer-likelihood evaluations on ILSP's Greek-translated ARC, MMLU, and TruthfulQA benchmarks. ARC normalized accuracy uses character-length normalization; MMLU accuracy is aggregated over questions. Demonstration selection uses seed 1337. All requested few-shot examples were used, and no questions were truncated.

Usage

This is a base text-completion model. Pretraining documents were separated by <eos>, so prepend <eos> to a prompt to start a fresh document. The tokenizer does not add it automatically.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ComputationalLinguisticsLab/LexisLM-500m"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
inputs = tokenizer("<eos>Η Ελλάδα είναι μια χώρα", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Citation

@misc{perifanos2026compactgreeklanguagemodels,
      title={Compact Greek Language Models from Scratch: Pretraining 500M- and 1B-Parameter Decoder-Only Transformers}, 
      author={Konstantinos Perifanos and Anne Laure Poite and Giannis Tsilikas and Ilias Katis and Stavros Deligiannidis and Dionysis Goutsos},
      year={2026},
      eprint={2611.10209},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2611.10209}, 
}