NULLXES-SVAROGV2

SVAROG is an experimental autoregressive language model developed by NULLXES.

This repository contains a pretrained base-model checkpoint. It is not an instruction-tuned assistant.

Architecture

Property	Value
Architecture	Decoder-only, LLaMA-inspired transformer
Parameters	131,697,408
Transformer layers	12
Hidden dimension	768
Attention heads	12
Feed-forward dimension	3,072
Position encoding	RoPE
Normalization	RMSNorm
Feed-forward activation	SwiGLU
Tokenizer	SentencePiece BPE
Vocabulary	24,000
Maximum context capacity	2,048 tokens
Training sequence length	256 tokens
Weight format	Safetensors

The repository identifier is NULLXES-SVAROGV2; the checkpoint uses the SVAROG V3 architecture configuration.

Training

The model was trained from scratch and then continued on a separate corpus.

• Initial pretraining: 3,000 optimizer steps.
• Continued pretraining: 1,000 additional optimizer steps.
• Continued-pretraining corpus: 45,000 documents from FineWeb2 Russian and Wikimedia Wikipedia Russian/English.
• Training hardware: NVIDIA Tesla T4.
• Precision: FP16 mixed precision.
• Optimizer: AdamW.

The continued-pretraining validation loss at step 1,000 was 3.9164, with perplexity 50.22. These results are specific to that validation set and are not general-purpose benchmarks.

Quick start

Install dependencies:

pip install torch transformers sentencepiece safetensors

Download this repository and run:

python inference.py --model-dir . --prompt "Искусственный интеллект — это" --max-new-tokens 64

Python usage:

from inference import load_svarog, generate

model, tokenizer = load_svarog(".", device="cpu")
print(generate(model, tokenizer, "Россия расположена", max_new_tokens=48))

The model uses a custom architecture implementation provided in model.py. Standard Hugging Face AutoModelForCausalLM.from_pretrained() loading is not configured for this release.

Chat template

chat_template.py provides role-based message serialization using textual delimiters:

• [im_start]system
• [im_start]user
• [im_start]assistant
• [im_end]

These delimiters are encoded using the existing SentencePiece vocabulary; they are not dedicated special token IDs.

The released weights have not been supervised-fine-tuned on this chat template. It is included for subsequent instruction-tuning development, not as a claim of chat capability.

Limitations

This is an early-stage base language model. Generated text may be incoherent, repetitive, factually incorrect, or unrelated to the prompt.

The model is not suitable for safety-critical decisions or reliable factual question answering. No comprehensive instruction-following, reasoning, safety, or multilingual benchmark results are claimed.

The model was trained with a maximum sequence length of 256 tokens despite the architecture supporting up to 2,048 positions. Longer-context performance has not been validated.

Data and redistribution

Training sources include FineWeb2 and Wikimedia Wikipedia. Dataset licenses, attribution requirements, and redistribution obligations must be reviewed before public distribution. This model card does not grant rights to third-party training data.

Developer

NULLXES

Experimental language-model research and development.