AlbinoRacoon/martin-1.3b-base

M.A.R.T.I.N.-1.3B

A 1.34B-parameter language model trained entirely from scratch on Slovenian and English text, Python code and ROS2 robotics code.

M.A.R.T.I.N. is a research project about building a small, domain-focused language model for the ROS2 robotics ecosystem and for Slovenian, a low-resource language, within an academic compute budget. This repository holds the base model at an intermediate checkpoint (≈ 31.8 B tokens of a planned 100 B). It continues text; it is not a chat assistant and not ready for production.

Povzetek. M.A.R.T.I.N.-1.3B je jezikovni model z 1,34 milijarde parametrov, ki smo ga naučili povsem od začetka, brez predhodno naučenih uteži. Učni korpus združuje slovensko in angleško besedilo, kodo v Pythonu ter kodo za robotski sistem ROS2, skupaj približno 31,8 milijarde tokenov. Gre za osnovni model, ki nadaljuje besedilo, ne za pogovornega asistenta. Zaradi stroškov smo učenje zavestno ustavili pri približno tretjini načrtovanih 100 milijard tokenov, zato je model namenjen raziskavam in ne produkcijski rabi.

At a glance

Type	Dense decoder-only transformer (Llama architecture), base model
Parameters	1,338,853,376
Layers / hidden size / heads	20 / 2048 / 16 (MLP 5632, RoPE θ = 10,000, tied embeddings)
Context length	4,096 tokens
Initialization	Random. Not distilled from, and not continued from, any other model
Training	≈ 31.8 B tokens (step 1,939,850), about 31.8 % of the planned 100 B; stopped early for cost reasons, before any learning-rate decay
Compute	≈ 740 GPU-hours on NVIDIA A100 40 GB (2 GPUs per job), HPC Vega (IZUM, Slovenia)
Tokenizer	Reused from Qwen/Qwen3-Coder-30B-A3B-Instruct (151,669 tokens; embedding matrix padded to 151,936). Not trained from scratch.
Languages	Slovenian, English, Python, ROS/ROS2 code
Weights	bfloat16 safetensors, 2.7 GB
License	Apache-2.0 (see the note on training-data terms below)

Quick start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "AlbinoRacoon/martin-1.3b-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16)

inputs = tok("Slovenija je država v srednji Evropi, ki", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=80)
print(tok.decode(output[0], skip_special_tokens=True))

generation_config.json already sets the decoding that worked best for us: sampling with temperature=0.8, top_p=0.9, repetition_penalty=1.2. In our tests greedy decoding tends to repeat sentences.

An unedited sample (sampling, seed 0):

Slovenija je država v srednji Evropi, ki spada med države, kjer živijo številne etnično in kulturne razlike. Državo Sloveniji predstavlja približno 68% prebivalcev. Leta 1950 se je število prebivalcev Sloven…

The text is fluent and grammatical, but the content is invented. This is typical of the model.

Training data

Sampling mixture of the planned run:

Source	Share
HuggingFaceFW/fineweb-edu (English web text)	50 %
bigcode/starcoderdata, Python subset	20 %
HuggingFaceFW/fineweb-2, slv_Latn (Slovenian web text)	14 %
cjvt/Wikipedia-translated (machine-translated Wikipedia in Slovenian)	8 %
cjvt Nemotron translated subsets (Slovenian)	3 %
Own ROS/ROS2 corpus from GitHub repositories (only Apache-2.0, MIT and BSD-2/3 licensed), oversampled	5 %

Slovenian therefore makes up about a quarter of the mixture. The upstream datasets keep their own terms (for example ODC-By for FineWeb and the StarCoder data terms). The weights are released under Apache-2.0, but you are responsible for checking that your use is compatible with those terms.

What it can and cannot do

The evaluations come from a master's thesis and are small and informal. None of them is a rigorous benchmark.

What we observed

• Grammatical Slovenian and English under sampling. Text drifts off topic after one or two sentences.
• Compared with other models of about the same size (TinyLlama, Qwen2.5-1.5B, GaMS-2B) on eight prompts, it more reliably answered in the requested programming language, used the modern ROS2 API instead of legacy ROS1 rospy, and produced fewer invented Slovenian words.
• Plausible Python/ROS2 syntax, but not reliable programs. It did not produce structurally complete ROS2 nodes in our formal tests, and neither did any of the other small models. Those tests used the earlier generation config (EOS id 2) and were not repeated after the fix described below.

What it does badly

• Worst or among the worst of the four small models on 7 of 8 general (non-ROS) tasks: facts, arithmetic, code logic, following instructions.
• Outputs often contain fabricated facts. Slovenian fluency does not mean factual accuracy.

Human preference (pilot). In a blind A/B arena against GaMS-2B (4 voters, very small sample) it won 7, lost 9 and tied 9 comparisons (4 skipped). This is indicative only.

Intended use and limitations

Intended: research on pretraining low-resource-language and domain-specific models under a small budget, and a starting point for continued pretraining or instruction tuning. Instruction tuning is the clearly missing step.

Not suitable for: assistants, factual question answering, safety-critical use, controlling robots, or any production deployment. Generated code must be treated as unverified.

The model can produce incorrect, fabricated, repetitive, biased or unsafe text. The checkpoint is under-trained and was saved while the learning rate was still high.

Technical notes

• End of document. Documents were separated by <|im_end|> (id 151645) during training, and generation_config.json uses it as the EOS token. An earlier research configuration used id 2, which is the character # in this vocabulary, and has been corrected here.
• Tokenizer. The chat_template was removed from tokenizer_config.json, because the model was never trained on a chat format. With some transformers versions a warning about a Mistral regex appears. For this Qwen tokenizer it did not change the token ids on the strings we compared.
• Software. Converted and tested with transformers 4.57.6. The architecture is a standard LlamaForCausalLM.
• Checksum. model.safetensors SHA-256 b234fa096768c6620fffac903237e261183b1d450b37efd2849a25e71776e64b

Acknowledgements and related models

Developed as part of a master's thesis project at UM FERI (University of Maribor, Slovenia) for the FarmBeast agricultural robot. Computing resources were provided by HPC Vega (IZUM). The tokenizer comes from the Qwen team (Apache-2.0).

Companion model from the same project: AlbinoRacoon/velik-martin-27b-lora, a LoRA adapter for Qwen3.6-27B.