NeuronUz/NeuronLM-103M-SFT-v1

NeuronLM 103.5M — Uzbek SFT v1

This is a first supervised fine-tuning baseline, not a reliable general assistant. The run completed and the model is technically loadable, but evaluation shows substantial repetition, weak instruction following and poor arithmetic.

Training

Base: the newly pretrained 103,488,128-parameter Uzbek model, exported to native Qwen3ForCausalLM. Its original pretraining checkpoint and export are preserved. SFT used this independent project, not the older NeuronLM SFT implementation.

• 78,857 training conversations and 1,626 validation conversations.
• Sources: UAzimov/uzbek-instruct-llm, behbudiy/alpaca-cleaned-uz, MLDataScientist/SlimOrca-Dedup-Uzbek-cleaned, and NeuronUz/gsm8k-uz.
• 7,251 GSM8K-Uz training examples; 166 validation examples from its train split.
• 16,964,817 input tokens; 10,162,091 assistant targets in the training data.
• One full fine-tuning epoch, 1,233 updates; learning rate 2e-5, cosine decay, 3% warmup, batch 8 × accumulation 8, BF16 computation and FP32 parameters.
• Context 1,024; maximum-length conversations dropped rather than truncated.
• Assistant content and end-of-turn supervised; system/user/header/PAD masked.
• Export selects checkpoint 1200, with lowest observed validation loss 2.351750. This assistant-only loss is on different data from the pretraining loss.

Source revisions and file hashes are in the project's data provenance files. The datasets include machine translations and residual incorrect answers. Structural checks do not establish factual correctness of every training example.

Evaluation

The official 1,308-example GSM8K-Uz test split was excluded from training and checkpoint selection. Exact normalized question overlap was checked; semantic paraphrase overlap across public datasets is not ruled out.

Selected checkpoint, vLLM BF16, greedy decoding, maximum 256 new tokens:

• Strict final-answer accuracy: 18 / 1308 = 1.38%.
• Extraction: last explicit Javob: numeric answer; absent marker is incorrect.
• 740 generations stopped; 568 reached the token limit.

An initial small-batch Transformers evaluation was abandoned for speed before an aggregate result was produced. The complete reported evaluation uses vLLM on the unchanged selected checkpoint. No model was selected using that test score.

On ten separate qualitative Uzbek chat probes, 6/10 emitted an end-of-turn token within 96 new tokens. These ten prompts are not a representative accuracy benchmark. Observed failures include answering 3 + 2 as 6, listing multiple cities for the capital question, violating requested output format, and repeating phrases. All ten raw outputs are provided in the run report.

Technical verification and deployment

• All weights finite; input/output embeddings remain tied.
• CPU FP32 cached decoding agrees with full forward logits within tolerance.
• Four native vLLM chat cases match CPU Transformers chat-template token IDs and greedy continuation tokens, including a multi-turn case.
• The complete math evaluation also exercised vLLM BF16 inference.
• Vocabulary remains 48,000. Chat start 4, end 5, PAD 3, BOS 0.
• Saved tokenizer includes the custom Uzbek normalizer/decoder and chat template.
• Config permits 2,048 positions; deployment validation used a 1,024-token limit.

No FPGA execution, quantization quality or GGUF tokenizer compatibility is claimed. This repository publishes the selected SFT v1 checkpoint.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "NeuronUz/NeuronLM-103M-SFT-v1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Salom!"}],
    add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=96, do_sample=False, eos_token_id=5, pad_token_id=3)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Native Qwen3ForCausalLM; no trust_remote_code required. Tested with Transformers 5.14.1 and vLLM 0.32.0. Use a 1,024-token deployment limit for initial testing.

vllm serve NeuronUz/NeuronLM-103M-SFT-v1 --dtype bfloat16 --max-model-len 1024

See evaluation_summary.json and source_provenance.json for recorded results and pinned source snapshots.