Susu — Super small unit

Susu is a 10,228,480-parameter experimental Chinese conversational language model. This is a reproducible research checkpoint, not a fluent or reliable chatbot. It still produces malformed Chinese, irrelevant responses, repetition and incorrect contextual recall. No smallest-model, state-of-the-art or benchmark leadership claim is made.

中文：Susu 的名字意为 Super small unit。本次公开的是小模型训练实验与失败分析，尚未达到稳定自然聊天和可靠上下文记忆的目标。

Use

Download this repository, then run:

pip install -r requirements.txt
python run_chat.py
python run_chat.py --device cuda --prompt "你好，小素。"

Use /reset to clear the in-process conversation and /exit to quit. CPU is the default. CUDA requires a compatible PyTorch installation; the training workstation used PyTorch 2.10.0+cu130 and RTX 5090. No remote teacher or API is used by this script. This custom PyTorch architecture is not registered with Transformers AutoModel and does not promise hosted inference-provider support.

Architecture and artifact

Item	Value
Total parameters, including tied embedding/output	10,228,480
Byte-level BPE vocabulary	6,144
Transformer blocks	5
Hidden / feed-forward width	384 / 1,024
Query / KV heads	6 / 2
Training and supported inference context	512 tokens
Export	FP32 safetensors, exact tensor round-trip verified

The configuration permits a longer architectural context, but training and this inference runner use 512 tokens. Oldest complete turns are dropped to fit. This is prompt-history conditioning, not persistent memory. Quantization is not included. The runtime source is provided locally; trust_remote_code is not required.

Final training and selection

The final consolidation used 740 approved training conversations and 76 validation conversations, batch 4, accumulation 2, learning rate 3e-5, seed 41474, a 120-step ceiling and validation every 20 steps. It stopped after 60 steps with no improvement. The selected best checkpoint was step 0 of this final stage, identical to the prior stage's best step 160. We did not publish the worse final-step weights as an improvement.

Final-stage processed tokens: 41,130; supervised tokens: 25,140; training-loop time: 1.972 s; mean training loss: 1.3622; best validation loss: 3.01629; training-loop throughput: 20,852 tokens/s. Total stage wall time was 10.933 s, including validation and checkpoint work. Process peak allocated CUDA memory was 377,634,304 bytes. Whole-device peak utilization was 21%, maximum sampled temperature 40°C and peak device-used memory 14,083 MiB; these device figures include other processes. These are final-stage metrics, not lifetime training cost or unique-corpus size.

Frozen evaluation: limitations are the result

After selection, one set of 12 five-turn conversations was authored and run once: six ordinary conversations and six contextual-recall cases. Sampling used temperature 0.7, top-p 0.9, seed 928371 and a 48-new-token cap. No training followed this evaluation. Prompts were authored after the freeze, but common phrases may overlap training; this is not an externally curated benchmark or independent blind test.

• EOS termination: 57/60 responses.
• Repeated four-character substring at least three times: 3/60 responses. This narrow detector misses other repetitive behavior.
• Final-turn memory keyword matches: 2/6; stricter semantic review: 1/6. The other keyword hit added an unsupported time and repeated until the length cap.
• All 60 responses were read by the coding assistant. Multiple irrelevant and malformed responses remain; no calibrated human fluency score was collected.

All outputs, including failures, are in EVALUATION.md and evaluation.json. See PAPER.zh-CN.md for the technical report draft.

Data, teacher and rights

Language pretraining used filtered LCCC and XDailyDialog-derived prose. Later SFT includes original persona/dialogue seeds, locally generated and reviewed teacher responses, and refreshed responses to licensed source prompts. Historical teacher versions were mixed; later generations used Ornith 1.5 9B through the local SUAI service. This release is sequence-level synthetic-response training, not verified logit distillation. DFlash and 300 tokens/s teacher decoding are not established by this experiment.

Pending BELLE candidates were not added to this final training. Exact and linked-group isolation was implemented, but original LCCC conversation-tree/user identifiers were absent, so perfect original-tree isolation cannot be claimed. The 76-conversation validation set was used repeatedly during development.

Use is limited to noncommercial research under LICENSE.md. Source-specific restrictions and attributions remain relevant; no blanket MIT/Apache permission for all training data or weights is asserted. Raw training conversations and optimizer states are not redistributed. See DATA_PROVENANCE.md.

Reproduction boundary

training_source.zip contains the custom architecture, tokenizer, training/preparation code and tests from the local source snapshot. The raw corpora and the complete historical experiment environment are not bundled, so this is an inference-reproducible checkpoint plus research evidence, not a claim of fully reproducible end-to-end corpus reconstruction. Model development and this documentation were assisted by an AI coding agent. The report is an unpublished draft, not a peer-reviewed paper.