Driw0x/minimind-rocm-pytorch

MiniMind ROCm — PyTorch checkpoints

Languages: English · Français · 中文

Dense MiniMind language-model checkpoints trained by Driw0x on Windows with an AMD Radeon RX 7800 XT, using PyTorch and ROCm.

This repository collects four checkpoints covering pretraining and full-parameter supervised fine-tuning (SFT), using the Mini and complete T2T datasets provided by the MiniMind project. The models are intended for learning, experimentation, and comparison of training stages.

ROCm identifies the training backend. The files use MiniMind's native PyTorch .pth checkpoint format and load with the MiniMind implementation and tokenizer.

Upstream and training code

• Original MiniMind project: jingyaogong/minimind.
• Training fork and experiment documentation: Driw0x/minimind.
• Official Hugging Face collection: MiniMind by jingyaogong.
• Official PyTorch checkpoint repository, for reference: jingyaogong/minimind-3-pytorch.
• Official training dataset: jingyaogong/minimind_dataset.

The checkpoints in this repository are Driw0x's training outputs. Credit for the original architecture, tokenizer, training framework, and dataset preparation belongs to the upstream MiniMind project and its contributors.

Available checkpoints

All four checkpoints use the same dense architecture, with approximately 63.91 million parameters, a hidden size of 768, and 8 transformer layers.

Checkpoint	Training stage	Dataset used at this stage	Initialization	--weight argument
pretrain_mini_768.pth	Pretraining on Mini data	pretrain_t2t_mini.jsonl	From scratch	pretrain_mini
full_sft_mini_768.pth	Full-parameter SFT on Mini data	sft_t2t_mini.jsonl	pretrain_mini_768.pth	full_sft_mini
pretrain_768.pth	Pretraining on complete data	pretrain_t2t.jsonl	From scratch	pretrain
full_sft_768.pth	Full-parameter SFT on complete data	sft_t2t.jsonl	pretrain_768.pth	full_sft

mini identifies the reduced training dataset; the model architecture stays the same. full_sft means that all model parameters are fine-tuned. The two training branches remain separate: the Mini SFT checkpoint is initialized from the Mini pretraining checkpoint, while the complete-data SFT checkpoint is initialized from the complete-data pretraining checkpoint.

For assistant-style generation, start with full_sft_768.pth. Keep the pretraining checkpoints as language-modeling baselines when comparing training stages.

Model configuration

Property	Value
Architecture	Dense MiniMind causal language model
Parameters	Approximately 63.91M
Hidden size	768
Transformer layers	8
Tokenizer vocabulary	6,400 tokens
Attention heads	8 query heads, 4 key/value heads
Mixture of experts	Disabled (use_moe=0)
Checkpoint format	PyTorch .pth
Documented training hardware	AMD Radeon RX 7800 XT
Documented platform	Windows, PyTorch with ROCm

Configuration and loading instructions refer to the verified fork checkout. The exact historical Git revision for each training run was not recorded separately.

Dataset provenance

Training data comes from the upstream MiniMind dataset repository, with direct links to the four relevant JSONL files in the checkpoint table above. See the dataset card for the upstream data descriptions and source information.

The Mini checkpoints are evaluated on the Mini datasets, and the complete-data checkpoints are evaluated on their corresponding complete datasets. SHA-256 hashes of the local dataset files are recorded below; the upstream dataset revision has not been separately recorded.

Local dataset file	SHA-256
pretrain_t2t_mini.jsonl	6DD6716C84AB36897BDBFC7F88E04F4441C48C1AB7ECEE88CE0B0E7D4685560C
sft_t2t_mini.jsonl	0B188AF3C70CCD42EE6A67C0A41FF07BC4A99BD52ED5D8BD274CEE36436EF28B
pretrain_t2t.jsonl	31EFC9A6FA7430769C0E78CDE1C8EC0273AC7BBAD20614C0EE58BCCEF327CC9D
sft_t2t.jsonl	063E7CAB1DC67CED030CBEDA2CE4D816BAF2F9CCDB4CDCAA28B8781FE891E935

Training commands and lineage

The Dense checkpoints were produced with the following training commands. The Mini branch uses the Mini pretraining checkpoint as the SFT initialization, while the complete-data branch uses the complete-data pretraining checkpoint.

Mini branch

cd trainer
python train_pretrain.py --save_weight pretrain_mini
python train_full_sft.py --save_weight full_sft_mini --from_weight pretrain_mini

pretrain_t2t_mini.jsonl
        ↓
pretrain_mini_768.pth
        ↓
sft_t2t_mini.jsonl
        ↓
full_sft_mini_768.pth

Complete-data branch

cd trainer
python train_pretrain.py --data_path ../dataset/pretrain_t2t.jsonl
python train_full_sft.py --data_path ../dataset/sft_t2t.jsonl

pretrain_t2t.jsonl
        ↓
pretrain_768.pth
        ↓
sft_t2t.jsonl
        ↓
full_sft_768.pth

Training settings

All four training runs used the defaults of the supplied trainer scripts, except for the command-line arguments explicitly shown above:

Setting	Dense Pretrain (Mini and complete)	Dense Full SFT (Mini and complete)
Configured epochs	2	2
Per-device batch size	32	16
Gradient accumulation steps	8	1
Effective batch size (1 GPU)	256	16
Initial learning rate	5e-4	1e-5
Maximum sequence length	340	768
Training mixed precision	bfloat16	bfloat16
Optimizer	AdamW	AdamW
Gradient clipping	1.0	1.0
DataLoader workers	8	8
Random seed	42	42
Mixture of experts	Disabled (use_moe=0)	Disabled (use_moe=0)
torch.compile	Disabled	Disabled

The trainers save their standalone .pth model weights as float16 tensors on CPU, even though training uses BF16 autocast. Learning rate is updated during training using the trainers' get_lr(...) schedule; any optimizer LR stored in a resume checkpoint represents a saved value, not the initial LR. Two epochs is the configured training target; retained resume checkpoint step counts are not used to infer a different run configuration.

Reported evaluation

These results come from the project's diagnostic tooling. The Mini branch and complete-data branch are evaluated separately on the datasets used for their respective training stages.

Mini checkpoints

Evaluation protocol: seed=42, 256 sampled records per dataset, batch size 4, pretraining sequence-length limit 340, SFT sequence-length limit 768.

Checkpoint	Pretrain loss	Pretrain PPL	SFT loss	SFT PPL
pretrain_mini_768.pth	1.8275	6.22	2.5207	12.44
full_sft_mini_768.pth	1.9992	7.38	1.5663	4.79

SFT changes relative to Mini Pretrain:

• Pretrain loss delta: +0.1717
• SFT loss delta: -0.9543

Mini dataset sizes observed by the diagnostic:

• Pretrain: 1,270,238 samples
• SFT: 895,718 samples

Complete-data checkpoints

Evaluation protocol: seed=42, 256 sampled records per dataset, batch size 4, pretraining sequence-length limit 340, SFT sequence-length limit 768.

Checkpoint	Pretrain loss	Pretrain PPL	SFT loss	SFT PPL
pretrain_768.pth	1.6362	5.14	3.0943	22.07
full_sft_768.pth	1.8598	6.42	1.3369	3.81

SFT changes relative to complete-data Pretrain:

• Pretrain loss delta: +0.2236
• SFT loss delta: -1.7573

Complete dataset sizes observed by the diagnostic:

• Pretrain: 8,468,827 samples
• SFT: 5,099,432 samples

These are training/reference-corpus diagnostics, not an independent held-out benchmark. Because the Mini and complete-data branches are evaluated on different datasets, their absolute loss values should not be treated as a direct apples-to-apples benchmark comparison.

Generation checks

The project uses two distinct checks:

1. Upstream automatic generation test: eight built-in prompts in eval_llm.py, reviewed qualitatively. This path uses sampling and a new random seed for each prompt, so repeated runs can produce different answers.
2. Additional diagnostic prompts: eight fixed Chinese prompts in scripts/diagnose_validation.py, using greedy decoding. This check reports a custom character 3-gram repetition ratio.

Checkpoint	Reported mean repetition ratio
pretrain_mini_768.pth	0.795
full_sft_mini_768.pth	0.361
pretrain_768.pth	Not retained as reliable
full_sft_768.pth	0.244

These ratios come from separate historical runs. A controlled comparison between Mini and full-data checkpoints requires matching generation limits and stopping settings.

The full-data external diagnostic used max_new_tokens=256. Its Full Pretrain aggregate is excluded because several generations reached the token limit while decoding to empty visible responses. The diagnostic uses a chat template for every model, which must be considered when interpreting a pretraining checkpoint. The repetition ratio measures a specific form of degeneration; it does not measure factual correctness or reasoning accuracy.

Manual review found more structured and stable assistant responses in the SFT checkpoints. The Full SFT checkpoint still makes factual, arithmetic, and code-correction errors and can fail strict formatting instructions. No standardized benchmark accuracy is reported here.

Load a checkpoint

Use the fork's environment setup instructions to prepare the PyTorch/ROCm environment.

For a fresh checkout of the inspected implementation:

git clone https://github.com/Driw0x/minimind.git
cd minimind
git checkout 3e849b863eab276f2829deeefc9b0cecd7bf4970

Download the desired .pth file from this repository's Files and versions tab and place it in the checkout's out/ directory. Use the tokenizer provided in the checkout's model/ directory.

From the project root, load the Full SFT checkpoint:

python eval_llm.py --load_from model --save_dir out --weight full_sft --hidden_size 768 --num_hidden_layers 8 --use_moe 0 --device cuda:0 --max_new_tokens 512

Replace full_sft with the argument shown in the checkpoint table to load another file. The recorded ROCm setup uses cuda:0; check the detected GPU and the fork's device-selection instructions if the system also exposes integrated graphics.

At the prompt, select 0 (自动测试) to run the eight built-in tests. The loader checks that checkpoint keys match the model with strict=True. The 512-token limit above is a suggested bound for this loading and generation check, rather than a claim about the historical evaluation settings.

Reproduce the local diagnostic

Evaluate the Mini branch on the Mini datasets:

python -m scripts.diagnose_validation `
  --weights pretrain_mini full_sft_mini `
  --mode local --device cuda:0 `
  --hidden_size 768 --num_hidden_layers 8 --use_moe 0 `
  --pretrain_data dataset/pretrain_t2t_mini.jsonl `
  --sft_data dataset/sft_t2t_mini.jsonl `
  --pretrain_max_seq_len 340 --sft_max_seq_len 768 `
  --eval_samples 256 --stat_samples 256 --batch_size 4 --seed 42

Evaluate the complete-data branch on the complete datasets:

python -m scripts.diagnose_validation `
  --weights pretrain full_sft `
  --mode local --device cuda:0 `
  --hidden_size 768 --num_hidden_layers 8 --use_moe 0 `
  --pretrain_data dataset/pretrain_t2t.jsonl `
  --sft_data dataset/sft_t2t.jsonl `
  --pretrain_max_seq_len 340 --sft_max_seq_len 768 `
  --eval_samples 256 --stat_samples 256 --batch_size 4 --seed 42

The script reports loss, perplexity, supervised-token counts, dataset statistics, and available checkpoint metadata.

Release integrity and training environment

The author confirmed that the same Python/PyTorch/ROCm environment was used for training and evaluation; it has not changed between those runs and verification.

OS: Windows 11 Pro (10.0.26200, 64-bit)
Python: 3.13.1
PyTorch: 2.13.0+rocm10.0.0
ROCm/HIP: 7.15.26333
Transformers: 4.57.6
GPU: AMD Radeon RX 7800 XT (gfx1101)
MIOpen: 3.6.0
CPU: AMD Ryzen 5 7600X

SHA-256 hashes of the released checkpoint files:

Checkpoint	SHA-256
pretrain_mini_768.pth	BE7CDE9E7C6C39DE89B97CA062E23417C5B44B29694CD8FE8FCE08F8D6DE1D83
full_sft_mini_768.pth	DEE002B748CAEB3D0925700F4C27F883E8899B59EAB4FB1270F25F952A118CC2
pretrain_768.pth	B8F9A8A16DAD52BBF269C06851F1A7BBF669E4DED09C8FCBA983FCAC8FAB934D
full_sft_768.pth	E7713F514AE21F6F19EA4BAFDE5A93F2213A3EECEA576C232939BFB23979D429

Provenance notes

The source checkout inspected and verified on 13 October 2026 was commit 3e849b863eab276f2829deeefc9b0cecd7bf4970, with a clean working tree. This confirms the current repository state, not independently the historical commit used to start each training run. The exact historical code and tokenizer revisions have not been separately recorded.

The local loss and perplexity results above were retained from the evaluation console outputs on 13 October 2026. Original raw evaluation log files were not archived; no tests were rerun solely to generate logs. The training commands, configuration, environment, local dataset hashes, and checkpoint hashes are documented here. Training duration and peak GPU memory were not supplied.

Limitations and intended use

These small educational models are useful for studying pretraining, SFT, and training on AMD hardware. Generated answers can be fluent and incorrect. The reported checks cover a small set of Chinese prompts and reference samples; they do not establish broad multilingual quality, reliable reasoning, production readiness, or a validated maximum context length.

License and acknowledgements

This repository declares Apache-2.0 in its model-card metadata. The upstream MiniMind project also provides its code under Apache-2.0.

The upstream dataset card lists Apache-2.0 and CC-BY-NC-2.0. Dataset contents retain their source-specific terms; this repository's license metadata does not replace them or establish blanket commercial clearance for the underlying data.

Thanks to jingyaogong and the MiniMind contributors for the original project, training resources, tokenizer, and datasets.