CrowdMind/lamb

Model Card: simple_20261101_curriculum

Model summary

simple_20261101_curriculum is an experimental, decoder-only GPT-style language model trained with the nanochat codebase. This card describes the base pretrained checkpoint saved for this run. The run script also contains supervised fine-tuning (SFT) and chat evaluation commands, but the available artifacts contain only the base checkpoint and its base-model evaluation; no SFT checkpoint is present.

Field	Details
Model type	Autoregressive transformer language model
Training stage covered here	Base pretraining
Run identifier	simple_20261101_curriculum
Checkpoint	output/models/simple_20261101_curriculum/base_checkpoints/simple_20261101_curriculum/model_001679.pt
Training code	scripts/base_train.py via runs/simple.sh
Tokenizer	nanochat BPE tokenizer, 32,768-token vocabulary
Maximum sequence length	8,196 tokens
Parameters	12 layers, 768-dimensional embeddings, 6 attention heads and 6 KV heads; SSSL window pattern
Training step in saved checkpoint	1,679
Training hardware recorded in log	One NVIDIA GeForce RTX 4060 Ti; BF16 compute

Intended use

This checkpoint is intended for research, experimentation, and evaluation of a small pretrained language model in the nanochat framework. It can be used to study language modeling, prompt completion, and as a starting point for further fine-tuning.

This is a base model, not a verified assistant model. Do not assume it follows instructions reliably, behaves safely, or is suitable for user-facing or high-impact decisions. Fine-tuning and task-specific evaluation are needed for those uses.

Training data

The run's curriculum manifest records the following training sources and row counts, in order:

Dataset	Rows	Notes
CrowdMind/2048-pretraining	145,056	First curriculum segment
CrowdMind/6144-pretraining	187,702	Second curriculum segment
AIGym/code-pretraining-8196	1,632,309	Code-focused segment
CrowdMind/cve-assistant-text	50,000	Security and CVE-related text segment

The script describes the sequence-length curriculum as 2,048 → 6,144 → 8,196 tokens, with the final segment using the first 50,000 CVE examples. The manifest also identifies a validation file but does not describe its composition. The available run metadata does not establish dataset licenses, filtering details, or full provenance; consult the source dataset cards and the prepared curriculum before redistribution or commercial use.

Training procedure

The run used runs/simple.sh to prepare or reuse the curriculum, train the base model, and evaluate it. Recorded base-training settings include:

• Depth: 12 layers
• Maximum sequence length: 8,196 tokens
• Total batch size: 524,544 tokens
• Device batch size: 1
• Ordered data: enabled
• Target parameter-to-data ratio: 8
• Evaluation interval: 81,960 tokens
• Checkpoint metadata reports 1,679 completed steps

The checkpoint metadata records a final validation bits-per-byte (BPB) of 0.3356 and a best observed validation BPB of 0.3315. The run log reports about 27,802 seconds of training time. These are run-specific measurements, not independent reproduction results.

Evaluation

The saved base evaluation reports a CORE metric of 0.0521 at step 1,679. The accompanying task results are in base_model_001679.csv. Evaluation was run by the repository's scripts/base_eval.py; it should be treated as a snapshot from this training run, not as a comprehensive capability, safety, or robustness assessment.

Limitations and risks

• The model may generate inaccurate, fabricated, biased, offensive, or otherwise harmful text.
• It has limited capacity and should not be expected to match larger or instruction-tuned models.
• The checkpoint is pretrained and may not follow requests or maintain a useful conversational format.
• Code and security-related training data do not make generated code secure. Review and test outputs before use.
• Training data may contain personal, copyrighted, low-quality, or otherwise unsuitable material; the available manifest does not resolve these questions.
• The available evaluation covers only selected benchmark tasks and does not establish generalization, calibration, privacy, security, or safety.

Recommended precautions

Use human review for generated content, especially code, security guidance, and factual claims. Do not use the model as the sole basis for consequential decisions. Before deployment or redistribution, review source dataset terms, evaluate the intended application, and add appropriate fine-tuning and safeguards.

Provenance

Run artifacts and configuration used for this card:

• runs/simple.sh
• meta_001679.json
• curriculum_manifest.json
• simple_20261101_curriculum.log

The model was trained from the nanochat implementation in this repository. This card does not claim that the checkpoint has been externally audited or independently reproduced.