O1-Qwen05-Adapter — MT-LNN Adapter for Qwen2.5-0.5B-Instruct

An MT-LNN residual adapter attached to a frozen Qwen2.5-0.5B-Instruct.

The adapter adds multi-timescale liquid recurrence (selective decay) to six decoder layers of the frozen base model. It is initialised with init_scale=0.001 so that at step 0 it acts as a near-identity residual — the base model capability is fully preserved from the start, with MT-LNN dynamics gradually turning on as training progresses.

Files

File	Description
`llama_mt_adapter_000500.pt`	Final checkpoint (500 steps) — the deployed one
`llama_mt_adapter_000400.pt`	Intermediate checkpoint (400 steps)
`llama_mt_adapter_000200.pt`	Intermediate checkpoint (200 steps)

Each checkpoint embeds its training args, from which the six adapter mounting points are reconstructed at load time.

Configuration

Item	Value
Base model	`Qwen/Qwen2.5-0.5B-Instruct` (~494M params, frozen)
Adapter type	`MTResidualAdapter` (pre-norm residual)
Inserted at layers	3, 7, 11, 15, 19, 23 (every 4th of 24 → 6 adapters)
Trainable parameters	12,421,926 (~12.4M, 2.5% of base)
Protofilaments	13
Time scales	5
Map hidden dim	64
Init scale	0.001
Parallel scan	enabled (real causal recurrence)
Dropout	0.0

Training

Item	Value
Dataset	`Salesforce/wikitext` / `wikitext-2-raw-v1` (train)
Sequence length	512 tokens
Batch	4 × grad_accum 4 (effective 16)
Total steps	500 (≈ 4.1M tokens)
Learning rate	2e-4 (AdamW), weight decay 0.01, grad clip 1.0
Precision	bfloat16
Hardware	RTX 5060 Laptop (8 GB) — ~10 minutes total

Final-window loss: 2.5467 / 2.8433 / 2.5862 / 2.6644 (steps 470–500). The base Qwen2.5-0.5B-Instruct reaches ~2.3 loss on WikiText-2 after full pretraining; the adapter is within the expected range for this brief run — a lightly adapted model, not a converged one.

Honest status — what is and isn't validated

500 steps ≈ 4M tokens is deliberately short. Too short to show the full capability of MT-LNN dynamics on natural language. Treat these as research checkpoints.
Inference latency: +13% vs. the bare base model (six adapter forward passes). Training throughput is ~3–6× slower than the equivalent transformer path (the parallel scan is inherently more serial than matmul).
AVP (consciousness validation) fails at this scale — it needs 125M+ params and WikiText-103-scale training to be meaningful.
Bilingual capability comes from the Qwen base; the adapter does not degrade it.
No long-context or PPL-improvement claim is made for this adapter.

Usage

Loading requires the MT-LNN adapter code from the M1 repository (the adapter is not a PEFT module — vanilla peft cannot read it):

Code & loading recipes: AwareLiquid/M1 (recipes.load_mt_adapter_dir path in progress)
Live demo (CPU, free tier): Spaces — AwarenessO1
Architecture context: AwareLiquid/M2 · AwareLiquid/M1-128M