M1-TinyLlama-Adapter — MT-v2s liquid adapter for TinyLlama-1.1B

The production M-Series adapter — a frozen TinyLlama-1.1B-Chat with MT-v2s residual adapters. Serving today as M1 on awareliquid.ai.

It adds cross-window memory through a constant-size liquid state — multi-timescale selective-decay recurrence + fast-weight associative memory — to a frozen attention model, at ~0.76% additional parameters. The carried state streams across decode steps and does not grow with context.

Files

File	Description
llama_mt_adapter_v2s_003000.pt	Final checkpoint (3000 SFT steps) — the deployed one (25.9 MB)

The checkpoint carries its training args (layout reconstruction spec) plus the adapter state_dict and step counter.

Configuration (from checkpoint args)

Item	Value
Base model	TinyLlama/TinyLlama-1.1B-Chat-v1.0 (frozen)
Adapter	MT-v2s — v2 core + selective (input-dependent) decay
Insertion	every 4th decoder layer (mt_every=4)
Protofilaments	13
Time scales	5
Map hidden dim	64
Init scale	0.001 (near-identity residual at step 0)
v2 config	d_proto 64, rank 128, selective decay ON, fast-weight dim 64, 1 head
LoRA	r=8, alpha=16, dropout 0.05, targets q/k/v/o
Trainable params	~8.4M (0.76% of the 1.1B base)

Training (SFT)

Item	Value
Data	tatsu-lab/alpaca + shibing624/alpaca-zh (bilingual EN/ZH)
Sequence length	768
Batch	2 × grad_accum 8 (effective 16)
Steps	3000
Learning rate	2e-4 (AdamW), weight decay 0.01, grad clip 1.0

Capability card (honest)

Claim	Status	Evidence
General QA / reasoning of its base class	✔ unchanged	capability suite ±1.2 pt vs base
Cross-window recall through the liquid state	✔ unique	0.56 ± 0.09 (seeds 0.621 / 0.434 / 0.621) vs 0.000 structural for attention/LoRA
Fast-weight matrix is the memory	✔ ablation	remove fast-weight → cross-window recall collapses 0.553 → 0.008
Selective decay beats static decay	✔ ablation	0.621 vs 0.553 (gap widens with training)
Snapshot → disk → fresh process → restore	✔ lossless	round-trip Δ +0.008 / +0.000; bit-exact unit test
Better perplexity than LoRA	✘ do not claim	attribution: MT adds ≈ 0 beyond LoRA
Long-context LM gains from the state	✘ do not claim	two null results; the state is episodic K→V memory, not a context replacement

The base model's attention does the in-window work; the liquid state carries what attention structurally cannot — discrete bindings across window boundaries (where LoRA/attention score exactly 0.000).

Usage

Loading requires the MT-LNN adapter code from the M1 repository (this is not a PEFT module — vanilla peft cannot read it):

• Code & recipes: AwareLiquid/M1 (serve/server_hf.py; HF-native recipes.load_mt_adapter_dir in progress)
• Architecture context: AwareLiquid/M2
• Also on HF: AwareLiquid/M1-128M (from-scratch hybrid) · AwareLiquid/M2-2B (byte-level research) · AwareLiquid/O1-Qwen05-Adapter (research adapter)

Papers

• MT-LNN architecture paper (PDF): https://huggingface.co/EverestAn/MT-LNN/resolve/main/mt_lnn_arxiv.pdf
• 中文版: https://huggingface.co/EverestAn/MT-LNN/resolve/main/mt_lnn_arxiv_zh.pdf

License

MIT (adapter weights). Base model: TinyLlama-1.1B-Chat-v1.0 (Apache-2.0).