AGILLM-4.4 — Detachable 49.1M AR Student

This repository releases a point-in-time, model-only export of the 49,097,218-parameter detachable causal-language-model student produced alongside the continuing 1,221,580,802-parameter AGILLM parent.

This initial repository does not contain the 1.22B parent weights, parent optimizer state, or live production training runtime. The released artifact was frozen from the student state associated with parent checkpoint step 2,560,770.

Adaptive General Intelligence Large Language Model v4.4 is the release boundary for AGILLM's checkpoint-compatible, Matryoshka-inspired teacher-student system:

• a mature 1,221,580,802-parameter parent that continues its existing AR/SAT/NAT DiffusionBlock training; and
• a separately optimized, independently exportable 49,097,218-parameter causal-AR student trained from detached parent representations.

AGILLM 4.4 does not replace or freeze the parent. The parent keeps training normally. After an accepted parent update, a stop-gradient copy of an already-computed mature AR hidden representation may train one attention or FFN sublayer in the detachable student.

accepted mature 1.22B DBlock commit
              |
              +-- detached final AR hidden state
                           |
                           +-- one 49.1M student sublayer update

The student has no authority to select parent batches, route parent layers, change parent gradients, retry a mature batch, or inject activations into the mature forward pass.

Matryoshka inspiration

This work is inspired by Matryoshka Language Model Suites, but it is not a faithful reproduction of that paper's shared-parameter, end-to-end nested suite.

AGILLM-4.4 retrofits a separate top-level student around an already mature parent. The student was initialized from deterministic read-only slices of the parent and then trained with detached teacher representations. Results reported for the referenced paper—including benchmark parity, compute savings, or speculative-decoding gains—are not results for this implementation.

Why this is 4.4

AGILLM 4.3 established the mature 28-layer DiffusionBlock/MoE training line. AGILLM 4.4 adds a new checkpointed component with its own architecture, optimizer, scaler, RNG, counters, schedule, export format, and failure boundary. Internal agillm43.* checkpoint identifiers are intentionally retained where required for backward compatibility; the product/release boundary is 4.4.

Architecture

Component	Configuration
Mature parent	1,221,580,802 parameters; width 1,280; 28 layers; 20 heads; tied K/V; MoE; 14 DBlocks
Parent objectives	Committed AR/SAT/NAT schedule with full-stack anchor passes
Detachable student core	49,096,704 unique parameters; width 256; 22 layers; 4 heads; rank 64; dense FFN; tied K/V
Standalone student export	49,097,218 unique parameters, including an inert 514-parameter SAT compatibility gate
Training bridge	LayerNorm(256) plus biasless 256-to-1,280 projection
Student plus bridge	49,425,410 parameters
Student objective	Exact streamed causal CE plus normalized-hidden/cosine distillation
Student optimizer	Separate PagedAdamW8bit state; separate scaler and RNG
Live update mode	Batch 1; 64 target tokens; one attention/FFN sublayer at a time; rare full-stack student anchor every 128 attempts

The standalone file is below 50 million parameters. The bridge is training-only and is excluded from the independently usable export.

Initial public artifact

The first AGILLM 4.4 repository artifact is exported from the atomic joint checkpoint below:

Field	Value
Parent checkpoint step	2,560,770
Parent observed-token counter at snapshot	248,383,631,360
Parent checkpoint main SHA-256	c5fa1674b09645f9a8e3005b463ddd4b973ec6a9ddf5d2b5485fe2511902f6ca
Immutable pointer record SHA-256	88b71c71e3a011faf65d1d7ca649ac4a6579630b0b0b8e6f63172dcdcd4bc9cf
Student commits stored in checkpoint	2,900
Full-stack student anchors stored	22
Student OOM/error/overflow skips	0 / 0 / 0
Last stored train-crop student CE	30.82943344116211
Last stored hidden distillation loss	0.0004056104808114469

Artifact:

artifacts/detachable-50m/detachable_50m_step02560770.pt

• Size: 182,879,879 bytes
• SHA-256: 46103c67118eadb6a265c3c8ba871b6d4732344c567ee5247af224f22e0f8a4a
• Exact unique parameter count: 49,097,218
• Embedded tokenizer: restored successfully
• ALiBi contract: corrected, scale 0.0
• Native AGILLM AR loader test: passed on CPU and generated one token

The one-token loader result is a compatibility smoke test, not a quality or throughput benchmark. The parent observed-token counter belongs to the parent; it is not a claim that the student directly consumed 248B tokens.

Inference

This is a custom AGILLM checkpoint, not a transformers.AutoModelForCausalLM directory. Use the included model-only loader:

python agillm44_detachable_50m_loader.py \
  --checkpoint artifacts/detachable-50m/detachable_50m_step02560770.pt \
  --expected-sha256 46103c67118eadb6a265c3c8ba871b6d4732344c567ee5247af224f22e0f8a4a \
  --expected-tokenizer-sha256 0e546c51529290ed4b18dc8b3e481a9ea39a2df2175b65ecb281af6bac79af73 \
  --device cpu \
  --prompt "The capital of France is" \
  --max-new-tokens 64 \
  --greedy

The loader contains only the detachable model definition, bounded zstd decoding, exact contract validation, embedded-tokenizer restoration, and AR generation. Both the outer wrapper and decompressed payload use torch.load(..., weights_only=True) with no unsafe fallback. It has no training, network, credential, supervisor, checkpoint-retention, or production-provenance code.

The full production single-file runtime remains deliberately excluded. Although no literal credential was found, it contains private-infrastructure discovery, supervisor/checkpoint-retention operations, absolute deployment paths, and detailed process provenance.

The verified public-loader test used Python 3.10, PyTorch, tokenizers==0.21.0, and zstandard support. The historical production loader also passed with transformers==4.48.0 and tokenizers==0.21.0. GPU training additionally uses bitsandbytes/PagedAdamW8bit.

What is and is not established

Established:

• the mature parent continues training while the student trains;
• teacher tensors are stop-gradient and transient;
• parent parameters are absent from the student optimizer;
• a student failure cannot retry or undo an accepted mature update;
• parent and student state can be saved together and the student can be exported independently;
• the exported artifact passes the normal AGILLM AR load path.

Not established:

• no claim is made yet that parent-guided training improves held-out quality per GPU-second versus a matched student-only baseline;
• train-crop CE is not a held-out evaluation and should not be treated as a standardized benchmark;
• no broad safety, factuality, instruction-following, or production-serving evaluation has been completed;
• the inert SAT compatibility gate does not make the standalone student a trained SAT/NAT model;
• hackathon or competition eligibility is a separate rules question.

A decrease in online hidden-distillation training loss is telemetry, not a held-out evaluation or an acceleration result.

Checkpoints and repository scope

This primary repository is deliberately bounded. It includes the standalone student, a sanitized model-only loader, the exact tokenizer bundle, release metadata, and documentation. The production runtime and full joint parent/student checkpoint are withheld from this first public commit. The full training checkpoint is a 62-shard package of roughly 7.5 GB and is not duplicated into this primary repo. See CHECKPOINTS.md.

License

The repository uses the same license: other classification as the existing AGILLM 4.3 model repository. No additional license grant is implied by publication; review the repository terms before redistribution or commercial use.