MANTA — Memory-Augmented Neural Tensor Assembly

Status: Architecture proposal / research specification, version 0.1 (2026-11-02). No trained checkpoint, validated benchmark result, or functioning 128K implementation is claimed.

A small, persistent language-and-reasoning core that dynamically assembles task-specialized neural capacity from an external, potentially very large memory of learned alignments.

Model overview

Property	Proposed target
Name	MANTA — Memory-Augmented Neural Tensor Assembly
Model family	Autoregressive, decoder-only, hybrid recurrent/sparse-attention language model with conditional neural assembly
Persistent neural core	~200M parameters (design target; not yet measured)
External alignment bank	Up to ~180 GB planned SSD-backed storage, expandable independently of core
Simultaneously selected alignments	Top-5 maximum per routing event
Effective context	128K tokens target, using stateful chunk processing + selective recall; not a verified 128K full-attention window
Tokenizer / vocabulary	32,768 tokens, tied input/output embeddings, candidate design
Primary training GPU	NVIDIA RTX 3050, 8 GB VRAM
Optional teacher/judge GPU	Quadro RTX 3000, 6 GB VRAM on a separate machine
Training language (initial)	English; multilingual extension later
Intended objective	General reasoning, in-context adaptation, reusable learned skills, adaptive modular composition
Training data	Existing public datasets, no new large-scale hand-authored corpus required

Motivation

MANTA explores a different scaling axis from a monolithic language model. Rather than attempting to compress a large amount of task-specific knowledge into a fixed 200M-parameter checkpoint, it separates:

General cognitive substrate — a compact always-loaded core for language, contextual integration, hypothesis construction and reasoning.
Durable knowledge and skills — external alignment records that can contain search keys, latent codes, executable neural modifications, provenance and validation scores.
Neural assembly — a router and composer that select up to five alignment records and dynamically configure task-relevant computations.
Experience-driven writeback — candidate skills formed from successful or instructive problem-solving episodes, then tested before promotion into persistent memory.

The central hypothesis is not that 200M+200M parameters automatically equal a much larger foundation model. It is that conditional reuse and combination of separately learned task-specialized modules may yield a favorable quality / active compute / resident VRAM trade-off in selected domains.

Architecture at a glance

flowchart LR
  C[User turns / input tokens] --> CE[Context-state encoder]
  C --> E[Embedding + 20-layer hybrid decoder]
  CE --> R[ANN router]
  DB[(SSD alignment bank\nup to 180 GB)] --> R
  R --> K[Top-5 alignment records]
  K --> NC[Neural composer / hypernetwork]
  NC --> A[Conditional adapters / optional expansion modules]
  A --> E
  E --> LC[LM head / token output]
  E --> W[Episode monitor / proposed memory write]
  W --> J[Verifier / teacher / tests]
  J --> P{Checkpoint-worthy?}
  P -->|Candidate| MW[Memory writer]
  MW --> AB[A/B and transfer evaluation]
  AB -->|Improves new tasks| DB
  AB -->|No gain / interference| X[Reject / quarantine]

A. Always-resident neural core

Target configuration (subject to engineering verification)

Item	Value / proposal
Decoder blocks	20, ordered as 5 × (3 Gated DeltaNet + 1 sparse attention)
Hidden size	768
Vocabulary	32,768
Embedding / LM head	Shared (tied)
Attention query heads in sparse blocks	12
Sparse-block KV heads	3 (GQA)
Sparse head size	64
DeltaNet state heads	12, candidate 64-wide state channels; exact GDN dimensions TBD
FFN	SwiGLU, intermediate size 2,560
Normalization	Pre-RMSNorm; final RMSNorm
Positional encoding	RoPE for sparse-attention branch
Residuals	Residual plus proposed 2-branch gated residual routing (parameter cost TBD)
Latent reasoning	Optional shared recurrent-refinement block, 1–4 passes; separate FLOP budget
MTP head	Optional experimental head, off for baseline
Long-context state	Gated DeltaNet recurrent state + selective sparse-KV memory + external recall

Each block: RMSNorm → (Gated DeltaNet or sparse attention) → gated residual integration → RMSNorm → SwiGLU → gated residual integration. Only 5 of the 20 attention blocks use the sparse path; the other 15 use recurrent state updates. The sparse backend may be QSA-inspired but is not claimed to be an implemented copy of Qwen QSA.

Parameter accounting and honesty

For an all-GQA proxy with d=768, d_ff=2560, 20 blocks and tied 32,768-token embeddings:

Embedding table: 32768 × 768 = 25,165,824 parameters.
One conventional GQA projection set (12 Q, 3 KV, 64/head): 2d² + 2d(3×64) = 1,474,560 parameters.
One SwiGLU feed-forward: 3 × 768 × 2560 = 5,898,240 parameters.
20 proxy blocks plus embeddings and basic norms: approximately 172.7M parameters.

This is only an accounting sanity check, not the MANTA parameter count. Actual Gated DeltaNet projections/state gates, widened residuals, optional shared latent-refiner, writer/router and neural-composition layers add or change parameters. The ~200M total is a design budget that must be enforced by a real model-construction script and num_parameters() assertion. If the architecture exceeds budget, reduce FFN width or core depth before claiming the name MANTA-200M.

B. External neural alignment bank

An alignment is a reusable, task-relevant neural skill or knowledge configuration, not merely a text chunk. Two supported representations are proposed:

B1. Compact neural code (default)

AlignmentRecord {
  id: UUID
  retrieval_key: int8[256]       # normalized, trained retrieval space
  latent_code: fp16[128]         # compact composition instructions
  adapter_reference: optional    # low-rank factors / shared-basis coefficients
  evidence_reference: URI or record ID
  prerequisite_ids: list[UUID]
  supported_tasks: list[str]
  confidence: float
  transfer_score: float
  interference_score: float
  encoder_version: str
  core_checkpoint_id: str
  status: candidate | validated | quarantined | retired
}

The stored neural code is not guaranteed to preserve arbitrary factual detail. Source references or text should remain accessible for accurate names, numbers, provenance and correction.

B2. Expandable neural skill module (optional high-capacity mode)

A skill can also refer to an independently trained real parameter module, possibly millions or hundreds of millions of parameters, stored on SSD and loaded only when selected. These may have their own layer stacks or FFN branches, with explicit input/output projection into the 768-dimensional core state.

Important parameter-count distinction: applying a 200M full-size delta by addition to a 200M matrix does not create a 400M active network; it changes an effective 200M set of weights. Running a separate 200M skill branch in addition to the core does increase the separately stored/active parameter count to approximately 400M. MANTA supports both mechanisms and must report the distinction in benchmarks.

B3. Storage and offload

The ~180 GB target is an external storage capacity, not an active VRAM allocation or a measured checkpoint. ANN search and metadata can reside in host RAM / SSD; selected adapter weights or latent codes are staged to GPU. Random SSD accesses and module swaps may dominate latency; benchmark page-cache-hit and cold-cache regimes separately. A small, always-loaded projection basis may synthesize much smaller adapters from retrieved 128-dimensional codes.

C. Context-conditioned neural assembly

The route/compose pipeline operates at a conversation-turn boundary initially, then optionally at reasoning checkpoints.

Encode current conversation state and task into query q_t = E_router(context_t).
Retrieve candidates by approximate nearest-neighbor search on retrieval_key.
Re-rank candidates by expected task utility, compatibility, validation evidence and prerequisite relations (not semantic similarity alone).
Select at most 5 records.
Construct a coherent layerwise alignment rather than naively summing arbitrary weights.
Keep a versioned active-alignment plan throughout the forward pass; refresh safely on later turns.

For compact alignment codes, a possible per-layer modulation is:

[ \Delta W_{\ell,t}=U_\ell\operatorname{diag}(g_{\ell,t})V_\ell^\top,\quad g_{\ell,t}=G_{\ell}(q_t, \sum_{i\in\operatorname{Top5}}\alpha_i z_i). ]

For a true modular expansion, selected skills run as conditional branches:

[ h_{\ell+1}=F_\ell(h_\ell)+\sum_{i\in\operatorname{Top5}}\alpha_{i,\ell},P^{out}{i,\ell} E{i,\ell}(P^{in}_{i,\ell}h_\ell). ]

This avoids falsely describing an additive delta as an extra 200M active computation. Initial integration targets FFN / latent-refiner layers, preserving the consistency of attention KV caches when the active alignment changes.

Hierarchical skills

Alignment dependencies may form a graph, for example:

mathematical reasoning
  ├─ algebra
  │    └─ factoring
  └─ combinatorics

The selector should retrieve specific required prerequisites, but Top-5 is a global per-routing-event cap, not five independent skills per layer. Hierarchical skill composition, functional compatibility and interference are research questions.

D. 128K context architecture

Target: 131,072 input tokens of effective contextual access using three distinct mechanisms:

Recurrent stream state (Gated DeltaNet-style) to carry compact history between chunks.
Selective exact token access (sparse attention + cached or reloaded KV blocks).
External episodic and semantic alignment memory for reusable knowledge across conversations/tasks.

128K effective recall must not be conflated with 128K full-attention or differentiable 128K BPTT. An 8GB training device likely requires truncated backpropagation, chunking, state detachment and/or precomputed memory representations. These trade-offs may prevent perfect recall or training on dependencies crossing chunk boundaries.

The target should be validated by 4K, 16K, 32K, 64K, and 128K tasks: needle finding, distributed multi-hop logic, context-only rule induction, distractor resistance and cross-turn follow-up, under both warm and cold storage caches.

E. Self-written experience memory

Episode

Each training episode records: problem/question(s), visible context, student hypothesis/attempt(s), output, external verifier/judge feedback, attempted correction, outcome, relevant evidence and transfer tests.

Allow mistakes and speculative hypotheses. A correct guess receives an outcome reward, but durable skill promotion requires evidence that the associated memory generalizes beyond a single lucky result. Partial/incorrect reasoning may still be retained as candidate experience, not deployed as validated knowledge.

Memory writer

A learned writer H_ψ(episode, feedback) emits a retrieval key, latent alignment code, optional skill-module update, and provenance links. Store the candidate separately from production memory.

Promotion gate: compare the same frozen core with the candidate alignment enabled vs disabled (and with random or non-relevant alignments as a compute-matched placebo). Evaluate on different, related examples. Promote only if transfer gain exceeds noise and interference remains bounded. Preserve old core and memory snapshot identifiers for reproducibility. Roll back harmful records.

Core vs alignment learning

Fast: add/update episodic information and candidate codes; no core-weight SGD required.
Medium: optimize writer, retriever, composer and small adapters using differentiable teacher feedback.
Slow: improve core language and general reasoning through supervised or verifiable reward-based training.

No claim is made that a single correct answer uniquely determines an optimal neural weight state. The learned writer and measured transfer gain are central unresolved hypotheses.

F. Training objectives

Proposed multi-objective loss (terms activated by stage):

[ \mathcal L = \mathcal L_{LM} + \lambda_{SFT}\mathcal L_{correction} + \lambda_R\mathcal L_{retrieval} +\lambda_U\mathcal L_{utility}+ \lambda_I\mathcal L_{interference}+\lambda_C\mathcal L_{compute}. ]

L_LM: standard next-token cross-entropy for language and structure.
L_correction: correct answer conditioned on task, previous answer and graded feedback; mask input tokens from target loss.
L_retrieval: supervised or contrastive key/query alignment using known relevant evidence.
L_utility: predict improvement from selecting a memory record, measured by held-out counterfactual A/B trials.
L_interference: penalize performance degradation on unrelated tasks when combining alignments.
L_compute: regularize number of active modules, transfer bytes and inference latency.

A discrete SSD ANN top-k operation is not inherently differentiable: initially train the query encoder on offline candidate sets, then train soft reranking / composition over retrieved candidates. All large-scale online RL claims are deferred.

G. Developmental / episodic training curriculum

Language substrate: educational natural text + small-model instruction examples.
Explicit reasoning: logic, elementary math, examples with full evidence.
Multi-hop integration: scattered facts, naturalistic reasoning and distractors.
Correction episodes: answer → grading → feedback → revision → transfer test.
Evidence withdrawal: fewer explicit premises; classify certainty versus hypotheses, and reward well-calibrated guesses appropriately.
Neural memory writing: extract reusable codes from passed checkpoints and verify actual benefit before promotion.
Long-context progression: progressively larger chunks and retrieval distances to the 128K effective-context target.

The full exam sheet is a learning episode, not an end in itself. Curriculum and teacher may use pre-existing datasets rather than new manually authored examples.

H. Hardware and execution

RTX 3050 8 GB: train student/core with mixed precision, small microbatches, gradient accumulation, activation checkpointing and/or memory-efficient optimizers; profile real peak VRAM. Full optimizer states for 200M parameters are a meaningful but not exclusive cost; long-sequence activations and sparse/recurrent kernels are likely harder. No claimed training throughput yet.

Quadro RTX 3000 6 GB: separate, quantized teacher/judge inference and dataset grading over a network queue. Teacher must fit its total resident/offloaded weights and KV cache; a MoE's small active parameter count does not imply the entire checkpoint fits 6GB VRAM/48GB RAM. Use deterministic verifiers wherever possible.

SSD: persisted alignment weights, approximate-nearest-neighbor index, episode evidence and checkpoints. Do not equate SSD capacity with GPU memory bandwidth.

I. Evaluation and counterfactual experimental design

For each frozen student checkpoint / task / prompt / seed:

Arm	Configuration	Purpose
A	Core only, memory disabled	Baseline reasoning
B	Core + standard text RAG	Value of externally supplied evidence
C	Core + retrieved single neural alignment	Value of neural modulation
D	Core + composed Top-5 neural alignments	Value of composition
E	Core + random or irrelevant Top-5 modules	Tests if extra FLOPs alone explain gain
F	Core + Top-5, matched compute/latency limit	Practical utility comparison

Measurements: exact-match / task score, held-out loss, transfer to unseen variants, hallucination vs grounded claims, calibration, retrieval usefulness, active parameter count, memory usage, cold/warm cache latency, GPU-hours, SSD reads and catastrophic-interference score. Report paired confidence intervals and repeat random seeds.

Critical data split: hold out task templates, rule families, source documents and benchmark instances; never allow test answers or test-specific writes into the alignment bank. Maintain a fixed bank snapshot for comparisons. Online adaptation must be evaluated separately from static generalization.

Primary falsifiable research claims

Composed neural memory improves held-out task score over core-only and text RAG at matched resource budgets.
New validated skills can be stored without core retraining, then transferred to genuinely new tasks.
Hierarchical composition gives more than selecting a single specialized module.
The 128K stateful path retrieves and integrates distant evidence better than shorter-context baselines.

If these are not supported empirically, the architecture should be simplified rather than presented as a breakthrough.

J. Existing datasets (not a claim that they were used)

Stage	Public source	Intended use
Pretraining	HuggingFaceFW/fineweb-edu	General language / explanatory text
Early language	roneneldan/TinyStories	Small-vocabulary developmental pilot
Small-model SFT	HuggingFaceTB/smol-smoltalk	Instruction and turn-taking
Logic	tasksource/FOL-nli	Formal inference and undecidability
Revised hypotheses	tasksource/defeasible-nli	Strengtheners/weakener evidence updates
Compositionality	CLUTRR/v1	Relation composition, distractors
Multi-hop retrieval	hotpotqa/hotpot_qa	Evidence-grounded, distributed QA
Math reasoning	open-r1/OpenR1-Math-220k	Verified-answer reasoning / correction
Step feedback	trl-lib/prm800k	Process-step labels / verifier training
Long-context testing	RMT-team/babilong	128K distributed facts, hold-out evaluation
Hard long-context testing	zai-org/LongBench-v2	503 challenging long-context problems, evaluation only
General tests	TIGER-Lab/MMLU-Pro	Held-out broad QA, evaluation only

Rights: dataset-level licenses and underlying source/teacher restrictions must be reviewed before training or redistributing derivatives. Avoid accidental training on held-out benchmarks. A ready-made reasoning dataset does not contain the proposed neural alignment codes; the memory writer must produce these from training episodes during the actual run. This is model-internal state generation, not authoring a new human dataset.

K. Limitations and risks

200M core may be insufficient to learn reliable general reasoning or robust in-context learning at the intended scale.
Alignments may interfere, especially when five independently trained modules are combined.
Retrieval similarity is not the same as downstream utility; router failures may dominate.
SSD offload may yield unacceptable latency even if storage capacity is abundant.
128K effective context does not imply 128K exact recall or backpropagation.
Automatic writeback can preserve errors, contaminated answers or private data unless gated by explicit policy, provenance, consent and evaluation.
Distilled outputs must comply with each source model's licensing/usage terms; public downloadability is not blanket permission to retrain/redistribute.
This proposal has no verified intelligence level, no measured superiority to MoE/RAG, and no benchmark scores.

L. Minimum roadmap

M0: implement core-only 20-layer baseline and assert actual param counts / VRAM.

M1: implement neural alignment record, SSD ANN key index and fixed Top-5 read path.

M2: implement composer with frozen core; train on existing compositional / logic datasets; A/B core vs memory.

M3: train memory writer on verified reasoning episodes; candidate-bank / promotion / rollback.

M4: counterfactual A/B + out-of-family transfer and interference tests.

M5: add chunked 128K effective context and rigorous long-context benchmark suite.

M6: only after ablation success, consider larger 200M expert branches and publication claims.

References and research inspirations

Qwen3.8-Flash-Next official model card — hybrid DeltaNet / QSA / gated-residual reference, not an identical MANTA implementation.
SmolLM2 small-model training data.
BABILong evaluation.
LongBench v2.

Project status and benchmarks will be updated as implementations and real measurements become available.

Downloads last month: -; Downloads are not tracked for this model. How to track