SecurMoE-Latent-8B (v0.1.0-preview)

1. Overview & Abstract

SecurMoE-Latent-8B (v0.1.0-preview) is an open research prototype investigating action-conditioned continuous latent steering directly inside the residual stream of an 8-billion parameter causal language model (Qwen/Qwen3-8B).

The Core Hypothesis

In standard language model inference, conditional generation requires textual steering: prompt instructions must explicitly state the desired task or transformation tokens (e.g., "Apply action ADD 5"). In contrast, SecurMoE-Latent evaluates whether an autoregressive model receiving an identical, uninformative prompt ("blind prompt"):

Current state value: 64. Apply the target transition and provide the calculation and resulting state value.

can be steered toward distinct counterfactual arithmetic trajectories purely by injecting a continuous perturbation vector into its internal Transformer hidden state at Layer 18.

Scientific Scope, Baseline Parity & Caveats

Experimental Research Preview: This release is a controlled scientific prototype evaluated on a synthetic four-action arithmetic operator domain.

No General Superiority Claim: SecurMoE-Latent does not claim superiority over the unmodified Qwen/Qwen3-8B base model in general arithmetic or multi-step reasoning. The foundation model natively solves standard arithmetic when prompted explicitly. This project investigates whether internal representation intervention alone can reliably steer generative branching under neutral/blind prompts.

Not a Sparse Mixture-of-Experts: Despite the historical repository name SecurMoE, this system does not utilize a conventional sparse MoE routing or gating network across expert FFNs. It operates via continuous activation-addition vector steering into the residual stream.

Non-Production Status: The prototype is released strictly for peer review, mechanistic interpretability exploration, and collaborative scientific investigation. It must not be deployed in production or safety-critical workflows.

2. Architecture Details

The SecurMoE-Latent-8B system couples three core architectural components:

                      +-------------------------------------------------------------+
                      |   Blind Prompt: "Current state value: 64. Apply target..."   |
                      +------------------------------+------------------------------+
                                                     |
                                            Token Embedding
                                                     |
                                           [ Layers 0 .. 17 ]
                                                     |
  Action ID: a in {0,1,2,3}                          v
            |                         Residual Stream at Layer 18:
            v                                       h_18
  [ Embedding 64-d ]                                 |
            |                                        |
            v                                        v
  [ MLP: 64 -> 1024 -> 4096 ] ----( + Delta h_a )---> [ Layer 18 Pre-Hook ]
            |                                                |
         * alpha                                             v
  (Scale = 0.1059)                                  [ Layers 19 .. 35 ]
                                                             |
                                                             v
                                               Autoregressive Generation

Component Breakdown

Base Foundation Causal LM (Qwen/Qwen3-8B):
- Pinned Repository Revision: b968826d9c46dd6066d109eabc6255188de91218
- Model Dimension ($d_{\text{model}}$): $4096$
- Layer Count: $36$ Transformer blocks
- Quantization: 4-bit NormalFloat (NF4) via bitsandbytes with double quantization (bnb_4bit_use_double_quant=True) and native FP16 compute (bnb_4bit_compute_dtype="float16"). The base model weights remain strictly frozen.
Task Adaptation LoRA (PeftModel):
- Rank ($r$): $16$
- Scaling Factor ($\alpha$): $32$ ($\text{LoRA Scale} = \frac{32}{16} = 2.0$)
- Dropout: $0.05$
- Target Modules: All 7 linear projection layers across all 36 blocks: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
- Status: Strictly frozen during steering inference.
Latent Transition Module (LatentTransitionModule):
- Total Parameter Count: Exactly $4,265,217$ parameters ($\approx 0.05%$ of foundation model parameter count).
- Discrete Action Embedding: $E \in \mathbb{R}^{4 \times 64}$, mapping discrete action token $a \in {0, 1, 2, 3}$ to continuous vector $e_a \in \mathbb{R}^{64}$ ($256$ parameters).
- Projection MLP: Two-layer projection network:
  - Layer 1: Linear $W_1 \in \mathbb{R}^{64 \to 1024}$, bias $b_1 \in \mathbb{R}^{1024}$ ($66,560$ parameters) with GELU activation.
  - Layer 2: Linear $W_2 \in \mathbb{R}^{1024 \to 4096}$, bias $b_2 \in \mathbb{R}^{4096}$ ($4,198,400$ parameters).
- Learnable Scaling Parameter ($\alpha$): 1 scalar parameter, initialized to $0.1000$, empirically converged during training to $0.1059$.
- Injected Perturbation: $$\Delta h_a = \alpha \cdot \left( W_2 \cdot \text{GELU}(W_1 e_a + b_1) + b_2 \right) \in \mathbb{R}^{4096}$$
Residual Stream Injection Pre-Hook:
- Attached directly as a PyTorch forward pre-hook to Transformer Layer 18 (model.base_model.model.model.layers[18]).
- The perturbation vector $\Delta h_a$ is broadcast across token positions and added to the residual stream hidden states: $$\tilde{h}{18} = h{18} + \Delta h_a$$
- The hook cleanly detaches after generation, preserving deterministic state across queries.

3. Action Space & Operators

The model maps four discrete action indices to specific arithmetic state transformations:

Action ID	Canonical Operator Name	Mathematical Operation	Example Input	Target Output Expression
0	`ADD 5`	$f(x) = x + 5$	$64$	`Action: ADD 5`, Resulting State: $69$
1	`SUBTRACT 5`	$f(x) = x - 5$	$64$	`Action: SUBTRACT 5`, Resulting State: $59$
2	`MULTIPLY BY 2`	$f(x) = x \times 2$	$64$	`Action: MULTIPLY BY 2`, Resulting State: $128$
3	`ADD 20`	$f(x) = x + 20$	$64$	`Action: ADD 20`, Resulting State: $84$

Blind Prompting Protocol

The user text prompt contains no cues regarding which operator to execute:

<|im_start|>user
Current state value: {value}. Apply the target transition and provide the calculation and resulting state value.<|im_end|>
<|im_start|>assistant

The operator selection is determined exclusively through the continuous vector injected into the residual stream at Layer 18.

4. Quickstart & Minimal Inference Code

The model provides a high-level Python API through the SecurMoELatent class. Complete examples are available in examples/minimal_inference.py and examples/quickstart.py.

Method A: Direct Autoregressive Generation (`model.generate()`)

from securmoe import SecurMoELatent

# Load model from local directory or Hugging Face Hub (e.g. YOUR_HF_NAMESPACE/SecurMoE-Latent-8B)
model = SecurMoELatent.from_pretrained(
    "YOUR_HF_NAMESPACE/SecurMoE-Latent-8B",
    load_in_4bit=True,
    device_map="auto",
    verify_checksums=True,
)

# Steered generation with Action 0 (ADD 5) at Layer 18
result = model.generate(
    prompt="<|im_start|>user\nCurrent state value: 64. Calculate resulting state.<|im_end|>\n<|im_start|>assistant\n",
    action_id=0,
    max_new_tokens=64,
    do_sample=False,
    temperature=0.0,
    top_p=1.0,
)
print("Steered Output:\n", result)

Method B: Pure Neural Transition & Post-Hoc Evaluation (`model.generate_transition()`)

import torch
from securmoe import SecurMoELatent

# 1. Load the model from local directory or Hugging Face Hub
# Automatically loads Qwen/Qwen3-8B in 4-bit NF4, attaches the LoRA adapter,
# loads the LatentTransitionModule, and registers the Layer 18 forward pre-hook.
model = SecurMoELatent.from_pretrained(
    "YOUR_HF_ORG_OR_LOCAL_DIR/SecurMoE-Latent-8B",
    device_map="auto",
    load_in_4bit=True,
    verify_checksums=True,
)

# 2. Perform Pure Neural Generation under a Blind Prompt
# Input state: 64
# Injected Action: 0 ("ADD 5") -> Expected Result: 69
# The textual prompt passed to the model contains ZERO tokens mentioning "ADD 5".
gen_result = model.generate_transition(
    value=64,
    action_id=0,  # Discrete Action: 0=ADD 5, 1=SUBTRACT 5, 2=MULTIPLY BY 2, 3=ADD 20
    max_new_tokens=96,
)

print(f"Target Injected Action : {gen_result['action_name']} (ID: {gen_result['action_id']})")
print(f"Model Selected Action  : {gen_result['parsed_action']}")
print(f"Model Computed Value   : {gen_result['parsed_value']}")
print(f"Generation Status      : {gen_result['generation_status']}")
print("\n--- Raw Model Output ---")
print(gen_result["raw_generation"])

# 3. Independent Ground-Truth Evaluation (Post-hoc check, decoupled from generation)
eval_result = model.evaluate_transition(gen_result)
print("\n--- Evaluation Summary ---")
print(f"Expected Target Value  : {eval_result['expected_value']}")
print(f"Value Match            : {eval_result['expected_value'] == gen_result['parsed_value']}")
print(f"Exact Spec Match       : {eval_result['exact_transition_match']}")

5. Hardware & Installation Requirements

Hardware Specifications

GPU: Dedicated NVIDIA GPU with $\ge 8$ GB VRAM (e.g., RTX 3070, RTX 3080, RTX 4060, RTX 4070, RTX 4080, RTX 4090, Tesla T4, A10, A100, V100).
Architecture Support: Turing (sm_75), Ampere (sm_80, sm_86), Ada Lovelace (sm_89), or Hopper (sm_90).
Telemetry Baseline (Measured on NVIDIA Tesla T4 16 GB):
- Base Model + LoRA Weights (4-bit NF4): $5.83\text{ GB}$
- Latent Transition Module Weights: $+17.06\text{ MB}$
- Peak Inference VRAM: $6,139.5\text{ MB}$ to $7,470\text{ MB}$
- Headroom on 16 GB GPU: $> 8\text{ GB}$
Host System RAM: $\ge 16\text{ GB}$ recommended.

Consumer 8 GB GPU Notice: While peak VRAM on headless Linux is $\approx 6.1\text{ to }7.5\text{ GB}$, consumer operating systems with desktop window compositors (Windows DWM, macOS, or X11/Wayland) may allocate $0.5\text{ to }1.5\text{ GB}$ of dedicated VRAM, leaving less available memory on strict 8 GB cards.

Software Dependencies

Python $\ge 3.10$
PyTorch $\ge 2.2.0$ (CUDA $\ge 12.1$ recommended)
Transformers $\ge 4.51.0$ (required for official Qwen3-8B architecture support)
PEFT $\ge 0.10.0$ (tested on $\ge 0.21.2$)
BitsAndBytes $\ge 0.43.0$ (tested on $\ge 0.50.2$)
Accelerate $\ge 0.28.0$ (tested on $\ge 1.15.0$)
Safetensors $\ge 0.4.2$ (tested on $\ge 0.8.0$)

# Minimal installation
pip install torch>=2.2.0 "transformers>=4.51.0" "peft>=0.21.0" "bitsandbytes>=0.50.0" accelerate safetensors

6. Checkpoint Verification & Cryptographic Ledger

All model checkpoint files must match the following SHA-256 digests and file sizes:

Artifact File	Size (Bytes)	SHA-256 Checksum	Purpose
`adapter_model.safetensors`	$174,655,536$	`bed27fb0941c9973de3795d40675ffee874f0a830ade232cc22120e2af9fd721`	Fine-tuned LoRA adapter weights ($r=16, \alpha=32$)
`latent_transition_module.safetensors`	$17,061,364$	`4bac1f67cd32b229180917430036683c52f8549332c94ee07770052f87480d0e`	Layer 18 `LatentTransitionModule` weights ($4,265,217$ params)
`adapter_config.json`	$1,169$	`a8be324c9d9919b3dd9598cae776a8419f57423eb2d15d9cdb74b14645b1f01b`	Hugging Face PEFT configuration
`securmoe_config.json`	$1,344$	`cb93ef33fdde0af5f20494825c5989714bfe506e5d7538999de9718b37fb262c`	SecurMoE runtime steering architecture metadata

Automated Hash Verification in Python

import hashlib
from pathlib import Path

EXPECTED_DIGESTS = {
    "adapter_model.safetensors": "bed27fb0941c9973de3795d40675ffee874f0a830ade232cc22120e2af9fd721",
    "latent_transition_module.safetensors": "4bac1f67cd32b229180917430036683c52f8549332c94ee07770052f87480d0e",
    "adapter_config.json": "a8be324c9d9919b3dd9598cae776a8419f57423eb2d15d9cdb74b14645b1f01b",
    "securmoe_config.json": "cb93ef33fdde0af5f20494825c5989714bfe506e5d7538999de9718b37fb262c",
}

for filename, expected_hash in EXPECTED_DIGESTS.items():
    path = Path(filename)
    if path.is_file():
        computed = hashlib.sha256(path.read_bytes()).hexdigest()
        assert computed == expected_hash, f"Hash mismatch for {filename}: {computed} != {expected_hash}"
        print(f"Verified {filename}: OK")

7. Empirical Calibration & Scientific Limitations

TRANSPARENT SCIENTIFIC DISCLOSURE (AUDITED NOVEMBER 6, 2026): Following rigorous empirical preflight benchmarking on a physical NVIDIA Tesla T4 GPU and subsequent forensic parser audits, the research team reports the following empirical findings and caveats without omission.

7.1 Physical Tesla T4 Preflight Benchmarking

A 6-trial hardware preflight verification battery was executed on a physical NVIDIA Tesla T4 GPU (15.10 GB VRAM, Turing architecture). The GPU loaded the model in $114\text{ seconds}$ and completed 6 sequential inference runs in $22\text{ seconds}$ with stable memory allocation ($6,139.5\text{ MB}$ peak VRAM).

7.2 Trial 3: Demonstration of Physical Latent Steering

In Trial 3 (cond3_latent_only), the model was given an entirely neutral, uninformative prompt containing zero textual tokens or instructions about arithmetic operations:

<|im_start|>user
Calculate the resulting state value for current state: 64.<|im_end|>
<|im_start|>assistant

The Layer 18 forward pre-hook injected the continuous steering vector for Action 0 (ADD 5) with average delta norm $\bar{|\Delta|} = 21.1934$. The model emitted:

Action: ADD 5
Resulting State: 69

This trial demonstrated physical latent steering inside the residual stream: the internal perturbation vector alone, absent any text cues, caused the autoregressive language model to emit the designated action token (ADD 5) and calculate the correct resulting state ($64 + 5 = 69$).

7.3 Trial 6: Magnitude Drift Failure Mode

In Trial 6 (cond_d_perm), an external harness mapping tested Action 0 mapped to internal steered Action 1 (SUBTRACT 5, target: $64 - 5 = 59$). The model emitted:

Action: SUBTRACT 1
Resulting State: 63

Diagnostic Finding: While the model followed the correct arithmetic operator sign (subtraction), it experienced magnitude drift from $5$ to $1$ ($64 - 1 = 63$). The emitted action string SUBTRACT 1 was not part of the registered 4-action ontology.
Scientific Conclusion: Reliable, robust magnitude control across all four discrete actions is NOT established.

7.4 Strict Exact Match (0/6 under Frozen v2.2 Specification)

Under the frozen preflight specification (v2.2):

A trial is scored as a strict exact match ($E_{\text{succ}}$) only if both the final numerical value is correct ($V_{\text{valid}} = \text{True}$) and an explicit step-by-step calculation formula line (Calculation: A [op] B = C) is generated ($C_{\text{valid}} = \text{True}$).
In physical GPU execution, the model emitted concise answers (Action: ADD 5\nResulting State: 69), omitting the calculation equation line.
Consequently, while numerical value validity ($V_{\text{valid}}$) passed in Trials 2, 3, and 4 ($69 = 69$), the strict exact match scorecard remains 0/6 under the frozen specification due to absent formula lines ($C_{\text{valid}} = \text{False}$).

Trial	Condition	Injected Steering Vector	Target Value	Model Output Text	Value Match ($V_{\text{valid}}$)	Formula Line ($C_{\text{valid}}$)	Strict Match ($E_{\text{succ}}$)
T1	Baseline unsteered	None ($\bar{\|\Delta\|}=0$)	None	Generic reasoning text	False	False	FAIL
T2	Text guidance only	None ($\bar{\|\Delta\|}=0$)	$69$	`Action: ADD 5\nResulting State: 69`	True	False	FAIL
T3	Latent steering only	Action 0 ($\bar{\|\Delta\|}=21.19$)	$69$	`Action: ADD 5\nResulting State: 69`	True	False	FAIL
T4	Aligned text + latent	Action 0 ($\bar{\|\Delta\|}=21.19$)	$69$	`Action: ADD 5\nResulting State: 69`	True	False	FAIL
T5	Conflicting text vs latent	Action 1 ($\bar{\|\Delta\|}=12.01$)	$59$	`Action ADD 5... Resulting state is 69.`	False	False	FAIL
T6	Steering permutation	Action 1 ($\bar{\|\Delta\|}=12.01$)	$59$	`Action: SUBTRACT 1\nResulting State: 63`	False	False	FAIL

7.5 Current Status: 4,000-Run Replication Campaign On Hold

Because four-action magnitude control is not universally established and output formatting remains sensitive to decoding parameters, the planned 4,000-run large-scale empirical campaign is STRICTLY ON HOLD (NO-GO).

The v0.1.0-preview checkpoint is released openly to facilitate:

Independent verification of the Layer 18 residual intervention mechanism.
Exploration of steering vector normalization and magnitude calibration.
Collaborative training of multi-layer or multi-action transition controllers.

8. License & Attribution

Software & Weights License

This model card, the SecurMoE-Latent-8B runtime code, the LoRA adapter weights, and the LatentTransitionModule weights are released under the Apache License, Version 2.0.

Copyright 2026 Vaibhav Singh and the SecurMoE Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Upstream Foundation Model Attribution

SecurMoE-Latent-8B is built upon the Qwen/Qwen3-8B base foundation language model developed by the Qwen Team / Alibaba Cloud, licensed under the Apache License, Version 2.0:

Upstream Model: Qwen/Qwen3-8B
Upstream License: Apache License 2.0
Pinned Commit: b968826d9c46dd6066d109eabc6255188de91218

We gratefully acknowledge the Qwen Team at Alibaba Cloud for releasing Qwen/Qwen3-8B under open and permissive licensing.

9. Citation

If you utilize SecurMoE-Latent-8B in your research, cite this work as follows:

@misc{singh2026securmoelatent8b,
  author       = {Singh, Vaibhav},
  title        = {SecurMoE-Latent-8B: Action-Conditioned Residual Stream Steering in Autoregressive Language Models},
  year         = {2026},
  month        = nov,
  howpublished = {Hugging Face Model Hub},
  url          = {https://huggingface.co/Vaib202/SecurMoE-Latent-8B},
  note         = {Research Preview v0.1.0-preview}
}

Human-Readable Citation:

Singh, V. (2026). SecurMoE-Latent-8B: Action-Conditioned Residual Stream Steering in Autoregressive Language Models. Research Preview v0.1.0-preview. Hugging Face.

Downloads last month: 41

Model tree for Vaib202/SecurMoE-Latent-8B

Base model

Qwen/Qwen3-8B-Base

Finetuned

Qwen/Qwen3-8B

Adapter

(2,454)

this model