Instructions to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Topovoid/TopoVoid-0.2B-Orthogonal-Preview", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Topovoid/TopoVoid-0.2B-Orthogonal-Preview", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Topovoid/TopoVoid-0.2B-Orthogonal-Preview" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Topovoid/TopoVoid-0.2B-Orthogonal-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview
- SGLang
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Topovoid/TopoVoid-0.2B-Orthogonal-Preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Topovoid/TopoVoid-0.2B-Orthogonal-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Topovoid/TopoVoid-0.2B-Orthogonal-Preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Topovoid/TopoVoid-0.2B-Orthogonal-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with Docker Model Runner:
docker model run hf.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview
- TopoVoid-0.2B-Orthogonal-Preview
- 1. Overview
- 2. Quick Start & Open-Dataset Benchmark
- ChatML & Long-Context Generation (
AutoModelForCausalLM) - Closed-Form Null-Space Write (
load_topovoid) - Summary of Open-Dataset Results
- 2.1 CounterFact (1-Hop Weight Write)
- 2.2 Stacking Modules
- 2.3 MQuAKE-CF-3k (2-Hop)
- 2.4 Text Classification (
AG News&DBpedia-14) - 2.5 External MidCBM Retrieval Index + Copy Head
- 2.6 Zero-Shot Commonsense Benchmarks, ChatML Dialogue & Overflow Recall
- ChatML & Long-Context Generation (
- 3. Further Research & Next-Gen Multimodal Results
- 4. Limitations
- 5. License & Attributions
TopoVoid-0.2B-Orthogonal-Preview
- Organization: Topovoid (topovoid.com)
- Maintainers: tirpitzia, 0kaguya
- License: Apache-2.0
1. Overview
This repository contains the 0.2B research preview of the TopoVoid architecture. It is a 212M-parameter bilingual (EN/ZH) causal language model pretrained from scratch with explicit middle-layer concept bottlenecks, paired with a closed-form solver that writes new facts into Attention and MLP weights inside the orthogonal null space of those concept layers, and a concept-indexed overflow memory that recalls evicted turns beyond the native 4,096-token RoPE window.
How the closed-form write works (closed_form_write.py):
- Trigger contexts: Each fact (
subject, declarativeprompt,target, optionalquestion) is expanded into declarative and ChatML QA prompt variants. Multi-token targets are unrolled into sequential token continuation steps. - Pullback target ($R$): For each edited layer ($\ell \in {8, 9, 10, 11}$, 1-based), the model runs forward to layer $\ell$ and takes three normalized autograd steps on the hidden activation to find a residual $R$ that increases the target token probability. Autograd is used to compute this activation target before the linear solve; no optimizer step or gradient descent updates the model weights.
- Null-space projection ($P_\perp$): The residual is projected through
$$P_\perp = I_{1024} - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T, \qquad P_\perp Q_{\mathrm{cbm}} = 0$$
where $Q_{\mathrm{cbm}} \in \mathbb{R}^{1024 \times 176}$ is the orthonormal basis spanning the read and write matrices of the L7
MidCBM, L7SubspaceTopKCBM, and L12Highway CBM. Of the1024residual dimensions,176(17.2%) are protected and848(82.8%) form the writable null space ($\max |P_\perp Q_{\mathrm{cbm}}| = 1.34 \cdot 10^{-7}$ infp32). - Closed-form linear solve ($V$): Trigger keys, natural-text covariance (
eval_c4,eval_zhinvocab_map_49152_chatml.npz), and negative/continuation suppression keys are combined in a regularized linear system to solve for $V$, yielding low-rank weight increments $\Delta W = P_\perp R V$ on the Attention output projection (proj) and MLP down-projection (out). - Mounting and unmounting: During multi-layer compilation, earlier layers' increments are applied before solving later layers. At inference, a compiled module is added directly to the base weights (
W + ΔW), adding zero prompt tokens.
How default-on MidCBM overflow recall works (modeling_topovoid.py, midcbm_infinite_context = true):
- Sequences up to
4,096tokens execute the exact native causal forward pass (0.0logit difference). - When prefill or decoding crosses
4,096tokens, middle turns/spans are segmented and encoded once through frozenL1..L7into a64-d(24-d MidCBM + 40-d SubspaceTopKCBM) late-interaction concept bank, recalling top-scoring chunks into a bounded active window (<= 768compressed prefill tokens with3,328native RoPE decode steps reserved before the next rollover).
Architecture
| Item | Value |
|---|---|
| Parameters | 212,011,264 (tied input and output embeddings) |
Layers, d_model, d_ff |
16, 1024, 2432 (SwiGLU) |
| Attention | 16 query heads, 4 KV heads (GQA), per-head QK-RMSNorm |
| Vocabulary | 49,152 active tokens, pruned from 151,936 (~82M unused embedding parameters removed) |
| Positional encoding | RoPE (theta = 5e5, 4,096-position native table; default-on MidCBM overflow recall above 4,096) |
L7 MidCBM |
24 dimensions (35:1 concept SNR design) |
L7 SubspaceTopKCBM |
40-d subspace, 256 dictionary atoms, Top-8 |
L12 Highway CBM |
24 dimensions |
Files in This Repository
| File | Description |
|---|---|
model.safetensors |
Pretrained fp32 weights (809 MB, 189 tensors); head.weight is tied to emb.weight on load |
config.json, generation_config.json |
Model config (including default midcbm_* overflow parameters) and default sampling config |
tokenizer.json, tokenizer_config.json |
Qwen-based fast tokenizer with ChatML chat_template (<|im_start|> / <|im_end|>) |
configuration_topovoid.py, modeling_topovoid.py |
Hugging Face TopoVoidConfig, TopoVoidForCausalLM, MidCBMInfiniteContext, MidCBMChatSession, and load_topovoid |
vocab_map_49152_chatml.npz |
Active-vocabulary index map (orig_to_compact, compact_to_orig) and 64 × 256 calibration token rows (eval_c4, eval_zh) |
closed_form_write.py |
Closed-form null-space weight writer ($\Delta W = P_\perp R V$) |
demo_write.py |
Verification script (subspace split, closed-form write, cosine retention, and declarative/ChatML generation) |
demo_chat.py |
Multi-turn ChatML and > 4,096-token MidCBMChatSession overflow recall demo |
2. Quick Start & Open-Dataset Benchmark
Tested with torch >= 2.8, transformers 4.57+ and 5.x, safetensors, and numpy on one GPU.
ChatML & Long-Context Generation (AutoModelForCausalLM)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Topovoid/TopoVoid-0.2B-Orthogonal-Preview" # or local path "."
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to("cuda").eval()
messages = [
{"role": "system", "content": "You are a logical, concise assistant. Answer the user question directly and accurately."},
{"role": "user", "content": "What is your model name? Answer briefly."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
prompt_ids = torch.tensor([model.encode_orig_ids(tok, prompt)], device="cuda")
with torch.autocast("cuda", dtype=torch.bfloat16):
out = model.generate(prompt_ids, max_new_tokens=128)
print(tok.decode(out[0, prompt_ids.shape[1]:], skip_special_tokens=False))
Closed-Form Null-Space Write (load_topovoid)
from modeling_topovoid import load_topovoid
from closed_form_write import concept_null_projector
model, tok, o2c, c2o = load_topovoid(".")
P_perp, Q_cbm = concept_null_projector(model)
Run the closed-form write demo (demo_write.py) or the multi-turn overflow recall demo (demo_chat.py):
python demo_write.py --model-dir .
python demo_write.py --model-dir . --counterfact counterfact.json --n 12
python demo_chat.py --model-dir .
Summary of Open-Dataset Results
The evaluation tracks below test distinct mechanisms—direct weight writing, frozen-feature linear readout, external concept-index retrieval, and long-context overflow recall—and should not be conflated:
| Track & Dataset | Measured Result | Conditions |
|---|---|---|
Direct weight write (CounterFact, 100 individual / 12 & 40 batch edits) |
Declarative rewrite Top-1: 0.00% → 100.00%; Prefilled ChatML stem Top-1: 0.00% → 100.00%; mean single-fact compile time 1.44 s (0.093 MB factor file) |
MidCBM & Sparse-256 cosine retention = 1.000000 (max_abs_diff = 0.0). Unprefilled free ChatML answer recall is 74.0% (single, with SYS_CHATML_EN), 41.7% (batch-12), and 45.0% (batch-40). On the Stage 10 pretraining checkpoint, paraphrase first-token Top-1 is 35.00% and neighborhood bleed is 44.90%. |
Frozen representation readout (AG News, 10 labels/class) |
Full L7 (1024-d): 75.37%; CBM (320-d): 67.32%; Word+Char TF-IDF SVM: 56.70% |
Stage 10 pretraining checkpoint (40 total labels, pre-fixed $\lambda=0.1, C=1$, 0 validation labels, 5 seeds; +18.67 pp L7 over SVM, 95% CI [+15.75, +21.60] pp). Transferring the Stage 10 L7 head to the shipped weights without re-fitting scores 76.15% (320-d CBM heads should be fitted directly on the shipped weights rather than transferred un-refitted across stages, which scores 25.02%). |
External MidCBM index (ARC, 399 new question paraphrases) |
33.33% → 68.42% (+35.09 pp, recall@1 = 66.17%) |
Base weights unchanged (3,307 indexed science items + zero-parameter copy head; 28.57% → 72.93% on the Stage 10 pretraining checkpoint; TF-IDF + copy head: 81.20%). |
Default-on MidCBM overflow recall (20 archive sessions, 6,477–11,389 tokens) |
Target chunk Recall@TopK: 20 / 20 (100.0%); exact value recall: 17 / 20 (85.0%: 8/10 EN, 9/10 ZH) |
Mean input length 8,947 tokens (> 4,096 native RoPE limit), default GenerationConfig (seed = 20261110), 0 OOM/RoPE errors, 0.0 logit diff at <= 4,096 tokens. |
Complete verbatim stdout of python demo_write.py --model-dir . (single CounterFact edit Sunrise Party → Toronto, unpenalized greedy, max_new_tokens = 64):
[1] Orthogonal Subspace Split: protected=176/1024 (17.2%), null=848/1024 (82.8%), max|P_perp Q_cbm|=1.34e-07
[2] Closed-Form Write (1 fact(s)): 0 optimizer steps, 1.56 s, module=0.093 MB
[3] Feature Cosine Retention: MidCBM=1.000000, Sparse-256=1.000000
[4] declarative: unpenalized greedy, max_new_tokens=64; verbatim text including EOS
Input:
Sunrise Party is headquartered in
Before write:
the United Kingdom. It is a political party that is run by the party's leader, the party's leader, and the party's leader. The party's main political party is the National Party. The party's main political party is the National Party.
History
The party was founded in 1994
After write:
Toronto, Ontario, Canada. The party is a non-profit, non-partisan, non-partisan political party that advocates for the rights of all Canadians. The party's website is www.thesecommons.org.
History
The party was founded in 1994 by the Toronto-based Toronto Star newspaper
[4] chatml_stem: unpenalized greedy, max_new_tokens=64; verbatim text including EOS
Input:
<|im_start|>user
Where is Sunrise Party headquartered?<|im_end|>
<|im_start|>assistant
Sunrise Party is headquartered in
Before write:
Paris, France. It is a non-profit organization that promotes the interests of the French people and promotes the interests of the European Union.<|im_end|>
After write:
Toronto, Ontario, Canada.<|im_end|>
[4] chatml_free: unpenalized greedy, max_new_tokens=64; verbatim text including EOS
Input:
<|im_start|>user
Where is Sunrise Party headquartered?<|im_end|>
<|im_start|>assistant
Before write:
Sunrise Party is a political party based in the United States. It is a non-profit, non-partisan, non-partisan organization that promotes the interests of the American people. It operates under the auspices of the National Association for the Advancement of Colored People (NAACP).<|im_end|>
After write:
Sunrise Party is a political party founded in 1994 by Sunrise M. M. M. M. Krasnaya. It is a non-profit, non-partisan, non-partisan political party in India. It is a member of the National Assembly of India.<|im_end|>
(With the canonical system prompt SYS_CHATML_EN on the 40-fact CounterFact harness query "Sunrise Party is headquartered in?", unprefilled greedy generation outputs Sunrise Party is headquartered in Toronto, Ontario, Canada.<|im_end|>.)
2.1 CounterFact (1-Hop Weight Write)
First edit layer: Layer 8 (1-based). On the shipped checkpoint, 1 (×100), 12, and 40 CounterFact edits achieve 0.00% → 100.00% rewrite Top-1 and 0.00% → 100.00% prefilled ChatML stem Top-1 (1.000000 MidCBM and Sparse-256 cosine retention). Full scaling from 1 to 640 facts in a single module (measured on the Stage 10 pretraining checkpoint using first-target-token proxies of ES / PS / NS; official ROME scores full target sequences):
- ES: $P(\mathrm{new}) > P(\mathrm{true})$ on the rewrite prompt.
- PS: $P(\mathrm{new}) > P(\mathrm{true})$ on paraphrase prompts.
- NS: $P(\mathrm{true}) > P(\mathrm{new})$ on neighborhood prompts (higher means less bleed into related subjects).
- Top-1: First-token top-1 hit rate of the new target on rewrite, paraphrase, and neighborhood prompts.
| Facts in one module | Rewrite top-1 (base → edited) | ES | PS | Paraphrase top-1 | NS | Neighbor top-1 on new target | C4 PPL ratio | Compile seconds | Factor MB |
|---|---|---|---|---|---|---|---|---|---|
| 1 (×100 median) | 1.00% → 100.00% |
100.00% |
75.50% |
35.00% |
19.70% |
44.90% |
1.002 |
1.37 |
0.093 |
| 12 | 0.00% → 100.00% |
100.00% |
91.67% |
41.67% |
13.33% |
56.67% |
1.015 |
2.80 |
1.06 |
| 40 | 0.00% → 100.00% |
100.00% |
67.50% |
20.00% |
31.25% |
27.50% |
1.043 |
6.53 |
3.53 |
| 160 | 0.63% → 95.63% |
100.00% |
59.06% |
21.25% |
41.69% |
23.31% |
1.115 |
22.24 |
14.45 |
| 640 | 0.31% → 78.28% |
94.38% |
54.69% |
16.95% |
46.77% |
17.87% |
1.148 |
48.55 |
57.51 |
2.2 Stacking Modules
Each module contains 40 CounterFact facts compiled independently from the same base checkpoint and added linearly (W + Σ ΔW_i, Stage 10 pretraining checkpoint):
| Modules stacked | Top-1 alone | Top-1 stacked | NS alone | NS stacked | C4 PPL ratio | MidCBM cosine | Sparse-atom cosine | Highway cosine (mean, min) |
|---|---|---|---|---|---|---|---|---|
| 2 (80 facts) | 98.75% |
96.25% |
30.25% |
31.62% |
1.112 |
1.0 |
1.0 |
0.997886, 0.966447 |
| 4 (160 facts) | 98.75% |
55.62% |
32.25% |
37.19% |
1.406 |
1.0 |
1.0 |
0.995213, 0.940090 |
| 8 (320 facts) | 97.81% |
34.69% |
32.41% |
40.56% |
3.096 |
1.0 |
1.0 |
0.989003, 0.887467 |
Note on multi-module decay and the 1B compiler upgrade: In this 0.2B preview repository, closed_form_write.py only projects each module individually against the protected concept basis $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) and does not include a cross-module Gram-Schmidt joint orthogonalization operator across different modules' right-hand key subspaces. As a result, 2 stacked modules (80 facts) retain 96.25% Top-1 (0.997886 Highway cosine), whereas summing 4 or 8 independently solved modules causes key collisions inside the null space (55.62% and 34.69%). Cross-module Gram-Schmidt joint compilation is implemented in our internal 1B commercial compiler (see Section 3).
2.3 MQuAKE-CF-3k (2-Hop)
Evaluated on 150 sampled cases (149 allow the unrelated-subject control; Stage 10 pretraining checkpoint). These numbers do not establish multi-hop reasoning on the base checkpoint:
| Measure | Value |
|---|---|
| Individual hop edits succeeded | 227 / 228 |
| Edited answer on multi-hop questions (before → after) | 13 / 150 → 117 / 150 |
| Edited answer on matched treated questions | 116 / 149 |
| Same answer on unrelated-subject control (before → after) | 12 / 149 → 118 / 149 |
| Cases where base knew all hops before edit | 6 / 150 |
2.4 Text Classification (AG News & DBpedia-14)
A separate ridge classifier head is fitted on frozen representations without changing the language model weights or generation path (Stage 10 pretraining checkpoint). Full L7 (1024-d) and CBM (320-d, MidCBM + SubspaceTopKCBM) are reported separately; Full L7 results cannot be attributed to the concept bottleneck alone.
Fixed-Hyperparameter 40-Label Test (AG News, 10 Labels/Class, 0 Validation Labels)
With ridge $\lambda = 0.1$ and LinearSVC $C = 1.0$ fixed in advance across 5 sampling seeds (20261203–20261207) and no validation labels used for selection:
- Full L7 (
1024-d):75.37% - CBM (
320-d):67.32% - Word (1–2 gram) + Character
char_wb(3–5 gram) TF-IDF LinearSVC:56.70% - Difference (Full L7 − SVM):
+18.67pp (95% paired seed-$t$ CI[+15.75, +21.60]pp; Holm-corrected $p = 0.0002975$).
Five-Seed Low-Label Curve (With 8,000-Example Validation Selection)
Each method uses the exact same training subsets (seeds 20261203–20261207) and selects hyperparameters on the 8,000-item validation split:
| Dataset | Fit labels / class | CBM (320-d) |
Full L7 (1024-d) |
Word + Char TF-IDF SVM | Full L7 − SVM (pp) |
|---|---|---|---|---|---|
AG News |
10 |
71.70% |
76.84% |
56.97% |
+19.88 |
AG News |
100 |
79.63% |
83.97% |
80.90% |
+3.07 |
AG News |
1000 |
84.19% |
87.61% |
87.80% |
-0.19 |
DBpedia-14 |
10 |
72.23% |
85.78% |
85.30% |
+0.47 |
DBpedia-14 |
100 |
88.09% |
94.27% |
94.67% |
-0.40 |
Full-Dataset Readout & Perturbation Checks
AG News uses 112,000 fit / 8,000 val / 7,600 test items. DBpedia-14 uses 120,000 fit / 8,000 val / 70,000 test items (10,000-item test subset for zero-shot rows; on that same 10,000-item subset, CBM / Full L7 / TF-IDF score 90.96% / 96.53% / 98.23%):
| Method | AG News | DBpedia-14 |
|---|---|---|
0.2B zero-shot (initial 4-template search) |
65.39% |
48.43% |
0.2B zero-shot (validation-selected prompt search: raw / calibrated / top-12 ensemble) |
81.41% / 81.79% / 83.30% |
— |
Readout: CBM (320-d) |
85.32% |
91.20% |
Readout: Full L7 (1024-d) |
89.00% |
96.84% |
| TF-IDF + Logistic Regression | 91.95% |
98.31% |
Qwen3-8B zero-shot (initial 4-template reference) |
81.97% |
88.38% |
| Full L7 readout with shuffled training labels | 24.92% |
7.64% |
Under surface perturbations on AG News (evaluated without retraining the head): Original 89.00% (TF-IDF 91.82%), All lowercase 86.45% (91.82%), Strip Reuters/AP datelines 88.83% (91.45%), First 32 tokens 86.09% (90.75%). The 4-class L7 head is 40 KiB (32 KiB float64 weights + 8 KiB float32 mean/std). Full-split tokenization takes 15.89 s, feature extraction 373.46 s, and ridge fitting 1.75 s per $\lambda$ candidate.
2.5 External MidCBM Retrieval Index + Copy Head
On 399 unseen paraphrases of 3,307 stored ARC science questions, indexing frozen MidCBM and SubspaceTopKCBM states in an external retrieval index with a zero-parameter copy head raises accuracy on the shipped checkpoint from 33.33% (0.2B alone) to 68.42% (+35.09 pp, recall@1 = 66.17%; on the Stage 10 pretraining checkpoint: 28.57% → 72.93%, +44.36 pp, 95% CI [+39.36, +49.38] pp; TF-IDF retrieval + copy head: 81.20%).
2.6 Zero-Shot Commonsense Benchmarks, ChatML Dialogue & Overflow Recall
Zero-shot multiple-choice log-likelihood accuracy across 9 open commonsense and QA datasets (7,068 items total; C4-512 PPL 40.09, zh_clean-512 PPL 58.91):
PIQA (995) |
HellaSwag (1,000) |
ARC-E (558) |
ARC-C (296) |
WinoGrande (998) |
CSQA (1,221) |
OpenBookQA (500) |
BoolQ-1k (1,000) |
XCOPA-zh (500) |
9-Task Macro (7,068) |
|---|---|---|---|---|---|---|---|---|---|
61.01%* |
28.10% |
37.63% |
24.66% |
50.70% |
28.67% |
24.80% |
58.70% |
50.40% |
40.52% |
*Per-task bf16 batching scores 607/995 (61.01%); joint 3,847-item core clean batching scores 608/995 (61.11%, 5-task macro 40.44%).
On 32 held-out bilingual ChatML prompts (max_new_tokens = 256), natural <|im_end|> termination is 32/32 (100%) with 0/32 severe loops under both unpenalized greedy (rep_4 = 0.0361) and default sampling (rep_4 = 0.0039). Verbatim raw outputs across long-context MidCBM recall (6,477–11,389 tokens) and open-domain QA (showing both grounded hits and 212M parametric limitations):
[gate4_en_01 | 6,477 input tokens | needle buried in early turn: "copper willow" | default]
Answer: copper willow<|im_end|>
[gate4_zh_10 | 11,380 input tokens | needle buried in early turn: "细雨竹窗" | default]
答案:细雨竹窗<|im_end|>
[gate4_en_09 | 10,868 input tokens | needle: "7316" — chunk recalled, last digit misread | default]
Answer: 7319<|im_end|>
[qa_zh_10 | live-sensor abstention, no system prompt | default]
User: 我没有提供所在城市或任何传感器信息,你能直接知道我现在窗外有没有下雨吗?
Assistant: 我无法提供您家中降雨量(如:大雨、雪)的实时数据。因此,我无法为您提供实时天气报告或本地天气数据。<|im_end|>
[qa_en_07 | unindexed code synthesis — 0.2B parametric failure | default]
User: Write a short Python function clamp(x, low, high) that keeps a number within [low, high], and show one example.
Assistant: Here is an example of the function clamp:
```python
def count_evens(data):
return sum(1, 2)
```
This function creates a new string that contains all the numbers in the input list. The result is a list with the most common elements and the least common ones. For example, `[0:9]` takes the number from 0 to 10 and adds 1 for every possible element. The resulting string then returns the remaining list element.<|im_end|>
3. Further Research & Next-Gen Multimodal Results
Based on this early 0.2B orthogonal framework, our team has completed further internal work on cross-modal representation alignment and industrial compilation toolchains (including cross-module Gram-Schmidt joint orthogonalization), scaling this formulation to 1B / 3B / 7B multimodal architectures.
On our 1B model (currently in testing and rollout across production workflows), mounting closed-form orthogonal modules without fine-tuning base weights yields:
- Enterprise Email Archiving & Routing:
65.8% → 98.5% - Complex Scanned Document Classification:
64.2% → 97.6% - SAR / Satellite Imagery Classification (used by OSU researchers):
64.7% → 98.3%
Technical details and industrial pipelines (CAD parsing and Tech Pack breakdown) are available at https://topovoid.com.
4. Limitations
212Mparametric capacity. While ChatML post-training supports clean<|im_end|>turn stopping and grounded span extraction, a0.2Bmodel pretrained from scratch does not reliably recall unindexed world facts, execute multi-step arithmetic, or synthesize general code (see Section 2.6).MidCBMoverflow recall is selective chunk retrieval, not full attention above4,096tokens. Only top-scoring evicted chunks (<= 256recall tokens, up to4chunks) plus the prefix sink and recent tail remain in the active window; global aggregation across all evicted turns, near-neighbor numeric/antonym copy (7316 → 7319,深蓝色 → 浅蓝色), or degenerate inputs repeating identical filler text can still fail.- Readout heads are separate modules. A classification head reads L7 features and outputs class scores directly. It does not change what the language model generates and does not show general reasoning gains.
- Cosine
1.0on L7MidCBMcomes from causal order as well as projection. With the first edit at Layer 8, L7MidCBMandSubspaceTopKCBMactivations are computed upstream of the edited weights and cannot change. The L12Highway CBMis downstream of layers 8–11 and is not strictly invariant (0.9979 → 0.9890mean cosine across 2 to 8 stacked modules). - Module stacking interferes beyond 2 modules in this
0.2Bpreview.closed_form_write.pyonly performs single-module orthogonal projection against $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) without a cross-module Gram-Schmidt joint orthogonalization operator. While2stacked modules (80facts) retain96.25%Top-1, adding4or8independently compiled modules degrades Top-1 to55.62%and34.69%and raises perplexity. Cross-module joint orthogonal compilation is part of our internal1Bcommercial compiler. - Report paraphrase, neighborhood, and unprefilled QA metrics beside rewrite Top-1. Declarative and stem-prefilled ChatML prompts reach
100.0%Top-1 on1–40edits, whereas paraphrase generalization (35.0%–41.7%Top-1), unprefilled free-form ChatML recall (41.7%–74.0%), and neighborhood bleed (27.5%–56.7%) are weaker. - Multi-hop hits require the control. On
MQuAKE-CF-3k,117/150multi-hop questions hit the edited answer, but118/149unrelated-subject control questions also trigger the same answer via surface question templates. - Writes raise perplexity on unrelated text, scaling with the number of facts written into a single module.
- The external memory index only recalls stored items, and can copy a wrong stored answer when retrieval is confident but incorrect.
- Pruned vocabulary and no safety tuning. Unsupported tokenizer IDs map to compact ID
0unless decomposed viaencode_orig_ids/closed_form_write.py. Pretraining stages used web text without spam or adult-content filtering, and no safety alignment was applied. Do not deploy directly in untrusted user-facing production.
5. License & Attributions
License
Code and TopoVoid-0.2B-Orthogonal-Preview weights in this repository are released under the Apache License 2.0 (see LICENSE).
- Organization: https://huggingface.co/Topovoid
- Developers: https://huggingface.co/tirpitzia, https://huggingface.co/0kaguya
Attributions
- Tokenizer and distillation teachers: Qwen tokenizer and Qwen teacher models (Qwen, Apache-2.0)
- Pretraining, post-training, and calibration text: FineWeb, FineWeb2 (ODC-By 1.0), C4 (ODC-By 1.0), SmolTalk (Apache-2.0), and Smoltalk-chinese (Apache-2.0)
- Evaluation benchmarks: CounterFact (MIT), MQuAKE (MIT), AG News, DBpedia-14 (CC-BY-SA-3.0), AI2 ARC (CC-BY-SA-4.0), PIQA, HellaSwag, WinoGrande, CommonsenseQA, OpenBookQA, BoolQ, and XCOPA
Citation
@misc{topovoid2026orthogonal,
title = {TopoVoid-0.2B-Orthogonal-Preview: Native Orthogonal Subspace Architecture and Closed-Form Null-Space Weight Compilation},
author = {tirpitzia and 0kaguya and {TopoVoid Research}},
year = {2026},
howpublished = {\url{https://huggingface.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview}},
note = {Research Preview, \url{https://topovoid.com}}
}
- Downloads last month
- 628