cloverx-id/XoneLM-1.0-Paper

XoneLM & LuminaV Research Papers

(What Happens When You Code at 3 AM: The Architecture & The Optimizer)

Check the 'Files and versions' tab to find the source code.

1. Architecture Paper XoneLM Architecture (54M) Read XoneLM.pdf

2. Optimizer Paper LuminaV Optimizer Theory Read LuminaV.pdf

Click on each image to read or download its official PDF file.

Research Paper Overview

Abstract

XoneLM is an experimental 54.07M-parameter language model constructed by integrating Stiefel manifold QR decomposition, degree-180 Chebyshev polynomial positional encodings, topological soliton wave tracking, rational power-law attention decay, Poincare hyperbolic routing, and low-rank latent Key-Value compression (MLA).

The entire system was trained from scratch on 32.01M tokens using the custom LuminaV tanh-bounded optimizer on a single consumer GPU in under two hours. The training run achieved monotonic convergence (Final Loss: 1.7355, Perplexity: 5.67) with zero loss spikes, zero gradient explosions, and zero arithmetic underflow errors.

Empirical Telemetry & Training Specs

Metric	Value
Total Active Parameters	54,073,344 (54.07M)
Architecture Backbone	12 Layers, 8 Attention Heads, Dim 512
Key-Value Latent Dimension	64 (Low-Rank Joint MLA)
Working Memory Slots	512 (256 Static + 256 Dynamic)
Optimizer	LuminaV-2B (Learning Rate: 8e-4)
Precision	Mixed Precision FP16
Training Tokens	32,009,639 Tokens (TinyStories)
Wall-Clock Training Time	117.88 minutes (1.96 hours)
Peak Training Throughput	9,868 tokens/second
Peak VRAM Usage	7.29 GB / 14.56 GB (Tesla T4)
Hub Diversity Z-Loss	0.0014
Final Loss / Perplexity	1.7355 / 5.67

Qualitative Analysis: The Box vs. Ball Case Study

During unconditioned zero-shot evaluation on the base pre-trained model:

• Prompt: "Once upon a time, Lily found a Box."
• Output: "It was a big, round ball. She was so excited to play with it..."

The model produced syntactically perfect English and dialogue quotation marks without infinite looping. However, the pre-training distribution prior of the TinyStories corpus (which heavily features children playing with balls in parks) overrode the prompt keyword. This highlights that unconditioned base models act as probabilistic continuation engines, and Supervised Fine-Tuning (SFT) is necessary for strict instruction adherence.

---

Architectural Components

1. Stiefel QR Working Memory Hub: An orthogonal working memory bank initialized via QR factorization on Stiefel manifolds to enforce metric stability from step zero.
2. PolyHoPE Positional Encodings: Degree-180 Chebyshev polynomials of the first kind evaluated on normalized token intervals to guarantee continuous variance without periodic decay.
3. Topological Soliton State Tracking: Discrete nonlinear sech-squared wave updates derived from collisionless plasma dynamics to maintain latent memory state stability.
4. LinHoPE Attention Decay: Heavy-tail Cauchy power-law decay combined with geometric recency bias to mitigate early token context amnesia.
5. Poincare Hyperbolic Routing: Episodic memory cluster assignment on Riemannian conformal unit disks.
6. Latent KV Compression: Low-rank Key-Value joint compression (dkv = 64) minimizing memory bandwidth during autoregressive decoding.
7. LuminaV Optimizer: Master-weight-free parameter optimization featuring a hyperbolic tangent bounding envelope, central innovation variance tracking, and cautious directional masking.

---

Quickstart: Training and Inference

1. Installation

git clone https://huggingface.co/cloverx-id/XoneLM-1.0-Papper
cd XoneLM-1.0-Papper
pip install -r requirements.txt

2. Running a Training Step with LuminaV

import torch
from tokenizer import build_xonelm_tokenizer
from modeling_xonelm import XoneLM
from luminav import LuminaV

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = build_xonelm_tokenizer()
vocab_size = len(tokenizer)

model = XoneLM(
    vocab_size=vocab_size,
    dim=512,
    num_layers=12,
    num_heads=8,
    kv_latent_dim=64,
    hub_size=512,
    num_terminals=32,
    slots_per_terminal=16
).to(device)

optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,
    betas=(0.9, 0.999),
    tau=0.8,
    buffer=2,
    cautious=True,
    execution="auto"
)

dummy_tokens = torch.randint(0, vocab_size, (2, 512), device=device)
dummy_labels = torch.randint(0, vocab_size, (2, 512), device=device)

optimizer.zero_grad()
output = model(dummy_tokens, labels=dummy_labels)
loss = output.loss
loss.backward()
optimizer.step()

print(f"Training step successful. Loss: {loss.item():.4f}")

3. Running Autoregressive Generation (DRY + Min-P)

from generate import generate_response

prompt = "Once upon a time, Lily found a Box."
result = generate_response(
    model=model,
    tokenizer=tokenizer,
    prompt_or_messages=prompt,
    max_new_tokens=64,
    temperature=0.45,
    min_p=0.08
)
print("Output:", result)

---

Citation

@misc{xonemi2026xonelm,
  title={XoneLM: An Over-Engineered 54M Language Model with Non-Euclidean Memory, Polynomial Encodings, and Bounded Optimizers},
  author={XoneMi},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/cloverx-id/XoneLM-1.0-Papper}}
}

License

All code and architecture assets (including PDFs) are released under the Apache-2.0 License.

---