QuadOrbit-40M

QuadOrbit-40M is the public research checkpoint evaluated in the paper "QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models." It adds a small bounded complex recurrent memory branch to each Transformer block while retaining causal attention and SwiGLU layers.

This is a base next-token language model. It is not instruction tuned and it should not be presented as a production chatbot.

Model details

Property	Value
Parameters	39,999,240
Hidden width	512
Layers	8
Query heads	8
Key/value heads	1
Orbit width	8 per layer
Context length	512 tokens
Vocabulary	32,768 byte-level BPE tokens
Training step	2,999
Seed	2026

The uploaded weights are the renamed orbitoid_v2_stable_complex_orbit_attention_lm checkpoint. The mathematical model is unchanged. Only the public name is now QuadOrbit.

Usage

Install the dependencies:

pip install "torch>=2.4" "transformers>=5.0" "tokenizers>=0.20" "safetensors>=0.4"

Load and generate:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Argo1-OOAS/QuadOrbit-40M"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device)

inputs = tokenizer("The future of language models", return_tensors="pt").to(device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        use_cache=False,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

The repository contains custom model code, so loading requires trust_remote_code=True. Review modeling_quadorbit.py before loading remote code. This portable release uses the clear PyTorch recurrence and does not require Triton. Generation is faster on a CUDA GPU.

Training data

The model was trained from scratch on a 209,715,200-token mixture:

Source	Share
FineWeb-Edu	45%
DCLM-Edu	25%
Stack-Edu	15%
FineMath-4+	6%
InfiWebMath-4+	5%
Cosmopedia v2	4%

Training used 3,000 updates with 262,144 sampled tokens per update, or about 786.4 million token presentations. Because the prepared corpus contains 209.7 million tokens, examples were sampled more than once during training.

Evaluation

The full validation evaluation used 104,448 consecutive target tokens in FP32.

Model	Validation loss	Perplexity
Parameter-matched Transformer	3.7383	42.02
QuadOrbit-40M	3.7398	42.09

The difference is small and comes from one training seed. It does not establish an improvement over the Transformer at this scale. See quadorbit.pdf for the full method, smaller-model result, proof, and limitations.

Intended use

This release is intended for architecture research, reproducibility, analysis, and small-scale experimentation. Users may study the recurrence, reproduce the reported evaluation, or continue training under the license terms.

Limitations

• It is a small base model and is not suitable for reliable factual assistance.
• It has not been safety tuned or instruction tuned.
• It may generate incorrect, biased, repetitive, or offensive text.
• The context length is limited to 512 tokens.
• The evaluation uses one corpus and one seed.
• The portable recurrence does not implement a generation KV cache.

Do not use this model for medical, legal, financial, safety-critical, or other high-impact decisions.

Licenses

The model weights and repository code are licensed under the Apache License 2.0. The included research paper is licensed under CC BY 4.0. Dataset content is not redistributed here and remains under the terms of its original providers.

Citation

@misc{argo1ooas2026quadorbit,
  title  = {QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models},
  author = {Argo1-OOAS},
  year   = {2026},
  url    = {https://huggingface.co/Argo1-OOAS/QuadOrbit}
}