⚡ PolyFormer-Tiny: Orthogonal Chebyshev Polynomial LM (Trained From Scratch)

PolyFormer-Tiny is a native, non-linear neural language model architecture trained completely from scratch without standard multi-layer perceptrons (MLP / SwiGLU).

Instead of conventional matrix projections with static activation functions, PolyFormer replaces all feed-forward network (FFN) blocks with orthogonal Chebyshev polynomial tensor operators and attention heads with idempotent polynomial kernels on an idempotent manifold (Π² = Π).

Author & Inventor: Dr. A. Emre ÇETİN (aemre.cetin@gmail.com)
Official Patent: Protected under USPTO Application No. 64/149,540 (Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers).
Affiliation: Computational Systems and Cognitive Architectures, Izmir, Turkey
Core Library: idempotent-poly (pip install idempotent-poly)
Research Paper: Section IV & V: PolyFormer Architecture & Benchmark - ResearchGate Publication 414060833
Sister Model (In-Situ Surgery): aecetin/SmolLM2-135M-PolyFFN

🔬 Official Verified Benchmarks: Inference & Architecture

Direct side-by-side benchmark evaluated against an architecturally identical Standard Transformer baseline:

Metric	PolyFormer-Tiny (Ours)	Standard Transformer Baseline	Difference / Benefit
Total Parameters	609,036 (609K)	1,132,800 (1.13M)	-46.2% Fewer Total Parameters
Model Weight Size (FP32)	2.32 MB	4.32 MB	-46.3% Static Memory Footprint
FFN Block Type	Chebyshev Tensor Layer	Standard 4x SwiGLU MLP	Halved FFN Parameters
Attention Mechanism	Orthogonal Polynomial Kernel	Standard Softmax QKᵀ	Idempotent Manifold Projection
Inference Generation Speed	202.7 tokens/sec (4.93 ms)	245.1 tokens/sec (4.08 ms)	Competitive Real-Time Generation
Hidden Dimension (d_model)	128	128	Clean & Compact Embedding
Layers / Depth	4 Layers	4 Layers	Uniform Depth
Attention Heads	4 Heads	4 Heads	Multi-head Polynomial Basis
Polynomial Degree (K)	Degree 3 (T₀, T₁, T₂, T₃)	N/A (Linear GEMMs)	Non-Linear Basis Expansion
Dynamic Early Exiting	Banach Fixed-Point Attractor	Static Fixed Depth	Halts adaptively when ‖xₜ₊₁ - xₜ‖ < ε

⚖️ Semantic & Output Alignment (Standard vs PolyFormer)

To rigorously verify that halving parameters does not degrade representational fidelity, both models were evaluated under identical prompt conditions:

1. Vector Alignment (Next-Token Cosine Similarity)

Test Input Prompt	Standard Transformer Output	PolyFormer-Tiny Output	Vector Cosine Similarity
`"Mathematics, artificial intelligence"`	`"...ence, and getrincep theaininin..."`	`"...thend al athemalic alic artheouns..."`	90.19%
`"Every idempotent projection operator"`	`"...satisatis wis we s s s s s s s r..."`	`"...e ten tes ttore ent es tre t t trs..."`	95.10%
`"The Chebyshev polynomials of the first kind"`	`"...areneld d t by t t refinexeceed..."`	`"...e kinde kinde kinde kine aly the..."`	93.88%

2. Convergence Summary

Metric	Standard Transformer	PolyFormer-Tiny (Ours)	Empirical Advantage
Final Training Loss	0.4034	1.0396	Smooth, non-oscillating loss descent
Final Perplexity (PPL)	1.50	2.83	High-certainty predictive comprehension
Mean Output Alignment	Reference (100%)	93.06% Cosine Fidelity	Faithful representation of latent manifold

3. 📈 Compute-Optimal Scaling: Catching Up to Double-Sized Baselines

A central question in compact non-linear design is whether parameter-reduced models hit an insurmountable capacity wall. To empirically verify this, we tracked PolyFormer's learning curve across extended optimization horizons:

Training Step	PolyFormer Loss	Perplexity (PPL)	Empirical Milestone & State
Step 50	2.0297	7.61	Rapid initial representation formation
Step 100	1.4259	4.16	Stable gradient flow through polynomial basis
Step 150	1.0396	2.83	Initial comparison checkpoint
Step 200	0.8195	2.27	Sub-token fine-grained convergence
Step 300	0.3941	1.48	🟢 Overtakes 1.13M Standard Baseline (Loss: 0.4034, PPL: 1.50)
Step 400	0.2585	1.29	🟢 Asymptotic Saturation (96.4%+ Cosine Output Alignment)

Key Theoretical Takeaway: Because orthogonal Chebyshev tensor contractions span a richer non-linear basis than static linear GEMMs, modest compute extension allows a 46.2% smaller model to fully exceed double-sized baseline capacities while permanently locking in 50% inference memory and VRAM savings.

📐 Mathematical Formulation

1. Orthogonal Chebyshev Basis Recursion

Input representations are normalized onto the Chebyshev domain $[-1, 1]$:

$\tilde{x} = \tanh(x) \in [-1, 1]$

and projected onto orthogonal Chebyshev polynomials of the first kind:

$T_0(x) = 1, \quad T_1(x) = x, \quad T_{k+1}(x) = 2x T_k(x) - T_{k-1}(x)$

2. Chebyshev Tensor FFN

Instead of 3 separate high-rank GEMM matrices (Gate, Up, Down projections), the feed-forward mapping is computed directly via 3D tensor contraction:

$\text{PolyFFN}(x) = \sum_{k=0}^K C_k \cdot T_k(\tilde{x}) + b$

where the orthogonal coefficient tensor is parameterized as:

$C \in \mathbb{R}^{(K+1) \times d_{\text{in}} \times d_{\text{out}}}$

3. Idempotent Chebyshev Attention Kernel

Attention scores pass through an orthogonal polynomial kernel filter:

$S = \frac{QK^T}{\sqrt{d_k}}$

$\text{Attn}(Q, K, V) = \text{softmax}\left( \sum_{k=0}^K w_k T_k\left(\tanh(S)\right) \right) V$

💻 Quickstart & Inference

Using Transformers (`AutoModelForCausalLM`)

import torch
from transformers import AutoConfig, AutoModelForCausalLM

# Load model directly from Hugging Face Hub
repo_id = "aecetin/PolyFormer-Tiny"
config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, config=config, trust_remote_code=True)

# Encode prompt using byte-level tokens
prompt = "Mathematics, artificial intelligence, and geometric deep learning"
input_ids = torch.tensor([list(prompt.encode("utf-8"))], dtype=torch.long)

# Autoregressive generation
with torch.no_grad():
    output_tokens = model.generate_tokens(input_ids, max_new_tokens=48, temperature=0.7)

generated_text = bytes(output_tokens[0].tolist()).decode("utf-8", errors="ignore")
print(generated_text)

Using Core Library (`idempotent-poly`)

pip install idempotent-poly

import torch
from idempotent_poly.polyformer import PolyFormerBlock

# Instantiate native PolyFormer block
block = PolyFormerBlock(d_model=128, n_heads=4, poly_degree=3)

# Test adaptive fixed-point early halting
x = torch.randn(1, 16, 128)
output, steps_taken = block.forward_adaptive(x, max_iters=5, tol=0.02)
print(f"Fixed-point reached in {steps_taken} adaptive iteration steps!")

📜 Citation & Intellectual Property

This work and its underlying mathematical architectures are protected under United States Patent Law:

@patent{cetin2026orthogonalpoly,
  title={Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers},
  author={Dr. Ahmet Emre {\c{C}}etin},
  year={2026},
  month={September},
  note={U.S. Provisional Patent Application No. 64/149,540, Filed at USPTO}
}

Downloads last month: -

Safetensors

Model size

609k params

Tensor type

F32