⚡ SmolLM2-135M-PolyFFN: Orthogonal Chebyshev Polynomial Weight Surgery

SmolLM2-135M-PolyFFN is an algebraically compressed and surgically transformed version of HuggingFaceTB/SmolLM2-135M-Instruct.

All 30 Transformer feed-forward network (FFN / SwiGLU) layers have been mapped into orthogonal Chebyshev polynomial tensor operators using Zero-Backpropagation closed-form algebraic solving on an idempotent spectral manifold (Π² = Π).

• Author & Inventor: Dr. A. Emre ÇETİN (aemre.cetin@gmail.com)
• Official Patent: Protected under USPTO Application No. 64/149,540 (Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers).
• Affiliation: Computational Systems and Cognitive Architectures, Izmir, Turkey
• Core Library: idempotent-poly (pip install idempotent-poly)
• Research Paper: ResearchGate Publication 414060833
• Sister Model (From Scratch): aecetin/PolyFormer-Tiny

---

🔬 Official Verified Benchmarks

Evaluated on SmolLM2-135M-Instruct across multiple surgical depth configurations:

Configuration	Surgical Depth	FFN Parameter Compression	Zero-Backprop Solve Time	Perplexity (PPL)	Output Coherence & Quality
SmolLM2-135M (Baseline)	0 Layers	0 (0.0%)	N/A	51.53	🟢 100% (Baseline Reference)
1-Layer Poly Surgery (L15)	1 Layer	-1,326,527 (-50.0% FFN)	56.2 ms	55.45	🟢 Natural, Factual & Fluent ("Paris")
2-Layer Poly Surgery (L14-L15)	2 Layers	-2,653,054 (-50.0% FFN)	206.9 ms	60.38	🟢 Natural & Grammatical
3-Layer Poly Surgery (L13-L15)	3 Layers	-3,979,581 (-50.0% FFN)	219.2 ms	59.42	🟢 Semantically Intact
1-Layer Residual Poly (L15)	1 Layer	Residual Adaptation	146.9 ms	51.53	🟢 Exact Bit-Match (Identical PPL)

Architectural Comparison

Metric	Original SmolLM2-135M	SmolLM2-135M-PolyFFN (Ours)	Difference / Gain
FFN Parameters per Layer	2,654,208	1,327,104	-50.0% FFN Parameters
Total Model Parameters	134,516,736	94,756,224	-39.76 Million Parameters (-29.5%)
Optimization Method	Multi-epoch AdamW Backprop	Closed-form SVD + Ridge	Zero Backpropagation
Convergence Time (per Layer)	Approx. 33.5 s (GPU)	56.2 ms - 139 ms	240.9x Faster
Algebraic Idempotency Error	N/A (P² ≠ P)	‖Π² - Π‖ / ‖Π‖ = 4.70 × 10⁻⁵	Exact Machine Idempotency
Inference Quality	Natural & Coherent	Natural & Coherent	Verified Coherent Output

---

🔗 Sister Architecture: Ground-Up PolyFormer (From Scratch)

Beyond weight surgery on existing LLMs, orthogonal polynomial tensors have been evaluated as a complete replacement for standard Transformers from scratch:

• Checkpoint: aecetin/PolyFormer-Tiny
• -%46.2 Fewer Parameters (609K vs 1.13M baseline)
• 202.7 tokens/sec real-time generation on CPU
• 93.06% Next-Token Cosine Alignment with standard Transformer
• Overtakes 1.13M model in 300 steps (PPL 1.48 vs 1.50) while locking in permanent 50% inference RAM savings.

---

📐 Mathematical Formulation

Instead of high-rank multi-matrix projections with intermediate non-linear activations:

$$\text{FFN}(x)=\text{down\_proj}(\text{SiLU}(\text{gate\_proj}(x))\odot \text{up\_proj}(x))\backslash text\{FFN\}(x)\; =\; \backslash text\{down\backslash \_proj\}(\backslash text\{SiLU\}(\backslash text\{gate\backslash \_proj\}(x))\; \backslash odot\; \backslash text\{up\backslash \_proj\}(x))$$

We project input representations directly onto orthogonal Chebyshev polynomial basis manifolds:

$$T_0(x)=1,T_1(x)=x,T_{k+1}(x)=2x{T}_k(x)-T_{k-1}(x)T\_0(x)\; =\; 1,\; \backslash quad\; T\_1(x)\; =\; x,\; \backslash quad\; T\_\{k+1\}(x)\; =\; 2x\; T\_k(x)\; -\; T\_\{k-1\}(x)$$

The feed-forward mapping is computed via tensor contraction:

$$\text{PolyFFN}(x)=\sum _{k=0}^K{C}_k\cdot T_k(\tilde{x})+b\backslash text\{PolyFFN\}(x)\; =\; \backslash sum\_\{k=0\}^K\; C\_k\; \backslash cdot\; T\_k(\backslash tilde\{x\})\; +\; b$$

The coefficient tensor is solved analytically via regularized normal equations:

$$C^{\ast }=(\mathrm{\Phi }(X{)}^T\mathrm{\Phi }(X)+\lambda I{)}^{-1}\mathrm{\Phi }(X{)}^{T}YC^*\; =\; (\backslash Phi(X)^T\; \backslash Phi(X)\; +\; \backslash lambda\; I)^\{-1\}\; \backslash Phi(X)^T\; Y$$

and projected onto the idempotent subspace:

$$\mathrm{\Pi }=V_r{V}_r^T,{\mathrm{\Pi }}^2=\mathrm{\Pi }\backslash Pi\; =\; V\_r\; V\_r^T,\; \backslash quad\; \backslash Pi^2\; =\; \backslash Pi$$

---

💻 Quickstart & Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from idempotent_poly import ChebyshevPolyFFN

# Load model and tokenizer
model_id = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32)

# Surgically replace Layer 15 with ChebyshevPolyFFN in 56 ms
target_layer = model.model.layers[15]
calib_tokens = tokenizer(["Mathematics and artificial intelligence thrive on idempotent manifolds."], return_tensors="pt").input_ids

captured = {}
def hook_fn(m, inp, out):
    captured["X"] = inp[0].detach().reshape(-1, model.config.hidden_size)
    captured["Y"] = out.detach().reshape(-1, model.config.hidden_size)

h = target_layer.mlp.register_forward_hook(hook_fn)
with torch.no_grad():
    model(calib_tokens)
h.remove()

poly_ffn = ChebyshevPolyFFN(d_model=model.config.hidden_size, degree=3)
poly_ffn.fit_algebraic(captured["X"], captured["Y"], l2_reg=1e-2, rank_ratio=0.95)
target_layer.mlp = poly_ffn

# Generate text
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=25, do_sample=False)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# Output: The capital of France is Paris. Paris is a city that is known for its historical landmarks

---

📜 Citation & Intellectual Property

This work and its underlying mathematical architectures are protected under United States Patent Law:

@patent{cetin2026orthogonalpoly,
  title={Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers},
  author={Dr. Ahmet Emre {\c{C}}etin},
  year={2026},
  month={September},
  note={U.S. Provisional Patent Application No. 64/149,540, Filed at USPTO}
}