YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
DPM-15L: Dual-Polynomial Attention (25M Parameters)
This repository contains the weights for DPM-15L, a 15-layer, 25M parameter causal language model utilizing a novel Dual-Polynomial Attention (DPA) mechanism and a fused No-Softmax (NSM) loss landscape.
This model completely replaces the standard Softmax attention and loss determination. The model uses hard zero-floored ReLU activation for raw attention scores and loss determination, which provides mathematical sparsity while maintaining stable gradient flow through its learned polynomial parameters.
Empirical Proof & Pareto Frontier
In test runs, DPM outperforms the standard Softmax baseline in both in-domain rank-order accuracy and zero-shot generalization. Since
Training Setup (OpenWebText)
- Architecture: 15 Layers, 4 Heads, 256 Embed Dim, 1024 FFN, RoPE Positional Embeddings
- Activation: Pure ReLU (Zero-Floored Sparsity)
- Parameters: 25 Million
- Training: 1 Epoch on OpenWebText
Zero-Shot Generalization (Wikitext-103)
A Note on Validation Loss: Because DPM and standard Softmax operate on fundamentally different mathematical loss frameworks (a zero-floored polynomial landscape vs. a dense exponential fraction), their raw NLL Validation Loss values cannot be directly mapped 1:1 against each other. To provide a rigorous, objective comparison, Next-Token Accuracy and Wikitext-103 Zero-Shot Generalization serve as the framework-agnostic benchmarks used to rate and compare model performance.
Testing Results included in the table.
| Model / Seed | Batch Size | OWT Val Loss | OWT Next-Token Acc | Wiki-103 Perplexity | Wiki-103 Zero-Shot Acc | HellaSwag |
|---|---|---|---|---|---|---|
| Softmax Baseline | 32 | 4.05 | 31.14% | 178.44 | 23.72% | 25.22% |
| DPM (Seed 42) | 32 | 3.17 | 40.01% | 81.28 | 28.30% | 25.12% |
| DPM (Seed 105) | 16 | 2.81 | 43.76% | 59.81 | 33.66% | 25.19% |
| DPM (Seed 42) | 16 | 2.71 | 44.81% | 44.67 | 34.17% | 25.67% |
| DPM (Seed 999 - Uploaded) | 16 | 2.69 | 44.26% | 44.05 | 33.15% | 25.85% |
Note: Dropping the batch size from 32 to 16 for Seed 42 unlocked a +4.8% jump in Next-Token Accuracy and cut zero-shot perplexity nearly in half, proving DPM's reliance on stochastic batch noise for optimal convergence.
Usage
You can run a zero-shot evaluation of these weights using the automated Google Colab script available in the TheToyChest GitHub Repository. Note that continuous_abla must be set to False when evaluating this specific checkpoint.