Title: Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization
Primary area: optimization
Keywords: Differentiable entropy regularization

Abstract: We introduce a principled, differentiable measure of computational complexity derived from range-partition entropy. As a regularizer, it trains neural networks to find structured, algorithmically efficient representations without architectural modifications, provably improving the trade-off between model performance and computational resources. In geometry, a learned preprocessor (EntropyNet) reduces input entropy and yields 4–5× speedups on convex hulls and triangulations with <0.2\% error. In Transformers, entropy regularization induces structured attention sparsity with kernel-agnostic FLOP and latency reductions, competitive accuracy at 70–80\% sparsity, interpretable masks (IoU gains vs. L1), and improved robustness on corrupted datasets. We provide uniform approximation bounds, JL/metric robustness, and a stability recipe (anchors $k$, temperature $\alpha$, weight $\lambda$) that makes performance robust. Results hold across ViT/BERT/GPT/LLaMA families and ImageNet/CIFAR/GLUE/WikiText settings, with practical training overheads mitigated to \~2–3.5\% via FAISS and loss scheduling.