Title: A Fast and Scalable Extented Hoyer Projection for Structured Neural Network Sparsity
Primary area: optimization
Keywords: Optimisation of Neural networks sparsity, Structured  Sparsity, Extent Hoyer score, Extent Hoyer score geometry, Fast Hoyer Projection

Abstract: This paper introduces a novel and efficient projection method based on the extended Hoyer score, designed to induce sparsity in neural networks. While the classical $\ell_1$ regularizer is almost everywhere differentiable and admits efficient linear-time projections, it lacks scale invariance. In contrast, the Hoyer score combines the advantages of differentiability and scale invariance, but its associated non-convex optimization problem is challenging.

To address this issue, we define the extended Hoyer score and analyze the properties of the corresponding space. 
Building on a first lemma showing that the extended Hoyer surface forms a hypercone, we propose a naïve projection algorithm. 
Then, leveraging a second lemma that provides an efficient thresholding rule, we derive a fast projection algorithm requiring only a single hypercone projection. 
Furthermore, we extend the classical bi-level $\ell_{1,\infty}$ projection to the more general $\ell_{H,\infty}$ projection, enabling improved column-wise structured sparsity.

Experimental results demonstrate that our extended Hoyer projection achieves linear complexity. We apply the proposed bi-level Hoyer projection to classification tasks using autoencoder and transformer architectures. The experiments show that our method not only directly determines the number of selected features but also improves the accuracy-sparsity trade-off.

TL;DR: This paper introduces a novel and efficient projection method based on the extented Hoyer score, designed to induce sparsity in neural networks.