Title: Spark Transformer: Reactivating Sparsity in Transformer FFN and Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

activation sparsity
Abstract: The discovery of the *lazy neuron phenomenon* (Li et al., 2022), where fewer than 10% of the feedforward networks (FFN) parameters in trained Transformers are activated per token, has spurred significant interests in *activation sparsity* for enhancing large model efficiency. While notable progress has been made in translating such sparsity to wall-time benefits across CPUs, GPUs, and TPUs, modern Transformers have moved away from the ReLU activation function crucial to this phenomenon. Existing efforts on re-introducing activation sparsity, e.g., by reverting to ReLU or applying top-k masking, often degrade model quality, increase parameter count, or complicate training. Sparse attention, the application of sparse activation to the attention mechanism, often face similar challenges.
    
This paper introduces the Spark Transformer, a novel architecture that achieves high activation sparsity in both FFN and the attention mechanism while maintaining model quality, parameter count, and standard training procedures. Our method realizes sparsity via top-$k$ masking for explicit control over sparsity level. Crucially, we introduce *statistical top-k*, a hardware-accelerator-friendly, linear-time approximate algorithm that avoids costly sorting and mitigates significant training slowdown from standard top-k operators. Furthermore, Spark Transformer reallocates existing FFN parameters and attention key embeddings to form a low-cost predictor for identifying activated entries. This design not only mitigates quality loss from enforced sparsity, but also enhances wall-time benefit. Pretrained with the Gemma-2 recipe, Spark Transformer demonstrates competitive performance on standard benchmarks while exhibiting significant sparsity: only 8\% of FFN neurons are activated, and each token attends to a maximum of 256 tokens. This translates to a 2.5x reduction in FLOPs, leading to decoding wall-time speedups of up to 1.79x on CPU and 1.40xon GPU.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed Spark Transformer consists of two parts: Spark FFN and Spark Attention. The authors dedicate/repurpose a portion of hidden states to estimate the importance of weights and KV vectors against the given hidden state (e.g., K and V vectors). Further, Spark Transformer identifies top-K elements in an approximate manner, not requiring the costly process of sorting. On the CPU and low-profile GPU machines, the proposed Spark Transformer shows notable speedup ranging from 1.25x to 1.79x.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The core ideas of embedding sparsity information within hidden states and avoiding the cost of sorting by approximately estimating top-k elements are well taken.\n+ The paper is well written and structured organizationally.\n+ Presenting results for CPUs and GPUs is another plus.\n\nWeaknesses:\n- Spark Transformer seems to target low-profile devices, where the degree of batching is not likely high (e.g., mobile devices). In this case, speculative decoding is gaining popularity, where the effective batch size of a ‘target’ model  increases. This paper does not even mention speculative decoding though.\n- It is not clear if the proposed idea can be highly effective to the cloud environments, which use more powerful GPUs/TPUs and where the batch size is neither extremely high (violating goodput in servers) or low (running without speculative decoding in mobile devices), thus, it would be limited by the memory bandwidth of fetching weights from main memory (e.g., HBM). In these scenarios, saving computation might not lead to any considerable improvement in overall performance and energy efficiency as the serving system is memory-bound and communication dominates computation in energy consumption. More discussion in these deployment scenarios will further strengthen the paper.\n\nQUESTIONS:\n* Can you further articulate the interaction of Spark Transformer and speculative decoding?\n* Can you further quantify or qualitatively discuss the effectiveness of Spark Transformer on conventional serving environments on the cloud?\n* Can you discuss the interplay between Spark Transformer and quantization techniques? (e.g., sensitivity of its accuracy while using different data types.)\n\nNot questions, but minor comments:\n*  Section vs. section: be consistent.\n* “an” NVIDIA GPU\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1  Summary\nThe paper proposes Spark Transformer, an architecture that re‑introduces and exploits activation sparsity in both feed‑forward networks (FFN) and multi‑head attention. The key technical pieces are\n* Spark FFN – splits the input,  locate the k most relevant neurons via the \"statistical top‑k operator\", and performs the remaining matmuls sparsely.\n* Spark Attention – mirrors the same idea for query–key scores, limiting each token to at most 256 attended tokens.\n* Statistical top‑k – a linear‑time, differentiable approximation of top‑k based on Gaussian quantiles coupled with soft‑thresholding; analysis (Thm 3.1/3.2) bounds the expected error and shows differentiability.\n* An implementation on Gemma‑2 (2 B) shows ≈ 8 % active FFN neurons and ≤ 256 attention edges, yielding 2.5× theoretical FLOP reduction and 1.86× CPU / 1.25–1.40× GPU decode speed‑ups with negligible quality loss on standard LLM benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Novel formulation utilizing activation sparsity which does not requires  Deja-vu-style \"predictor\".\n2. Linear‑time top‑k backed by theory and ablation showing <1 % training slow‑down versus >10× for JAX approx‑max‑k.\n3. End‑to‑end pre‑training at scale (2 T tokens) with near‑neutral accuracy on 8 diverse downstream tasks.\t\n4. Evidence of wall‑time wins on commodity 16‑core CPUs and a low‑profile NVIDIA T4 GPU.\n\n## Weaknesses\n1. GPU baseline ambiguity – Fig. 5 compares “optimized vs. un‑optimized” Spark kernels; the paper does not show a direct comparison against a dense Gemma‑2 baseline, making the reported 1.25–1.40× speed‑up hard to interpret.\n2. Regularisation ablation missing – Sec. 3.2 advocates Huber regularisation for smoothing statistical top‑k, but no experiment contrasts Huber with the common ReLU (or no) regulariser to isolate its benefit.\n3. Comparison to predictor‑based sparse methods – the paper qualitatively cites Deja‑Vu, PowerInfer, etc., yet does not provide a quantitative comparison on speed‑up vs. additional parameters / training cost, making it hard to gauge relative efficiency.\n\nQUESTIONS:\n1. GPU speed‑up baseline – Could you provide absolute tokens/s for Spark versus a dense Gemma‑2 implementation on the same T4, so the community can reproduce the 1.25–1.40× figure? (Fig. 5’s GPU bars compare Spark without optimization  vs. Spark with optimization, not Spark vs. dense Gemma‑2)\n2. Huber versus ReLU – Have you tried dropping Huber (δ → 0) or replacing it with the ReLU‑style hard threshold? How does this affect convergence and final quality?\n3. Predictor overhead comparison – When integrating a Deja‑Vu‑style low‑rank predictor into a standard FFN (no splitting), what is the FLOP/latency trade‑off relative to Spark FFN?\n4. Attention sparsity generality – Did you evaluate Spark Attention on tasks requiring long‑range reasoning (e.g., PG‑19, LongBench) to ensure the 256‑edge cap does not hurt quality?\n5. Training stability – Were any gradient‑scaling or learning‑rate adjustments needed because of statistical top‑k, or does the recipe match Gemma‑2 exactly?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper modifies the transformer architecture into a new Spark Transformer model, leveraging sparsity in the model activations through predictors that decide which modules in the model are activated during inference. Sparsity is introduced in the training process, without any additional post-training modifications to the model. To efficiently identify sparse parts of the model, the authors introduce a novel statistical top-k operator. Spark Transformer version of Gemma-2B performs close to the original model using 40% of its FLOPS and provides 1.8 inference speedup on CPU and 1.4 inference speedup on GPU.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. The paper introduces a method to directly integrate activation sparsity into the model via jointly trained sparsity predictors, which avoids the need to apply any post-training modifications and leads to an elegant solution.\n\n2. The proposed top-k operator is a novel and efficient alternative to standard top-k selection.\n\n3. Unlike many prior works that focus solely on FFN sparsity, this paper applies activation sparsity to attention mechanisms as well.\n\n4. The method supports efficient batched inference, as shown in Appendix C.3, which is not always given in sparse activation papers.\n\n5. Writing is very clear and easy to follow.\n\n# Weaknesses\n\n1. The empirical results are limited to a single model (Gemma-2B), and there are no results that would provide evidence that the method generalizes across architectures, model sizes, or tasks.\n\n2. The paper mentions related works on activation sparsity and provides a comparison between their approach and existing alternatives in Table D.1. However, this table contains only main results scraped from the other papers, not experimental results reproduced by the authors. Moreover, these results are not really compatible, as they refer to different models. Therefore, the paper does not provide sufficient results to convincingly position the method with respect to the other existing alternatives. Additionally, while wall-clock speedups for Spark Transformer are mentioned in the paper, the table comparing Spark with other methods includes only FLOPS, which do not directly translate to the inference efficiency.\n\n3. Paper emphasizes CPU speedups and does not provide results for common GPUs like A100, which makes it hard to judge the practical impact of the method for the most common deep learning accelerators.\n\n4. The paper gives a fairly detailed description of the architectural modifications, including hyperparameters, training setup, and implementation details. However, the authors do not share the code, which greatly affects the reproducibility of their work.\n\n# Summary\n\nWhile the paper proposes an interesting and potentially impactful idea, the lack of experimental evaluation makes it impossible to judge the impact of the proposed method in practice and judge whether it provides a significant efficiency improvements compared to already existing alternatives.\n\nQUESTIONS:\nWhy is the idea only evaluated for a single, relatively small LLM? The authors justification for not providing the training code (“Training data and code for Gemma-2 are proprietary.”) indicates that the authors should have access to sufficient computational resources. If the authors cannot provide the code, the paper should at least contain strong experimental evidence that allows the reader to adequately compare the proposed method to the alternatives and judge its impact of wider set of models. However, all the experiments are contained to a single architecture (Gemma 2B) and the authors do not perform sufficient experiments to compare existing alternatives to their method even in this single setting.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Spark Transformer, a novel architecture designed to re-enable activation sparsity in both feedforward networks (FFNs) and attention mechanisms of Transformer models. The authors propose a statistical top-k operator—an efficient, differentiable approximation to top-k masking that avoids expensive sorting. They also design a unified predictor framework using reallocated parameters to estimate activation importance without additional overhead. The Spark Transformer achieves high sparsity (e.g., 8% FFN activation, ≤256 attended tokens), significantly reducing FLOPs while maintaining near-baseline quality. Extensive experiments on Gemma-2 show 1.4x decoding speedups on GPU, demonstrating the practicality and efficiency of their approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. The Spark Transformer introduces a unified architecture for both FFN and attention sparsity using reallocated model parameters as low-cost predictors, avoiding extra modules or parameters and maintaining training simplicity.\n2. The proposed statistical top-k operator provides an efficient, differentiable, and hardware-friendly alternative to traditional top-k methods, enabling sparse activation without significant training overhead.\n3. The model achieves significant FLOPs and wall-time reductions (up to 1.79× on CPU, 1.4× on GPU) while maintaining competitive accuracy on standard downstream benchmarks compared to dense baselines like Gemma-2 2B.\n\n**Weaknesses**:\n1. The experiments are conducted solely on the Gemma-2 2B model, leaving it unclear how well the proposed method scales to larger models or generalizes to more complex scenarios such as instruction-tuned, or multimodal settings.\n2. The paper introduces statistical top-k and shows that it outperforms standard top-k in empirical results. However, the paper lacks an in-depth analysis of why statistical top-k performs better. Is the improvement due to better differentiability? This is particularly important since the non-differentiability of sparse top-k has long been a bottleneck in training sparse MoE models, with many works proposing improved gating mechanisms. If statistical top-k indeed provides both performance gains and differentiability, it would be valuable to explore its potential applicability to MoE training, which the paper does not address.\n\nQUESTIONS:\nNo other questions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}