Title: EUGens: Efficient, Unified and General Dense Layers
Venue: NeurIPS 2025
Accepted: yes
Keywords: feedforward fully-connected layers

MLPs

random features

general dense layers

linear time complexity layers
Abstract: Efficient neural networks are essential for scaling machine learning models to real-time applications and resource-constrained environments. Fully-connected feedforward layers (FFLs) introduce computation and parameter count bottlenecks within neural network architectures.  To address this challenge, in this work, we propose a new class of dense layers that generalize standard fully-connected feedforward layers, $\textbf{E}$fficient, $\textbf{U}$nified and $\textbf{Gen}$eral dense layers (EUGens).  EUGens leverage random features to approximate standard FFLs and go beyond them by incorporating a direct dependence on the input norms in their computations. The proposed layers unify existing efficient FFL extensions and improve efficiency by reducing inference complexity from quadratic to linear time. They also lead to $\textbf{the first}$ unbiased algorithms approximating FFLs with arbitrary polynomial activation functions. Furthermore, EuGens reduce the parameter count and computational overhead while preserving the expressive power and adaptability of FFLs. We also present a layer-wise knowledge transfer technique that bypasses backpropagation, enabling efficient adaptation of EUGens to pre-trained models. Empirically, we observe that integrating EUGens into Transformers and MLPs yields substantial improvements in inference speed (up to $\textbf{27}$\%) and memory efficiency (up to $\textbf{30}$\%) across a range of tasks, including image classification, language model pre-training, and 3D scene reconstruction. Overall, our results highlight the potential of EUGens for the scalable deployment of large-scale neural networks in real-world scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new type of neural network layer called \"EUGene\", using random feature methods that speed up the inference time and reduce the training parameters of the network. The authors also provide a theoretical analysis of EUGen's estimation with the concentration bounds. The authors validate their approach by demonstrating how to integrate EUGen layers in popular architectures like tranformers and Neural Radiance Fields with experiments on LLM pre-training, vision transformers, and 3D image reconstruction.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n * S1) Zero-shot integration is demonstrated on many important and commonly used architectures such as transformers and neural radiance fields, showing that the inference speed decreases without retraining with the classical back-propagation algorithm. \n * S2) Theoretical analysis including EUGene’s approximation results and concentration bounds is presented. \n * S3) Many architectures use feed-forward neural networks as building blocks and thus, this work can potentially impact many applications. \n * S4) Writing is clear and easy to follow.\n\nWeaknesses:\n * W1) Important future work directions and limitations are not adequately detailed. \n * W2) (small weakness): notation in line 103 is confusing (w+, x+), because \"+\" is usually used to denote \"pseudo inverse\". Maybe \"w_1\" and \"x_1\" can be used to denote appending ones.\n * W3) l105: \"the vectors\" G are not defined. What are they? Trainable parameters, derived from W? Not clear at this point.\n * W4) The theoretical analysis is based entirely on the assumption that the modelled network layer has a polynomial activation function. this is excluded explicitly in universal approximation theorems (see questions).\n\nQUESTIONS:\n1) Figure 1: Why does the loss increase with more EUGene layers? Is this for the same reason as outlined in the experiment 2? If so, I would encourage the authors to address this in the earlier experiment than the one that comes later. \n2) Figure 1: How does EUGene with more layers has fewer inference parameters (lower fraction of inference parameters)? \n3) What are the important future work directions in the authors' views? \n4) Why is theorem 3.1 relevant for neural network approximations? Polynomial activation functions are always explicitly excluded in all universal approximation theorems, because they lead to polynomial approximations - and these are extremely ill conditioned, i.e. do not really model classical neural network approximations. Also, as (l117) \"in practice we use k <= 3\", and l164 \"k <= 2\", so the polynomial approximation certainly is not \"arbitrarily accurate\" (for which k has to approach infinity). All theorems seem to be based on the polynomial activation assumption, which makes it very important to clarify this point. I may have missed an important argument for this assumption and am willing to raise the score if it can be explained properly. Simply stating that \"Bernstein polynomials can approximate arbitrary continuous functions\" is not enough, since k is very small.\n5) l105: m is not defined, is it equal to l? Probably not, since l117 assumes m << min(l,d).\n\nLIMITATIONS:\nI would encourage the authors to make a separate section on limitations and future work and elaborate on the important points. This will help the reader. The limitations are mentioned, but are hidden in the details of the experiments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The authors propose a novel approximation to the dense layer that has improved compute complexity in terms of layer size. Extensive experiments show end-to-end speed up over fully connected layers.

STRENGTHS AND WEAKNESSES:
## Strengths:
* Extensive experiments.

* Practically relevant problem.

* Easy to interpret trade-off plots (e.g. Fig. 3b)


## Weaknesses:
* No code is given, making verification of the author's claims unreasonably time-consuming.  

* Line 87 states that the key distinguishing feature of EUGens over SNNK is that "[SNNK] cannot be applied to unbounded activation functions (e.g. ReLU)", but Fig. 2 shows results for SNNK on a ReLU network --> can the authors please explain this contradiction?

* The claim  that model size can be reduced seems to reference line 210, and Tab. 7 -- Here we see that the 30% reduction in parameters causes >0.5dB degradation in output quality. This is far from equal qualtiy. I suggest the authors relativize this claim and encourage them to remove it from the abstract without more detailed explanation.

* Despite the misleading claims in the abstract, this method seems to reduce compute only (not memory use; or if so, this has not been convincingly proven). In this context it would be appropriate to explain that commonly e.g. LLMs are memory rather than compute bound, limiting the utility of this approach.

QUESTIONS:
For the speed-up experiment: How are the standard linear and EUGen layer implemented?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents EUGens, a new class of dense layers designed to approximate standard feedforward layers (FFLs) using random features. The approach is backed by theoretical guarantees showing unbiased approximation for polynomial activations, and the layers can be plugged into existing architectures like Transformers and NeRFs to reduce computational cost. The authors also propose a lightweight distillation method that allows swapping in EUGens without needing full backpropagation, which is a practical feature for real-world deployment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The theoretical contributions are solid and well-presented. EUGens are shown to be unbiased approximators of FFLs with polynomial activations, supported by detailed variance and concentration bounds.\n- The method is architecture-agnostic and works in diverse domains—from NLP and vision to 3D reconstruction—showing versatility.\n- The empirical results demonstrate improvements in inference speed and memory usage, often without substantial quality degradation.\n- The distillation technique that allows layer-wise replacement without backprop is clever and could be very useful for adapting pretrained models efficiently.\n\nWeaknesses\n- One concern I have is that the experimental results don't clearly demonstrate EUGens as a superior alternative. Figures 3, 4, and 5 all suggest that EUGens offer a trade-off between model size and performance. But it’s hard to judge how favorable this trade-off really is, since reducing the number of inference parameters generally comes at the cost of accuracy. To show that EUGens are truly competitive replacements for existing FFLs, I think it would help to include an experiment where the EUGen model is scaled up to match the inference parameter count of the baseline—so that we can directly compare performance under equal capacity.\n\n- Another point that left me puzzled is the use of 2, 4, 6, and 11 EUGen layers in the experiments. This doesn’t feel like a clean or scalable setup. In particular, Figure 4 shows that the 11-layer configuration performs significantly worse, which raises the question: is EUGen's benefit limited in deeper models? If 11 layers already hurt performance—and they probably still make up a small part of the network—it might mean that EUGens don’t scale that well in depth. I’d appreciate more clarity on why these particular layer counts were chosen, and whether EUGens are actually viable when used more extensively in deeper architectures.\n\n- Additionally, while the paper focuses a lot on inference efficiency, it doesn't say much about training cost—especially when the Gi_j matrices are learned. If EUGens are supposed to be efficient alternatives, the cost of training them should be addressed too.\n\n- Finally, the distillation method assumes you can store and access the input-output pairs of the layers you want to replace, which might not always be feasible for large models or datasets.\n\nQUESTIONS:\nSee Weakness Section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EUGens (Efficient, Unified, and General dense layers), a new class of dense layers that generalize standard fully-connected feedforward layers (FFLs) by leveraging random features. EUGens aim to improve the efficiency of neural networks by approximating FFLs and reducing their inference complexity from quadratic to linear time. The method goes beyond traditional FFLs by incorporating direct dependence on input norms, which enhances their expressive power. Additionally, EUGens can approximate FFLs with arbitrary polynomial activation functions, offering significant reductions in parameters and computational overhead. The paper also presents a layer-wise knowledge transfer technique that bypasses backpropagation for more efficient adaptation of EUGens to pre-trained models. Empirically, integrating EUGens into architectures like Transformers and MLPs yields improvements in inference speed (up to 27%) and memory efficiency (up to 30%) across tasks such as image classification, language model pre-training, and 3D scene reconstruction.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nNovel Contribution: The introduction of EUGens provides an efficient alternative to traditional FFLs by incorporating random features and input norm dependencies. This novel approach reduces computational complexity, making it well-suited for large-scale neural networks.\n\nTheoretical Foundation: The paper provides a thorough theoretical analysis, including proofs and concentration bounds that demonstrate EUGens' capability to approximate FFLs with polynomial activation functions. This strengthens the validity of the proposed method.\n\nEmpirical Validation: Extensive experiments show that EUGens outperform previous techniques (e.g., low-rank approximations) in terms of approximation quality (MSE loss), speedup, and memory usage. The method is applied successfully across various architectures, including Transformers, MLPs, and NeRF (Neural Radiance Fields), showing significant real-world applicability.\n\nKnowledge Transfer and Distillation: The introduction of a distillation framework for layer-wise knowledge transfer without retraining is a significant innovation, improving deployment efficiency, especially for pretrained models.\n\nScalability and Practical Relevance: The paper highlights EUGens' potential for deploying large-scale neural networks in real-time, resource-constrained environments, making them highly relevant for practical applications.\n\nWeaknesses:\nLimited Task Diversity: The experimental evaluation mainly focuses on image classification, language models, and 3D scene reconstruction. While these are essential tasks, the paper could benefit from evaluating the method on a broader range of tasks (e.g., NLP beyond GPT-2, or tasks with sequential or graph-based data).\n\nImpact of Input Norms: The introduction of direct dependence on input norms enhances expressiveness but may lead to inefficiencies in certain applications. Further discussion on the trade-offs between performance and computational cost when using this feature could help clarify its limitations.\n\nEffect of Hyperparameters: While the paper evaluates the performance of EUGens with various hyperparameters (e.g., number of random features, polynomial order), a more detailed analysis of the sensitivity to these hyperparameters would provide deeper insights into the method's robustness.\n\nComputational Complexity of Large Models: Although the method improves inference speed, the evaluation does not fully explore the scaling of EUGens with respect to very large models (e.g., extremely deep Transformers or large-scale NeRF setups). The impact of increasing model size or network depth on the gains provided by EUGens remains unclear.\n\nQUESTIONS:\nTrade-off Between Expressiveness and Efficiency: The introduction of input norm dependence is a key feature of EUGens. How does this affect the trade-off between expressiveness and computational cost in practice? Are there specific scenarios where this dependence may be detrimental?\n\nHyperparameter Sensitivity: How sensitive is the performance of EUGens to changes in hyperparameters, such as the number of random features (m) and polynomial order (k)? Can the method adapt dynamically to different network sizes and tasks?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}