EUGens: Efficient, Unified and General Dense Layers
Authors: Sang Min Kim
Byeongchan Kim
Arijit Sehanobish
Somnath Basu Roy Chowdhury
Rahul Kidambi
Dongseok Shim
Kumar Avinava Dubey
Snigdha Chaturvedi
Min-hwan Oh
Krzysztof Marcin Choromanski
Venue: NeurIPS 2025 poster ()
Keywords: feedforward fully-connected layers
MLPs
random features
general dense layers
linear time complexity layers
Primary area: probabilistic_methods
TL;DR: This paper introduces EUGens: Efficient, Unified and General Dense Layers, capable of accurately approximating general feedforward fully connected layers, but of linear rather than quadratic time complexity.

Abstract: Efficient neural networks are essential for scaling machine learning models to real-time applications and resource-constrained environments. Fully-connected feedforward layers (FFLs) introduce computation and parameter count bottlenecks within neural network architectures.  To address this challenge, in this work, we propose a new class of dense layers that generalize standard fully-connected feedforward layers, $\textbf{E}$fficient, $\textbf{U}$nified and $\textbf{Gen}$eral dense layers (EUGens).  EUGens leverage random features to approximate standard FFLs and go beyond them by incorporating a direct dependence on the input norms in their computations. The proposed layers unify existing efficient FFL extensions and improve efficiency by reducing inference complexity from quadratic to linear time. They also lead to $\textbf{the first}$ unbiased algorithms approximating FFLs with arbitrary polynomial activation functions. Furthermore, EuGens reduce the parameter count and computational overhead while preserving the expressive power and adaptability of FFLs. We also present a layer-wise knowledge transfer technique that bypasses backpropagation, enabling efficient adaptation of EUGens to pre-trained models. Empirically, we observe that integrating EUGens into Transformers and MLPs yields substantial improvements in inference speed (up to $\textbf{27}$\%) and memory efficiency (up to $\textbf{30}$\%) across a range of tasks, including image classification, language model pre-training, and 3D scene reconstruction. Overall, our results highlight the potential of EUGens for the scalable deployment of large-scale neural networks in real-world scenarios.

EUGens: Efficient, Unified, and General Dense Layers
Sang Min Kim
Byeongchan Kim"!
Arijit Sehanobish
Somnath Basu Roy Chowdhury +3
Rahul Kidambi*
Dongseok Shim'
Avinava Dubey*3 Snigdha Chaturvedi
Min-hwan Oh'
Krzysztof Choromanski*54
Seouul National University
'Independent
*Google Research
4UNC Chapel Hill
Google DeepMind
"Columbia University
Abstract
Efficient neural networks are essential for scaling machine learning models real-time applications and resource-constrained environments. Fully-connected feedforward layers (FFLs) introduce computation and parameter count bottlenecks within neural network architectures. To address this challenge, in this work, we propose a new class of dense layers that generalize standard fully-connected feedforward layers, Efficient, Unified and General dense layers (BUGens). EUGens leverage random features to approximate standard FFLs and go beyond them by incorporating direct dependence the input norms in their computations. The proposed layers unify existing efficient FFL. extensions and improve efficiency by reducing inference complexity from quadratic linear time. They also lead the first unbiased algorithms approximating FFLs with arbitrary polynomial activation functions. Furthermore, EuGens reduce the parameter count and computational overhead while preserving the expressive power and adaptability of FFLs. We also present layer-wise knowledge transfer technique that bypasses backpropagation, enabling efficient adaptation of EUGens to pre-trained models. Empirically, we observe that integrating EUGens into Transformers and MLPs yields substantial improvements in inference speed (up to 27%) and memory efficiency (up to 30%) across range of tasks, including image classification, language model pre-training, and 3D scene reconstruction. Overall, our results highlight the potential of EUGens for the scalable deployment of large-scale neural networks real-world scenarios.
1 Introduction
Recent advances in machine learning (ML) have revolutionized the design and deployment of intelligent systems, with applications in robotics [8, 49), computer vision [40], natural language processing (NLP) [18, 91, 94, 1]. and 3D scene understanding [61]. These advancements have been primarily driven by large-scale machine learning models, which capable of solving increasingly challenging tasks. Despite the effectiveness of these models, computational inefficiencies often hinder their real-world deployment. Therefore, improving the efficiency of both training and inference of such models is essential to applying these methods in fast- paced, safety- cafetclcritical domains.
A major source of computational cost in modern ML models is the use of fully connected feedforward layers (FFLs), which are central to many architectures. FFLs are integral to Transformers [97]-the backbone of Large Language Models (LLMs) and Vision Transformers ViTs)-and to implicit neural representations (INRs) like Neural Radiance Fields (NeRF) [61] for 3D scene modeling. These layers account for substantial portion of the model parameters and computation. In Transformers, fully connected feedforward layers (FFLs) are major contributors to the overall model size. Similarly,
Equal Contribution Correspondence to: tkdals90820smu. ac. kr. arijit. sehanobi shg@aaill1hhmmmmmm1mm com, kchero@google.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

N
f(Wx)
g(W) f(x)
EUGen
g (W)
Layer Norm
f(x)
Wx
g
Masked Multi- head Attention
W
W
Layer Norm
Standard FFL
EUGen
Transformer
Figure I: Schematic diagram showcasing the workflow of standard fully connected (left) and EUGen layer (middle). In EUGen, both the input, x, and weight, W, are transformed by non-linear mapping f and g. These operations produce low-dimensional matrices whose multiplication reduces computational cost. Different dimensions of the new representation different monomials in the polynomial approximation of the activation function f. The representation can also directly depend on ||x||2 (see: Sec. 3). Such EUGen layers are introduced in Transformer blocks (right) to improve the efficiency of the overall network.
the input x correspond to
architectures like NeRF and iSDF [69] use FFLs encode nigh-dimensional spatial representations, supporting real-time tasks such as photorealistic rendering. Therefore, reducing the computational complexity of FFLs is essential for enabling the practical deployment of these architectures.
In this work, we present a new class of efficient dense layers that generalize their regular fully- connected feedforward counterparts (FFLs), unify other proposed extensions, and provide efficiency by replacing quadratic- with linear-time inference. These Efficient, Unified, and General dense layers (EUGens) can be seamlessly integrated into implicit neural representations and Transformer archi- tectures, significantly reducing computational complexity and parameter count without sacrificing expressivity. The key innovation of EUGens lies their ability disentangle the processing of weights and inputs, connecting them through simple linear operations (as shown in Fig. 1). Inspired by the seminal work of Rahimi and Recht [73] on random features (RFs), EUGens employ RF transformations to process the inputs. They effectively act as kernel methods, enabling efficient approximation complex interactions high- sigh-diiensiooall data. EUGens can approximate regular FFLs and even go beyond them by introducing direct dependence on the inputs' norms.
By incorporating EUGens into Transformers and NeRFs, show substantial improvements in inference speed (up to 27%) and memory usage (up to 30%) across diverse tasks: image classification, NLP, and 3D scene reconstruction. We also propose layer-wise knowledge transfer technique that bypasses backpropagation, enabling efficient adaptation of BUGens into pre-trained models. Our results highlight the potential of BUGens for scalable deployment real-world scenarios. To summarize, our main contributions are as follows:
We propose new class dense layers, EUGens, which leverages random features approximate feedforward layers and provide unbiased estimation for polynomial activation functions (Section 3).
We present an extensive theoretical analysis of FUGen's is estimation with the concentration bounds (Theorem 3.2 and 3.3) and propose additional improvements via QMC techniques (Section 3.
We empirically show that EUGens accurately approximate FFLs with common activations (e.g.. ReLU, GeLU, Softplus), significantly outperforming baseline methods (Section 4)..
We integrate EUGens into Transformers and NeRFs as FFL replacements, enabling faster deploy- ment across diverse NLP, vision, and 3D scene modeling tasks (Section 4.2, 4.3, and 4.4).
We propose distillation framework that replaces existing feedforward layers with EUGens, which can be trained analytically without backpropagation. This approach enables zero-shot integration with pretrained models, improving inference speed without retraining (Section 4.5).
The proofs of the theorems (Theorem 3.1 introducing EUGens that give unbiased estimations for the FFL.s with polynomial activations, Theorem 3.2 and Theorem 3.3 providing concentration results and Theorem 3.4 presenting extension to general continuous activations) are given in the Appendix.

---

2 Related Work
significant challenge in training neural networks is the quadratic scaling of their space and time complexity with the size of the hidden layers. To address this challenge, many works have focused on reducing these computational bottlenecks. One of the popular ways to tackle this issue is via dimensionality reduction à la Luaison-Linddsstruus. Transform (or JLT) [22, 21, 2] or via kernel methods with random features (RF) [73]. Since the influential work of Rahimi and Recht [73], substantial effort has been dedicated to improving the accuracy of RF-based methods by introducing entangled projections [119, 17, 16]. Notably, kernels of the form K := f(x y) where f exp [15, 53] or f is a polynomial or power series with positive coefficients [47, 100, 101, 99, 3, 46] have been extensively studied. However, these approaches are limited to positive definite kernels with only subset [15, 53, 3] targeting the attention module in the Transformer.
Kernel methods have been used to linearize two-layer neural networks [13, 14] and to analyze general networks using Neural Tangent Kernels (NTK) [44] and Path Kernels [26]. The popular NTKKbased methods are suitable for explaining neural network training during the fine-tuning phase when the weights do not deviate much from the pre-trained weights [107, 60, 92]. Therefore, none of these methods can be applied directly reduce the complexity of FFL layers when training from scratch. Closest to our approach, Sehanobish et al. [80] proposed the Universal Random Feature (URF), which disentangles weights and inputs in specific MLP layers. However, their method relies on Fourier transforms of activation functions, limiting its applicability to commonly used activation functions in machine learning. In particular, it cannot be applied to unbounded activation functions (e.g. ReLU).
Beyond kernel methods, common approaches to reduce the computational complexity of dense-layer models are via pruning [85, 59], quantization [111, 57, 124], knowledge distillation [51, 83, 9], or replacing dense matrices with structured ones [70, 108] all widely used in Transformers. In the context of NeRFs, various specialized methods have been proposed to enhance their efficiency [89, 112, 84, 75, 64, 42, 76]. Similarly, techniques tailored to Signed Distance Fields (SDF) architectures have been developed to enable their application weal-world scenarios [69, 72, 103, 106].
In contrast to the above solutions, EUGen layers provide a ceneral-purpose mechanism applicable across diverse architectures, including Transformers, NeRFs. and incremental SDF models. The formulation of EUGens also allows post-training compression and fast distillation using a closed- form solution. Finally, would like to highlight that EUGens is orthogonal to common efficiency techniques like sparse sampling, pruning, etc., and can be combined with such to further reduce the computational footprint. For completeness, we discuss additional related techniques in Appendix C.
3 Efficient, Unified, and General Dense Layers (EUGens)
In this section, we formally define EUGen layers. Take weight matrix row E Rd (in the column format) and input vector E Rd. The kth EUGen layer EUGen
(w, (w,x) is given follows:
(W,Gen"(w,x) -(v (concaa
i G
i
concat
II
G
(I) Here w and x are obtained from and x by concatenating with 1 and ||x|| respectively, stands for the dot-product, II defines Hadamard-product (i.e. element wise multiplication), concat denotes concatenation of the vectors, G E (d+1) (for = O,... k. = I,....0 i), and V, Φ :: C + R (M) & are applied element-wise). For the weight matrix W E Rixd, the EUGen layer of Rtth order EUGen" (W,x) x) is given by concatenating values EUGen" " (w, x) for different rows w of W.
Intuitively speaking, order k controls the expressivenes of the layer. Indeed, (Theorem 3.1), even for identity functions V and tb EUGens are capable of accurately approximating fully-connected feedforward layers (FFLs) with complex activation functions for the appropriately tuned hyperparameter k.
will later
Note that EUGen layers disentangle W from only to connect them in the final computations of the dot-product (lincar) kernels (shown in Fig. 1). Indeed, EUGen (w. x) can be re- re-written as:
3

---

EUGen" (w,x) = f(x)) where functions f.g are defined as follows:
f(x)= = connaat
i
((())= = [cconcat
[
Inference time complexity: Here only matrix products Gjxt need to be computed (for the pre-computed g(w)). Under an assumption that applying per-entry takes constant-time, inference can be conducted in time (mmkk? ml). In practice significant gains can be obtained, as compared to the brute-force O(d² + dl) variant, since time complexity becomes linear in d. We demonstrate it in Sec. 4.2, 4.3, and 4.4. If drop dependence on i the definition of Gj (i.e. G = Gj), time complexity can be further improved to O + ml).
small & < 3), thus for < min(d.) 1)
We ready formulate main theoretical result. In the theorem below, show that EUGens are capable of unbiasedly approximating FFLs with arbitrary polynomial activation functions f and with easy to construct matrices G To the best of our knowledge, this is the first such result. The most relevant previous results from [80] assumed the existence of the Fourier Transform (FT) f.
Theorem 3.1 (EUGens unbiasedly approximate FFLs with polynomial activations). Take the FFL defined as follows for a weight matrix W Rixd and an input vector E Rd; FFL(W, x) f(Wx) (note that this is the most general form, since the bias term can always be absorbed by adding extra column to matrix W and extra entry to input x), where the activation function f IR + R is given
f(z)
ax
(3)
1=
Choose V. as identity functions. Take some zero-mean distributions: Dj E P(R) = 0.... and j = 11...... i, with standard deviations ozz- Assume that the last columns of the matrices Gj are all zero and entries of the other columns of matrices Gj are sampled independently as:
G)(,))~
Oigmā aa मं (2i)) Dj,
(4)
where (t( E C satisfies: (t) = sgn(ai). Then the following holds:
E [EUGen( W, x)] = ((Wx)
(5)
The results holds also G3 does depend on i, i.e. G = G5, as well as when rows of matrices G are dependent, as long as entries within each row are chosen independently.
By setting the last columns of the matrices GJ zeros in Theorem 3.1, we do not leverage the opportunity, that EUGens give, provide direct dependence ||x||2- That suffices unbiasedly and accurately approximate FFLs with polynomial activation functions (and in practice any continuous since, by the Bernstein Theorem [78], any continuous function defined on a closed interval can be approximated with arbitrary accuracy by a polynomial). Adding direct dependence on ||x||2 enables EUGens provide computational models beyond the scope of regular FFLs.
Concentration results for the approximation mechanism presented Theorem 3.1, in the form of the variance formula, are given in Theorem 3.2. Additional results, providing exponentially small probabilities large divergence from the mean and leveraging Azuma's inequality, are given in Theorem 3.3. We complement our theoretical analysis with the result showing how EUGens with polynomial activation functions can be used to approximate FFLs with continuous activations (Theorem 3.4). Those results leverage polynomial approximation of the continuous functions.
The sketch of the proof of Theorem 3.3 is as follows. observe that the EUGens are effectively conducting an unbiased Monte Carlo approximation of the output values of the regular FFL. Under the assumptions of the theorem, this is achieved by averaging over independent and bounded random variables. We can then apply standard concentration inequalities, such as Azuma's Inequality.

---

Theorem 3.2 (Concentration results of EUGens: part 1). Under the assumption from Theorem 3.1 (the entries of all G for O...., k. j = 1,.... sampled independentl), the variance of the estimation Z FFL xi[ui of FFF(LW, x][u] for u L..., given as follows for Tt.j denoting the fourth moment of the random variable sampled from on. th row of W (in the column format) and p; (w(u) x
Di Di w(u) denoting the
((
Var(2) (2)
-
1
Σ
(2(w(w)
+l (T 3)
Σ w
Pi
a
m7 ==0
==1
(6)
Theorem 3.3 (Concentration results of EUGens: part II). Consider the setting from Theorem 3.2 and assume furthermore that the absolute values of the entries of random variables taken from m 25 Dj are upper-bounded by e for some c Then the following holds for I
max (w(()))) [. 772 = max;=0,.... |
([())l))) and any E > 0:
mE²
F[FFL[W x) [u] (FFLWWx))(ue 2 2 h(e) 2 2exp
2((11 + 12
Remark I:
The upper bounds on failure probabilities in Theorem 3.3 and Theorem 3.4 exponen- tially small in the number of random features m. Consequently, strong guarantees regarding accurate be directly obtained via the union bound trick.
approximation in several points simultaneously
Remark practice, BUGens are often initialized to provide an unbiased approximation of the well-defined regular FFL layers, but then matrices Gj are usually trained.
Remark III: Although pure polynomial-action activvation networks (PNNs) have limited expressiveness unless the degree k is large, this limitation does not directly apply when polynomial layers are combined with standard activation layers. Our models use hybrid architectures that mix EUGen layers with regular feedforward layers, so the PNN theory no longer applies. Empirically, across multiple modalities, -dggre hybrids match the performance of standard models while offering substantial computational benefits (see Secs. 4.2, 4.3, and 4.4). Accordingly. our approximation results should be interpreted in the hybrid setting rather than in the context pure PNNs.
Remark IV: Since the BUGen layers disentangle the weights and the inputs, they can be trivially combined with a subsequent linear layer. This enables us compress models like Transformers and NeRFs where an EuGen layer is naturally followed by linear layers (more details presented in Fig. and in Sec. B.
3.1 Quasi Monte Carlo (QMC) EUGens
In this section, we assume that m S (which has to be the case anyway for computational gains). Since in this section, as in Theorem 3.1, the last columns of G will be zeroed, without loss of generality we will assume that: G E d The standard initial choices for the matrices G are Gaussian matrices. Note that for the special case of k 1, EUGens effectively conduct JLT dimensionality reduction with projection matrices G a well-known fact that structured random matrices with correlated rows can reduce the variance of the JLT estimation of the dot dot-product kernel.
In particular, Gaussian orthogonal matrices (GOMs) [119] from Rdd (with Gaussian marginal distributions of rows, but exactly orthogonal rows), truncated to their first rows provably reduce that variance [17]. We applied this approach to EUGens with general (k > 1) and empirically confirm its effectivvnness, also in the setting with EUGens directly depending on ||x||2 (see: Sec. 4.1).
5

---

Theorem 3.4 (EUGens approximating FFLs with general continuous activations). Take an FFL. using continuous activation f with the associated modulus of continuity function w, defined as:
w(x) sup 11-10/5
- s(22)
(8)
Assume that (w(u) x S for some O. Then there exists C(E) > 0 such that the following is true. For any k > there exists hthh -order EUGen layer such that for any given 0
and u E {I.....}, the probability that [FFL(W. x FFL(W, x)[u] 2 C(E)
1
vi
upper-bounded by Poound min(p1.P2) where (for h(-) in Theorem 3.3):
Var
FFL(WW, x)
(9)
P
2
P2 h (c).
ReLU
SoftPlus
GELU
0.05
MSE Loss
0.04-
0.4
0,03
0.03
0.2
0.02
0.02
0.0
0.01
BUGen
SNNK Low Rank
EUGen
SNNK
Low Rank
BUGen
SNNK Low Rank
Figure 2: Approximation capability of our EUGee layer using (left) ReLU, (middle) Softplus and (right) GELU activation functions. EUGens provide superior approximation results compared to the baselines (SNNK and Low-Rank methods) with the same number of parameters.
4 Experiments
In this section, we outline the experimental setup and evaluate the performance of BUGen. Specifically, we design experiments to answer the following research questions:
(RQ1) How effective are EUGens in approximating
layers (FFLs)?
(RQ2) How well do large-scale neural networks (e.g.. Transformers & INRs) with EUGens perform?
(RQ3) How much speedup do EUGens achieve in large-scale neural networks?
Next, answer those research questions by evaluating EUGens in variety of architectures, such as Transformers (e.g., LLMs and ViTs) and NeRFs. We applied EUGens of order in all settings (further ablations higher i: can be found in Appendix E.6.) We provide more details about experimental setup in Appendix D.
4.1 Approximation Quality of EUGen in Synthetic Settings
In this experiment, we evaluate the approximation quality of EUGen layers. Specifically, we test whether EUGen layers can approximate the outputs of a fully-connected feedforward layer (FFL), Y f(Wx b), where f is an activation function. We compare EUGen's performance with low-rank approximation and SNNK [80] special case of EUGen, for three different activation functions (ReLU, SoftPlus, and GELU). We report the results in Fig. 2, where we compute the MSE loss between the FFL output, Y. and EUGen prediction, Y. We observe that EUGens achieve significantly lower MSE loss across all activations, showcasing superior approximation quality of FFL.s. This provides the answer (RQI), showing that FUGens are effective in approximating FFL.s. We conducted additional experiments in synthetic setups (see Appendix D.I for more details).

---

GPT-2
3.00
Ideal
EUGen (2 layers)
GPT-2
LowRank GPT-2
EUGen layers)
EUGen GPT-2 [2 ayers)
Validation Loss
3:05
Low Rank layers)
EUGen GPT 2 (4 layers)
EUGen (6 layers)
Validation
4
EUGen GPT-2 [6 layers)
3.10
Low Rank (6 layers)
EUGen GPT-2 (11 layers)
3.15
3.20
EUGen (11 layers)
Low Rank (12 layers)
31
10K
60
70
80
90
100
Iterations
Fraction of Inference Parameters (%)
Figure 3: Evaluation result of EUGen for language model pre-training using GPT-2 (86M parameters). (Left) We report the validation loss of GPT-2 with different numbers of EUGen layers during pre-training. (Right) Tradcoff plot between the number of inference parameters and validation loss. Overall, we observe that EUGens outperform LowRank variations while enabling significant speedups with minimal impact on validation loss.
ImageNet
Places365 EUCer, FL
Risen
0.80
Validation Accuracy
Validation Accuracy
0.56
EuGan PL
Ja
0.75
0.54
0.52
0.70
* VIT Base Rank
VIT Base Law Rank EUGen
0.50
0.65
EUGen 32)
0.48
50
70
80
90
100
50
60
80
90
100
Fraction of Inference Parameters
Fraction Inference Parameters (%)
Figure 4: Evaluation results of EUGen for image classification tasks: (left) ImageNet and (right) Places365 datasets using ViT (86M parameters). We observe that EUGens can match the performance of vanilla ViTs with significantly smaller fraction of parameters than the Low-Rank baseline.
4.2 EUGens in LLM Pre-training
In this experiment, we evaluate the performance of EUGen in the context of LLM pre-training. We consider the GPT-2 Transformer architecture (with 124M parameters) and replace the FFLs with EUGen layers. We pre-train this architecture on ~36.8B tokens from Open TebText [35] dataset over 50K iterations. We compare approach against the default GPT-2, GPT-2 with FFLs replaced by low-rank matrices, and baselines with varying numbers of EUGen layers.
We report the pre-training results in Fig. 3 (left). We observe that EUGens are good approximations of FFL., achieving validation loss similar to vanilla GPT-2. We also notice that increasing the number of EUGen layers slightly increases the loss, which is expected due to the error accumulation effect. In Fig. 3 (right), we report the trade-off between the validation loss and the number of inference parameters (as fraction of the vanilla GPT parameters). EUGens achieve good validation loss (y-axis is reversed) while using significantly fewer parameters. This provides the answer to RQ2 RQ3, showing that EUGens, in the context of LLM pre-training, achieve good performance using significantly smaller number parameters.
4.3 EUGens in Vision Transformers
In this experiment, we evaluate the efficacy of EUGens Vision Transformers (ViTs) [29]. Similar to the last setting (Section 4.2), we replace fully connected feedforward layers (FFL.) with EUGens and use the resultant ViT architecture for image classification. In this setting, we systematically replace all the FFLs with EUGen and evaluate it the ImageNet [23] and Places365 [125] datascts.
7

---

In Figure 4, we report the accuracy vs inference parameter tradeoff of EUGen and baseline methods. We observe that EUGens achieve the best trade-off curve, closely matching the performance of the vanilla VIT while using significantly fewer parameters (e.g., replacing six layers reduces the number of inference parameters by nearly 30%). This provides further evidence to answer RQ2 & RQ3, showcasing that EUGens are effective in achieving good image classification performance while using a fraction of trainable parameters. Additional results for image classífication (including results with ViT-L) and language processing with Transformers are presented in Appendix D.3 and D.4.
NeRF
D-NeRF
Zip-NeRF
Mip-NeRI 360
35.0
35.0
52.5
EUGen
50.0
MLP
32.5
32.5
30.0
30.0
30.0
27.5
27.5
-2..1%
27.5
17.2%
25.0
+6.256
25.0
127
5%%
25.0
1.75
2,00
2.25
1.20
1.25 Inference Time (8)
3.8
4.0
27.5
30.0
32,5
Figure 5: Quantitative results for NeRF experiments including NeRF, D-NeRF, Zip-NeRF, and Mip-NeRF 360 showing PSNR versus inference time. Our models achieve similar PSNR scores while achieving at least 24% improvement in speeds for implicit representation models (NeRF and Mip-NeRF) as well as speeding the efficient hybrid models D-NeRF and Zip-NeRF by at least 6%.
4.4 EUGens for Neural 3D Reconstruction
In this section, show how EUGens be seamlessly injected into various neural 3D scene reconstruction methods including NeRF [61] with demonstrated photo-realistic scene generation capability, Mip-NeRF 360 [4] the state-of-the-art (SOTA) architecture for novel video synthesis, Zip-NeRF [5]. D-NeRF [71] and iSDF [69], real-time module designed to reconstruct SDF from depth sequences. In all cases, replaced up to three layers with HUGens (details in Appendix D). We applied EUGens explicitly using input's 's length |x||2 (see: Sec. 3).
In Fig. 5, we present the quantitative results of this replacement, plotting PSNR with inference time. For NeRFs [61], EUGen significantly reduces both inference time by 24% (Fig. 5 (left)) and model size by 30% (Tab. 8), respectively, with virtually no loss in reconstruction quality. For Mip-NeRF, on 360 V2 dataset [4] we achieve an 27% (Fig. 5 (right))increas in inference speed while maintaining a similar reconstruction quality. We show qualitative results in Fig. 6.
For iSDE, in Fig. 7, we compare the original iSDF with EUGen replacing MLP layers (see Fig. 21 for all results and Table 4 for quantitative results). EUGens achieve similar performance to iSDF while being 22.6% faster during inference and achieving a 5% improvement in training speed. When we visualize slices of the reconstructed SDFs on the xy-plane at 70 cm, the reconstruction quality remains quite similar. Similar to the previous section, these results answer questions: RQ2 and RQ3.
4.5 Knowledge Distillation with EUGens
In the previous sections, we showed that EUGen layers can significantly accelerate the inference time of implicit neural representations. However, achieving this acceleration typically requires retraining the model from scratch, which is computationally expensive and time-consuming. In this section, we show results on directly replacing the FFL layers with simple layer-wise distillation. This approach climinates the need for per-scene retraining, enabling the acceleration of existing pretrained models without the overhead of rebuilding them from scratch. By storing the inputs and outputs of the target hidden layer in trained implicit neural representations and then optimizing the EUGen layer using a mean squared error objective, we can efficiently replicate the behavior of the original model.
This optimization has a closed-form solution and can be performed without buckprepagation when G are sampled from a fixed distribution. We will refer to this EUGen variant as the 'Analytic' variant (see Appendix B.2). For NeRF, we show that we can recover the quality of the original model while improving inference speeds by up to 26% (Fig. 8 and Table 5 in Appendix E.4).
8

---

Drums
Ground Trush
NeRF
NURe
Ficus
Ground Truth
NeRT
EUGen-NeRF
Figure 6: Results of rendering through NeRF versus EUGen-NeRI: drums (left) and ficus (right). The Ground Truth column shows the reference image, followed by the rendered results. We observe that the EUGen-NeRF renderings are indistinguishable from the NeRF renderings.
2 2ov
apt 2 2b
Ground Truth
BuG MLP
ML.P
8.24
Li Norm (cm)
EUGen
7,81
EUGen-iSDF
7.2 7.6
1.0 1.1
1.3
14
Training Time
Inference
T Time
Signed Distance
-20
Figure 7: (Left) Training and inference comparisons between iSDF and EUGen-iSDE EUGen-iSDF achieves 5% faster training and 23% faster inference with comparable reconstruction accuracy. (Right) Visualizations of the SDF reconstructions on ReplicaCAD scenes using iSDF (colormap below).
32-
31.31
31.25
2.4-
Distillation Time (8)
28.54
Inference Time
PSNR
24
2.0
1.84
1.84
0.12
16
1.6
Bascline Analytic EUGen
Baseline
Analytic
BUGen
Analytic
BUGen
Figure 8: Distillation results for NeRF comparing the analytic and our EUGen variant with the bascline. (Left): Comparing PSNR of the baseline model with the distilled versions. (Middle): Comparing inference speed of baseline our distilled models. (Right): Comparing the distillation speed of analytic vs the EUGen variant.
4.6 Ablation Studies
In this section, we analyze how EUGens' key design choices impact performance. Specifically, we investigate the impact of trainable projection matrices (G)),, the number of random features, and compare EUGens with dimensionality reduction baselines. See Appendix E for detailed ablations.
Trainable Projection Matrices Improve Performance. We study the impact of trainable vs. fixed G matrices in the iSDF setting, measuring reconstruction accuracy and speed. Consistent with prior findings [19, 121J), we observe that trainable Gj matrices enhances performance across all downstream tasks shown in Fig. 9 (left & middle).
Number of Random Features (m.) Fig. 9 (right) reveals the trade-off between computational speed and reconstruction quality with increasing number of random features. We find that increasing the random features improves performance (lower distance) while increasing inference time. We perform additional ba-ed ablations in Appendix E.2 (see Table 8, 10, Fig. 26, 27).
EUGen Models vs. Dimensionality Reduction. Reducing hidden dimensions (d) a common approach to accelerate training. To ensure fair comparison, decrease the hidden channels of

---

Chamfer Distance (ci))
LI Nomo (cm)
4.17
4.28
Training: Time (8)
2,4
BUD (129)
4.0
7.81
EUG ISD wio tranable G (129)
3,5
7.8-
7.74
2.2
BllGee-iStrr
[SU OUGGe--8DD momobll
2.0
BUGen
wro trainable
BUGen
w/o trainable
0.9
1.0
1.1
1.2
1.3
Inference Time (ms)
Figure 9: (Left & Middle) Comparing the performance of EUGen-iSDF with or without trainable G
The non-trainable G is faster in training but shows slightly worse performance. (Right) Trade-off between reconstruction quality and speed for EUGen-iSDF models with 32, 64, 128 random features.
baseline models to match the training parameters their EUGen counterparts. As shown in Fig. 10, the EUGen models consistently outperform these dimensionality reduction baseline models.
5 Conclusion
We introduce a novel type of neural net- work layer, termed EUGen, which lever- ages random features to efficiently approx- imate the computations of fully connected feedforward layers. By integrating EUGen into LLMs, Transformers, Neural Radiance Fields (NeRF), and Signed Distance Fields (SDF) models, we achieve reduction training parameters while preserving ex- pressivity. Our approach enhances infer- ence speed across all downstream appli- cations for wide range of model archi- tectures. Additionally, the straightforward design of EUGens facilitates layer-wise
Baseline
Paremoter-matched Baseline
Elitien
NeRF
Chamfeer Dislance (cm)
iSDF
30.8
2.16
PSNR
30.4
2.08
2.00
1.8
2.0
2.2
2.4
1.2
1.3
Inference Time (8)
Inference Time (ms)
Figure 10: We compare the performance of EUGen models with baselines using (left) NeRF and (right) iSDF architectures. In both settings, EUGen models achieve better reconstruction quality than the baselines with the same parameter count while being much faster. knowledge distillation without requiring backpropagation. We demonstrate that this distillation process leads to efficient inference. Overall, EUGens are a key contribution towards improving the efficiency of scrge machine learning systems without impacting their expressivity.
Acknowledgements
BK and MO were supported by the National Research Foundation of Korea (NRF) grant funded by the Korea gOVTM (No. RS-2022-NR07 1853 and RS-2023- 00222663).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprini
2023.
[2] Nir Ailon and Edo Liberty. An almost optimal unrestricted fast jonnson-llndeesstteeee transform. ACM Trans. Algorithms, 9(3), 2013. ISSN 1,54963255. doi: 10. (0.11155228319922443700 URL https::////// :/dttp:::/ddiii org/ 100 1145/2483699 2483701.
[3] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-hrroughhut tradeoff. arXiv:2402.18660 2024.
[4] Jonathan T. Barron, Ben Mildenhall, Verbin, Pratul Srinivasan, and Peter Hedman. Mirfnerf 360: Unbounded anti-aliase neural radiance fields. CVPR, 2022.
10

---

[5] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[6] Etienne Boursier, Loucas Fillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow reLU networks for square loss and orthogonal inputs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. URL https://opeereviiii
net/ worumodd-L74------....---------d 1=L74c-1UxQ111
Guy Bresler and Dheeraj Nagaraj. corrective view of neural networks: Representation, memorization and learning. Conference on Learning Theory, pages 848-901. PMLR, 2020.
[8] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge robotic control, arXiv preprint arXiv:2307.15818, 2023.
[9] Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws. arXiv preprint arXiv:2502.086606 2025.
[10] Devendra Singh Chaplot, Deepak Pathak, and Jitendra Malik. Differentiable spatial planning using transformers, 2021. URI.ttps : //arxvv...viv////a///ii org/ httg/abs//1122/ 2112. 01010.
[11] Anpei Chen, Zexiang Xu, Andreas Geiger. Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
[12] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering mobile architectures. In The Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[13] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Ad- vances Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009. URL
http://prrceddiin...e.eeereeieee neurips iirriis..oo cttcccc/paaaa cc/paper_ppp. waarrr/lllsssppp/aee/ee 7775eeeeeedtaab555552eeee00000.0 Paper. Paper.pdf.
[14] Youngmin Cho and Lawrence K. Saul. Analysis and extension arc-cosine kernels for large margin classification, 2011. URL https:///aaxiii orgg berrg/abs/tie 11///11111.71
[15] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2020.
[16] Krzysztof Marcin Choromanski, Mark Rowland, and Adrian Weller. The unreasonable effectiveness of structured random orthogonal embeddings. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Anmual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 219-228, 2017. URL https: http:////roceeeeeeee///e newrips pttpppppe//////aas// 2ff229696773004700offoooo aww.e4447700ddfffff9e2e522 www.e//araaaaaaaaaaatteaa.a. Abstract html.
[17] Krzysztof Marein Choromanski, Mark Rowland, and Adrian Weller. The unreasonable effectiveness of structured random orthogonal embeddings. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 219 228, 2017. URL https: //wp://edio./......oooooaeca. newni ssarips.co cc/ httpp/appp//////a/ 11E2299988738004700ffffdeeeee htw.119ffaa2f---saaaatttt html.
[18] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways, 2022. arXiv preprint 22X2v2220..02311, 2022.
11

---

[19] Sankalan Pal Chowdhury, Adamos Solomou, Kumar Avinava Dubey, and Mrinmaya Sachan. Learning the transformer kernel. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URI. https::p//// / openreview
[20] Nikhil Das and Michael Yip. Learning- Searning-based proxy collision detection for robot motion planning applications. IEEE Transactions Robotics, 36(4):1096-111 11100-99611,,.1111110011111 2020. doi: 10.1 TR0.2020.2974094.
[21] Anirban Dasgupta, Ravi Kumar, and Tamás Sarlos. A sparse johnson:
Lindenstrauss
transform. Proceedings of the Forty-Second ACM Symposium on Theory of Comput- ing, STOC '10, page 341-350, New York, NY, USA, 2010. Association for Comput- ing Machinery. ISBN 9781450300506. doi: 10.1145/1806689, 1806737.
URL https:
//doi.org org/ 10. 1806688 1806737.
[22] Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures Algorithms, 22(1):60-66 65, 2003. doi: https://oi..or 10. (0002//sa.1007)) httpt:///////ll http///onnneeibbaarrrrrr wi wiley. com/ /ts///s////////////e 10bbs/10.1000 10001002//500 10073.
[23] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255 2009. doi: 10. N0.109/CVPR.2209922666 520064448...
[24] Junyuan Deng, Qi Wu, Xieyuanli Chen, Songpengcheng Xia, Zhen Sun, Guoqing Liu, Wenxian Yu, and Ling Pei. Nerf-loam: Neural implicit representation for large-scale incremental lidar odometry and mapping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8218-8227, 2023.
[25] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. N..18853/1/N11-14222 1423. URL https:
//aclanthology org/N19-14223 1423/.
[26] Pedro Domingos. Every model learned by gradient descent is approximately a kernel machine, 2020. URL. https://arxii:i http://xxivvoorrrbraa/r// wttggaab/22012// 2002200012..
[27] Jing Dong, Mustafa Mukadam, Frank Dellaert, and Byron Boots. Motion planning as prob- abilistic inference using Gaussian processes and factor graphs In Proceedings of Robotics: Science and Systems (RSS), 2016.
[28] Jing Dong, Mustafa Mukadam, Byron Boots, and Frank Dellaert. Sparse Gaussian processes on matrix Lie groups: A unified framework for optimizing continuous-time trajectories. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6497-6504. IEEE, 2018
[29] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representattonss 2021. URL https://openreve wwt..e//oooom/////////aaaaaa
[30] Mark Nicholas Finean, Wolfgang Merkt, and Ioannis Havoutis. Predicted composite signed- distance fields for real-time motion planning in dynamic environments, 2021. URL https:
// /arxii.oorrg///................t has///bs//00000 00969.
[31] Sara Fridovich- Keil, Alex Yu, Mathhew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5501-5510, 2022.
12

---

[32] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. arXiv preprint arXiv:2103.10388 2021.
[33] Shuang Geng, Qianhao Wang. Lei Xie. Chao Xu, Yanjun Cao, and Fei Gao. Robo-centric esdf: A fast and accurate whole-body collision evaluation tool for any-shape robotic planning. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 290 1900-297,00 2023. doi: 10. 10//10005555...22.... 21025555555200222000222777 10342074.
[34] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension, 2020. URL http::/:///aax..ooraabbbb/////// 12191.
[35] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007, wttp://ii....iiiiiieeeeeeee htt..///pppeeetTeettrrppppp.sss 2019.
[36] Chun Gu. zipnerf-pytorch. httppp//ggtthh..ca////////gatg/p 2023.
[37] Kang Han, Wei Xiang, and Lu Yu. Volume feature rendering for fast neural radiance field reconstruction. Advances in Neural Information Processing Systems, 36, 2024.
[38] Luxin Han, Fei Gao, Boyu Zhou, and Shaojie Shen. Fiesta: Fast incremental euclidean distance fields for online motion planning of aerial robots. In 2019 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 4423 4430, 2019. doi: 10. 10.1109//00440997.
2019.899 18888999999
[39] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVE international conference on computer vision, pages 5875-5884, 2021.
[40] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[41] Daniel Hsu, Clayton H Sanford, Rocco Servedio, and Emmanouil Vasileios Vlatakis- Gkaragkounis. On the approximation power two-layer networks of random relus. Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 1423-2461. PMLR, 15-19 Aug 2021. URL https http://pooceeee...cee/ mlr press/ ttees//v134/ssu/// html.
[42] Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12902-12911, 2022.
[43] Wenbo Hu, Yuling Wang, Lin Ma. Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri- miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19774-19783, 2023.
[44] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 1580-8589, Red Hook, NY, USA, Curran Associates Inc.
2018.
[45] Mohammad Mahdi Johari, Camilla Carta, and François Fleuret. Eslam: Efficient dense slam system based on hybrid representation of signed distance fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 17408-17419, 2023.
[46] Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Polysketchformer: Fast transformers via sketches for polynomial kernels, 2024. URI. YkCjojDG31.
13

---

[47] Purushottam Kar and Harish Karnick. Random feature maps for dot product kernels. In Neil D. Lawrence and Mark Girolami, editors. Proceedings of the Fifteenth International Conference Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 583-591, La Palma, Canary Islands, 21-23 Apr 2012. PMLR. URL. https: / proceedings.
mlr. press/ 2eess/v2//// 1/22/arr2..2.
[48] Diederik P Kingma. Adam: method for stochastic optimization. arXiv preprint arXiv:/412.6988 2014.
[49] Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, al. Sara-rt: Scaling up robotics transformers with self-adaptive robust attention. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6920 6920.6992.000 IEEE, 2024.
[50] Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning. CoRR, 0os/1911.030990 2019. URL http://arxiv orgg abs/ 1911.03090.
[51] Lei Li, Yongfeng Zhang, and Li Chen. Prompt distillation for efficient lIm-based recom- mendation. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, pages 1348-1357, 2023.
[52] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8456-884 8465,445,55 2023.
[53] Valerii Likhosherstov, Krzysztof Marcin Choromanski, Kumar Avinava Dubey, Frederick Liu, Tamas Sarlos, and Adrian Weller. Chefs' random tables: Non-trigonometric random features. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: https://openreview net/forum? Id=vRsCvlvdBel.
[54] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering, In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14556-145655 2021.
[55] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33: 33:15651-15663, 2020.
[56] Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan. Effi cientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[57] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm 2024.
quantization with learned rotations. arXiv preprint
[58] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019, URL https::/t//tteeeee htpps:// net/forum? Bd=Bkg6RicqY7.
[59] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023,
[60] Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. kernel-based view of language model fine-tuning, In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceed- ings of the 40th International Conference Machine Learning, volume 202 of Proceed- ings of Machine Learning Research, pages 23610-23641 PMLR, 23-29 Jul 2023. URL https: http:///ppcceeiiggs.. mlr, press/ vv0ss//v22//// 20002/mmlladiii html.
14

---

[61] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[62] Alireza MousaviiHHssseing Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat Erdogdu. Neural networks efficiently learn cow-dimensional representations with SGD. In The Eleventh International Conference on Learning Representations, 2023, URL https:
// //oppnrevvee... net/ cere/fforum't wrrmmidd-tayykgeee
[63] Mustafa Mukadam, Jing Dong, Xinyan Yan, Frank Dellaert, and Byron Boots, Continuous- time Gaussian process motion planning via probabilistic inference. In The International Journal of Robotics Research (I/RR), volume 37, pages 1319-1340, 2018.
[64] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM transactions on graphics (TOG), 41(): 1-15, 2022.
[65] Radford M. Neal. Priors for Infinite Networks, pages 29-53. Springer New York, New York, NY, 1996. ISBN 978-1-4612-0745-0, doi: 10.1007/978- sttps://doi. org/ 10.
1-4612-0745-0-2. 1-11788----12--755--- 10.10077978---611--74-----000
URL
[66] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgib- bon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium Mixed and Augmented Reality, pages 127-136, 2011. doi:
10. 100.////MA..001..0000
[67] Richard A. Newcombe, Dieter Fox, and Steven M. Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[68] Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto. Voxblox: Incremental 3d euclidean signed distance fields for on-board mav planning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.
[69] Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. isdf: Real-time neural signed distance fields for robot perception. In Robotics: Science and Systems, 2022.
[70] Andres Potapczynski, Shikai Qiu, Marc Anton Finzi, Christopher Ferri, Zixi Chen, Micah Goldblum, C. Bayan Bruss, Christopher De Sa, and Andrew Gordon Wilson. Searching for efficient linear layers over continuous space of structured matrices. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: // //openreview. wttt://ffrommndddfoooee cBBANWwdF.
[71] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[72] Ri-Zhao Qiu, Yafei Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye, Jiteng Mu, Ruihan Yang. Nikolay Atanasov, Sebastian Scherer, et al. Learning generalizable feature fields for mobile manipulation. arXiv preprint ar arXiv:2403.07563, 2024.
[73] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Proceed- ings of the 20th International Conference on Neural Information Processing Systems, NIPS 07, page 11 1177-1184, Red Hook, NY, USA, 2007. Curran Associates Inc. ISBN 675603300.
[74] Victor Reijgwart, Alexander Millane, Helen Oleynikova, Roland Siegwart, Cesar Cadena, and Juan Nieto. Voxgraph: Globally consistent, volumetric mapping using signed distance function submaps. IEEE Robotics and Automation Letters, 5(1):227-234, January 2020. ISSN 2377-3774. doi: 10.1109//ra.2019/ 2953859. LRA. 2019. 2953859.
URL http: / /xxx doi. org/10 2rg/10.110 1109/
15

---

[75] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps, 2021. URL ittpss////aax ort.//////bb...../. 13744.
[76] Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG), 42(4):1-12, 2023.
[77] Radu Alexandru Rosu and Sven Behnke. Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8466-8475, 2023.
[78] Walter Rudin. Principles of mathematical analysis/ Walter Rudin. International series in pure and applied mathematics. McGraw-Hill, New York, third edition. edition, 1976 1976. ISBN 007054235X.
[79] John Schulman, Yan Duan, Jonathan Ho, Alex Lee, Ibrahim Awwal, Henry Bradlow, Jia Pan, Sachin Patil, Ken Goldberg, and Pieter Abbeel. Motion planning with sequential convex optimization and convex collision checking. Int. J. Rob. Res., 33(9):255--1270, aug 2014. URL https://doft ottp//../////1/////1
ISSN 0278-3649. doi: 10.117700278364914528813 0278364914528132
[80] Arijit Sehanobish, Krzysztof Marcin Choromanski, YUNFAN ZHAO, Kumar Avinava Dubey, and Valerii Likhosherstov. Scalable neural network kernels. In The Twelfth International Conference on Learning Representations, 2024. URL id=4iPu1k1FWa.
www.eeewwweeeeetruuneeeeea net/forum?
[81] Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar. Structured transforms for small- footprint deep learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett, editors, Advances Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015. URL https: /proceedinn/ssse// neur heurips... 2015/file/ htt.f//le//111000e800 05000000000000000000000000666600
satper_ffles/pppple Paperr.ap.ppppe pdf.
[82] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
[83] Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, et al. Lim pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408.. 1796, 2024.
[84] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5459-5469, 2022.
[85] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference Learning Representations, 2024.
[86] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans. Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251-266 2021.
[87] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop. Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11358-11367, 2021.
[88] Towaki Takikawa, Thomas Müller, Merlin Nimier-David, Alex Evans, Sanja Fidler, Alec Jacobson, and Alexander Keller. Compact neural graphics primitives with learned hash probing. In SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
16

---

[89] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar, Block-nerf: Scalable large scene neural view synthesis. arXiv, 2022.
[90] Jiaxiang Tang. Torch-ngp: pytorch implementation of instant-ngp. 2022.
https://gthub..com http://gthtbb..oomuuuuwwwww/cho/
[91] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[92] Akiyoshi Tomihari and Issei Sato. Understanding linear probing then fine-tuning language models from NTK perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:/ //ppnr/eeee/eeeeeeeeeeeee www/tfoummmiieeeeeeeeeee 1vgk-yyGf
[93] Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 1049-1056, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 605555561..... doi: 10.1 1411445553344155555555
URL https: /doi. org/oo//// 10. 1145/155337 1553508.
[94] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[95] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construc- tion of large-scale nerfs for virtual flytthroughs In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 12922 12931, June 2022.
[96] Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12375-12385 2023.
[97] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL /ttttss://p/oott h/proceeeoog/c/oooeo/o/oo neuriinngnnnnnu..ssee......... sourips.co 22aper/2011/hahh 3f 2
Abstracctt..t.a..a... html.
[98] Mrinal Verghese, Nikhil Das, Yuheng Zhi, and Michael Yip. Configuration space decomposi- tion for scalable proxy collision checking in robot planning and control. IEEE Robotics and Automation Letters, 7(2):381 0000:38113818888 2022. doi: 10. 1..100991AA.2222.14444488
[99] Jonas Wacker and Maurizio Filippone. Local random feature approximations of the gaussian kernel. arXiv preprint arXiv:2204.0556 2022.
[100] Jonas Wacker, Motonobu Kanagawa, and Maurizio Filippone. Improved random features for dot product kernels. arXiv preprint arXiv:2201.08712. 2022.
[101] Jonas Wacker, Ruben Ohana, and Maurizio Filippone. Complex-to to-real random features for polynomial kernels. arXiv preprint arXiv:2202.0203: 2022.
[102] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLPP: Analyzing and Interpreting Neural Networks for NLP, pages 353- 355. Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/V 14446. URL
[103] Fangjinhua Wang, Marie-Julie Rakotosaona, Michael Niemeyer, Richard Szeliski, Marc Pollefeys, and Federico Tombari. Unisdf: Unifying neural representations for high-fidelity 3d reconstruction of complex scenes with reflections. arXiv preprint arXiv:2312.13285, 2023.
17

---

[104] Peng Wang, Lingje Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint orXiv:2106. 10689, 2021.
[105] Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3295-
3306, 2023.
[106] Zian Wang, Tianchang Shen, Merlin Nimier-David, Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Müller, and Zan Gojeic. Adaptive shells for efficient neural radiance field rendering. arXiv preprint arXiv:2311. 10091, 2023.
[107] Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In International Conference Machine Learning, pages 23549-23588. PMLR, 2022.
[108] Xiuying Wei, Skander Moalla, Razvan Pascanu, and Caglar Gulcehre. Building on efficient foundations: Effective training of LLMs with structured feedforward layers. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview.oo
[109] Wei Wu, Jun Xu, Hang Li, and Satoshi Oyama. Asymmetric kernel learning. Technical Report MSR-TRR-2010-55, Microsoft Research Asia, June 2010. URL https: wttp://wcoooo....oommmoom http:/publlllaiiiiocc cattion /
com/ htt//enuss/rreeeee ch/ wwwphhppppllaacaeeeeeee
http://ppppp.ptttaaaatatttetttee wsyymetrii--keeeleee........ learning//
[110] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In European conference on computer vision, pages 106-122. Springer, 2022.
[111] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[112] Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin. Grid-guided neural radiance fields for large urban scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8296-8306, 2023.
[113] Zike Yan, Haoxiang Yang, and Hongbin Zha. Active neural mapping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10981-10992, 2023.
[114] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492-2502, 2020.
[115] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-483 2021.
[116] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-9, 2023.
[117] Lin Yen-Chen. Nerf-pytorch. https: / thub.
innernfnnnnnnnneeneneeeeeenn htwp:echhheettteeeeetttttttth//h 2020.
[118] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761, 2021.
[119] Felix X. Yu, Ananda Theertha Suresh, Krzysztof Marcin Choromanski, Daniel N. Holtmann-Riee and Sanjiv Kumar. Orthogonal random features. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gamett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain,
18

---

pages 1975-19888 2016. URL https: www.igs...uiss....e wttp./ppppp//////esselll// Abstract. html.
[120] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems, 35:25018 -50332, 2022.
[121] Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Re. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations, 2024, URL Ad=4g0212N2Nx
peppenreview..etee
[122] Michael Zhang, Simran Arora, Rahul Chalamala, Benjamin Frederick Spector, Alan Wu, Krithik Ramesh, Aaryan Singhal, and Christopher Re. LoLCATs: On low-rank linearizing of large language models. In The Thirteenth International Conference on Learning Representa- tions, 2025. URL https://openr eviewwreview.om net weteeeooeeee ftr//oormmiee -o JytGeyJxx9.
[123] Youmin Zhang, Fabio Tosi, Stefano Mattoccia, and Matteo Poggi. Go-slam: Global optimiza- tion for consistent 3d instant reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3727-3737, 2023.
[124] Yilong Zhao, Chien-Yo Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikei. Atom: Low-bit quantization for efficient and accurate IIm serving. Proceedings of Machine Learning and Systems, 6:19-2209, 2024.
[125] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places:
10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1461) 2017.
[126] Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R Oswald, Andreas Geiger, and Mare Pollefeys. Nicer-slam: Neural implicit scene encoding for rgh slam. In 2024 International Conference on 3D Vision (3DV), pages 42-52. IEEE, 2024.
[127] Matt Zucker, Nathan Ratliff, Anca D. Dragan, Mihail Pivtoraiko, Matthew Klingensmith, Christopher M. Dellin, J. Andrew Bagnell, and Siddhartha S. Srinivasa. Chomp: Covariant
hamiltonian optimization for motion planning. The International Journal of Robotics Research, 1193, 2013. doi: 10.1 3488805. URL https: //dp::::///dii. org/10. 10. 1177/0278364913488800
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract accurately reflects the paper's contributions, including the proposal of EUGens, theoretical advancements (e.g., unbiased approximation polynomial FFLs), and empirical improvements in speed and memory across diverse tasks. All claims are supported by the content.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly discuss the limitations of our method, EUGen. In particular, we highlight performance-speed tradeoff depending the number of random features, and address the computational cost during the distillation process.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide the assumptions and proofs of EUGens in main paper and the
appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all necessary details to reproduce our main results, including architectural and training specifications in the main paper, along with the source code for EUGen layers and representative use cases for implicit neural representations.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
21

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly available datasets and baseline code, and own code is also provided with detailed instructions to support faithful reproduction of the results. When official implementations unavailable, used widely adopted unofficial PyTorch versions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //wips.coo wwtppbccc//ccccccccccccccccc gewidee/Coeeeeeeeeeeeeeeeeeeeeee
issiisss ssionnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All training and evaluation details are provided in the main paper and supple- mentary material, with source code included in the appendix for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report error bars for most experiments to provide measure of statistical significance.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:: We disclose the compute resources used for our experiments, including GPU type, memory, model size, and execution times.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurs htt.eeewwwpp...pppppppeeeeeeeeee cc/public//thhcccctteeee delines?
Answer: [Yes]
Justification: This paper fully adheres to the NeurlPS Code of Ethics. We have carefully reviewed the ethical guidelines and ensured that our research complies with them in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
23

---

Answer: [Yes]
Justification:
We have included the broader impact section in Appendix F.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any model or dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly available datasets and code as published the original papers, and we cite all the original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
24

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve any crowdsourcing research with human subjects.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did use LLMs in our work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / /tt////errpppp/pe/ /2tps///cccooeeeeeee/2///////// for what should or should not be described.
26

---

Appendix for EUGens: Efficient, Unified, and General Dense Layers
A Proofs of the Theoretical Results
In this section, we present the proofs of all our theoretical results.
A.1 Proof of Theorem
Proof. Note that, by the definition of the EUGen layer, have the following (under the condi- tions of the theorem) for g) E Rmxd with mean-zero entries of variance one each, and where [g]( (= is an independent set of random variables for every 1,..,, m. m:
$ Σ
EUGen(W. (w,,)
Σ
8) 1-1111
(10)
i=0
j=1
Thus for any w E Rdd we get:
d Σ
(E
Σ
d (5
E EEUGGen x)]
1
g
(11)
0m0
Since {g) )= is an independent of matrices for any O, ...y k. we can re-write:
Σ g (
EI Σ r=1j=1
FFFUGen (w x)]
(12)
m
We will now prove the following lemma:
Lemma A.1. The following is true:
d Σ Eggh
)] wT
(13)
Proof. The proof of the above is in fact completely analogous to the proof of the unbiasedness the regular Johnson-Lindenstinn Lindenstrauss Transform (JLT), but provide it again here for completeness. We have:
Σ 8 1:10 )
E
EE
Σ Lid2
(14)
Eggh Wt,
We used the fact that {g. (0)),,... is a set of independent random variables for any fixed i, and r 1, m and furthermore, each entry of g has zero mean and unit standard deviation.
We can thus conclude that:
k a. (w x)' = =s(w" -(wwx)
E[EUGen(w, x)]
a.
w
x)
Σ 1=0
(15)
0=0
r=
That completes the proof.
27

---

A.2 Proof of Theorem 3.2
Proof. We will leverage our analysis and notation from the proof of Theorem 3.1. We have:
Var(EUGennww.x)) m²²
Var
Σ
Tj
(16)
==0
for Ti defined as:
71 $ T. = 0:
Σ
gy
[r][0
(17)
r=1j=1
Note that by the conditions of the theorem, different T; are independent. Thus we can write:
Var(EUGen(w, x))
Var(T.)
(18)
m2
==0
We can rewrite: T; a; >m R4 where: Rg = I[}- EE 1|[l((
E> 8) bl4w)
Furthermore, again by the assumptions of the theorem, different R2 are independent. Thus have:
Var(T;) a
Var(R:)
(19)
Note that each R; has the same variance and thus we can write:
Var(EUGGn(ww x)) = Σ a Var(RR)
(20)
==
We have the following for r 1:
Var R1) = E[ (B[RR])²² =
d
2
27
[r|
Σ
(w TwTxx [x)x)==
1=1
(21)
d
|
E j=1
II
Σ
8
- (w xx)
suffices to prove that:
27
E
Σ trllx
Σ g
2(w xx) + ||
1221 ||
(Ti. - 3) Σ wix (22)
That however follows by the computations completely analogous those provided in the proof of Lemma 3.1 from [16].
A.3 Proof of Theorem 3.3
Proof. Note that, by our previous analysis, we can rewrite:
EUGen(w, x) f(w Txx x)
(23)
m
where each Y is defined as follows:
d
Y = a; II E
Extrrl0 1=1
aa,(wT x)
(24)
28

---

By our previous analysis, we know that B| = O. Also, by the assumptions of the theorem, we can conclude that for any fixed {- an independent set of random variables. Denote: K K² m Y Note that K is completely deterministic. Therefore, by the union bound, we
can write:
[[EUUGen(w,XX) x) - f(w x)] 2 c
l max Pa T
&
(25)
1=1
By the assumptions of the theorem, the Cauchy- -schway and Triangle Inequality, we have the following:
1
|a Io
laul( x)}
(26)
j=1
Now we will apply the following version of the Azuma's Inequality:
Lemma A.2 (Azuma's Inequality). Assume that random variables W Wi,.., Wm are independent and
of mean zero. Assume furthermore that for some nonnegative {Cr rrm..... l. we have: Wrl S C, with probability one. Then the following holds:
2
F[[W] + W 2 2exp(
(27)
> C
To complete the proof of Theorem 3.3, it suffices to apply the above Azuma's Inequality for random variable W; defined as: W, _Y? YP and Inequality 26.
A.4 Proof of Theorem 3.4
Proof. We will apply the following well-known result, showing how Bernstein polynomials can be used to approximate continuous functions:
Lemma A.3 (Bernstein Polynomials for Continuous Functions Approximation). Take some constant > O. Let f: A, A] + R be bounded. Then there exists C such that the following holds:
If - Ba(f)lo S Cw(
(28)
where Bf((f is the nth Bernstein polynomial of (in particular of degree n.).
To complete the proof of Theorem 3.4, it suffices to: (1) note that pt the upper bound of P|EUGen w, x) f(w x)] 2 e] (this is implied by Theorem 3.2 and Chebyshev's 's Inequality), apply Theorem 3.3 and Triangle Inequality.
29

---

EUGen w/ Linear Layer
Compressed EUGen Layer
g(W)
Wgg(W
W
g
f())g())
W
g( ()
W
Y2
[W'(()] (()
X
f(x)
f
f(x
Figure 11: Schematic diagram of our compression method. (Top row): A EUGen layer followed by a linear layer. (Bottom Row): : The weights can be multiplied together to create single weight matrix without affecting the inputs, effectively disentangling the weights and the inputs.
B Extended Functionalities of EUGens
In this section, we discuss additional properties of the EUGen layers and show how certain special instantiations generalize several well-known layers and methods.
B.1 Compressing Networks by Combining Layers
For training NeRF models and uptraining Transformers, we use an EUGen layer followed by linear layer. In this section, we explain how we can achieve improved efficiency by compressing linear layers with EUGens. Recall our EUGen layer is given by the following equation (following the PyTorch convention):
Y1 u
(29)
the EUGen layer is followed by the linear layer with weight W and bias b', then the results is:
Y₂ Y, T +b'
((X)u(W) T W T (X1 W +b',
+b'
where W-
W This simple compression technique allows us to collapse any EUGen layer followed by FFL in our experiments, allowing for more efficient inference (see Fig, 11).
B.2 Closed-Form Knowledge Distillation
The formulation of the EUGen layer makes knowledge distillation particularly simple. Specifically, want train EUGen layer EUGen (W. to match the outputs of trained FFL using MSE as the metric, For brevity, we will suppress & for the rest the subsection. If x (resp. y) is the input (resp. output of a trained FFL, we want to find the weights W, which minimize the following emror:
min |iEUee W, yll W
(30)
The optimal W has a closed form
W = (x' X''
(31)
where x' (x).
The above solution is unique iff x x' is invertible. In the case where T' is not invertible, the inverse can replaced by the Moore-Penrose pseudoinverse. Note that, there also exists closed form formula for regression with weight decay (I2-regularizatioo)) i.e. solution for the following optimization problem
[EGeen W) yl12 + A|| W
30

---

Thus, the problem of distilling specific input- pairs in the EUGen layer can be solved without backpropagation. Note that an analytic formula does not exist if the projection matrices G are trainable. However, in that case, the optimization is extremely lightweight and can be performed with minimal compute.
B.3 Connections with Low-Rank Layers and 2-layer Neural Networks
In this section, we will discuss connections with our layer with low rank matrices, asymmetric kernels, SNNKs and works on 2-layer Neural Networks, Finally we show some results on the expressivity of some special instantiations of EUGen layers.
Recall that our EUGen layer of the form BUGen (W, x) ( where G is (Gaus- sian) projection matrix. In practice, we learn both W and G. making the EUGen mechanism quite expressive. This setup naturally leads to a more general variant: EUGen (W, x) = // (W) (x) where H and G are different project matrices. While H may not be explicitly parameterized, its effect can be captured implicitly by learning W, since EUGen involves computing products like GW, Empirically, we find in some cases that using separate projection matrices for transforming W and x in the BUGen layer may be beneficial during training. Thus, we can consider the explicit variant: FUGen R (W, x) = H (W)&G (x), which allows for more flexible form of kernel learning through disentangled projections. similar idea explored in [122].
With this formulation, we are now ready to show how EUGen generalizes various well-known approaches.
We start by showing how low rank layers can be special case of our EUGen layers. this case, take T = ① = Id, H = I and h = 1, and in this case EUGen (UGGen(Wxx x) = WGx.
Then G low rank, EuGen degenerates into low rank layer. Note that these layers have been used in the training of neural networks [70, 108, 81J. Similar to observations in Wei et al. [108], we find that the low rank layers struggle to yield good performance, and using a non-linear d leads quality gains (almost 9%), even with vr as the identity mapping.
When & and V are related to the Fourier transform of the activation function of the feed-forward layer (see [80] for precise definitions of & and v and G = H. then EuGen becomes the SNNK layer. However, SNNKs cannot unbiasedly approximate FFLs with unbounded activations, which not the case for EuGeNs (Theorem 3.1). EuGeNs can accurately and unbiasedly approximate polynomial-activatio FFLs without any learning, simply by sampling matrices G from predefined distributions. During training, SNNKs may implicitly approximate various FFLs; however, such approximations remain biased.
Next, we explain why we view our EUGen layer as an asymmetric kernel between the space of weights and the space of inputs. We interpret the EUGen layer computing similarity measure between the weights and the inputs, after projecting them into a lower-dimensional space. The motivation for the lower dimensionality of the weights comes from Mousavi-Hosser et al. [62] and this can lead to better generalizability.
For convenience of the reader, following Wu et al. [109], we define asymmetric kernels.
Definition B.1. (Asymmetric Kernel). Let X' and y be two input spaces, and 7 be feature space assumed to be a Hilbert space. Asymmetric kernel is a function : x ) - R, satisfying k(x, v) = (xxx(x)), ( for all E X' and V, where ox and dy are mapping functions from X' and y to H. respectively.
From the definition, space of weights and the space of inputs.
clear that our EUGen-layers are indeed asymmetric kernels between the
As explained earlier, even simple V, for example, the identity function can lead fairly expressive networks which we will now show below by incorporating deep and powerful results on 2-layer ReLU networks. For simplicity of the discussion, let assume EUGen layer is of the form EUGen(W: x) >"
[ x)), wi E R. g; E Rdd, and f is the ReLU activation, i.e. is the identity function, H - and (x) ReL.U(Gx).
In this special case, EUGen corresponds to the first-order Taylor expansion of 2-layer neural network with respect to the top-layer weights W.
31

---

The following the results can be easily extended to IR
Proposition B.2. (1) Under certain conditions, EUGen layers without trainable G and r number of
random features can upproximate any polynomial of degree at most I, where [2+6 < S 12+5 for some 8 >> O.
(2) Every 1-Lipschitz, function can be approximated with respect to the 12 norm over [-1, 1]4 by EUGen-layer of poly(r) number of random features.
Proof. (1) proved in Theorem 1 in Ghorbani et al. [34], while (2) can be shown via Theorem and 2 in Hsu et al. [41].
When G is made trainable, even simple EUGen-layers effectively function as 2-layer ReLU networks, inheriting their strong expressive power (for ex: See Theorem 1 in Boursier et al. [6]). More- over, it is shown that the 2-layer networks are more expressive than their degree Taylor series counterparts [34].
Finally we would like clarify the differences between our work and the works on the linearization of 2-layer networks.
A lot of work has been focused on the lincarization of 2-layer networks of the form L(0,x)
W2f(W 1x), where 0 = (W1, W2). and f is some non-linear function and W is fixed and only W2 is trainable. Under various assumptions, one express L as kernel [14, 13, 34, 7, 65]. Note that our goal is fundamentally different as we are trying to linearize a single layer and not a 2-layer network of specific form. Moreover, allowing the projection matrices G be trainable, our EUGen layers do not fit in these above frameworks.
C Additional Related Works on Implicit Neural Representations
In this section, we discuss techniques commonly used to accelerate NeRFs and SDFs, Neural Radiance Fields (NeRF) [61] achieve remarkable quality in 3D reconstruction by leveraging MLPs to map spatial points to volume densities and colors. However, both training and rendering processes remain time-consuming. prompting extensive research aimed at accelerating these tasks, by integrating explicit representations [55, 1 18, 42, 89, 87, 11, 84, 64, 31, 112, 5, 88, 43), dividing scene into smaller blocks [75, 89, 95, 110, 96], caching (baking) the implicit functions [32, 39, 12, 76), and devising some novel tweaks [54, 37].
Signed Distance Fields (SDF) are scalar fields that represent the distance to the nearest surface point, crucial in robotics for tasks such environment mapping, collision avoidance, and trajectory optimization [127, 93, 79, 63, 28, 27, 10, 20, 98, 123, 45, 24, 113, 126]. In computer vision, Truncated Signed Distance Fields (TSDF) are employed in SLAM systems, as traditional SDF computations can be prohibitively expensive in real time [66, 67]. Several approaches have emerged to facilitate real-time SDF construction due to importance in robotics [69, 38, 68), while others have addressed challenges in constructing SDF for complex environments [30, 74, 33, 72]. To get a photo-realistic quality of scene reconstruction, recent works replace the volume density in NeRF formulations with SDF, using well-designed SDF-to-volumg acceleration is achieved by combining explicit grid structures with implicit SDF [120, 105, 52, 103, 77], populating thin shells near the zero level set [106], or baking SDF into a high-quality triangle mesh [1]6].
density conversion formulas [114, 104, 115]. Further
32

---

D Experiments
Contents
D.1 Synthetic experiments.
33
D.2 Detailed results for 3D scene reconstructions
33
D.3 Image Classification Experiments
37
D.4 NLP Experiments
40
D.5 LLM Pre-training
40
In this section, provide additional details for the experiments in our main paper. Our code for the EUGen layer implementation and related experiments can be found at https: htpp::///hi //ggttuu.b... com/ arij jtttttgatamee//ee /EUGen/.
D.1 Synthetic experiments.
For these experiments, 10,000 samples drawn randomly from the range (0, 1) with dimension of 512, and are then passed through a Linear layer of shape (512, 512). Non-linearity of ReLU (resp. Softplus) is applied the outputs of the linear layer. Our aim is show that our EUGen-layer can accurately approximate these outputs.
We use the SNNK layers [80] and Low Rank approximation as basclines. We train EUGen and SNNK layers for 3k epochs with 128 random features using the Adam optimizer [48]. We repeat this experiment 10 times and report the MSE loss across the repli- cates. This experiment is run on a single NVIDIA RTX 4090.
We present an additional experiment on image fitting by replacing FFLs in the Siren network [82] which is a MLP with sine activations. The Siren network for the image fitting experi- ment is layer neural network with hidden dimension matrices sizes 256 x 256. It was shown in the orig- inal work that Siren network can not only fit the image but also aceu- rately represent the derivatives of the signal. We replace the hidden layer with our EUGen layer with 32 random features, resulting in modest compu- tational gains. We show that like the original Siren network, our EUGen- Siren network can also accurately model the derivatives of the signal. It should be noted that our EUGen-Sinee Siren- network is about 3 the size of the baseline Siren network.
u
Figure 12: Top to bottom The baseline Siren network the first
row, fitting not only the image, but also gradients. (Middle) SNNK-Sireh (Bottom) EuGen-Siren on the bottom row produces an accurate approximation of the above.
D.2 Detailed results for 3D scene reconstructions
In this subsection, we provide details for all reconstruction experiments in the main paper.
NeRF Experiments. For these experiments, we use default configuration of the Pytorch implementation of NeRF 17). The inference time is computed using single NVIDIA RTX 4090 and an AMD Ryzen 7700 8-Core Processor. tests are conducted on the Synthetic NeRF dataset [61], and the details of our architecture choices are illustrated in Figure 13 (top row). In our
33

---

29
(a) Baseline NeRF architecture
(b) EUGen-NeRF architecture
rlad 255
255
rod
250
250
2:0
2bb
(c) Baseline iSDF architecture
(d) EUGen LLUGen-iSDDPaa architecture
Figure 13: (Top): Network architecture of the baseline NeRF and our EUGen-NeRL model. Gray denotes Linear ReLU layer, Orange is Lincar without activation, Pink is the EUGen layer, + is concatenation, and the activation for the RGB output is Softmax. (Bottom) Network architecture of the SDF prediction network in iSDE. (b) Our modified EUGen-iSDF architecture. Note that both cases, EUGen layer followed by the Linear layer can be compressed more efficient network inference.
experiments, the number of random features to 256, with additional results for the different number random features provided Section E.2.
We report the detailed results for each scene in Table 1. We achieve an almost 24% increase in inference speed and lower storage costs, while matching the performance of the baseline.
iSDF Experiments. Figure 13 (bottom row) shows the baseline iSDF [69] and the modified EUGen- iSDF architectures. use six ReplicaCAD [86] scenes, following original iSDF [69] paper. In all experiments, we use the default configuration of public available iSDF code while replacing only the SDF network with our EUGen-iSDF.
To evaluate efficiency, we report two time-related metrics: training time and inference time. However, it's important note that, as iSDF an incremental mapping module, the number of keyframes can vary between different experiment runs. Because the sample size is proportional to the number of keyframes, training time may not serve perfect measure of the network's efficiency. To account for this, we also report inference time, which is measured by sampling 1000 random rays and recording the time taken for single forward pass through the network.
In Figure 21, we visualize the reconstructed SDFs for all ReplicaCAD dataset and report the quantitative results in Table 4. We achieve 23% increase in inference speed while matching the quality the baseline.
Mip-NeRF 360 Experiments. Mip-NeRF 360 [4] is of SOTA models for novel view synthesis. We use the NeRF-Factory PyTorch implementation of Mip-NeRF 360. To fit within memory constraints, reduce the training batch-size from 4096 2048. Experiments are conducted on the 360 v2 dataset, replacing three network layers with EUGen layers, each using 1024 random features.
Inference is performed
single NVIDIA RTX 4090 GPU with an AMD Ryzen 7700 CPU.
We report the detailed results for each scene in Table 2. EUGen layers produce comparable results while being 27% faster.
Zip-NeRF Experiments. We build on prior work that accelerates NeRF by combining explicit hash- grid structures with MLPs [11, 84, 64, 5], enabling more efficient spatial encoding and reducing MLP overhead. However, these improvements We tackle this gap using EUGen layers.
not address inefficiencies within the ML.Ps themselves.
34

---

y0.0
128
128
128
128
&x
ylktt
128
128
128
128
Ax
76
76
(a) Baseline D-NeRF architecture
(b) EUGen-D-GWi D-NeRF architecture
Figure 14: (a) Architecture of the deformation network D-NeRF [71, 90]. (b) Our modified EUGen-DD-Neke architecture. Gray blocks denote Linear ReLU layers, Pink blocks are the EUGen layers, and there no nonlinear activation (ReLU) for Ax.
We focus on Zip-NeRF [5], a state-of-the-art method for unbounded scenes that uses shallow MLPs to decode grid-based features. Its architecture contains two proposal ML.s for sample placement and a NeRF MLP for producing color outputs. Due to the proposal ML.Ps' low dimensionality (6 + 64 + 1), instead target the NeRF MLP, specifically its view-dependent component composed of two linear ReLU layers. We replace the first of these with our EUGen layer, resulting in BUGen-ZZp-NeRF-a minimal intervention that serves as strong case study for BUGen's effectiveness.
All experiments use the default Py PyTorch implementation of Zip-NeRF [36] on the Mip-NeRF 360 dataset [4]. To match NeRF rendering benchmarks, we use consistent hardware settings but reduce the ray batch size from 65,536 to 4,096 due to memory constraints. We use 64 random features for the EUGen layer.
All rendered images from Mip-NeRF 360 dataset can be shown in Figure 16. Despite this minimal change, we maintain similar rendering quality while improving the rendering speed by 6%.
Dynamic NeRF Experiments. In this experiment, evaluate method on D-NeRF [71], which synthesizes novel views of dynamic scenes with complex, non-rigid motion. D-NeRF extends NeRF by introducing deformation network that takes position and time inputs and outputs 3D offset, effectively bending rays to model scene motion over time.
We use the PyTorch implementation of D-NeRF [90], which incorporates hash-grid structure [64] to accelerate training and rendering. As a result, the network shallower than the original D-NeRE To apply our method, we replace two layers in the deformation network with EUGen layers (see Figure 14).
For fair comparison with the baseline, we reduce randomness by using the preload configuration and disabling other acceleration options, ensuring consistent settings across all experiments.
We present the rendered images Fig. 19. We achieve 7% speedup in inference while maintaining similar reconstruction quality.
Table 1: Comparison between NeRF and our EUGen-NeRF on the Synthetic NeRF dataset [61]. J. We report PSNR, SSIM, and LPIPS per scene, and also include the averaged results along with inference time and storage. A higher PSNR/SSIM and a lower LPIPS/Time/Strrage is preferred.
Chair
Hotdog
Lego
Materials
Ship
Avg.
Storage
NeRF
33.70
25.46
26.40
35.81
31.31
29.00
33.11
28.79
30.45
2.36s
3.27MB
PSNR
EUGen-NoRF
33.09
25.18
28.40
35.47
30.60
28.88
32.63
28.66
30.36
1.79s
2.27MB
w/o trainable
32.76
24.92
28.11
35,22
30.14
28.62
32.42
28.61
30.10
1.79s
2.27MB
NoRF
0.978
0.930
0.939
0.979
0.965
0.956
0.979
0.870
0.950
SSIM
BUGen-NeRF
0.974
0.954
w/o trainable
0.972
0.922
0.958
0.975
0.952
0.951
0.975
0.857
0.945
LPIPS
BLGGen NeRI trainable
0.016
0.053
0.023
0.015
0.022
0.023
0.022
0.079
0.032
Distillation Results.
We show complete distillation results for NeRF Table 5 on the lego dataset.
35

---

EUGen-NeRF w/o trainable G
Ground Truth
NeRF
BUGen-NeRF
chair
drums
ficus
hotdog
lego
materials
mic
ship
Figure 15: Rendered images from the Synthetic NeRF dataset [61]. For each scene, the images from left to right show the Ground Truth, NeRF [61], and EUGen-NeRF.
36

---

Table 2: PSNR of rendered images for Mip-NeRF 360 in 360 V2 dataset. all settings, higher PSNR & SSIM, and lower LPIPS is preferred. We observe that EUGen achieves comparable performance to Mip-NeRF while being 27% faster.
Scene
Bicycle
Bonsai
Counter
Garden
Kitchen
Room
Stump
Mip-NeRF 360
22.31
30.59
27.77
24.65
29.87
31.18
24.49
PSNR (1)
BUGGen-Mip-Nand 360
22.10
29.99
27.29
24.30
29.34
30.60
24.04
Mip-NeRF 360
0.466
0.897
0.809
0.626
0.879
0.885
0.569
SSIM (T)
EUGen-Mip-NeRF 360
0.435
0.882
0.793
0.593
0,859
0.869
0.538
Mip-NeRF 360
0.529
0.230
0.301
0.364
0.169
0.262
0.488
LPIPS (U)
EUGen-Mip-NeRFF 360
0.557
0.263
0.328
0.389
0.196
0.295
0.516
Table 3: Detailed hyperparameters for EUGen-Vin experiments ImageNet and Places365 datasets.
LR stands for learning rate.
Hyperparameter
ImageNet (1.28M)
Places365 (1.8M)
Num. layers
12
12
Num. heads
12
12
Hidden size
768
768
MLP dim.
3072
3072
Num. patches
16 16
16 16
Batch size
4096
4096
Epochs/ / Total Steps
cf7300 epochs
80, 000 steps
Base LR
10-3
10 3
Final LR
10-a
10 -
Optimizer
Adam
Adam
LR schedule
Linear warmup (10* steps Imagenet), 500 steps Places365), constant, cosine decay
In Figure 20 (left), we visualize the rendered image with the distilled Zip-NeRF model. We set the number of random features to 64, with ablations provided in Appendix E.2 (Fig. 27 and Table 10). Furthermore, to make the distillation more efficient, we adopt several strategies: (1) we sample only 1/16 of the training images, (2) we render them at 1/8 of the original resolution, and (3) we randomly sub-sample 10% of the resulting input-output pairs. These choices significantly reduce the memory footprint and allow the distillation process to run efficiently within our hardware constraints.
We also test the distillation in the last hidden layer iSDF [69]. In Fig. 20 (right), we visualize the reconstructed SDF and the extracted zero level-set mesh from apt 2 nav dataset. Our results indicate that the reconstruction quality of the distilled EUGen-iSDE is comparable to that of the baseline iSDF, with 9% improvement in inference speed.
D.3 Image Classification Experiments
+ BAGeaT
In this section we list the experiment setup of all image classification experiments. First we provide the hyperparameters and experiment setup for Ima- geNet and P1aces365. We replace a part of the Feed- Forward Network (FFN) block in Transformers with the EUGen layer, namely the expansion layer with the GeLU activation. Note that the EUGen layer then be combined with the following linear layer to create a single linear layer of size [r, 768]. where the number random features. This allows for a significant compression of the ViT model.
Figure 18: Flops vs parameters. EUG-ViT ab- tains 30% reduction in flops.
The hyperparameters and model architecture are shown in Tab. 3. Fig. 4 illustrates that we achieve approximately the same accuracy as VIT full (86M parameters) by replacing its top-6 MI.P layers with EUGen, leading to 30% reduction in inference parameters.
37

---

Table 4: Quantitative results of EUGen-iSDE, i the number random features, Lw is calculated across all SDF levels, and Lurf is for ground truth surface points. Unin refers to iteration time (including backward pass), and infer measures forward pass inference time. Results are averaged across 6 ReplicaCAD [86] sequences with 3 random seeds. Our EUGen-iSDF model achieves less than similar quality the baseline while being 23% faster.
Distance (cm)
Time (ms)
Method
L1 avg +
Ll surf
CD +
Irrain +
linfer +
iSDF
8.99
4.08
2.02
8.24
1.37
EUGen-iSDF
9.37
4.17
2.10
7.81
1.06
BUGen CeGGen--iipNeRRE trainable
Ground Truth
Zip-NeRF
EUGen-Zip-NoRF
bicycle
garden
stamp
counter
kitchen
bonsai
Figure 16: Rendered images from the Mip-NeRF 360 dataset [4]. For each scene, the images from left to right show the Ground Truth, Zip-NeRF [5], and EUGen-Zip-NoRF.
38

---

ImageNet
Places365
0.80
Validation Accuracy
Validation Accuracy
0.75
0.54
0.70
ViT-Base Rank
ViT-Base
0.50
EUGen (nx, 32)
EUGen (rx
EUGen U/ 64)
EUGen (ra 64)
0.48
60
80
100
80
100
Fraction Inference Parameters (%)
Fraction of Inference Parameters (%)
Figure 17: Evaluation results of EUGen for image classification tasks: (left) ImageNet and (right) Places365 datasets using ViT (86M parameters). We observe that EUGens match the perfor- mance of vanilla ViTs with a significantly smaller fraction of parameters than the Low-Rank baseline.
Table 5: Distillation results for NeRF in lego dataset. Trainable G produces a leaner efficient model while being almost 26% faster. Both analytic and trainable uses 128 random features except the last row which uses 64 random features.
Method
PSNR
SSIM
LPIPS 4
Inference Speed (s)
Baseline
31.31
0.965
0.017
2.36
Analytic
28.54
0.948
0.027
1.84
Trainable
31.25
0.965
0.017
1.84
Trainable G (64)
31.09
0.964
0.017
1.75
Moreover in Fig. 17, show that improve accuracy by increasing the number of random features and our models greatly outperform the low rank variants.
Next show results additional image classification tasks, namely Cifar-10, Cifar- 100, DTD and Oxford Pets. For these set experiments r = 16. Fig. 22 shows that we can maintain accuracy across wide range of image classification tasks using ViT while reducing the size of the model by almost 40% and improving inference speeds by 34%. We present detailed analysis of flops in Fig. 18 as successively replace MLP layers starting from the top layer.
To summarize, by replacing top-6 layer's MLP blocks with EUGen reduces the size of the VIT model from 346 Mb to 196.85 Mb. This speeds up inference by almost 35% and has minimal impact on accuracy. We use the AdamW [58] optimizer with a weight decay 10-5 and learning rate of 3 x 10-5 and constant scheduler with warmup steps 6% of the total number of training steps with batch size 16.
Moreover our method can be combined with other efficient architectures like EfficientViT [56]. showcasing the versatility of our method.
Scaling EUGens to Deeper Models this subsection, we will show that EUGens can be used in deeper models like VIT-L. which has 24 layers. Table 6 shows that replacing most ViT-L MLP blocks with EUGens yields over a 50% reduction inference parameters, matching performance on Places365 and incurring only small drop on ImageNet.
Table 6: Top-1 accuracy (%) and parameter counts for baseline and EUGen variants. EUGen-r corresponds to æ MLP blocks in Vit-L replaced by EUGen.
Dataset
Baseline
EUGen-18
EUGen-21
Params (B / /18 21)
Places365
58.60
57.47
57.04
304M / 176M 150M
ImageNet
82.20
78.91
79.13
304M / 176M 150M
39

---

Table 7: EuGen experiments on GLUE benchmarks on Linearizing the Tanh Pooler Layer MCC
score is reported for CoLA, Fl score is reported for MRPC and QQP, Spearman correlation is reported for STSB, and accuracy scores are reported for the other tasks. All results are an average of seeds. Our EUGen model beats all the other baselines on all the GLUU tasks.
Dataset
# Training Parameters
RTE
MRPC
QNLI
QQP
SST-2
MNLI
STSB
COLA
Pooler + Classifier
~ 59k
57.5
81.5
74.5
72.0
84.9
56.4
78.1
29.4
Linear Probe
1k
57.8
81.2
69.2
62.2
83.1
43.4
69.9
35.4
SNNK [80]
2k
60.1
81.5
72.2
70.2
83.0
53.7
71.8
32.8
EuGen (ours)
2k
61.7
83.8
75.5
72.8
86.8
57.4
82.8
45.2
D.4 NLP Experiments
In this section, we provide additional NL.P experiments, namely uptraining BERT [25] on the GLUE benchmark [102].
Linearizing the Feed-Forward Networks Similar to the GPT setting, we replace a part of the Feed-Forward Network (FFN) block in Transformers with the BUGen layer, namely the expansion layer with the GeLU activation. Our compression trick allows for significant reduction of parameters of the BERT model. In all our experiments r = 32. We report the MCC score for CoLA, F1 score for MRPC and QQP, Spearman correlation for STSB, and accuracy scores for the other tasks.in Fig. 22 as we successively replace MLP layers starting from the top layer. We use the AdamW optimizer [58] with weight decay of 10 and a learning rate of 3 x 10-5, and constant scheduler with warmup steps 6% of the total number of training steps with a batch size of 16.
We also present the detailed results for each GLUE task in Figure 24. Our EUGen- BERT model matches the baseline model across all the 8 tasks while being almost 26% smaller. We also show that EUGens can accelerate smaller language models like DistilBERT. Our results show that we can further compress DistiBBERT by 21%. Thus our method can complement other well-known methods for model compression like knowledge distillation and pruming.
Linearizing the Pooler Layer Many encoder only pretrained transformers like BERT use a special token called [CLS] that is appended in front of the input sequence which can be used for classification tasks. After the transformer blocks, these encoder models employ pooling layer on the final hidden state of the [CLS) token as the final representation for the sequence which is then passed to a classifier layer for a classification or regression task. The pooler layer is linear layer of size the hidden dimension of the transformer with a tanh activation (in this case d = 768). A EuGen layer can be used as a drop-in replacement for this pooler layer. For this experiment, the base model is frozen and only the pooler and the classifier weights are tuned. We beat the SNNK layer (which has the same number training parameters as ours) as well as the baseline model (Results for classifier and pooler are taken from [50] which has significantly more training parameters on all the 8 tasks.
d. where d is
D.5 LLM Pre-training
In this section, we provide the implementation details of the LLM pre-training exper- iment. Our GPT-2 implementation is based on the open-source repository available https://github.com httpp://gghub..ooo/aaaaaatt//// coo.//aaaaatttynnaoGG....... We performed pre-training on NVIDIA A6000 GPUs on OpenWeb'Ter dataset. In this experiment, we used 64 random features, 40 gradient accumulation steps, 0.1 weight decay, batch size of 36, and train the model for 50K iterations. We set the gradient clipping parameter to 1.0 for vanilla GPT-2 and set it to 0.0 for EUGen GPT-2. In Fig. 3, we present the results with to 11 (out of 12) GPT layers replaced with EUGens, as we observed that replacing all layers affects the convergence.
40

---

JUUGen-D-NeRF w/o trainable G
Ground Truth
D-NeRF
F-GGen-D-NeRF
bouncing balls
hell warrior
hook
jumping jacks
lego
mutant
stand
1-rex
Figure 19: Rendered images from the D-NeRF dataset [71]. For each scene, the images from left to right show the Ground Truth, D-NeRF [71, 90], EUGen-D-NeRF without trainable G, and BUGen-D- NeRF We observe that EUGens generate realistic renderings in settings.
41

---

Pretrained Zip-NoRF
2-15em
2- .2
2-2.8m
Level-sct mash
Ground Truth
EUGen Distillation
iSDF
EUGen Distillation
w/o Trainable G
EUGen Distillation w/o Trainable
Figure 20: Visualization EUGen distillation. (Left): Rendering Zip-NeRF, (Right): Visualization of
reconstructed SDFs. The last columa shows the zoom-in rendering of zero level-set mesh.
2
opt
nov
obj
opr 3 HUP
nov
apt 3 obj
Ground Truth
iSDF
EUGen-iSDD
Signed Distance [m]
-15
Figure 21: Reconstructed SDFs in iSDF and our EUGen-iSDF for six ReplicaCAD sequences.
33.2%
65
75
75
75
(3071
Paramrater
Figure 22: Accuracy vs Inference parameters trade-off plot for uptraining ViT by successively replacing MLP layers with the EUGen layers from the bottom. We show results for CIFAR-10. 10, CIFAR- 100, DTD, and Pets. We achieve almost 34% reduction in parameters, while matching the performance of the baseline models in all cases.
CIFAR-100
RTE
SST-2
Accuracy
93
Finetuned EfficienVVI
Accuracy
Accuracy
90- EUGen
60
Finetuned DistilIBERT
90
87-
ElGGen
58
55
Inference Parameters (x10*)
Inference Parameters (x10%)
Inference Parameters (x10%)
Figure 23: Uptraining experiments other datasets and architectures. (Left) Uptraining Effi ViT on CiFAR-100, (Middle and Right): Uptraining DistilBERT on SST2 and RTE datasets. Our model can achieve similar performance various baseline models while being significantly faster, thus becoming complementary technique to distillation and other efficient architectures.
42

---

RTE
MRPC
CoLA
SST-2
72
96
Accuracy
FI Score
Accuracy
68
92
MCC
60
92
+25.7%
88
100
110
90
100
110
90
100
110
90
100
110
STSB
QQP
MNLI
QNLI
Spearmun Correlation
92
Fl Score
Accuraey
Accuracy
88
84
Fineouned BERT
EUGen BERT
90
100
110
90
100
110
90
100
110
90
100
110
Inference Parmmeters (x10%)
Figure 24: Detailed evaluation results of EUGens the GLUE benchmark. MCC score is reported for CoLA, Fl score is reported for MRPC and QQP, Spearman correlation is reported for STSB, and accuracy scores are reported for the other tasks. EUGen matches the baseline performance while being almost 26% smaller.
43

---

E
Ablation Studies
In this section, we present additional experiments on (a) justifying the use of ReLU over Softplus, (b) Effect of the number of random features in various models, (c) justify the choice of using ORF in our EUGen layers, (d) justify the choice training projection matrices and (e) finally, show that introducing non non-linearity is essential the performance of EUGen layers
100
Average
100
Surface (s - 0cm)
100
0cm <mm 10cm
100
10cm <m s 20cm
SJP error [em]
relu
SJF eirr [cm)
20
sotplus
10
10
05.11
26
29 47
0.5
0.5
26
39
47
57
0.5.1
26
39
Keyframe times
20cm
50cm
100
50cm
100cmm
100.
100cm <c
SDF error [cm]
50:
50
50:
20
Keyframe
Keyframe
Keyfrome
Figure 25: Visualization of the error when comparing ground truth and reconstructed SDF using various activation functions. The first column displays the average error. The subsequent rows illustrate errors across different ranges of ground truth SDF values. In robotics scenarios such as grasping or manipulation, achieving more precise reconstruction, particularly for lower SDF values as shown in the first row, is crucial.
E.1 ReLU over Softplus in iSDF
As our goal is to explore how far we can accelerate using EUGen layers, we first investigate potential speed improvements the baseline model before making modifications. We found that replacing Softplus with ReL.U results in faster inference with minimal quality degradation. Specifically, this change causes a slight performance decrease for distances 2 100 cm (see Figure 25). However, since surface information is critical for robotics tasks such as manipulation or navigation, we exploit ReLU activation in our iSDF experiments both baseline (iSDF) and EUGen-iSDD to prioritize network speed.
Table Comparison between NeRF and our random feature mechanism on the drums dataset. The inference time refers to the approximate time for forward passes of each model to render a single 400x400 image.
Rendering Quality
Efficiency
Method
# RF
PSNR 1
SSIM t
LPIPS L
finter (s)
4
Model Size (MB)
NeRF
25.46
0.930
0.049
2.36
3.27
256
24.92
0.922
0.056
1.79
2.27
128
24.64
0.918
0.059
1.47
1.39
EUGen-NeRF
64
24.39
0.913
0.064
1.32
0.96
w/o trainable
32
23.90
0.903
0.076
1.22
0.74
16
23.17
0.886
0.098
1.19
0.63
E.2 Effect of Number of Random Features in 3D Reconstruction Tasks
Effeet of Number of Random Features EUGen-NeRF. In this experiment, we investigate how the number of random features affects the quality of EUGen-NeRF. We evaluate three scenes: holdog, materials, and drums, which represent easy, medium, and hard cases, respectively, based on rendering quality in Table I. We test models with 16, 32, 64, 128, and 256 random features.
44

---

We present the qualitative and quantitative results of the number of random features in EUGen-Neri in Figure 26 and Table 8. Using small number of random features, like 16, results in approximately a two-fold acceleration compared to the baseline model.
Drums
Ground Truth
256
128
64
32
16
Hotdog
Materials
Figure 26: Comparison of rendering results with varying numbers random features in EUGen-NeRF The numbers at the top indicate the number of random features used for each specific EUGen-NeRF model. The ground truth image is on the left, while the right shows zoomed-in views of each reconstruction.
Finally, in extreme setting, increasing the number of random features to match the baseline leads to improved results, demonstrating that EUGen-NeRF learns better representations than the baseline NeRF
Table 9: Matching the baseline's 's parameter count by increasing random features leads to improved performance.
PSNR t
SSIM f
LPIPS 1
baseline EUGen (same # params)
30.45
0.950
0.032
30.54
0.950
0.030
Effect of Number of Random Features in Zip-NeRF Distillation.
In Figure 27 and Table 10, we present an ablation study illustrate how the number of random features affects the distillation process. The number of random features balances the trade-off between quality and inference speeds and storage costs.
Effect on number random features for iSDF experiments. In this subsection, we show how the number of random features affect the results in our iSDF experiments. For this study, use only the
45

---

Pretrained Zip-NeRF
Pretrained
32
128 2.66 512
Figure 27: Comparison of rendering results with varying numbers of random features in Zip-NeRI distillation. The numbers at the top indicate the number of random features. Rendering from the larget pretrained model is on the left, with zoomed-in views of each reconstruction on the right.
Table 10: PSNR of rendered images and inference time for pretrained Zip-NeRF and layer-wiss distillation for our analytic variant using various number of random features on the bicycle dataset [4].
Pretrained
m 32
64
m
128
256
m 512
PSNR f
23.66
20.72
21.06
21.55
21.72
22.06
Inference time (s)
4.03
3.76
3.78
3.86
4.06
4.43
10
10"
EUGen
EUGea-ORF
Loss
10
Loss
10-
Loss
10
10a
10
10a
16
32
64
128
256
512
128 256
512
16
32
128
Random
Feutures
Random Features
Random Features
Figure 28: Comparing the approximation capability of our EUGen layer with orthogonal random features vs random features. (Left) Approximating a ReLU-Lincar layer. (Middle) Approximating Softplus-Lineat layer. (Right) Approximating GELU-Linear layer.
non-trainable G variant. As before, show that we can tradeoff accuracy speed by controlling the number of random features (Table 11). Qualitative results are shown in Figure 30.
E.3 Orthogonal Random Features in EUGen
In this subsection, we show that ORFs in EUGen outperform random projections in approximating ReLU, Softplus, and GELU linear layers (Fig. 28).
E.4 Trainable G improves performance
Recent works [19, 121] have shown that training the projection matrix instead of using fixed probability distribution can lead to improved results. We validate this hypothesis in the context of various 3D scene rendering tasks for NeRF, D-NeRF, and Zip-NeRF (Fig. 29).
NeRF
D-NeRF
Zip-NeRF
30.4
30.36
30.4
30.32
30.26
28.0
27.94
PSNR
30.10
PSNR
PSNR
27.81
30.0
27.8
29.6
29.6
27.6
EUGen
w/o trainable G
EUGen
w/o trainable G
EUGen
w/o trainable G
Figure Trainable G improves performance for 3D scene rendering tasks for various architectures. (Left): NeRF, (Middle): D-NeRF, and (Right) ZiR-NERF
46

---

Distillation Experiments with Trainable Projection Matrix. In this subsection, we show that using trainable G also improves the quality the distilled models. However, this case, optimizing the EUGen layer to mimic the outputs of an FFI. in pretrained INR can not be done analytically. So we use gradient descent train this EUGen layer. Table 12 shows the results for various loss functions that can be used to compare the output of an EUGen layer with the target FFL. We observe that the Huber loss performs better than MSE as it is robust to outliers. Adding a trainable bias improves performance, but does not add any extra latency during inference. That is why the trainable G with bias and Huber loss the preferred choice for all the distillation experiments.
apt 2 mnp
apt
apt obj
apt mnp
opt 3
apt obj
128
64
32
Signed Distance [m]
-2.0
-3.5
-1.0
-25
0.0
0.5
1.0
35
Figure 30: Reconstructed SDFs in iSDF for six ReplicaCAD scenes with varying numbers of random features. Each row corresponds to a different random feature count (indicated on the left) and follows the colormap shown at the bottom. Trainable G is not used for this experiment.
Table 11:
Quantitative results of EUGen-iSDE: RF the number of random features, LI is calculated across all SDF levels, and surt is for ground truth surface points. tuain refers to iteration time (including backward pass), and finter measures forward pass inference time. Results are averaged across 6 ReplicaCAD [86] sequences with 3 random seeds. For this ablation, the variant without trainable G is used.
Distance (cm)
Time (ms)
RFs
LLavg (L)
La (L)
CD (1)
tuain ())
fineer (L)
128
9.62
4.28
2.12
7.74
1.06
64
9,39
4.46
2.21
7.40
0.93
32
9.96
4.38
2.40
7.13
0.84
Table 12: Performance comparison of methods based on PSNR, SSIM, and LPIPS.
Method
Loss Funetion
PSNR +
SSIM f
LPIPS
Baseline
N/A
31.31
0.965
0.017
Analytic (no backprop)
MSE
28.54
0.948
0.027
Trainable G
MSE
31.07
0.964
0.017
Trainable G
Huber loss
31.11
0.964
0.017
Trainable G + bias
MSE
31.21
0.965
0.017
Trainable G + bias
Huber loss
31.25
0.965
0.017
E.5
Effect of Nonlinearities
Non-linearity in EuGen arises from at least of Φb, V,
2. As discussed in Appendix B.3, a simplified variant reduces to a low-rank layer. We show that such low-rank layers consistently under-perform, underscoring the necessity of non-linearity.
Fig. 31 further illustrates this effect. We train various INR models with low rank layers having the same number of training parameters EUGen-modee for fair comparison. The performance
47

---

gap widens as more layers are replaced, with respective layer replacement counts of (3, 2, 2, 1) for (NeRF, iSDF. D-NeRF, Zip-NeRF).
iSDF
NeRF
D-NeRF
Zip-NeRF
LJ Distance (em)
12
30.60
30.4
30.32
28.0
27.94
27.90
11.29
PSN
30
30.13
30.0
27.8
10
9.37
28.25
29.6
27.6
EUGen Low Rank
EUGen Low Rank
EUGen Low Rank
EUGin Low Rank
Figure 31: The importance of non-linearity in EUGen layers. Low rank variants underperform compared to EUGen models in various INRs.
We train a BERT model by replacing the expansion block of the final Feed Forward Networks by low rank matrices with same number of training parameters as that of EUG-BERT, On 3 datasets, namely MRPC, COLA, and SST2 our method outperforms the low rank one (Fig 32).
92.7
90.1
90
88.7
Score
75
70
65
61.1
58.3
MRPC
SST-2
CoLA
Figure 32: Comparsion of parameter matched low rank BERT with our BUGGnn-BRRY For all the datasets, EUGen-BERR BERT outperforms the low rank variant.
E.6 Increasing the Polynomial degree
In this subsection, will show that increasing the polynomial degree k (5 5) can improve performance the cost of inference speed. The experiments are run for GLUE language tasks (eight datasets) and CIFAR-100 100 from the vision-suite of tasks.
Table 13:
Increasing the polynomial degree improves performance the GLUE benchmark. MCC score is reported for CoLAA, F1 score is reported for MRPC and QQP, Spearman correlation is reported for STSB, and accuracy scores are reported for the other tasks.
Dataset
Baseline
EUGen, k<5
EUGen, k=5
RTE
57.5
61.7
62.1
MRPC
81.5
83.8
84.5
COLA
29.4
45.2
44.4
SST-2
84.9
86.8
87.4
STSB
78.1
82.8
82.9
QQP
72.0
72.8
73.4
MNLI
56.4
57.4
58.1
QNLI
74.5
75.5
75.6
Average score the GLUE benchmark while using lower-degree EUGen layers is 70. 70.75 and that of counterpart applying polynomials of degree 71.05. The latter one is 3% slower in training and 2% slower in inference.
48

---

On CIFAR-100, EUGen layers of degree 3 achieve an accuracy of 89.53, while increasing the degree raises 89.81. Similar to GLUE results, we find that the higher-degree polynomial variant achieves better quality, at the price of the small speed loss (5% slower in training and 3% slower in inference).
We thus see here a classic speed-quality tradeoff, as well as improved quality for higher-degree polynomials, as expected.
F Broader Impact & Limitations
Societal Impact. This paper proposes an efficient neural network layer that lowers the computation footprint of fully connected feedforward layers (FFLs). The experiments with GPT-2, ViTs, NeRF, and iSDF illustrate how EUGens can lower the inference speed while remaining competitive with the baselines. We believe EUGens can enable the deployment of large-scale models in real-time systems. The application of EUGens will also help alleviate computational bottlenecks, thereby reducing the carbon footprint associated with running these models.
Limitations. Although EUGens yield significant improvements in inference speed and memory usage, more work is needed comprehensively assess trade-off's while exploring the space of all possible and T. Furthermore, it is an open research question to design an efficient unbiased algorithm for approximating FFLs with general activation functions (beyond polynomials).
49


Reviews:
[{"review_id": "1agoBLJkkn", "reviewer": "Reviewer_DUMo", "summary": "This paper introduces EUGens (Efficient, Unified, and General dense layers), a new class of dense layers that generalize standard fully-connected feedforward layers (FFLs) by leveraging random features. EUGens aim to improve the efficiency of neural networks by approximating FFLs and reducing their inference complexity from quadratic to linear time. The method goes beyond traditional FFLs by incorporating direct dependence on input norms, which enhances their expressive power. Additionally, EUGens can approximate FFLs with arbitrary polynomial activation functions, offering significant reductions in parameters and computational overhead. The paper also presents a layer-wise knowledge transfer technique that bypasses backpropagation for more efficient adaptation of EUGens to pre-trained models. Empirically, integrating EUGens into architectures like Transformers and MLPs yields improvements in inference speed (up to 27%) and memory efficiency (up to 30%) across tasks such as image classification, language model pre-training, and 3D scene reconstruction.", "questions": "Trade-off Between Expressiveness and Efficiency: The introduction of input norm dependence is a key feature of EUGens. How does this affect the trade-off between expressiveness and computational cost in practice? Are there specific scenarios where this dependence may be detrimental?\n\nHyperparameter Sensitivity: How sensitive is the performance of EUGens to changes in hyperparameters, such as the number of random features (m) and polynomial order (k)? Can the method adapt dynamically to different network sizes and tasks?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nNovel Contribution: The introduction of EUGens provides an efficient alternative to traditional FFLs by incorporating random features and input norm dependencies. This novel approach reduces computational complexity, making it well-suited for large-scale neural networks.\n\nTheoretical Foundation: The paper provides a thorough theoretical analysis, including proofs and concentration bounds that demonstrate EUGens' capability to approximate FFLs with polynomial activation functions. This strengthens the validity of the proposed method.\n\nEmpirical Validation: Extensive experiments show that EUGens outperform previous techniques (e.g., low-rank approximations) in terms of approximation quality (MSE loss), speedup, and memory usage. The method is applied successfully across various architectures, including Transformers, MLPs, and NeRF (Neural Radiance Fields), showing significant real-world applicability.\n\nKnowledge Transfer and Distillation: The introduction of a distillation framework for layer-wise knowledge transfer without retraining is a significant innovation, improving deployment efficiency, especially for pretrained models.\n\nScalability and Practical Relevance: The paper highlights EUGens' potential for deploying large-scale neural networks in real-time, resource-constrained environments, making them highly relevant for practical applications.\n\nWeaknesses:\nLimited Task Diversity: The experimental evaluation mainly focuses on image classification, language models, and 3D scene reconstruction. While these are essential tasks, the paper could benefit from evaluating the method on a broader range of tasks (e.g., NLP beyond GPT-2, or tasks with sequential or graph-based data).\n\nImpact of Input Norms: The introduction of direct dependence on input norms enhances expressiveness but may lead to inefficiencies in certain applications. Further discussion on the trade-offs between performance and computational cost when using this feature could help clarify its limitations.\n\nEffect of Hyperparameters: While the paper evaluates the performance of EUGens with various hyperparameters (e.g., number of random features, polynomial order), a more detailed analysis of the sensitivity to these hyperparameters would provide deeper insights into the method's robustness.\n\nComputational Complexity of Large Models: Although the method improves inference speed, the evaluation does not fully explore the scaling of EUGens with respect to very large models (e.g., extremely deep Transformers or large-scale NeRF setups). The impact of increasing model size or network depth on the gains provided by EUGens remains unclear.", "was_revised": false, "final_justification": "", "rebuttal": "We are happy to hear that the Reviewer thinks that the paper provides a thorough theoretical analysis and extensive experiments complementing it.\n\n> Task diversity:\n\nWe would like to sincerely thank the Reviewer for the thoughtful comment. As the Reviewer notes, our paper already targets a broad range of applications, including image classification, language models, and 3D scene reconstruction. In particular, we have conducted extensive experiments on image classification (ImageNet, Places365, CIFAR-10, CIFAR-100, DTD, PETS), NLP tasks (language modeling on 36.8 billion tokens as well as text classification using the GLUE benchmark), and 3D reconstruction (including unbounded and dynamic scenes as well as depth estimation). We believe these experiments span a wide breadth of topics relevant to the machine learning community. Furthermore, we have demonstrated that our method is truly general, as it can be integrated into various architectures (GPT, BERT, ViT, NeRF, Siren, EfficientViT), including large-scale models such as ViT-L. Taken together, we feel this already provides a solid diversity of tasks and architectures to support our claims.\n\n> Impact of input norms / Trade-off Between Expressiveness and Efficiency: \n\nThank you very much for an excellent question. As shown in Eq. 1, the norm operator is applied independently on the weight-vectors and inputs. Since those weight-vectors and input-vectors are processed independently via nonlinear transformations, only to be fused in the final dot-product computations of the EUGen layers, the computational complexity of the variants with norms turned on is the same as for the variants that do not apply norms, up to the negligible time-complexity norm computation. That computation is indeed negligible as compared to the overall computations since it requires only one dot-product calculation (per weight-vector or input-vector; namely the dot-product of the vector with itself), whereas projections with matrices G involve several dot-product calculations.  \nWe will clarify it in the final version of the paper.\n\n> Effect of hyperparameters:\n\nThank you very much for a great comment. In the paper,  we have already conducted detailed ablations on the number of random features (m). In l. 239, we discuss the effect of the number of random features on the quality and the computational footprint of the model, with results provided in Fig. 9 (right). This is for the 3D reconstruction task. We have also conducted those ablations for ViTs (see: Fig. 4). For additional results, please see: section E2 for other 3D reconstruction tasks (Fig. 26, 27, 30 and Table 7, 8) and Fig 28 for synthetic tasks. Our results show that the number of random features provides a classic trade-off between a model's computational footprint and quality. \n\nFollowing Reviewer’s suggestion, for the rebuttal we have also conducted additional experiments, increasing the degree of the polynomial to k = 5. The experiments were run for GLUE language tasks (eight datasets) and CIFAR-100 from the vision-suite of tasks.  The results are presented below :\n\nGLUE : \n\n|Dataset\t|baseline\t|EUGen, k<5 | EUGen, k=5|\n|----------------|-----------|------|----------| \n|RTE\t|\t57.5\t|61.7|\t62.1 |\n|MRPC |\t81.5|\t83.8|\t84.5 |\n|COLA\t |\t29.4|\t45.2|\t44.4 |\n|SST-2\t|\t84.9|\t86.8|\t87.4 |\n|STSB\t|\t78.1|\t82.8|\t82.9 |\n|QQP |\t\t72|\t72.8|\t73.4 |\n|MNLI\t|\t56.4|\t57.4|\t58.1 |\n|QNLI\t|\t74.5\t|75.5|\t75.6 |\n\nAverage score for the deep NN using lower-degree EUGen layers is 70.75 and that of its counterpart applying polynomials of degree k=5 is 71.05. The latter one is 3% slower in training and 2% slower in inference. We thus see here a classic speed-quality tradeoff, as well as improved quality for higher-degree polynomials, as expected.\n\n\nCiFAR - 100 : \n\n|EuGen with k<4 | EuGen with k=4 |\n|---------------------------|--------------------------|\n|89.53 | 89.81 | \n\n\nFor CIFAR-100 the conclusions are similar to those for GLUE tasks: the higher-degree polynomial variant achieves better quality, at the price of the small speed loss (5% slower in training and 3% slower in inference).\n\n\n> EUGens in deeper models:\n\nWe would like to thank the Reviewer for an excellent question. Following Reviewer’s comment, we conducted additional experiments by training the **24-layer ViT-L model**. By replacing up to **21 layers** with our proposed EUGen modules and training from scratch, we achieve performance at par with the ViT-L model across both the ImageNet and Places365 benchmarks. These lead to a **50.67% reduction in inference parameters**."}, {"review_id": "SkCKOu5ZRC", "reviewer": "Reviewer_NaGy", "summary": "This paper presents EUGens, a new class of dense layers designed to approximate standard feedforward layers (FFLs) using random features. The approach is backed by theoretical guarantees showing unbiased approximation for polynomial activations, and the layers can be plugged into existing architectures like Transformers and NeRFs to reduce computational cost. The authors also propose a lightweight distillation method that allows swapping in EUGens without needing full backpropagation, which is a practical feature for real-world deployment.", "questions": "See Weakness Section.", "limitations": "NA", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The theoretical contributions are solid and well-presented. EUGens are shown to be unbiased approximators of FFLs with polynomial activations, supported by detailed variance and concentration bounds.\n- The method is architecture-agnostic and works in diverse domains—from NLP and vision to 3D reconstruction—showing versatility.\n- The empirical results demonstrate improvements in inference speed and memory usage, often without substantial quality degradation.\n- The distillation technique that allows layer-wise replacement without backprop is clever and could be very useful for adapting pretrained models efficiently.\n\nWeaknesses\n- One concern I have is that the experimental results don't clearly demonstrate EUGens as a superior alternative. Figures 3, 4, and 5 all suggest that EUGens offer a trade-off between model size and performance. But it’s hard to judge how favorable this trade-off really is, since reducing the number of inference parameters generally comes at the cost of accuracy. To show that EUGens are truly competitive replacements for existing FFLs, I think it would help to include an experiment where the EUGen model is scaled up to match the inference parameter count of the baseline—so that we can directly compare performance under equal capacity.\n\n- Another point that left me puzzled is the use of 2, 4, 6, and 11 EUGen layers in the experiments. This doesn’t feel like a clean or scalable setup. In particular, Figure 4 shows that the 11-layer configuration performs significantly worse, which raises the question: is EUGen's benefit limited in deeper models? If 11 layers already hurt performance—and they probably still make up a small part of the network—it might mean that EUGens don’t scale that well in depth. I’d appreciate more clarity on why these particular layer counts were chosen, and whether EUGens are actually viable when used more extensively in deeper architectures.\n\n- Additionally, while the paper focuses a lot on inference efficiency, it doesn't say much about training cost—especially when the Gi_j matrices are learned. If EUGens are supposed to be efficient alternatives, the cost of training them should be addressed too.\n\n- Finally, the distillation method assumes you can store and access the input-output pairs of the layers you want to replace, which might not always be feasible for large models or datasets.", "was_revised": true, "final_justification": "Based on the points raised in my comments on the rebuttal, I have decided to lower my score from 4 to 3. While I acknowledge that the paper is grounded in strong theoretical foundations — which is certainly a strength — I believe the experimental validation requires further work to fully substantiate the claims.", "rebuttal": "We are happy to hear that the Reviewer thinks that theoretical contributions of the paper are solid and well-presented and that the method works in diverse domains - from NLP and vision to 3D reconstruction, showing versatility. \n\n> Trade-off between model size and performance for EUGen (an experiment where the EUGen model is scaled up to match the inference parameter count of the baseline): \n\nAs requested by the Reviewer, we have added an experiment where we matched exactly the number of the parameters of the baseline, while training the variant applying EUGen layers. The task was NerF model training. The NerF variant leveraging EUGen layers turned out to be **more accurate** than the baseline in terms of PSNR (higher is better), and virtually indistinguishable on others (SSIM (higher is better) and LPIPS (lower is better)). We provide details below.\n\n|                                 | PSNR  | SSIM  | LPIPS |\n|--------------------|-------|-------|-------|\n| baseline                        | 30.45 | 0.950  | 0.032 |\n| Eugen (same param number) | **30.54** | 0.950 | **0.030** |\n\n\n\n> EUGen scaling with depth (how scalable EuGen’s setup is):\n\nThank you very much for the question. We apologize for the confusion. Fig 17 and18 in the Appendix present the expanded versions of the results from Fig 3. These experiments were conducted using a ViT-B model, which has a total of 12 layers. Our experiments show that we can replace up to 11 layers by EuGens with minimal drop in performance and have a **40%** parameter reduction. The FFLs in Transformers contain most of the training parameters. The increase in loss is expected as an increased number of EUGen layers corresponds to more approximation error in the network. We will clarify it in the final version of the paper.\n\nFollowing Reviewer’s comment, we conducted additional experiments by training the 24-layer ViT-L model. By replacing up to 21 layers with our proposed EUGen modules and training from scratch, we achieved performance at par with the ViT-L model across both the ImageNet and Places365 benchmarks. These led to a 50.67% reduction in inference parameters without sacrificing accuracy. This shows that our method can also be successfully applied to deeper models.\n\n\n> Cost of training:\n\nThe projection matrices $G_{ij}$ are very small (to be more specific, for all fine-tuning experiments with ViT and BERT, we use 32 projections and  for most 3D reconstruction tasks  we apply 64 projections). Finally,  they incur only negligible extra cost in training. We will clarify it in the final version of the paper. Please see: Fig. 7 and Table 4 for training times of the iSDF models. We also demonstrated that iSDF training time can be improved by 5% with EUGens. This clearly shows that matrices $G_{ij}$ do not incur any significant extra cost in training. Table 9 in the Appendix further shows the training time as a function of the number of random features m. Models with smaller m are characterized by shorter training time. \n\n> Distillation:\n\nIn practice, performing distillation using EUGens is not expensive because of the following reasons:\n\nWe can perform forward passes using the teacher model during training and access the input activations in a batch-wise fashion. This strategy prevents storing large amounts of data in an offline setup.\nFor specific models like Zip-NeRF, even the above strategy can be expensive due to the large number of samples required per image. To mitigate this, we adopt several strategies: (1) we sample only 1/16 of the training images, (2) we render them at 1/8 of the original resolution, and (3) we randomly sub-sample 10% of the resulting input-output pairs. These choices significantly reduce the memory footprint and allow the distillation process to run efficiently within our hardware constraints."}, {"review_id": "qBrAV1LqUf", "reviewer": "Reviewer_QK92", "summary": "The authors propose a novel approximation to the dense layer that has improved compute complexity in terms of layer size. Extensive experiments show end-to-end speed up over fully connected layers.", "questions": "For the speed-up experiment: How are the standard linear and EUGen layer implemented?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n* Extensive experiments.\n\n* Practically relevant problem.\n\n* Easy to interpret trade-off plots (e.g. Fig. 3b)\n\n\n## Weaknesses:\n* No code is given, making verification of the author's claims unreasonably time-consuming.  \n\n* Line 87 states that the key distinguishing feature of EUGens over SNNK is that \"[SNNK] cannot be applied to unbounded activation functions (e.g. ReLU)\", but Fig. 2 shows results for SNNK on a ReLU network --> can the authors please explain this contradiction?\n\n* The claim  that model size can be reduced seems to reference line 210, and Tab. 7 -- Here we see that the 30% reduction in parameters causes >0.5dB degradation in output quality. This is far from equal qualtiy. I suggest the authors relativize this claim and encourage them to remove it from the abstract without more detailed explanation.\n\n* Despite the misleading claims in the abstract, this method seems to reduce compute only (not memory use; or if so, this has not been convincingly proven). In this context it would be appropriate to explain that commonly e.g. LLMs are memory rather than compute bound, limiting the utility of this approach.", "was_revised": true, "final_justification": "The most relevant of my concerns were adequately addressed in the rebuttal, so I'm raising my score. Yes, this paper does not fully detail whether EUGENS are helpful for all kinds of neural networks, but I am sufficiently convinced they can be beneficial for some of them to recommend acceptance.", "rebuttal": "We are happy to hear that the Reviewer thinks that our paper presents extensive experiments, showcasing the capabilities of our method.\n\n> Code\n\nWe have provided the pointers to anonymous repositories at the bottom of Page 24 (Appendix).\n\n> SNNK for ReLU: \n\nThank you very much for this question. SNNKs indeed cannot unbiasedly approximate FFLs with unbounded activations, which is not the case for EuGens, with instantiations providing unbiased activations of the FFLs with arbitrary polynomial activations (via random feature mechanisms), as stated in Theorem 3.1. Thus EuGeNs can accurately and unbiasedly approximate polynomial-activation FFLs without any learning, simply by sampling matrices $G$ form the pre-defined distributions defined in Theorem 3.1 (explicit approximation). To summarize, when SNNKs replace FFLs with unbounded activations, explicitly they provide a replacement unrelated to ReLU-activations. During the process of training, SNNKs are capable of approximating layers with ReLU-activations (implicit approximation), however this approximation still will not be unbiased. We agree with the Reviewer that this requires clarification, and thus we will provide the above explanation in the final version of the paper.\n\n> Reduction in model size and impact in quality, memory footprint:\n\nWe would like to thank the Reviewer for the question regarding memory footprint. As the Reviewer suggests, we will provide detailed quality vs memory savings trade-off early on in the paper and update the abstract accordingly. Thank you very much for the comment ! \n\nOur EUGen-models are characterized by the smaller number of parameters than their regular counterparts and thus indeed do have a lower memory footprint. For example, by replacing top 6 FFLs with EUGen layers, we managed to drastically reduce the size of the ViT model from **346 Mb to 196.85 Mb**. Furthermore, we show memory footprint improvements of the NeRF models applying EUGens in Table 7. We see that models with a smaller number of random features have a smaller corresponding memory footprint. Additional experiments that we have conducted for the rebuttal on GPT2 show memory footprint compression from 1.4 GB to 1.2GB, via EUGens. \nOf course we are subject to the classic quality-compute tradeoff here, but since EUGen layers can be applied surgically to replace selected layers in regular models (all experiments we have conducted in the paper are for **hybrid architectures** with a combination of regular FFLs and EUGens), we can customize the extent of memory footprint reduction at the cost of some quality loss to the particular application. \n\nWe conducted additional experiments by training the **24-layer ViT-L** model. By replacing up to 21 layers with our proposed EUGen modules and training from scratch, we achieve performance similar to ViT-L model across both the ImageNet and Places365 benchmarks. These lead to a **50.67%** reduction in inference parameters. The memory required also reduces from 1.2GB to 0.6GB.\n\n\nWe will clarify it in the final version of the paper.\n\n> Implementation of standard linear and EuGen layers\n\nThank you very much for the comment. The linear layer is implemented via nn.Linear and the EuGen layer is implemented, as described in Fig 1. In a nutshell, we train two small networks: one to transform the weights and the other to transform the inputs into a common latent space where the weights and inputs are fused via standard matrix multiplications. Please see the code for more details. We will clarify it in the final version of the paper."}, {"review_id": "3Gt47nNugR", "reviewer": "Reviewer_4fkV", "summary": "The authors propose a new type of neural network layer called \"EUGene\", using random feature methods that speed up the inference time and reduce the training parameters of the network. The authors also provide a theoretical analysis of EUGen's estimation with the concentration bounds. The authors validate their approach by demonstrating how to integrate EUGen layers in popular architectures like tranformers and Neural Radiance Fields with experiments on LLM pre-training, vision transformers, and 3D image reconstruction.", "questions": "1) Figure 1: Why does the loss increase with more EUGene layers? Is this for the same reason as outlined in the experiment 2? If so, I would encourage the authors to address this in the earlier experiment than the one that comes later. \n2) Figure 1: How does EUGene with more layers has fewer inference parameters (lower fraction of inference parameters)? \n3) What are the important future work directions in the authors' views? \n4) Why is theorem 3.1 relevant for neural network approximations? Polynomial activation functions are always explicitly excluded in all universal approximation theorems, because they lead to polynomial approximations - and these are extremely ill conditioned, i.e. do not really model classical neural network approximations. Also, as (l117) \"in practice we use k <= 3\", and l164 \"k <= 2\", so the polynomial approximation certainly is not \"arbitrarily accurate\" (for which k has to approach infinity). All theorems seem to be based on the polynomial activation assumption, which makes it very important to clarify this point. I may have missed an important argument for this assumption and am willing to raise the score if it can be explained properly. Simply stating that \"Bernstein polynomials can approximate arbitrary continuous functions\" is not enough, since k is very small.\n5) l105: m is not defined, is it equal to l? Probably not, since l117 assumes m << min(l,d).", "limitations": "I would encourage the authors to make a separate section on limitations and future work and elaborate on the important points. This will help the reader. The limitations are mentioned, but are hidden in the details of the experiments.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n * S1) Zero-shot integration is demonstrated on many important and commonly used architectures such as transformers and neural radiance fields, showing that the inference speed decreases without retraining with the classical back-propagation algorithm. \n * S2) Theoretical analysis including EUGene’s approximation results and concentration bounds is presented. \n * S3) Many architectures use feed-forward neural networks as building blocks and thus, this work can potentially impact many applications. \n * S4) Writing is clear and easy to follow.\n\nWeaknesses:\n * W1) Important future work directions and limitations are not adequately detailed. \n * W2) (small weakness): notation in line 103 is confusing (w+, x+), because \"+\" is usually used to denote \"pseudo inverse\". Maybe \"w_1\" and \"x_1\" can be used to denote appending ones.\n * W3) l105: \"the vectors\" G are not defined. What are they? Trainable parameters, derived from W? Not clear at this point.\n * W4) The theoretical analysis is based entirely on the assumption that the modelled network layer has a polynomial activation function. this is excluded explicitly in universal approximation theorems (see questions).", "was_revised": true, "final_justification": "The paper presents impressive inference time speedups in numerical experiments, for a large range of network architectures. Although the theoretical setting in the paper misses the key points of the numerical algorithm, it still warrants publication in my view. If accepted, the authors should clearly indicate this mismatch between numerical experiments and theory in the main paper.", "rebuttal": "We would like to sincerely thank the Reviewer for valuable comments and feedback. We address all the comments below. \n\n\n> Questions about Figure 1: \n\nWe believe the Reviewer is referring to Fig. 3 and 4. In our experiments, we keep the total number of layers in GPT-2 fixed (12 layers). The stated numbers L of layers refer to the numbers of layers replaced by EUGen layers. The increase in loss is expected as an increased number of EUGen layers corresponds to more approximation error in the network (we provided detailed results in the Appendix Fig. 17 & 18).\n\n> How does EUGene with more layers has fewer inference parameters?\n\nFor EuGen the number of inference parameters is smaller and below we provide a brief explanation. A EuGen layer is of the form $f(W)g(x)$ and thus can be combined with a subsequent linear layer. Please see Fig 11 and the exact derivations in App. B1. In Transformers, an MLP block applies an up-projection using a weight matrix of shape $[4d, d]$, with a GeLU non-linearity followed by a down-projection via a matrix of shape $[d, 4d]$. Note that when we apply EuGens, we replace the up-projection + GeLU with an EuGen layer which has a shape $[4d, m]$, where m is the number of random features. This weight matrix of the EuGen layer can be combined with the down-projection matrix to create a single weight matrix $[d, m]$ and thus the number of the parameters in our compressed models scales as $O(d)$ as opposed to $O(d^2)$ of the baseline. This is also explained in l. 870-877 (App. D3).\n\nWe will include all these clarifications in the paper.\n\n> vectors G and m: \n\nThe number of random features is denoted by $m$. The $\\mathbf{G}$’s are projection matrices with entries drawn randomly from a scalar Gaussian distribution, encoding corresponding $m$ Gaussian projections. We will make it clear in the updated draft.\n\n> notation in line 103 is confusing (w+, x+)\n\nThanks for pointing this out. We will update the notation in the final version.\n\n> Future work directions and limitations:\n\nWe believe we can improve EUGens via the following future research directions: \n\n* Currently EUGen provides unbiased estimation of the FFLs using polynomial activations. Extending this unbiasedness to arbitrary activation functions is an open problem (we already explicitly mention that limitation in Appendix F).\n\n* It would be interesting to see whether it is possible to apply polynomial function approximations using random features in both attention and MLP modules to maximize computational gains.\n\n* Finally, in the future work we also would like to explore efficient training strategies (and provide corresponding theoretical analysis) for models with all regular layers replaced with EUGen layers.   \n\nWe would like to note that such models are not polynomials neural networks in general, since in the most general definition of EuGens (Eq. 1), maps $\\Phi$ and $\\Psi$ explicitly depend on the norms of the input-vectors and weight-vectors respectively. Those models are more expressive than approximators of the regular FFLs with polynomial activations. We already successfully applied such general EUGen layers in this paper (look: comment in l.47-48, Sec. 4.4 and l.207 there).  \n\nAppendix F currently includes our Broader Impact and Limitations section; we will expand this section to address point 2 and 3 (we already discuss point 1 in Appendix F).\n\n\n> Relevance of Theorem 3.1 for neural network approximations:\n\nWe would like to sincerely thank the Reviewer for the question. \n\n* While pure polynomial-activation neural networks (PNNs) may have limited expressiveness unless the degree $k$ is very large, those results do not directly apply when polynomial-activation layers are mixed with non-polynomial activation layers. Our work uses hybrid architectures, which is a mixture of regular feedforward and EUGen layers. Our experiments show that our hybrid models, with a small degree $k$, achieve performance comparable to their standard counterparts while offering significant computational gains (see empirical evidence in Sec. 4.2 and 4.3). The theoretical analysis of the standard polynomial-activation NNs (PNNs) no longer holds. Thus our approximation theorems should be read in the context of the hybrid setting, where the corresponding layers are used, rather than pure PNNs.\n\n* Pure PNNs, despite their more limited expressiveness, are successfully used in several applications these days, e.g. (1) to replace deep convolution layers ([1]), (2) in dynamical system simulation, where their properties are particularly relevant for modeling ODE solutions and where they provide better accuracy and computational footprint that traditional numerical solvers ([2]), (3) in regression and classification tasks, where PNNs with additional tricks (such as the so-called product activations) turn out to be very strong learning models and are very relevant for Bayesian learning, as the authors explicitly admit, due to the corresponding closed-form calculations ([3]). Finally, in the recent work ([4]), lower bounds on the polynomial degree beyond which PNNs achieve maximum expressiveness are given via a rigorous theoretical analysis. Those theoretical results provide yet another evidence of the relevance of PNNs, even beyond hybrid architectures.\n\n* The above discussion regarding hybrid architectures and customized training procedures is analogous to the discussion on polynomial vs exponential functions in the attention setting. In that context, polynomial kernels (especially low-degree) were originally considered as not applicable due to their limited expressiveness (e.g. polynomial-capacity associated memory, instead of the exponential, as it is the case for the standard softmax-attention kernel). However low-degree polynomial kernels are nowadays used on the regular basis in attention modules of Transformers (see for instance: [5], where degree k=2 is applied or [6] where low-degree polynomials are used), providing comparable quality and significant reduction of the memory footprint. This is exactly due to new training procedures and hybrid strategies (e.g. up-training mechanisms, where regular Transformer is fine-tuned with some of the regular attention layers replaced by the polynomial ones). \n\n* Studying theoretical properties of PNNs is very important in the context of understanding regular deep NNs, as experts in the field directly admit. Because of that and the fact that their theoretical analysis is more tractable, several top-tier papers focus solely on analyzing their theoretical properties. A flagship paper is a manuscript of Kileel, Trager and Bruna ([7]), where the authors say: “...we believe polynomial networks can help us access a better understanding of deep nonlinear architectures, for which a precise theoretical analysis has been extremely difficult to obtain. Furthermore, polynomials can be used to approximate any continuous activation function over any compact support (Stone-Weierstrass theorem). For these reasons, developing a theory of deep polynomial networks is likely to pay dividends in building understanding of general neural networks…”. Our paper goes beyond theoretical analysis, by balancing theoretical results with strong empirical evidence (as the Reviewer mentions: “Zero-shot integration is demonstrated on many important and commonly used architectures such as transformers and neural radiance fields”). Because of that and the above comment, we do believe that the results on approximating polynomial-activation FFLs with random features (Theorems 3.1 - 3.4) are relevant and important, even outside of the scope of hybrid models.\n\nWe completely agree with the Reviewer that that should be clarified and thus will provide the above clarification in the final version of the paper.\n\n\n[1] P-nets: Deep Polynomial Neural Networks; Grigorios G. Chrysos et al.  CVPR 2020\n\n[2] Polynomial Neural Networks and Taylor Maps for Dynamical Systems Simulation and Learning; Andrei Ivanov et al. 2019  \n\n[3] Ladder Polynomial Neural Networks; Li-Ping Liu et al.  2021\n\n[4] Activation thresholds and expressiveness of polynomial neural networks; Bella Finkel et al., 2024\n\n[5] Simple linear attention language models balance the recall-throughput tradeoff; Simran Arora et al. ICML 2024\n\n[6]  PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels; Praneeth Kacham et al. ICML 2024\n\n[7] On the Expressive Power of Deep Polynomial Neural Networks; Joe Kileel et al. NeurIPS 2019\n\n\n\n> Higher-degree polynomials for EUGens:\n\nTo ground the discussion on higher-degree polynomials in EUGens, for the rebuttal we have conducted additional experiments, increasing the degree of the polynomial  up to k=5. The experiments were run for GLUE language tasks (8 datasets) and CIFAR-100 from the vision-suite of tasks.  The averaged results are presented below :\n\nGLUE : \n\n|baseline\t|EUGen k<5 | EUGen, k=5|\n|-|-|-| \n|66.79\t|70.75|\t71.05 |\n\nPlease find the full results in Reviewer DUMo’s response. We observe an improved quality for higher-degree polynomials using EUGens.\n\n\nCiFAR - 100 : \n\n|EuGen w/  k<4 | EuGen w/  k=4 |\n|-|-|\n|89.53 | 89.81 | \n\n\nFor CIFAR-100 the conclusions are similar to those for GLUE tasks: the higher-degree polynomial variant achieves better quality, at the price of the small speed loss (5% slower in training and 3% slower in inference).\n\nTo summarize, increasing degree helps slightly, but given the overall quality-speed tradeoff, we found the approach taken in the paper more optimal."}]

Author rebuttal:

Meta-review: The reviewers acknowledged the relevance and novelty of the proposed method, its theoretical justification, and its effectiveness in terms of speedup and memory requirements. Nevertheless, they raised some concerns about the limited diversity of the tasks explored, the tradeoff between model size and accuracy, and the influence of some hyperparameters. Generally speaking, the authors' feedback convincingly addressed the reviewers concerns, with the exception of NaGy, who downgraded their score to a Borderline Reject. While Reviewer NaGy acknowledges the methodological contribution of the work, they still have some concerns about the empirical evaluation. However, the AC agrees with the other reviewers that the current set of experiments, complemented by those provided in the rebuttal, is sufficient to warrant acceptance. The authors are nonetheless strongly encourages to incorporate their feedback in the final version of the paper.
Decision: Accept (poster)