Scalable Neural Network Kernels
Authors: Arijit Sehanobish
Krzysztof Marcin Choromanski
YUNFAN ZHAO
Kumar Avinava Dubey
Valerii Likhosherstov
Venue: ICLR 2024 poster (2024)
Keywords: scalable kernel methods
random features
deep neural networks
Primary area: metric learning, kernel learning, and sparse coding
TL;DR: We provide novel kernel methods that can be used to linearize MLP blocks leading to parameter efficient training and reduction in storage complexity.

Abstract: We introduce the concept of scalable neural network kernels (SNNKs), the replacements of regular feedforward layers (FFLs), capable of approximating the latter, but with favorable computational properties. SNNKs effectively disentangle the inputs from the parameters of the neural network in the FFL, only to connect them in the final computation via the dot-product kernel. 
They are also strictly more expressive, as allowing to model complicated relationships beyond the functions of the dot-products of parameter-input vectors. We also introduce the neural network bundling process that applies SNNKs to compactify deep neural network architectures, resulting in additional compression gains. In its extreme version, it leads to the fully bundled network whose optimal parameters can be expressed via explicit formulae for several loss functions (e.g. mean squared error), opening a possibility to bypass backpropagation. As a by-product of our analysis, we introduce the mechanism of the universal random features (or URFs), applied to instantiate several SNNK variants, and interesting on its own in the context of scalable kernel methods. We provide rigorous theoretical analysis of all these concepts as well as an extensive empirical evaluation, ranging from point-wise kernel estimation to Transformers' fine-tuning with novel adapter layers inspired by SNNKs. Our mechanism provides up to 5x reduction in the number of trainable parameters, while maintaining competitive accuracy.

Published as a conference paper at ICLR 2024
SCALABLE NEURAL NETWORK KERNELS
Arijit Sehanobish" Independent Researcher
Krzysztof Choromanski Google DeepMind & Columbia University
Yunfan Zhao Harvard University
Avinava Dubey"
Valerii Likhosherstoy Waymo
Google Research
ABSTRACT
We introduce the concept of scalable neural network kernels (SNNKs), the re- placements of regular feedforward layers (FFLs), capable of approximating the latter, but with favorable computational properties. SNNKs effectively disentangle the inputs from the parameters of the neural network in the FFL, only to connect them in the final computation via the dot dot-product kernel. They also strictly more expressive, as allowing to model complicated relationships beyond the func- tions of the dot-products of parameter-input vectors. We also introduce the neural network bundling process that applies SÑNKs to compactify deep neural network architectures, resulting in additional compression gains. In its extreme version, it leads to the fully bundled network whose optimal parameters can be expressed via explicit formulae for several loss functions (e.g. mean squared error), open- ing a possibility to bypass backpropagation. As by-product of our analysis, we introduce the mechanism of the universal random features (or URFs), applied instantiate several SNNK variants, and interesting on its own in the context of scalable kernel methods. We provide rigorous theoretical analysis of all these concepts well as extensive empirical evaluation, ranging from point-wise kernel estimation to Transformers' fine-tuning with novel adapter layers inspired by SNNKs. Our mechanism provides up 5x reduction in the number of trainable parameters, while maintaining competitive accuracy.
INTRODUCTION
Consider kernel function: K Rd IRd -+ R. taking as input two feature vectors encoding latent embeddings of their corresponding objects and returning their similarity. Kernel methods are among most theoretically principled approaches to statistical machine learning (ML) and have proven effective in numerous real-world problems (Schölkopf & Smola 2002 Kontorovich et al.) 2008). Despite their theoretical guarantees and applicability in a rich spectrum of ML settings, the main drawback of these techniques is high computational complexity, at least quadratic in the size N of the training dataset. For example, the kernel regression has time complexity ((N3).
To address this issue, Rahimi & Recht (2007) proposed to construct random feature (RF) map Rdd + R.m that transforms an input point z to a finite-dimensional feature vector (zz) E R such that: y(x,y) proximating general kernels K(x,y) via linear (dot-product) kernels K(x,y) y) A X y for 2 = (z) drastically changes computational complexity landscape, which is now dominated by the number m of random features, thus providing computational gains if m < N. Since their seminal work, there has been variety of works proposing random features for broad range of kernels like Gaussian, Matern (Choromanski et al. 2018) and polynomial (Kar & Karnick] 2012 Wacker et al.) 2022a b).
E (x)) (y)) (effectively approximately linearizing kernel function). Ap-
In the meantime, with the advances in: the optimization algorithms for deep ML architectures and the accelerators' hardware, neural network (NN) models (Goodfellow et al. 2016; [Schmidhuber) 2014 LeCun et al. 2015) have become predominant in machine learning.
"Equal Contribution

---

Published as a conference paper at ICLR 2024
The feedforward layer (FFL) is the core computational module of NNs and is of the following form:
Guusia
(b)
f(Wx + b))
for x R2 W E [R(xd b R (bias) and an activation function f: R + R (applied element-wise). The expressiveness of deep NNs, far surpassing standard kernel methods, comes from stacking together sev- eral FFLs, each encoding non-linear mapping with learnable W, b.
(c)
Figure 1: Pictorial representation of different NN layers discussed paper. Pink arrays represent NN weight matrices and grey ones, Gaussian projections matrices applied in SNNKs. Nonlin- ear transformations applied in mappings & and y are symbolically represented as functions y and A respectively. Upper left: Regular FFL with activation f. Upper right: SNNK applied to a single FPI.. Bottom: Bundling process using SNNKs and applied to a deep neural network.
In this work, we draw a deep connec- tion between scalable kernel methods and neural networks. We reinterpret the FFL. outputting the expected vector of dot-products of: (1) the la- tent embeddings of the input x and (2) the parameters: W, b of the FFL, effectively disentangling input from model's parameters in the computational graph, only connect them in the final computation via the dot-product kernel. To be more specific, we think about the FFL as the following transformation:
Ky(x, (W, b)) def (K5(x, (w", bo)). Ky(x, (w' bb-1))))
(wb
[y(x, (w, def (x) " Vy(w,b),
f(wTx+b where mappings: f R² RT" V IR x + R" satisfy: f(w x+6) = , (v) (w,, 6)] and w ...w'=1 are the transposed rows of W. Then, in the instantiation of the layer the expecta- tions are dropped out. Rewriting an FFL in terms of two towers: one corresponding the input and one to its learnable parameters has several advantages:
network compression: in the above formulation, instead of transforming layer parameters with V y, one can directly learn vectors V y b;) for i O,..., 2 - 1. Then the number of trainable parameters becomes (Im) rather than (ld) and for m < d the layer effectively has a reduced number of parameters.
2. computational savings: if RFs can be constructed in time o(dl) per point and m < d, the overall time complexity o(dl) of the FFL (given pre-computed embeddings y ((w(,,;)) )) sub- quadratic in layers' dimensionalities,
3. deep NN bundling process: two-tower representation can be used iteratively to compactify multiple FFLs of NNs, the process we refer to as neural network bundling (Sec. 3.3): this also leads to the computational gains.
4. deep NNs as scalable kernels: the extreme version of the bundling procedure, involving all the layers, provides two-tower factorization of the entire deep NN with several potential practical and theoretical implications (Sec. (.3). In particular, it leads to an explicit formula for the optimal parameters of the fully-bundled network under several loss objectives (e.g. mean squared loss), opening possibility to bypass backpropagation.
order to find mappings: 4 V from Eq. [ we develop a new bounded random feature map mechanism, called universal random features (or URFs) that leads to the unbiased estimation of f(w x x b) as long as ff has a well-defined Fourier Transform (FT), either in the classical Rieman- nian distributional sense. To derive URFs, combine Fourier analysis techniques with recent methods for softmax-kernn
estimation from Likhosherstov al. (002)).
Note: We do not put any additional assumptions regarding f. in particular f is not required to be differentiable. Furthermore, function Ky does not need to be positive semi-definite. This critical for applications in neural networks, where the activation function f usually does not correspond to a positive semi-definite kernel.
To summarize, main contributions in this paper are as follows:
2

---

Published as a conference paper at ICLR 2024
We introduce the scalable neural network kernel module (SNNK) as a replacement of traditional
FFL (Sec. 3) providing the disentanglement of the network's 's input and its parameter-ser before final dot-product computation, given in Eq. (see also: Fig. [
We accompany SNNKs with our universal random features mechanism (URFs) to efficiently: (1) construct mappings , and f from Eq. [ and consequently: (2) implement SNNKs (Sec. 3.1..
We provide explicit formulae for URFs for trigonometric maps. Those produce SNNK-based
replacements of the SIREN networks from Sitzmann et al. (2020).
We propose new NN-layers corresponding to the specific SNNK instantiation, called ReLU- SNNK (Sec. [3.2). that we found particularly effective in downstream applications (see: Sec. 4.3.2) We show that they are related to the class of the arc-cosine kernels (Cho & Saul, 2011). We also demonstrate using them that SNNKs are strictly more expressive than regular FFLs, as allowing to compute the functions of the inputs and parameters that cannot be defined as point-
wise transformed vectors of their dot- dot-products.
We introduce the neural network compactification process, that refer to as neural network bundling, leveraging SNNKs (see: Sec. and Fig. TT..
We provide exhaustive empirical evaluation of SNNKs, from point-wise kernel estimation to the adapter-based Transformers' fine-tuning, providing about 5x reduction of the number of train- able parameters (Sec.
Commonly used methods for compressing neural networks are pruning (Liagg et al. 2021 distil- lation (Gou et al. [202I) and quantization (Gholami et al., (021]. Compressing neural networks via SNNKs is novel and completely orthogonal to these methods and can be combined with such.
2
RELATED WORK
The literature random features is vast, yet most of the works focus on approximating positive definite kernels. The results on dimensionality reduction and the so-called Shinsonn-indenstraasss Transform (or JLT) (Dasgupta & Gupta 2003 Dasgupta et al.. 2010: Ailon & Liberty 2013) for the dot-produee kernel marked the birth of the subject as as an archetype mechanism that Rahimi & Recht (2007) extended from linear to non-linear shift-invariant kernels. A substantial effort was made to further improve the accuracy of RF-methods by entangling projections used to construct RFs (Choromanski et al. 2017 Yu et al. 2016; Choromanski et al. 2018 Rowland et al. 2018).
For certain classes of functions f. RF-mechanisms leading to the linearization of Ky have been already developed. In addition the rich recent literature on the approximation techniques for the softmax-kernel (x.
exp(x" y) (Likhosherstov et al. 2022 2023 Choromanski et al., 2.2T algorithms for analytic f with positive coefficients of their Taylor series expansion were given (Kar& & Karnick [2012). Other RF-methods assume that kernel inputs are taken from the unit-sphere (Scetbon & Harchaoui] 2021; Han et al., 2022). Both assumptions are unrealistic for the neural network applications as far as inputs x are concerned (interestingly, the latter one would be however more justifiable for the parameter-tower as long as bounded- norm weight matrices are considered, e.g. orthogonal neural networks (Helfrich et al.) 2018). We would like to emphasize that our two-tower mechanism, effectively leading to the linearization of the FFLs from Eq. can in principle work with various RF-algorithms, and not only our proposed URFs.
The kernels applied in connection to neural networks have been widely studied (Bengio & Lecun 2007]. Such kernels are generally constructed using dot-products of outputs of the shallow neural networks with various non-linearities like ReLU (Cho & Saul 2009 Bresler & Nagaraj 2020) and tanh (Williams) 1996) or the gradients of the network like the NTK kernel (Jacot et al. 2000).. Most of the work in linearizing NNs via kernels have been done in the case of 2-layer network where
J(x; 0) Σ fx w'). 0 = (a1, ON; w
w E N(d+1)
(3)
It is assumed that w' and if (non-linear activation) are fixed and scalars and a; are trainable. Under various assumptions, one can write compact linearized form of this neural network (Cho & Saul, 2009 2011 Ghorbani et al., 2020). Moreover, in the above setting, Jxx; 0) corresponds to the first-order Taylor expansion of J with respect to the top-layer weights a; which was first explored by (Neal, 1996). Even though our setting is fundamentally different, as our goal is to linearize build on the above intuition create SNNK-layers (see also: discussion in Appendix C). Note that NTK-based analysis, as leveraging Taylor-based linearization of the NN, is valid only for the mature stage of training/fneetunngg when
single layers to disentangle the weights and the inputs,
3

---

Published as a conference paper at ICLR 2024
weights do not change much and thus such linearization is accurate (Malldi al. 2022 SNNKs
do not rely on this assumption. Furthermore, SNNKs can be used also in the context of non-positive definite (Ong et al. 2004) and asymmetric (He et al. 2023) kernels since mappings & and v in principle are different (on expectation they can produce both symmetric and asymmetric functions).
Arc-cosine kernels were studied in the context of deep NNs before (Cho & Saul 2009). However, in (Cho & 2009). the weights are still entangled with the FFL--nppu, as the initial latent repre-
sentations of the inputs (for random parameters) are interpreted as RF's for the arc-cosine kernel.
SCALABLE NEURAL NETWORK KERNELS (SNNKs)
The scalable neural network kernel (SNNK) computational module is defined follows:
5NNK, (W, b)) def (SNNK) (w",bo)), SNNK (x, (w'-1, bb-11)))
(4)
SNNK (w, def (x T 1 (w, b),
for, x E R some mappings: f : Rd - R mh y R x R + Rm and transposed rows of 3.1 functions y V can be constructed in such a
W E RRixd w'.....'--, As we show in Sec. way that the SSNK module approximates particular FFL, i.: SNNK, (x, (W. bl) = f(Wx+b), but mechanisms that do not imitate known FFLs are also of interest (see: Sec. 3.2
Time complexity: If we denote by mm (d) time complexity for constructing an embedding: (x),, then time complexity for constructing SNNK y (x, (W, b)) (given the pre-computed v (w(', b.) for i == O...,, - 1) is: [m) = ml+ mm (d). In Sec. 3.1 we show an algorithms for constructing URFs in time t. (d)- = (md) and thus computational gains are provided as compared to the regular FFL (with time complexity (Id)) as long as m & min d).
FFL compression: As already mentioned in Sec. [] the key observation is that in the setting, where the layer learned (and thus w ...., w'-1 are learnable), mapping y does not even need
to be applied, since vectors w def y b) for j = O,..., (- can be interpreted as unstructured learnable vectors. Thus the number of trainable parameters of the SNNK layer is ((ml), instead of (dl) and consequently, the FFL is effectively compressed if m < d.
3.1 UNIVERSAL RANDOM FEATURES (URFs)
In this section, we show how to construct embeddings (x) and V , b) (additional intuition is provided in Sec. A). We denote by FT, the Fourier Transform of f, where i E C satisfies: 1² = =-1
FT J [()) [(x)exp( (xi(f222
(5)
If the integral does not exist in the classical Riemannian sense, we use its distributional in- terpretation. We rewrite F as: FT, = FT - F + iFTimit iFT) where: FTJO FTJ. .FTmm. ,FTim. : R + Rz0- Without loss of generality, we will assume that all four functions are not identically zero.
Let us denote by Po. P P2, P3 some probabilistic distribution on l. (e.g. Gaussian) and by Po. Pr. Dy : R + R2 their corresponding density functions. Furthermore, denote by
Po. PI,P2, P3 probabilistic distributions of densities: Po.Pi, P2. FT FT FT FIT respectively. We can then write:
R +
Rz0 proportional to:
pr(() (E) (E)
pg(E)
f(z) = FT,(()
Σ 1=0
Σ
cy BEWP
exp (2riE))))
(2(()
(6)
where: co = fa FT. FTT (r)dT, = -TT (7) = i FTj (r) and furthermore
C3 --iff (T)dr.
(T)dr. For IRd, b E R. let denote:
fxx b) CERE
(p((E) (E)(E)
exp (2 Tw + b)) cs CyTRews, [S) b) exp(X (E))(E)) (7)
for S)(5,b) (5)
exp(2righ), x(E) p(E)x, ((E) = n)(Ew, where ((E), C sat-
(,(
isfy: ((E))(E) = =2TiE. Inside the expectation in Eq. 7 we recognize the softmax-kemel value
4

---

Published as a conference paper at ICLR 2024
Kexp (E)), ((()) = (cp(""(5)W/) We thus disentangle x(g) from w there, by applying
softmax- neemml linearization mechanism from Likhosherstov et al. (2022): exp (()
(4g (W)). where Ag: Rd + R is defined as follows for A I 0:
|
Ag(z) = (1 - 4A)* exp( (Allll + vT 4Ag
(8)
Thus [x, (x, ( P. I (x) g.e( (w, b)] for r g.c (x), r (w, given as: (x) Ag(p(()x), (w b) =
(9)
That observation directly leads to the RF mechanism for the estimation of fx w, b). We can rewrite: fx(, w.b) = EB (x) T (w, b)] for (E1,gı), (Em gm) P, @ (V (O. and:
1
1 (T E1 (x) &m (x))), v (w,b) 6) = Vi (1 81 (w, b), 1 gm bj) (10)
d (x) = m
Several strategies can be used to construct samples (f1, (fI, (Ems gm). e.g. iid sampling or block-iid sampling with a fixed & used within a block, but constructed independently for different blocks. In the experiments, we also choose: P(E) 2 and m/{==1
The case of discrete P, with finite number of atoms: Assume that 5 is sequence of atoms with the corresponding positive probabilities: (pI....,kk Pk). Then one can also construct K pairs of RF-vectors (x'(x: kt), (w, b; k)) each obtained by replacing P, with a distribution corresponding to deterministic constant pk and get & (x), y (w, 6) by concatenating vectors from (+) (x; k and (W 6; k) respectively. This strategy is effective if K is small.
5k)
Note that: f(x T w b) = Σ ー 7 w, b) and thus (x()) and , (w, b) can be defined as: + (x) = coneat ((01 (x)))x)) (x)) y b)) concat ((V) (w, =))
(11)
for the vector concatenation operation concat, completing the description of the URF mechanism.
Remark 3.1 (boundedness) We observe that for upper-bounde ||x||2, [| (b), the entries of , and V y( ((w, b) are also upper-bounded long A < 0. This follows directly from the formula for Ag (z) in Eq.
Trigonometric activation functions: Let assume now that f(z) = sin(z) or f(2) cos(2). Note that even though none of them has a Fourier Transform in the classical Riemannian sense, both have trivial Fourier Transforms in the broader distributional sense. To that, we rewrite both (ex(=))eep((iiz and cos(2) cxp(-2)) Therefore the correspond-
activations as: sin(2) ing distributions used in the URF derivations above become binary distributions over {- This observation has interesting practical consequences, since it leads to the conceptually simple linearization of the FFLs applied in SIREN networks (see: Sec.
2n 2x }.
3.2 BEYOND REGULAR FFLS: THE CURIOUS CASE OF THE RELU-SNNK LAYER
We also propose another SNNK layer which is not directly inspired by any known FFL, but turss out to work very well in practice (see: Sec. 4.3.2). In this case, the mappings & and vi are defined (x) ReLU( ( Gx), ((w, b) ReL.U Vi Gw) for the Gaussian matrix: G Riidd with entries sampled independently at random from (0, 1). One can ask a question what kernel does this pair of maps correspond It turns out that the answer particularly elegant.
Theorem 3.2 (arc-cosine kernels; Cho & Saul (201T)) The nth-order arc-cosine kernel Kn Rd x Rd + R defined as: K, (x. y) = x
y y Jn where ax,y E [0, stands for an angle between x and and J(0) def (- (sin(()nn( (0))
9t
x-0 s Then, Kn can be linearized
26
as: (x,)) = TE T (y)] r,, def
ReLU((v w)") and w No).
We conclude that our proposed ReLU-SNNK layer is a scalable version of the FFL defined as: W Rixd b Kg (wl, x)..., 1 K1 (w1 x))
5

---

Published as a conference paper at ICLR 2024
Remark 3.3 The ReLU-SNNK layer is not regular FFL since the values of its output dimen- sions cannot be re-written as f(x w' + b,) for some f R. - finterestingly, after r-base pre- processing. it can be still interpreted dot dot-product kernel). This shows that the SSNK mechanism is capable of modeling relationships beyond those of regular FFLs.
3.3 BUNDLING NEURAL NETWORKS WITH SNNKS
We are ready to propose the neural network bundling process, relying on the SSNK-primitives. Consider the following deep NN module with input x = Xp E Rdd and output y = xx,, E RRd::
x f4i (W.x, + be); O...., 1-
(12)
Xa =
for: (I) matrices W. E 1R4411xd; (2) bias vectors: b, E Rd and (3) activations: fi R + R.
To understand how the bundling process works, we start by replacing first FFL in Eq. 12 with its SNNK analogoue. We obtain the following computational block:
R = f.n (W... . + b.) for = O,..,, 2-
Rn = (
(13)
Wo = Wi (Wo, bo); W; W.+ for = I.....I.. 1-2
f4t fe+z, b. = bit1 for i = 0, 2
In the system of equations above, (W ba) is matrix with transposed rows of the form: V (W:, 0r by), where w for j = O,... d1 - are the transposed rows of W, and bn = (bS. .... bg T. We have thus successfully replaced module of L feedforward layers with module (L - 1) feedforward layers. By continuing this procedure, we can ultimately get rid of all the FFLs and obtain an estimator y of y, given as: y Wx, where
fz (.-0,, 11 (W LWE-IVE / W. (WW, (WI ,,, (Wo, bo), b.)...,, bu)) E Rd..m
(14)
This has several important consequences. In inference, replacing matrices W .... WL-1 with one matrix W is effectively compression scheme (that does not necessarily need be applied to all the layers, but a particular consecutive set of layers of interest). If we apply bundling process to the entire deep neural network, effectively provide its two-tower factorization with input disen- tangled from the parameters. In training, directly learn it (see results in Appendix J.3] table Since the output y is now modeled as an action of the unstructured learnable matrix W on the pre-processed input x, for several loss functions there exists an explicit formula for the optimal W. This is the case in particular for the standard regression loss (see discussion in Appendix[.. J.3). If bundling is applied to particular module, backpropagation J..3).
treat W an unstructured parameter matrix and
through it is not necessary since there exists an explicit formula for the corresponding Jacobian.
EXPERIMENTS
We present an extensive empirical evaluation on SNNK on wide range of experiments. More details on each of the experiments can be found in the Appendix G
4.1 POINTWISE KERNEL ESTIMATION
As warm-up, test the accuracy of the applied RF-mechanisms synthetic data. We take = 2000 and 1. We consider: (a) a SIREN-FFL. with the activation function f(u) = sin(u) and bias 6 0.5, (b) arc-cosine-FFLL from Sec. 3.2 The entries of the weight vectors and the inputs to the layers are taken independently from
Vd Unif(0, 1). We report the mean relative error 500 instantiations of the RF-mechanism) made by the RF-
of the NN output (by averaging based estimator as well as the empirical standard deviation as a function of the number of random projections. This setup corresponds quantifying the accuracy of the kernel estimator pointwise. The results are presented in Fig. (g and h). Our SNNK provided an accurate approximation with a much smaller number of random projections than the dimensionality of the input vectors.
4.2 TOY EXPERIMENTS

---

Published as a conference paper at ICLR 2024
04-
BIREN-aye
SWNK
TTM
M4 PUNN
SWKK
Nan
Narm
120
projections
arccos-laye
SNI
aeseage
random projections
Figure 2: Architecture (a) SNNK layer (see Section C.. SNNK-Adpt layer (c) image fitting (SIREN),
MNIST UCI experiments, (d) SNNK-OPNN model, (e) SNNK-inspired Adapter- layer, (f) SNNK- inspired Adapter-BRRT layer. (g.h): The relative error (obtained by averaging over = 500 instantiations of the RF-mechanism) made by the RF-based estimator the particular entry of the output of the: (g) FIREN-FFI. and (h) arc-cosine-FFLL
function of the number of random projections (see: Sec. 4.1. The maximum p for (g) larger than (h), as (g) in theory produces larger variance per random projection. The corresponding
standard deviations are negligible: (g) 5 10 100
12 5 10 100 10 $_10 12 2.5 10 10 12 10
10 10-12 10 12 10 10 10 12 (h) 10 12 10 1008,3 333 10-8,1000000000 10
1,10 1,10012,1 2.2.2.11 10-9.
SNNKs are versatile and can be used as drop-in replacement for FFLs in a wide variety of NNs like the SIREN network Sitzmann et al. (2020), QPNN inspired Neural Network (PINN) to solve the Hamiltonian for quantum physical systems (Sehanobish et al.) 2021) and simple multi-layer per- ceptron (MLP) for classification on MNIST (LeCun & Cortes) 2010). We use the sine activation variant for the first two experiments and the ReLU variant for MNIST. 32 random fea- tures are used for the solution of the 2-body problem and MNIST and 64 random features for the image fitting problem. We match the performance of the baseline NNs on the 2-body and the image fitting problem (see figure and outperform the bascline MNIST (Figure 9) while incurring lower training costs. For additional details regarding these experiments, see Ap- pendix G.I
Physics-
Figure 3: (1) Left column Injecting SNNK in a PINN network to approximate the potential energy of the 2-body sytem. Top to bottom Ground truth potential, Learned potential by QPNN (Sc- hanobish al. 202T) and QPNN-SNNK. QPNN-SNNK potential function perfectly even using less trainable parame- ters than the baseline QPNN. (2) Rightmost three column Siren network the first row, fitting not only the image, but also the gradients. SNNK on the bottom produces accurate approx- imation the above.
leam
4.3 FINETUNING EXPERIMENTS
In this subsection, we show how SNNKs can be used for parameter efficient finetuning. For NLP experiments, we use the GLUE benchmark consisting of different natural language understanding --0 (Krizhevsky et al.) 2009) tasks (Wang et al. 2018). For vision tasks, we use CiFAR-10, CiFAR-
and ImageNet-IK Deng et al. 2009). BERT-base (Devlin et al.) 2019) is used as backbone for text experiments and ViT (Kolesnikov al. 2021) for image experiments. Our code is built top

---

Published as a conference paper at ICLR 2024
1000
CESPTER
93
ARLPPEC CIER-100
ARUPPER Imagrter
2.3M
10%
60
62.11%
Accurscy
83.21%6
83 13.44%
-2925
IM
56.57%
91.20%
0.3M1
Desedre
5M8
Deotine
58
Beseline
SNNNK
90
Boselre
SNNK
Baseine
SWAK
Figure 4: Comparison of trainable parameters between various layers/modules and the drop in replacement NNK layers. Results for CiFar-10, Cifar- 100 and ImageNet are for SNNK-Adapler models.
of Transformers (Wolf et al. 2020) and adapter Transformer library (Pleiffer et al. 2020). Detailed comparisons with various baselines can be found in Appendix K and additional experiments in Appendix[]
4.3.1 LINEARIZING THE POOLER LAYER IN TRANSFORMERS
For text classification tasks, SNNK layer can be used drop-in replacement for the pooler layer which is a linear layer with a tanh activation. For these set of experiments, the base model is frozen and only the pooler and the classifier weights are tuned. We get computational gains the number of random features employed by SNNK is smaller than that of the hidden size of the Transformers. More details are presented Appendix G.2
On GLUE dev set, our LANN-linearized pooler models outperform the baselines on 5 out of 8 tasks (Table (top half)). Additional results can be found in Appendix |
In this setting, the linearized pooler weights can be merged with the classifier weights to create weight matrix of size (# number of random features number of classes) and then one can simply store the newly merged layer instead of separately storing the trained classifier and pooler layers. This dramatically reduces the storage from 18.92 Megabit to only .02 Megabit leading to a compres- sion factor of 1/1000. More details are presented in Appendix D Ablation studies on the number of random parameters for this experimental setting are presented in Appendix !
Table SNNK experiments on GLUE benchmarks. MCC score is reported for CoLA, Fl score is reported for MRPC and QQP, Spearman correlation is reported for STSB. Accuracy scores are reported for the other tasks. All results are obtained by averaging over 5 seeds.
Datasel
RTF.
MRPC
QNLI
QQP
SST-2
MNLII
STSB
COLA
Bett-uscling [eee ct 2019
17.5
61.5
74.5
72.0
S1.
58.4
78.1
29.4
Coeine ShNNippistr (ober
61
1:15
82.07 + 1.07
735-022 0.2
20.13 0.17
86.21 + 0.31
52 + 0.32
78.93 +
35 0.00
Airgtes-sscling MMosuu 2003
N385
+ 1.4
86.5
1.07
90.63 0.00
88.12
01.74 6.36
K3.68 0.11
58.8 +..
56.1+
Ad Moosavi 2202
64.25
1.72
35,00
1.06
89.0 0.25
88.00 0.16
91.31 0.511
1
18.15
51441
KeLL-SNNKK Maperr (o0n)
124
91,26
0011
85.82
02:31
1.277
8200 0.17
88.81
58.21
4.3.2 SNNK-INSPIRED ADAPTER LAYERS
Adapters in Transformers were first introduced in (Houlsby et al. 2019) and there has been lot of work designing different architectures (Pfeiffer et al.] [020] Karimi Mahabadi et al. 2021 Moosavi et al. 2022) and unifying various paradigms (Moosavi Adapters are bottleneck MIPs which are (generally) added twice to each Transformer layer. In our
al. 2022 He et al.] 2022a
work, we replace each adapter block by a single SNNK layer (Figure 2 (c) and (I)) using only 16 random features resulting in a big drop of training parameters (see Figure 4.. Figure (b) shows the
architecture of SNNK-inspired adapter layers. Additional details presented Appendix D
As is customary for adapter experiments, base model is frozen and only the adapters and classifier are tuned. Table [ (bottom half) shows our results on using SNNK layers in place of adapters on the GLUE dev set. We outperform the bascline on 5 out of 8 datasets while employing only 1/3 of the training parameters.
MNLI, it is noted in (Houlsby et al., 2019), that using smaller adapter size causes worse performance and performance boost can be achieved by increasing the size of the adapter (256 is used in their case). Similar to this observation, we note that we can improve per- formance and match the baselines on large datasets (ex. MNLI, QNLI) as increase the number of random features (see Figure 5). Our method also produces competitive performance on image Detailed com- parisons with SOTA parameter efficient finetuning methods can be found in Table 7 (vision tasks) and in Table & (GLUE tasks).
datasets including Cifar- 10, Cifar- 100 and ImageNet. Figure 4 (right (right figures)).
8

---

Published as a conference paper at ICLR 2024
XC
MPC
CMM
Berckkoler
Figure 5: Ablation with different number of random features for the ReLU-SNNK- adapter experiments the GLUE dev set. AA the reported adaptable adapter numbers in Moosavi et al. 2022).
CONTATIAC
HNSOS 31M
0.02
3.5
NP
0.29
0:2
of Trttrabee Noremaders
Teieebie Perametess
* Temeeble Terameles
Figure 6: Comparison of CE loss for SNNK vs different sizes of MLP on UCI datasets.
Moreover, we note that our methods are completely orthogonal to techniques such as gating mecha- nism in (Mao al. 2022) or algorithms relying on dropping suitable adapter layers (Moosavi al.) 2022
Rücklé et al. 2021 Thus it can be easily combined with them.
4.4
UPTRAINING TRANSFORMERS
In this section, we report results of replacing part of the Feed-Forward Network (FFN) block in Transformers with SNNKs. Details are in the Appendix [.4] for brevity, here we summarize findings. We observe from Table replacing FFL blocks with SNNK layers reduces the number of parameters and FLOPS for both training and inference. This followed by bundling process leads to a large reduction in size and inference time of the bundle. For example, for BERT and ViT models, replacing top-6 Transformer layer's MLP block with SNNK reduces the size of the model from 440 Mb to 226.71, and 346 Mb to 176.42 Mb respectively. Figures [ demonstratee that reducing the model size and inference speed by 40-50% has minimal impact on accuracy for both NLP and Image classification tasks.
reppaain
4.5 EXPERIMENTS ON UCI DATASETS
We have conducted experiments with variety of real-world datasets found in the UCI Machine Learning Repository (UCI MLR) We trained three-layer MLP model as baseline (see Appendix Sec. |ec. for details). We varied the output of the middle-layer to train MLPs with different sizes. For method, replace the middle-layer with SNNK (Figure ( (c)). SNNK matches outperforms the baseline while using only a fraction of the training parameters (Figure 6
5 CONCLUSION
We present scalable neural network kernels (SNNK), novel efficient NN computational model that can be used to replace regular feedforwards layers in MLPs, where inputs and parameters are disentangled and connected only in the final computation via dot-produe kernel. We introduce a general mechanism of the universal random features (URFs) instantiate SNNKs, show that SNNKs are capable of encoding subtle relationships between parameter- and input-vector beyond functions of their dot-products and finally, explain how they lead to the compactification of the NN stack via the so-called bundling process. We complement our theoretical findings with the exhaustive empirical analysis, from pointwise kernel estimation to training Transformers with adapters.
https://archive: iss uci www.mdd/mmm//mm/ htt///idxxx../ html
http://minnnexx.eee

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This paper focuses mostly on the algorithmic properties of the techniques linearizing kemnels as- sociated with feedfoward layers' calculations for the computational gains. The experiments with adapter-ban based fine-tuning of Transfomers are presented to illustrate the main concepts. It should be noted though that Transformers should be used cautiously given their considerable computational footprint (improved though while adapters are applied) and the corresponding carbon footprint.
AUTHOR CONTRIBUTIONS
AS and KC led the project. AS ran several empirical studies on GLUE, CiFar-10 and CiFar-100 datasets and proposed several strategies for efficiently using SNNK-layers within Transformer mod- els. KC proposed FFL linearization-scenne linearization-schee schemes. YZ ran empirical studies on GLUE, CiFar-10 and CiFar- 100 datasets. AD implemented and ran UCI experiments, helped with GLUE/image experiments, proposed strat- egy for efficiently using SNNK-layers and created all figures in experiments. VL proposed an idea to linearize FFLs by disentangling inputs from weights. All authors contributed to the writing manuscript.
URFs, the bundling mechanism and implemented all
REPRODUCIBILITY STATEMENT
Hyperparameters to reproduce each experiment is detailed in section H The code is provided
REFERENCES
Nir Ailon and Edo Liberty. An almost optimal unrestricted fast johnson-llnndenstrauss transform. ACM Trans. Algorithms, 93):21:1-21:12, 2013, doi: 10.1 10.14552483699.24887711 URL httpst 1145/
//doi.
2483701
Yoshua Bengio and Yann Lecun. Scaling learning algorithms towards ai. In L. Bottou, O. Chapelle, D. DeCoste, and J. Weston (eds.), Large-scale kernel machines. MIT Press, 2007.
Guy Bresler and Dheeraj Nagaraj. A corrective view of neural networks: Representation, memoriza- tion and learning. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, 901. PMLR, 09-12 Jul 2020. URL http:::: wwr.mmmlr... wtt.eeeedd...........e.ssss.....
pp. 848-901.
bres 1eallee 1er2eer222. html
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205. 2022.
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Y. Bengio, D. Schuurmans, Lafferty, C. Williams, and A. Culotta (eds.). Advances Neu- ral Information Processing Systems, volume 22. Curran Associates, Inc., 2009. URL
wwwp://s..s.rrri.c..c........ www.erriis.cppppp....eepppp http:////cceeeeeeee///////////// 5 b75iecceba<faabb3332222888822233 Paper.
Youngmin Cho and Lawrence K. Saul. Analysis and extension arc-cosine kernels for large margin 3712
classification. CoRR, abs/1 00bs/11122 12.3771, 2011.
Krzysztof Choromanski. Mark Rowland, Tamás Sarlós, Vikas Sindhwani, Richard E. Turner, and Adrian Weller. The geometry of random features. In Amos J. Storkey and Fernando Pérez- Cruz (eds.), International Conference on Artificial Intelligence and Statistics. AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, pp. 1-9. PMLR, 2018. URL http:::/proceeeenggss .ml...niininmm.................. mlr. press/ v84 cho wowansko 8a httm)
10

---

Published as a conference paper at ICLR 2024
Krzysztof Choromanski, Haoxian Chen, Han Lin, Yuanzhe Ma, Arijit Sehanobish, Deepali Jain, Michael S Ryoo. Jake Varley, Andy Zeng, Valeri Likhosherstov, Dmitry Kalashnikov, Vikas Sindhwani, and Adrian Weller. Hybrid random features, 2022.
Krzysztof Marcin Choromanski, Mark Rowland, and Adrian Weller. The unreasonable effective- ness of structured random orthogonal embeddings. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 218-228, 2017. URL http:: http::::ss/ppp.oeeeedeeooo neurips neurips ps. CC/ 1152226667.6644000f66666111613--
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane. Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with per- formers. In 9th International Conference on Learning Representaatins, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URLltttps://ppenreeee Id=Ua6zukOWRH
net/ net/forunt forum?
Anirban Dasgupta, Ravi Kumar, and Tamás Sarlós. A sparse johnson-lindenstrauss transform. CoRR, abs/1004.4244, 2010. /ttpp:/://arxx
4240
Sanjoy Dasgupta and Anupam Gupta. An elementary proof of theorem of johnson and lin-
denstrauss. Random Struct. Algorithms, 22(1):60-655 65, 2003, doi: 10.1002/ssa.0003. URL
https://doo wrw.orggo...............o....... wttgg//..1000.... 2002/rsa.1007.
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Puttern Recognition, pp.
2009. doi: 10.1 109/CVPR.2009, 0506644..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-11 1423. URL https:
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao. Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. CoRR, abs/2103.13630, 2021.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension, 2020.
Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep Learning. Adaptive com- putation and machine learning. MIT Press, 2016. ISBN 978-0-262-03661-3. URL http:
// //www. www..eeeeeeannngnooo.o org/
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. Int. J. Comput. Vision, 12966):1789-1119, jun 2021. ISSN 0920-5691, doi: 10. sl 1112--021-01145--22 URL URUhttps::/doi http://.....11111111000111/1112/ http://11113-2211----1 021000---211-1-333--111
Insu Han, Amir Zandich, and Haim Avron. Random gegenbauer features for scalable kernel meth- ods, 2022.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor
and Graham Neubig. Towards unified view of parameter-efficien transfer learning. In International Conference on Learning Representations, 2022a. URL httttp:://oppreeettttt
Mingzhen He, Fan He, Lei Shi, Xiaolin Huang, and Johan A. K. Suykens. Learning with asymmetric kernels: Least squares and feature interpretation. IEEE Trans. Pattern Anal. Mach. Intell. 45(8): 10044-10050 2023. doi: 10. 10.1109//AAMM..233355533
URL https: /oi. orq/10..
11 1109/TPA /TTMM 2223. 3257351

---

Published as a conference paper at ICLR 2024
Xuchai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efftcient model adaptation for vision transformers. arXiv preprint arXiv:2203.16329, 2022b.
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018. Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1974-1983. PMLR, 2018. URL htt.httt:pppocceeeeeeee
mlr www.:pppssssssceeeee http///////eeee//eeeeeeee www.heelfhihhhhhe htthllll..mmlmm
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 09-15 Jun 2019. URL https: https::/ppoceeetteeess httpp//pcceenn..../// .mlr.
Sress/v97/her sbyl 9a html
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL. https: hnttts:://ppereeviiee net net/forum?
id=nZee/KeeFYf9
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks, 2020.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European Conference on Computer Vision (ECCV), 2022.
Purushottam Kar and Harish Karnick. Random feature maps for dot product kernels. In Neil D. Lawrence and Mark Girolami (eds.), Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Re- search, pp. 583-591, La Palma, Canary Islands, 21-23 Apr 2012. PMLR. URL https:
//proceedings mlr. wwwpseeeee.. wttpaa//v2/kaarr...m wttl/.rrrl12..... html
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetorrss. In Annual Meeting of the Association for Computational Linguistics, 2021.
Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Un- terthiner, and Xiaohua Zhai An image is worth 16x16 words: Transformers for image recognition at scale. In Ninth International Conference on Learning Representations. ICLR, 2021.
Leonid Kontorovich, Corinna Cortes, and Mehryar Mohri. Kernel methods for learning languages. Theor Comput. Sci., 405(3):223-236, 2008. doi: 00.10166..cs.2000.0663333 URL https:
//doi.org//i 10. 101 wtw.:////............o...o..e.et tes 2008. 06.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. C0 (canadian institute for advanced re- search), 2009. URL http::tttt////tt www.ww/www..oooooooooo wwwpoontto.dudtoee
Yann LeCun and Corinna Cortes.
MNIST
handwritten
digit database.
http://aamnleunn.cc///xxdmm/m//o 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nat., 521((553):436 444, httti/t/t:///do http:://o...rrr/....00/////rrrus
2015. doi: 10.1038//ature
Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning, 2019.
Tailin Liang, John Glossner, Lei Wang. Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey, 2021.
12

---

Published as a conference paper at ICLR 2024
Valerii Likhosherstov. Krzysztof Marcin Choromanski, Kumar Avinava Dubey, Frederick Liu, Tamás Sarlós, and Adrian Weller. Chefs' random tables: -onttrigonometric random features. In NeurIPS, 2022. URL http: wwwppperr...iip.ppeom.epe wwwp::iippp.ppppppeeoeeeemm cc/paper files/paper/2022 /
Valerii Likhosherstov, Krzysztof Choromanski, Avinava Dubey, Frederick Liu, Tamás Sarlós, and
Adrian Weller. Favor#: Sharp attention kernel approximations via new classes of positive random features. CoRR, 2t0p//2220.00777, 2023. doi: 10.4485500aXX1.22222000777 URL https://
4tt..orr/1...455///11...... C0787
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning, arXiv preprint arXiv:2210.05643 2022.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficient language model tuning, 2022.
Nafise Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. Adaptable adapters. In Proceedings the 2022 Conference the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 20023742-373 Seattle, United States, July 2022. Association for Computational Linguisties. URL https://aclamtee oggg 2022. naacl
Radford M. Neal. Priors for infinite networks, 1996. UBLLattps.//pp...eaattcccceett
org/CozpusID: 118117602121
Cheng Soon Ong, Xavier Mary, Stéphane Canu, and Alexander J. Smola. Learning with non non-positive kernels. In Carla E. Brodley (ed.), Machine Learning, Proceedings of the Twenty-first Interna- fional Conference (ICMI. 2004), Bansf, Alberta, Canada, July 4-8, 2004, volume 69 of ACM International Conference Proceeding Series. ACM, 2004, doi: 10.1145/10153301015544 URL
https://doi.
10 10.14451111131111111111
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sehastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transform- ers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020); Systems Demonstrations, pp. 46-54, Online, 2020. Association for Computational Linguistics. URL https: / www aclweb. www.://.....toooogggoooooooooo/e lem:1p-demos..
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.). Advances in Neural In- formation Processing Systems, volume 20. Curran Associates. Inc., 2007. URL
https:://poceee
Mark Rowland, Krzysztof Choromanski, François Chalus, Aldo Pacchiano, Tamás Sarlós, Richard E. Turner, and Adrian Weller. Geometrically coupled monte carlo sampling. Samy Bengio, Hanna M. Wallach, Hugo I Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Ro- man Garnett (cds.), Advances in Neural Information Processing Systems 31: Annual Confer- Neural Information Processing Systems 2018, NeurIPS 2018. December 3-8, 2018, Montréal, Canada, pp. 195-205, 2018. URL htpp:///OO/haaa/aaaaaeeeeeaaaaa
htRtaa/s::////ceeeeeeeeeeeee http:::://occcceeeeenceo htwpeediiiinnn...ngr... neurips....pp...... co/ .html
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers, 2021.
Meyer Scetbon and Zaid Harchaoui. spectral analysis of dot-product kernels. 2021.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. CoRR, abs/ 444.78222 2014.
URL htt.ttttp///aattt
7828
13

---

Published as a conference paper at ICLR 2024
Bernhard Schölkopf and Alexander J. Smola. Learning with kernels support vector machines, reg- ularization, optimization, and beyond. Adaptive computation and machine learning. MIT Press, 2002. [tRp:tt:://we WtL.ttpp//wwwww....aa.oooooo// htt.//aaa...oooocccc//////700000
Arijit Sehanobish, Hector H. Corzo, Onur Kara, and David van Dijk. Learning potentials of quantum systems using deep neural networks, 2021.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
Jonas Wacker, Motonobu Kanagawa, and Maurizio Filippone. Improved random features for dot product kernels. arXiv preprint Xxiv:2201.087/22 2022a.
Jonas Wacker, Ruben Ohana, and Maurizio Filippone. Complex-to-real random features for poly- nomial kernels, arXiv preprint arXiv:2202.02031, 2022b.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guisties. doi: 10.18653/v 1/W18-5446 5446.
hacttp:://claaathooooooy// org/ 5446
Christopher K. I. Williams. Computing with infinite networks. In Proceedings of the 9th Interna- fional Conference on Neural Information Processing Systems, NIPS' 96, pp. 295-301, Cambridge, MA, USA, 1996. MIT Press.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. http:: /tt::://wwwww. /www./w...mmmmmom aww.aclwwwbb.....
2020. emnlp-demos.e
Bruce X. B. Yu, Jianlong Chang, Lin Liu, Qi Tian, and Changan Chen. Towards unified view on visual parameter-efffcieet transfer learning. ArXiv, abs/2210.00788 2022. URL https:
//api. www.steeeennnttooeeeee ort./
Felix X. Yu, Ananda Theertha Suresh, Krzysztof Marcin Choromanski, Daniel N. Holtmann- Rice, and Sanjiv Kumar. Orthogonal random features. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference
Neural Infor-
mation Processing Systems 2016. December 5-10. 2016, Barcelona, Spain, pp. 1975- nerripss wttpppappe/221//aa...... 1983, 2016. URL https: Abstrrtr .ttml
53aca 33
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-bart masked amnuuage-modek 2022.
14

---

Published as a conference paper at ICLR 2024
A TOWARDS URFS: ADDITIONAL INTUITION
In this section, we provide an additional intuition that led us to the mechanism URFs. Our first observation is that if one has mechanism for the linearization of the softmax kernel K K(x,y)
exp(x' y). i.c a randomized mapping & IR + Rn such that: ((..)) = [[((x) Φ(y)), then this mechanism automatically provides an approximate linearization of the kernel defined as:
K(X, y) Σ exp(s;x y),
(15)
for some coefficients: Ci, C₁2 $1,... $t. To see that, note that one can simply define the random feature map V for that kernel
(z) concat (Vero( siz), vao( V vs2)),
(16)
where concat stands for the concatenation operator. Alternatively, addition CI,..., a > one can first sample the index & E {l,.... {} from the discrete distribution (Pi...,P)) with En=
Cr
and then define V as:
L (z)
Σ Cn (V(()) 12
(17)
The mechanism of URFs the natural extension this observation to the setting, where the kernel cannot be given by the sum from Eq. 15 but the formula with the sum replaced by the integral (which immediately leads to the representation of the function as the Fourier Transform of its inverse Fourier Transform vice versa: inverse Fourier Transform of its Fourier Transform). The only remaining step is to choose a right mapping for the estimation of the regular softmax kernel and here we have decided leverage the recently introduced improvement of the positive random feature map mechanism from Likhosherstov et al. (2022).
B PROPAGATION OF THE ERROR OVER THE NETWORK
In this section, we show how the error accumulates when we try bundle deep feedforward network.
The variance of the estimation of the kernel value x(x,y) is proportional to 1, where is the m number of random features. This is the case since its estimator can be rewritten as:
X
where each X, provides an unbiased estimation of the kernel value. Since random variables X; independent, using Azuma's Inequality, we can also conclude that if [X,] & c, then:
P[ K(X, 2 el S 2exp
(- (Smc=))
(18)
for any c > O. Recall from exp(x (x (E) Eg~A((oLa [Ag [W), where Ag. IRd - IR is
defined as follows for < 0:
||
Ag(z) (1 - 1A)* exp( A ||gll
2 + VT- 14g T
(19)
Note that the boundedness condition holds if A < O (see Remark 3.1).
w Now assume that the kernel function K satisfies (in the region of interest): [K(x, w) K(u, w) < 5(a), as long as ||x - u//1 < a for some function 8. Note that this is not strong assumption as any continuous function on a compact subset of TR uniformly continuous, i.e. satisfies the above condition for some 8, where 8(a) + 0 as + 0.
We then conclude that the probability that the approximate output of the bundled d-layer neural network differs from the exact output by at least + ō(e) + ... + - 8(8(E)) (composition of (d (d-1) 1) f-functions) in the L -norm is upper-bounded by the RHS from the Inequality 18 but with an extra multiplicative factor d (coming from the union-bound). We see that the number of random features needed for an accurate approximation is larger if К grows faster (c.g. is L-Lipschitz with larger constant T.).
15

---

Published as a conference paper at ICLR 2024
C SNNK LAYERS AND RELATION TO LINEARIZATION OF 2-LAYER NEURAL NETWORKS
In this subsection, we provide details on the training of the SNNK layers and how they can be injected in place of MLP layers. We use the PyTorch style of Linear layer in our notations, namely right multiplication by W Recall an MLP layer with weight matrix W, bias b and non-linear function a takes an input X and computes
(20) Rewriting A as (r b), one can think of W as fixed weights while A are trainable which is exactly equation 3 Moreover A has fewer parameters than W, resulting in parameter efficient training.
f(x) = (XW +b ~ ( Tb)
Using this above intuition, one can seamlessly plug SNNK in place of either pretrained network where the input to v will be pretrained weights or random weights in case of untrained networks. We do not train W in this case, but only A.
D SNNK-INSPIRED ADAPTER BLOCKS
We think of adapters (discarding the nonlinearity) as low rank factorization of a linear layer. If the weight matrix W of the linear layer is of size (d x d) and & is the low rank of the factorized matrices where k < d, the number of trainable parameters is 2 x k X d (again discarding the bias terms) as opposed to x d. However, we consider the problem of linearizing the entire block by one feature matrix of size (d x random features) and similarly transform the input tensors and compute the matrix multiplication in the feature space. In this setting, we ignore the bias term b as b is initialized as the zero vector and it does not change the initial feature matrix. Thus our implementation of linearized adapter looks like:
Y = (NNK(X) = ((X)V
(21)
where & and v are suitably chosen feature maps. As W is randomly initialized, we can treat V (W)) as random unstructured matrix and update A. We use residual connection as in (Houlsby et al.,
2019). It is well-known (Houlsby et al. 2019 He et al. [022a Pfeiffer et al. (020) that for stable training the adapter block should behave like the identity matrix at initialization. We introduce a vector that we call the gating vector the modulating vector that is initialized the zero vector. Thus the equations of our adapter block becomes: :
Y (v @ SNNK (X)) + X
(22)
stands for Hadamard or element-wise product.
At this point, we would like to give some motivation regarding the use of this vector and discuss initialization schemes. One simple way to initialize the entire block the identity, choose A as the O matrix but that in turn leads to optimization difficulties. Another initialization scheme would be initialize A from a Gaussian centered around with small variance, but does not lead to good performance. Other detailed initialization schemes are not studied and are beyond the scope of this work. Meanwhile, adding the gating vector allows us initialize the block as the identity matrix, leading to stable training.
Thus the number of training parameters in this regime is (d 1) x random features which is consid- erably lower than the adapters if the number of random features are small. For all our experiments, we only use 16 random features resulting in lowering the number of training parameters compared to the bascline adapters. Our AXNK-inspired adapters can be used in different configurations and can be also combined with SOTA adapter-base based methods and we leave that to future work.
E BUNDLING THE POOLER AND THE CLASSIFIER LAYERS
The bundling of the pooler and the classifier layer takes particularly simple form in this case: If W,, by (resp. We. b.) are the weight matrices and biases of the pooler and classifier layer resp and be the tanh activation function. Then
a (XW,) + W. bb ((X) TW? b
TW.
(23)
16

---

Published as a conference paper at ICLR 2024
of Parameterss
7.5K
-66K
of Paramn Parameters
-17K
7e5
~669K
50K
6e5
-18K
-4.5K
Se5
-423K
Baseline
Bascline
Baseline
Figure 7: Comparison of trainable parameters between the baseline models for toy experiments and the SNNK- counterparts.
(W, p. b,) is matrix of size d x k where d is the dimension of the transformer, & is the number of random features and k < d. We is a matrix of size x where e is the number of output classes. Thus, instead of storing the of size & X c, resulting in huge storage savings. Moreover during deploying, one can only use the smaller matrix as the output layer resulting in lower flops.
matrices, we can only store the product of the 2 matrices which is
DATASETS
We describe the dataset statistics used in different experiments. Glue a benchmark dataset in NLP comprising of 8 different natural language understanding (NLU) tasks (Wang al. 2018). Table 2 shows the train, dev splits for various tasks. CiFar- consists of 50k natural images for training split
Table 2: Statistics of Glue datasets
Dataset
RTE
MRPC
QNLI
QQP
SST-2
MNLI
STSB
COLA
Train/Dev
2.49k/277
3.67k/408
105k/5.46k
364k/40.4k
67.3k/872
393k/9.83k
5.75k/1.5k
8.55k/1 .04k
into 10 categories with 5k images in each eategory and 10k images for testing. CiFar- consists of 50k natural images for training split into 100 categories with 500 images in each category and 10k images for testing. MNIST is a dataset of handwritten digits, consisting of 60k training examples across 10 categories and 10k test examples. For these datasets, sampling from the training set to create a validation set which used to select the best model to run on the holdout set.
25% stratified random
We use three large UCI classification datasets CoverType]] 510K points, dim = 54), HIGGS (11 M points, dim 28) and HEPMASS (11M points, dim 28) evaluate SNNK.
G EXPERIMENTS
this section, we provide additional details for various experiments.
G.1 ADDITIONAL DETAILS ON Toy EXPERIMENTS
In the following subsections, provide additional details the toy experiments. The aim for conducting the toy experiments is to showcase the versality of the SNNK-layer that can be used as drop-in replacement for MLP layers in different optimization problems.
G.1.1 QPNN
QPNN is simple PINN-styled 3 layer neural network (Sehanobish al. 2021). learns the poten- tial energy of a quantum system in an unsupervised manner by learning the principle of conservation of energy (using Schrodinger's equation). The hidden layer of the QPNN of size 128 x 128 and in this work, we replace the hidden layer by our SNNK layer with 32 random features, resulting
https://archive.iess http:e///cciieeeeeuu.e.ee///aaae
17

---

Published as a conference paper at ICLR 2024
in efficient computation of the potential energy of the 2-body system. We use the cosine variant of SNNK for generate the plots in the main paper. We observe similar performance for the sine variant of SNNK.
G.1.2 SIREN
Siren network is a MLP with sine activations introduced in (Sitzmann et al. 2020). Siren network for the image fitting experiment is 3 layer neural network with hidden dimension matrices of sizes 256 x 256. As is shown in the original work, Siren network can not fit the image but can also represent the derivatives of the signal. We replace the hidden layer with sine variant of the SNNK layer with 64 random features, resulting in modest computational gains. We show that like the original Siren network, our SaNK-augmented Siren network can also accurately model the derivatives of the signal.
The cosine variant of SNNK performs similarly in this task. This is not surprising since the derivative of siren network is also siren network, and cosine function is a shifted sine function and so they behave similarly.
G. G.1.3 CLASSIFICATION ON MNIST
The baseline NN is simple 3-layer feedforward layer with hidden dimension to size 512 x 512 with ReLU activation. Like our previous experiments, we replace the hidden layer by our ReLU-variant of the SNNK layer. We train the baseline as well as SNNK-MLP with various number of random features for 25 epochs and measure their accuracy on the test set (see figure (left).
Moreover, we observe lower training losses and stable training for SNNK-MLP across different number of random features (see figure (middle)).
G.2 EXPERIMENTS LINEARIZING THE POOLER LAYER IN PRETRAINED TRANSFORMERS
Many encoder only pretrained transformers use a special token called [CLS] that is appended in front of the input sequence which can be used for classification tasks. After the transformer blocks, these encoder models employ a pooling layer on the final hidden state of the [CLS] token as the final representation for the sequence which is then passed to a classifier layer for classification or regression task. The pooling layer is linear layer of size d x d, where d is the hidden dimension of the transformer with a tanh activation. For the baseline results, the BERT layers are frozen and only the classifier and pooler is tuned.
We found the ANNK-variam with the cosine variant to be particularly performant in various scenar- ios we use it as proxy. Both the cosine and the tanh functions have the same range of values so we find no instabilities or difficulties in training. For these experiments, we replace the pooling layer by our NNK layer employing fewer random features than the hidden dimension. Detailed discus- sion about the bundling procedure explained in Appendix We would like to note that in image classification tasks, the pooler layer is not used, only the final [CLS] token is used for classification.
G.3 EXPERIMENTS ON SNNK-INSPIRED ADAPTERS
Adapters in transformers were first introduced for NLP tasks (Houlsby et al., 2019) and there has been a lot of work designing different architectures (Pfeiffer et al. 2020; K Karimi Mahabadi al. 2021 Moosavi et al., 2022) and unifying various paradigms (Moosavi al. [022] He al. 2022a). Adapters are bottleneck MLPs which are added twice to each Transformer layer, after the attention layer and once after the feedforward layer. For simplicity, we choose to work with the original Houlsby configuration in this work.
Following the success of adapters for NLP tasks, several authors have proposed various variants of adapters in ViT (Chen et al.) 2022 He et al.) 2022b Yu et al.) 2022). Like the BERT-adapter, we use the adapter implementation for ViT-adapters using the Houlsby configuration as described in (Pfeiffer al. 2020).
18

---

Published as a conference paper at ICLR 2024
H HYPERPARAMETERS
In this section, detailed hyperparameters for each experimental setting is presented. All experiments on the smaller dataset are run on Google Colab free version (T4 GPU), while experiments on the larger datasets used V100 GPU and 40Gb A100 GPUs in Google Colab. For the GLUU experiments, we use the sequence length of 128. For experiments on image datasets, we use the ViT checkpoint "google/vit-use-pateeee 2-tp-patchh-2222-nnn-n
which is ViT-base model pretrained on ImageNet-21Ik (Deng) et al. 2009). We use 224 x 224 resolution for the images for all the experiments. For experiments the on text datasets,
checkpoint.
H.1 ADAPTER FINETUNING
For these experiments, we use learning rate of Le 3, batch size of 64 and a constant scheduler with warmup steps 6% of the total number of training steps with AdamW optimizer (Loshchilov & Hutter 2019). All models converge within 5-30 epochs depending on the size of the dataset. For the baseline experiments,
choose adapter size of 48 for all experiments. For our SNNK-adapter experiments, we merely use the hyperparameters found in (Pfeiffer et al. (020).. We realize that such choice of pyrrarrameeter may not be suitable for optimal performance but detailed hyperparameter tuning is beyond the scope of the work.
For experiments on image datasets also use gradient clipping to clip the gradients to global norm 1. We set number of random features as 16 for all adapter finetuning experiments except ImageNet for which we used 32.
H.2 POOLER FINETUNING
For these experiments, we use learning rate of 1e - 3, batch size of 64 and weight decay of le with AdamW optimizer. All models converge within 5-20 epochs depending on the size of the dataset.
For the image experiments, we also use a constant scheduler with warmup steps 1% of the total number of training steps. We also divide the inputs to our SNNK layer as well the initial weights by d22 where d the hidden dimension the Transformer. Note that this is the same transformation that is used in (Choromanski et al.] 2021).
H.3 EXPERIMENTS ON BUNDLED NETWORKS
For these experiments, we hyperparameter search over learning rates in (1e-2, 5e-3, 2e-3, le-3 }. We use batch size of 64, Adam optimizer and do not employ any regularization techniques.
H.4 TOY EXPERIMENTS
For the experiments on image fitting and computing the potential energy of the 2-body system, we use the default parameters as used by the authors of the original papers. For MNIST experiment, use batch size of 32, learning rate .000 with Adam optimizer. We also use dropout of after every layer except the output layer.
H.5 UCI EXPERIMENTAL DETAILS
three layer MLP is used as a baseline. We use dropout and ReLU activation in the first two layers, with the layers being: 1) ReLu(Linear(dim 512)) and 2) ReLu(Linear(5112 dparam)) and the output layer being output = Linear( Linear(ddpuam.. numClasses), where dim is the dimension the input (data dependent), dparam is hyperparameter that vary to get all the relevant plots and numClasses the total number of classes to predicted. For our method, we replace the second layer with a SNNK layer.
19

---

Published as a conference paper at ICLR 2024
ONU
Cal-ooM--eerr
2
20
AYSE
30
Dors
Ders-Fodee
Eert-oocerr
Figure 8: Ablation with different number of random features for the Cosine-SNNK S-No-poolerrr experiments the GLUE dev set. Bert Pooler baseline from Lee al. 2019).
99.0
MiUST
60.0
SNNK
Uaseline
97.0
Handom Features
LE WFZ WNJ
PBC
Accuracies of SNNK-MLP for different number of random features. Middle Training losses for the corresponding models MNIST training set. Right: Starting with pretrained Pooler weights produce better results than randomly initialized layer for both image and text datasets.
Figure 9: Ablation results for MNIST dataset. Left
H.6 UPTRAINING EXPERIMENTS
We use random features, AdamW optimizer and linear learning schedule with warmup of 6% of the total optimization steps for all uptraining experiments. For the larger GLUE datasets (SST-2, MNLI, QNLI, QQP) we use a weight decay of .0001 and for the smaller datasets we use a weight decay of 11 and gradient clipping of 1. For image datasets, we use weight decay of .001. We train for 5-20 epochs depending on the dataset, and selecting the best model based on validation loss.
I ABLATION STUDIES
In this section, we present additional ablations in different experimental settings. Figure 9 (Left (Left and Middle) shows accuracy on MNIST test set and the training curves for SNNK-MLP for different number of random features.
We also show the importance of starting with the pretrained pooler weights over randomly ini- tialized layer. Our results show that the initial features coming from the pretrained pooler weights produce better results on a wide range of text and image datasets.
Finally we present detailed ablations on the number of random features for pooler experiments on the Glue tasks (see figure
J ADDITIONAL EXPERIMENTS
We show some additional experiments with different SNNK layers on various tasks.
J.1 POOLER EXPERIMENTS
In this section, we present additional pooler experiments using a lincarization of the tanh kernel as well as pooler experiments for vision transformers.
20

---

Published as a conference paper at ICLR 2024
96
95.06%
95.18%
87
Accuracy
94
96.22%
86.3215
92
90
Bascline
Figure 10: Results on linearized pooler experiments on CiFar-10 and CiFar-100.
J.1.1 EXPERIMENTS USING BRUTE FORCE LINEARIZATION OF TANH
Since tanh is the activation function used in pooler layers, we compute the linearization of the tanh layer via the method sketched in Appendix L.T] The sparse features produced by this kernel makes learning difficult and produces poor performance (see table 3). This result motivates the need for novel techniques that developed this work.
Table 3: Bank-SNNK experiments on some GLUE benchmarks. The baseline numbers for BERT- base are from (Lee et al. 2019). MCC score is reported for CoLA, F1 score reported for MRPC, Spearman correlation is reported for STSB, and accuracy score is reported for RTE. All results are an average seeds.
Dataset
RTE
MRPC
STSB
COLA
Bert-baseline
57.5
81.5
78.1
29.4
Tanh-SNNK-poole
52.08
68.71
63.49
20.72
J.1.2 LINEARIZING THE POOLER LAYER IN VISION TRANSFORMERS
Vision Transformers introduced by (olesnikov et al. 2021) introduces a [CLS] token which can be used for image classification. Like BERT, VIT also has pooling layer (a linear layer with tanh activation) that applied to the representation the [CLS] token coming from the final transformer layers. One similarly that representation for image classification. In that setting, apply our lincarization the pooler layer.Our method matches the baseline accuracy on Cifar 10 and outperforms Cifar-100 (see figure TO
J.2 SNNK-ADAPTER EXPERIMENTS
We present additional experiments on GLUE benchmarks using a Cosine-SNNK adapter layer. Even though sinusoid activation is not common in Transformers or adapters, we see that they perform rel- atively well beating the baseline on out of 8 tasks while using only 1/3 of the training parameters.
Table 4: Cosine-Si -Ninke experiments on GLUE benchmarks. The adapter baseline numbers are from (Moosavi et al., 2022). MCC score is reported for CoLA, Fl score reported for MRPC and
QQP, Spearman correlation is reported for STSB, and accuracy scores are reported for the other
tasks. All results are an average of seeds.
Dataset
RTE
MRPC
QNLI
QQP
SST-2
MNLI
STSB
COLA
Bert-Ad basetere
63.83
84.8
90.63
88.12
91.74
83.53
88.48
56.51
Dosine-SNNK-AAaptee
64.77
85.06
88.67
86.471
91.97
79.28
88.57
52.12
J.3 TRAINING BUNDLED NETWORK
We present proof-of-coneeee experiments training using bundled network. We use the setting of finetuning a BERT-basel model where the pretrained transformer layers are frozen and only the pooler and classifier layers tuned.
21

---

Published as a conference paper at ICLR 2024
Linear
Linear
GELU
M-WW by
Linear
Linear
SNNK based
SNNK Bundled
Figure 11: FFN and SNNK adapted FFN layer of BERT and VIT.
We linearize the pooler layer and combine it with the classifier layer to train matrix W of size (# random features classes), resulting dramatic drop training parameters. We maintain competitive performance while using 1024 random features thus training approximately 1/30 of the number of parameters the baseline model which trains both the pooler and classifier head. We also train a linear probe, where the pooler is also frozen and only the classifier head is trained. The results are presented in 5
We would like to emphasize that these results remarkable, particularly on large datasets like MNLI, where we are only about 2.7 points off the top row despite being about 1/30th smaller. Our results also show that the BERT representations generally can not linearly separate the downstream classes. Our novel methods introduce non-linearities without many additional parameters allowing for implicit construction of complicated decision boundaries. Moreover, our bundled networks can be trained stably at very high learning rates to 2c-2, allowing them to converge rapidly.
STSB dataset is a regression task and as explained in sec 3.3 we have closed form formula for W in that case. We computed the features of the BERT vectors via our random feature mechanism and calculated the coefficients of the linear regressor using the closed form formula, which gives us a Spearman correlation of 67. The entire fitting and evaluation takes barely a few seconds on a CPU. We get the same result when we import the linear regression function from sklearn.
On Cifar-10 0 and CiFar-100, we achieve an accuracy of 95.8 and 82.73 respectively, closely matching the baseline results.
Table 5: Mandled-SNNK experiments on GLUE benchmarks. MCC score is reported for CoLA, Fl score is reported for MRPC and QQP, Spearman correlation is reported for STSB, and accuracy scores reported for the other tasks. All results an average of 5 seeds.
Dataset
# Training Parameters
RTE
MRPC
QNLI
QQP
SST-2
MNLI
STSB
COLA
Pooler + Classifier [Lce et al. 2019
59k
57.5
81.5
74.5
72.0
84.9
56.4
78.
29.4
Linear Probe
1k
62.22
83.02
69.87
Bundled final- final-layers (ours)
2
60.01
81.48
72.21
70.19
82.97
53.69
71.84
32.79
J.4 UPTRAINING TRANSFORMERS
In this section we provide experimental details and results on how SNNKs can be used to replace certain FFN blocks in Transformers. It is well-known that the FFN blocks account for more than half of the training parameters and the storage size. If d is the hidden size of the transformer, the FFN block consists of 2 linear layers, one taking the input representation of dimension d 4d with GELU non-linearity and the other taking the intermediate representation down to dim d. Our aim here is to replace the first expansion linear layer with non-linearity by a small SNNK layer employing only 8 random features (Figure [.. After we have trained model, we can bundle the SNNK with the following linear layer entirely bypassing these large FFN blocks. particular, we replace f(x) Gelu(zW, + bi) + b2, where W1, W e gdx4d with f'( = (z)))A"))) AW, + b2. where = (W,,,,1) In our settings d = 768 and number of random features is Thus is of size 8 x 4 * d. As described by our bundling process, for forward pass, A W2 be precomputed into a matrix of size d x to save further memory and computational cost.
For both image and sequence classification tasks, we replace the top k FFN layers of BERT and ViT models by SNNKs using the procedure described above. Each successive replacement of FFN layer with SNNK leads to reduction of both number of parameters and flop count (see Table 6. The flops
22

---

Published as a conference paper at ICLR 2024
15
95
ON1/
400.1001
80
Acouracy
-0
50
3WW00unds
40
50
100
115
70
80
380
11.
75
70
90
100
115
70
80
150
11.
# Inferre vce Pammrnees (x10f)
Inference Parmereees (x1df)
# Inference Parameeers (x1of)
Inference Parameters (x10f)
AWA
5755
12
0014
acnueyy
22
Spearan
Pretuned BERT
Fastured BRRT
Protoned EEa?
Fruttand ECET
80
70
00
90
350
13c
75
70
8D
60
300
11c
70
90
1ón 110 40
70 A Infercnnce Poramcterr (x109)
00
90
10n 110
Info once Paramcterr 1x30P)
of Inforence Paramcters Ex109)
Inferrnnc Foramcters (K20P)
Figure 12: Uptraining experiments using BERT GLUE benchmark. MCC score is reported for CoLA, Fl score is reported for MRPC and QQP, Spearman correlation is reported for STSB, and accuracy scores are reported for the other tasks. Full column corresponds to full finetuning and the BERT results are sourced from (Pleiffer et al. (0200)
GIaRR
95
50
80
40
60
70
Inforence Poran etors (K10P)
Iiferrncc Faramaters bx10f)
Figure 13: VIT uptraining results on CIFAR-10 and CIFAR-100.
counted using an batch size of 64 with sequence length of 128 for BERT models and using batch of 32 images for the ViT models. Note that we can substantially reduce the size/flop counts during inference due to our novel bundling process. In fact, with 6 layers bundled, the BERT size is 226.71 MB down from 440, and for ViT, it is 176.42 Mb down from 346 Mb (almost a 50% reduction). We plot the change in accuracy as the model size and flops decrease Figure [2] The notable reduction in model size and FLOPS comes with only a slight decrease in accuracy.
On a free Google Colab, running batch size of 64 with sequence length of 128, takes our SNNK-BERT .32 secs compared to .44 seconds to the original BERT model. For SNNK-ViT, it takes .26 seconds to run inference on a batch of 32 images whereas ViT takes .33 seconds.
Table 6: Detailed analysis of parameters and flops of bundled transformers.
Model
Full
12
11
9
8
7
# Training parameters (millions)
110
107
100
93
86
79
72
BERT
# Inference parameters (millions)
110
104
97
90
83
76
69
# Training Flops (billions)
716
697
678
659
640
620
601
Inference Flops (billions)
716
677
639
600
561
523
484
# Training parameters (millions)
86
84
77
70
63
55
48
ViT
# Inference parameters (millions)
86
81
74
67
60
53
46
# Training Flops (billions)
563
548
533
519
504
489
475
# Inference Flops (billions)
563
533
503
474
444
414
384
K DETAILED COMPARISONS WITH ADDITIONAL BASELINES
We present detailed comparisons our SNNK-inspiree adapter methods with different SOTA pa- rameter efficient finetuning (PEFT) methods. table 7 results for BiTFit, Adapter, AropterDrop. LoRA, censformer-probingg tation are from He et al. (2022b), Visual Prompt Tuning results are from Jia et al. (2022) and that of AdaptFormer are from Chen et al. (2022). Adapter (Houlsby) is a baseline trained by us that uses the Houlsby configuration as described in (Pfeiffer et al., 2020). Our SNNK-inspired adapters per- form competitively amongst various parameter efficient methods. has a pooler layer but it is kept frozen, while BELU-SNNK-AGAA does not use a pooler layer as is commonly the case with ViTs.
LoRA-Fix, LayerNorm Tuning, LePE Tuning. RPB Tuning, KAdap-
LLUU-SNN--Aaaptee model
23

---

Published as a conference paper at ICLR 2024
Table 7: Comparisons with SoTA parameter efficient methods for Vision Models.
Training Parameters (in millions)
Methods
Cifar- 10
Cifar- 100
Full Finetune
85
98.95
91.67
BitFit
.36
92.3
81.0
Adapter (Houlsby)
1.8
99.1
91.2
Adapter
1.51
98.1
90.6
DropterDrop
.17
96.8
88.4
LoRA
.22
98.7
90.6
Transigrmer-probing
3.2
96.5
86.9
LoRA-Fix
.15
96.2
88.3
LayerNorm Tuning
.08
92.2
71.7
Attention Tuning
28.41
93.9
85.7
LePE Tuning
.17
93.7
90.8
RPB Tuning
87.0
KAdaptation
91.2
Visual Prompt Tuning
.08
90.97
AdaptFormer-64
1.26
91.86
ReLU-SNNK W.NSSNNNNApppae Adapter*
.3
98.87
92.11
S.LUU-SNN-AAAan
.
98.2
91.1
We now present baselines for PEFT methods for the GLUE tasks. The training parameters in the table refer to the additional training parameters injected to the frozen transformer models. Bit- Fit results are taken from /aken et al. (2022), Adapter (Houlsby and Pfeiffer) from [Peiiffer et al. own experiments using the same Lora hyperparameters
(2020). For the Lora baseline, we for Roberta-base (Hu et al. 2022).
Table Comparisons with SoTA parameter efficient methods on GLUE benchmarks. MCC score is reported for CoLA, F1 score is reported for MRPC and QQP, Spearman correlation is reported for STSB, and accuracy scores are reported for the other tasks. Final classifier parameters are not used for counting the total number of training parameters.
Training Parameters millions)
Datasel
RTE
MRPC
QNLI
QQP
SST-2
MNLI
STSB
COLA
Poll Finetune
110
66.2
90.5
91.3
88.3
92.6
84.1
88.8
59.5
Adapter buseline Mousavi al. 2022
.9
63.83
84.8
90.66
88.12
91.74
83.53
88.48
56.51
Adapter (Houlsby)
1.8
69.8
91.5
91.2
88.6
92.8
81.11
89.2
59.1
Adapter (Pleiffer)
.9
70.8
89.7
91.3
88.2
90.2
84.1
89.0
Adaptable Adapter Moosavi al.
2023
61.25
85.09
89.
88.09
91.31
82.89
88,25
51.41
LoRA
.3
72.5
9X._
91.5
88.5
92.7
84.1
89.1
59.2
BaFFt
72.3
90.4
961.2
81.
92.
82.2
89.2
58.8
Relu SNNK Adapter
.3
69.68
91.26
00.44
85.82
92.31
82.06
85.81
58.21
24

---

Published as a conference paper at ICLR 2024
L USING POLYNOMIAL KERNELS FOR BRUTE FORCE CONSTRUCTION OF RANDOM FEATURES
Kar & Karnick (2012) constructs random features for kernels of the form (( y) f(ay ) where
I has a power series expansion of the form f := Σ a, and a₂ 2 0. This can be easily extended the case where does not necessarily have positive Taylor coefficients. Write as f := fi f2.
where fi and f2 have positive coefficients, and (xx, y) K₁ (2.
(2, y). Each K; admits random feature mechanism & and thus K(x, y) { 2 [=)] (y) 22))) where
refers to concatenation of 2 vectors along their columns. The approximation error can be bounded
by the sum the approximation errors for each factor. We refer to this construction as the brute force variant.
L.I EXTENDING POLYNOMIAL KERNELS TANH
Tanh is an odd function so an 0, if is even. Moreover, the non-zero terms alternate in sign, i.e.
0,
if n 1(mod 4)
On < 0, if n 3(mod 4)
0 otherwise
Splitting it up in 2 power series with the positive and negative terms produces extremely sparse kernels 2, and this due to the external measure used in (Kar Karnick 2012). Even with rejection sampling, i.e. essentially sampling only odd terms still produces sparse kernels. This becomes a bottleneck in the pooler experiments where most of the accuracies are contributed by the pretrained features and applying sparse kernel distorts them. We use the same random Rademacher matrices for construction of both K₁ and K₂ as it reduces the variance (Choromanski et al. 2022).
Table shows some results using the above kernel for the pooler experiments. The results considerably worse for larger datasets. This negative results show the need for developing novel techniques discussed in this paper.
M FOURIER TRANSFORMS OF SOME COMMONLY USED ACTIVATION FUNCTIONS
In this section we compute the Fourier Transform of some well-known activation functions.
M.1 SINE
Sine activation was used by Sitzmann et al. (2020). This is well-known and is given by:
PT(s)n()))(()) -10()))))0(
1
1 )]
5(k
(24)
2m
2m
where 8 the Dirac delta distribution.
M.2 COSINE
The Fourier transform of cosine well known and given by
1 5 + 1
1
1
[=]cos(=)]((})
5(k
(25)
2
2
Note that the Fourier transform real cosine
function.
M.3
RELU
smoothed version of RELU and compute it's Fourier transform instead. This method is employed in (Bresler & Nagaraj. 2020 In fact our method can be seen generalization of the results presented in the above paper.
The Fourier transform of RELU function is not well behaved. So
25

---

Published as a conference paper at ICLR 2024
N
LIMITATIONS
The key ingredient in the computation of URFs is the computation of the Fourier Transform (FT) of the activation function. However, the FT of some of the activation functions used in practice is not well-behaved, e.g. ReLU, see (Bresler & Nagaraj 2020) for the derivation of the FT of "smooth" truncated ReLU. Smoothing and truncating the function incurs an error.
Furthermore, approximating regular feedforward layers with random feature techniques incurs other errors that might propagate from one SNNK layer to the next SNNK layer (see Section [ for detailed discussion).
As we can see from Section J.4 replacing multiple layers of FFL with SNNK results in trade-off between accuracy and efficiency. This trade-off is limitation of our current work and an open question for future research.
Even though the approximation of the kernel may depend on the 7, interrablity/smooonness of the activation function, we noticed that in practice taking a proxy function simulate the kernel and learning the weights work well as long they are not ill-conditioned (i.e. not too spiky or sparse).
26


Reviews:
[{"review_id": "Ioz9uV39qo", "reviewer": "Reviewer_zChr", "summary": "The paper introduces Scalable Neural Network Kernels (SNNKs), a novel alternative to regular feedforward layers (FFLs) in neural network architectures. These SNNKs, while approximating the behavior of FFLs, bring in computational advantages by separating the inputs from the neural network's parameters and then connecting them through a dot-product kernel.\n\nThe primary contribution is the conceptualization of SNNKs that can mimic FFLs but have better computational attributes. Unlike traditional FFLs, these kernels can capture complex relationships beyond just the functions of the dot-products of parameter-input vectors.\n\nThe authors propose a bundling process utilizing SNNKs to condense the architecture of deep neural networks. This leads to compression benefits, and when fully implemented, it results in a bundled network. Interestingly, for specific loss functions like mean squared error, optimal parameters for this bundled network can be explicitly derived, potentially bypassing the need for backpropagation.\n\nAn auxiliary outcome of the research is the introduction of a mechanism called \"universal random features,\" which is instrumental in formulating various SNNK variants. This mechanism also holds significance in scalable kernel methods.\n\nThe paper goes on to offer a thorough empirical evaluation of the proposed ideas, ranging from point-wise kernel estimation to the fine-tuning of Transformers using new adapter layers inspired by SNNKs. Remarkably, their method achieves up to a 5x reduction in trainable parameters while retaining competitive accuracy.", "questions": "The paper mentions that SNNKs can approximate FFLs. Could you provide more insight into the approximation error? In what scenarios might the approximation be suboptimal, and how does the error scale with the depth or complexity of the network?\n\nit would be helpful to understand more about the bundling process's efficiency. How does the network's performance vary as more layers are bundled, especially in deeper architectures?\n\nIn the experiments where SNNKs achieved up to a 5x reduction in trainable parameters, were there any notable trade-offs in terms of latency, inference time, or other metrics?\n\nCould you elaborate on the key differences between the \"universal random features\" mechanism and traditional random feature approaches? What are the primary advantages of this new mechanism?\n\nWhen applying SNNKs to Transformers, especially in the pooler layer linearization, were there any specific challenges or nuances encountered, given the attention mechanisms and positional encodings in such models?\n\nGiven the focus on computational efficiency, were there any hardware-specific optimizations or considerations when implementing SNNKs? How do SNNKs perform across different hardware platforms, like CPUs, GPUs, and TPUs?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper introduces the concept of Scalable Neural Network Kernels (SNNKs), a fresh take on neural network architecture. This novel approach to approximating regular feedforward layers (FFLs) with computational benefits showcases a high degree of originality. The \"neural network bundling process\" and the notion of a fully bundled network present innovative methods for condensing deep neural network architectures. The \"universal random features\" mechanism, which aids in the formulation of various SNNK variants, is another original contribution.\n\nThe research maintains a high standard of quality, underpinned by a combination of rigorous theoretical foundations and empirical evaluations. Extensive experiments have been conducted across various architectures and datasets, ensuring that the proposed methods are tested in diverse scenarios. The results, especially the reduction in trainable parameters without significant performance losses, stand testament to the quality of the work.\n\nThe paper is structured well, with a clear delineation between theoretical concepts, methodologies, and experimental results. While the document is dense with technical details, the authors have made efforts to explain concepts clearly, aided by visual representations where necessary. The inclusion of a comprehensive list of references and contextualization relative to prior work adds to the clarity, helping readers understand the evolution and significance of the presented ideas.\n\nThe versatility of SNNKs, as demonstrated by their applicability in various architectures (from PINNs to Transformers), signifies their broad utility. By addressing the computational challenges associated with traditional kernel methods and FFLs, the paper offers solutions that could pave the way for more efficient and scalable neural network models in the future.", "weaknesses": "The paper could benefit from a more direct comparison of SNNKs with other existing solutions or methods aimed at network compression or efficiency. Highlighting the unique advantages of SNNKs over these methods would further solidify its significance.\n\nThe paper could delve deeper into the robustness of the SNNK approach. Are there scenarios where the approximation might break down? Understanding the edge cases and potential pitfalls would be crucial for practitioners looking to adopt this method.\n\nProviding more explicit details about the implementation, hyperparameters used, or potential challenges faced during the experiments would be beneficial for researchers aiming to replicate or build upon the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j7zMZk8Rau", "reviewer": "Reviewer_uu7e", "summary": "The paper introduces Structurally Neural Network Kernels (SNNK), a novel approach to modeling interactions in neural networks. By exploiting the low-rank nature of neural networks, SNNKs offer a significant reduction in parameter count. When used in multilayer perceptrons and transformer models, SNNKs consistently outperformed traditional baselines across multiple datasets, including synthetic data, toy experiments, UCI datasets, GLUE, and CIFAR. A primary advantage is the reduction in storage requirements without compromising accuracy. The work suggests SNNK layers can be integrated seamlessly with other popular techniques, providing both efficiency and enhanced performance, making them a promising tool for neural network architectures.", "questions": "I have mentioned my concerns I the weakness. Authors can go over the points mentioned in the weakness and clarify them.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper introduces a new computational model, the scalable neural network kernels (SNNK), providing a novel approach to efficient neural network design, particularly for replacing feedforward layers in MLPs.\n\nThe design of SNNKs ensures that inputs and parameters are disentangled, leading to efficient final computations via a dot-product kernel, which can greatly reduce computational overhead.\n\nThe bundling process highlighted in the paper leads to the compactification of the neural network stack, suggesting potential storage savings and efficiency improvements.\n\nThe paper does not rely solely on theoretical claims but provides empirical analysis, spanning from pointwise kernel estimation to practical application scenarios like training Transformers with adapters, strengthening the validity of the proposed methodology.", "weaknesses": "The authors should provide some explanation or intuition why their model doesn’t work well in the some of the experiments they have performed.\n\nThe analysis of how deep of a feed forward network can be approximated using the proposed method should be analyzed in further details. \n\nCan scalable neural network kernel be applied in any scenario or there are some specific scenarios when SKNN won’t work well. Authors should discuss about such datasets/models. If there is none, then authors should also discuss that. They can include some of the experiments involving ImageNet datasets and other models to further support the claim.  \n\nIt would be great to analyse how well the SKNN generalizes to the unseen datasets or in general generalizability as compared to normal network network,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b9oeLQuzFa", "reviewer": "Reviewer_YEib", "summary": "This paper introduces scalable neural network kernels (SNNKs), which disentangle the inputs and parameters of a feedforward layer before connecting them via a dot product kernel. The key ideas are:\n\n- SNNKs approximate regular feedforward layers but with reduced parameters by replacing the weight matrix with a low-dimensional embedding. This allows compression of the layer. \n\n- They introduce universal random feature maps to instantiate different SNNK variants based on the Fourier transform of the activation function.\n\n- SNNKs can express more complex relationships beyond standard feedforward layers. They demonstrate this with a ReLU-SNNK layer related to arc-cosine kernels.\n\n- SNNKs enable a neural network bundling process to compactify model architectures. In the extreme case, the entire network can be expressed as a two-tower computation.\n\n- For certain losses like MSE, the optimal parameters of a fully bundled network can be solved in closed form, bypassing backpropagation.\n\n- Experiments validate SNNKs on tasks ranging from kernel approximation to Transformer fine-tuning. SNNK adapters match baseline accuracy with 5x fewer parameters.\n\n- Bundled SNNK models maintain accuracy while reducing parameters 30x. Closed-form solutions for regression produce strong results.\n\nIn summary, the paper provides a thorough theoretical analysis of SNNKs along with empirical validation. The ideas open interesting research directions in model compression, faster training, and expressive power beyond standard neural network layers.", "questions": "Please comment on the issues raised in the weaknesses part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Here are some of the main strengths of this paper:\n\n- It makes an insightful connection between scalable kernel methods and neural network layers, introducing a novel perspective on feedforward layers.\n\n- The concept of SNNKs is very clearly presented along with detailed theoretical analysis and constructions.\n\n- The Fourier transform based universal random feature mechanism to instantiate SNNKs is interesting and useful.\n\n- SNNKs provably increase expressive power over standard layers, as shown through the analysis of the ReLU-SNNK layer. \n\n- The neural network bundling process enabled by SNNKs is an impactful idea for model compression and acceleration.\n\n- The paper provides extensive empirical validation ranging from synthetic data to large Transformer models across vision and language.\n\n- Both model compression and training acceleration are demonstrated convincingly through the experiments.\n\n- The writing is clear, incremental, and easy to follow. Theoretical concepts are explained intuitively.\n\nOverall, the solid theoretical foundation, novel perspectives introduced, and thorough experimentation are major strengths. The paper makes well-motivated connections between areas leading to useful techniques for efficient deep learning.", "weaknesses": "Some potential weaknesses or limitations of this paper:\n\n- The focus is on feedforward fully-connected layers, not convolutional or recurrent layers commonly used in modern networks.\n\n- Experiments are limited to standard datasets and models; more complex domains like bioinformatics are not evaluated. \n\n- There is no investigation into how SNNKs affect representation learning or generalization. The emphasis is on compression.\n\n- Optimization and learning dynamics with SNNKs are not analyzed, apart from the fully bundled case.\n\n- The work does not connect to broader topics like kernel methods or metric learning. \n\n- Ablation studies teasing apart the contributions of different components could be more detailed.\n\n- The writing in parts of the theory and experiments lacks clarity or intuitive explanations.\n\n- Practical guidance on hyperparameter selection and tuning SNNKs is limited.\n\n- Applications beyond efficiency, like using SNNKs for privacy or interpretability, are not explored.\n\n- Potential negative results or limitations of SNNKs compared to standard layers are not discussed.\n\nIn summary, the lack of experiments on more complex data and models, limited analysis of learning dynamics, and minimal connections to related areas are notable weaknesses. However, the paper makes excellent contributions within its defined scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fcFsLxVQJc", "reviewer": "Reviewer_hP8B", "summary": "This paper proposes to approximate feedforward layers with kernels, thereby achieving better computational efficiency and sometimes better accuracy. This paper empirically verifies their claim on many vision and language datasets over various architectures.", "questions": "1. This paper talks a lot about achieving computational efficiency through dimensionality reduction (if m << d). Could I achieve the same effect by using a feed forward layer but simply reducing the latent dimension from d to m?\n2. Could the authors share empirical evidence that replacing feedforward layers with SNNK indeed results in faster training?\n3. In my understanding if SNNK were to replace feed forward layers then there should be an experiment whether the authors replace every feed forward network in Transformers with SNNK and report the results?\n4. Is there any particular reason as to why the SNNK adapted architectures in Figure 2 look the way they are? In other words, how might one understand the interplay between SNNK and feed forward layers in the same architecture?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written, easy to follow, and empirical results seem strong.", "weaknesses": "The main weakness of the paper is that I am not convinced by whether SNNK can practically replace feed forward layers in practice. See Questions for more details", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)