Approximation to Smooth Functions by Low-Rank Swish Networks
Authors: Zimeng Li
Hongjun LI
Jingyuan Wang
Ke Tang
Venue: ICML 2025 poster (2025)
Keywords: Neural Network
Swish Activation Function
Universal Approximation Theory
Network Compression
Low-Rank Compression
Low-Rank Factorization
Primary area: theory->learning_theory
TL;DR: We offer a theoretical basis for low-rank compression from the perspective of universal approximation theory by proving any Hölder function can be approximated by a Swish network with low-rank weight matrices.

Abstract: While deep learning has witnessed remarkable achievements in a wide range of applications, its substantial computational cost imposes limitations on application scenarios of neural networks. To alleviate this problem, low-rank compression is proposed as a class of efficient and hardware-friendly network compression methods, which reduce computation by replacing large matrices in neural networks with products of two small ones. In this paper, we implement low-rank networks by inserting a sufficiently narrow linear layer without bias between each of two adjacent nonlinear layers. We prove that low-rank Swish networks with a fixed depth are capable of approximating any function from the Hölder ball $\mathcal{C}^{\beta, R}([0,1]^d)$ within an arbitrarily small error where $\beta$ is the smooth parameter and $R$ is the radius. Our proposed constructive approximation ensures that the width of linear hidden layers required for approximation is no more than one-third of the width of nonlinear layers, which implies that the computational cost can be decreased by at least one-third compared with a network with the same depth and width of nonlinear layers but without narrow linear hidden layers. Our theoretical finding can offer a theoretical basis for low-rank compression from the perspective of universal approximation theory.

Approximation to Smooth Functions by Low-Rank Swish Networks
Zimeng Li Hongjun Li?
Jingyuan Wang 134 Ke Tang
Abstract
While deep learning has witnessed remarkable achievements in a wide range of applications, its substantial computational cost imposes limita- tions on application scenarios of neural networks. To alleviate this problem, low-rank compression is proposed as class of efficient and hardware- friendly network compression methods, which reduce computation by replacing large matrices in neural networks with products of two small ones. In this paper, we implement low-rank networks by inserting sufficiently narrow linear layer without bias between each of two adjacent nonlinear lay- ers. We prove that low-rank Swish networks with fixed depth are capable of approximating any function from the Hölder ball C ([0, 1)d) within an arbitrarily small eтror where 3 is the smooth parameter and R is the radius. Our proposed con- structive approximation ensures that the width of linear hidden layers required for approximation is more than one-third of the width of nonlinear layers, which implies that the computational cost can be decreased by at least one-third compared with a network with the same depth and width of nonlinear layers but without narrow linear hidden layers. Our theoretical finding can offer theo- retical basis for low-rank compression from the perspective of universal approximation theory.
(6) classical feofforward network
(b) a low-rank network
Figure 1. An illustration of the difference between classical feed- forward network and low-rank network. A classical feedforward network composed of several nonlinear hidden layers and linear output layer. A low-rank network is composed of several interleaved nonlinear and linear layers where the linear layer act as the output layer. A "A" stands for an input neuron, a "O" stands nonlinear neuron (i.e. neuron with activation function), and "O" stands for linear neuron (i.e. neuron without activation function).
of how neural networks can approximate a specific class of functions to an arbitrarily small error (Hornik et al., 1989; Cybenko, 1989; Leshno al., 1993; Yarotsky, 2017; Yarot- sky & Zhevnerchuk, 2020; Siegel, 2023; Li et al., 2024a), providing solid foundation for understanding their out- standing performance in a wide variety of fields such as computer vision (Tolstikhin et al.. 2021; Li et al., 2024b). speech recognition (Graves et al., 2013; Ren et al., 2019), natural language processing (Touvron et al., 2023; et al., 2023), intelligent healthcare (Ren et al., 2021; Shi al.,
1. Introduction
The universal approximation theory (UAT) for neural net- works mainly studies the quantitative and qualitative aspects
Sccooo of Computer Science and Engineering, Beihang Uni- versity, Beijing, China 'Institute of Economies (School of So- Sciences), Tsinghua University, Beijing, China 'School of Economics and Management, Beihang University, Beijing, China 'Engineering Research Center of Advanced Computer Applica- tion Technology, Ministry of Education, China. Correspondence to: Hongjun Li <hongjunllie tsinghua.cdu.cm Jingyuan Wang <jywang @ buaa.cdu.en...
2022), and smart city (Wang al., 2022a;b; Jiang al.,
2023; Ji et al., 2025).
In the realm of theoretical statistics, the upper bound of the approximation rate is one of the most important ingredients to derive the consistency and convergence rate of neural network estimates (Chen & White, 1999; Schmidt-Hieber 2020; Kohler & Langer, 2021; Farrell al., 2021). Based on
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Approximation to Smooth Functions by Low-Rank Swish Networks
the convergence rate, we could further obtain the pointwise asymptotic normality of neural network estimates and the asymptotic normality of functionals of neural network esti- mates, which is a key step towards constructing confidence intervals and conducting hypothesis testings (Shintani & Linton, 2004; Horel & Giesecke, 2020; Zhong et al., 2022). Evidently, the UAT for neural networks occupies a central position within the framework of learning theory.
be universal approximators. Their results indicate that only O (L.)) nonzero parameters are required achieve the op- timal approximation rate, where L is the depth and H. is the width. For quantization, Petersen & Voigtlaender(2118 (0118) proves that networks with parameters encoded by (log, ) bits can approximate piecewise smooth functions to e and Gühring & Raslan(2021) proves the same quantization con- dition for approximating smooth functions by networks with general smooth activation functions. As for knowledge dis- tillation methods, all research efforts involving upper and lower bounds of sizes of networks needed to achieve rapid approximation rate can offer valuable insights for the design of student networks (Shen et al., 2022b; Hon & Yang. 2022; Liu & Chen, 2024).
Numerous works focused UAT's of ReLU neural net- works (Yakotsykk,, 2017: 2018; Opschoor et al., 2022), owing to its highly efficient computation. However, due to the fact that the derivative of ReLU is zero on (-o0, 0), if the input of a ReLU neuron is less than zero, its related parameters cannot be trained, which is referred to as the "dying ReI.U" problem. Moreover, ReLU suffers from poor smoothness, as it merely possesses discontinuous weak derivative up to the first order and the higher order derivatives are zero al- most everywhere. Consequently, ReLU neural networks are unable to achieve universal approximation with higher-order Sobolev norms. In practice, problems in some fields such as ecology, economics, and engineering physies concern not only the estimation of the unknown function fu. but also the estimation of its high-order derivatives & Caswell, 2014; Wang & Werning, 2022), which cannot be estimated using ReLU networks.
Low-rank compression is a class of efficient and hardware- friendly neural network compression techniques that approx- imate weight matrices through matrix factorization (Denil et al., 2013; Sainath et al., 2013). A standard low-rank compression pipeline typically involves two key steps: first, decomposing weight matrices of the trained network into pairs of low-rank matrices, followed by fine-tuning the re- sulting low-rank network on the training dataset. Intuitively, whether low-rank compression can preserve performance without significant degradation depends on the ranks of the original weight matrices, which are inherently shaped by the training data distribution. Consequently, there is no univer- sal guarantee that low-rank compression will remain effec- tive across diverse tasks and domains. However, extensive empirical evidence suggests that low-rank compression can achieve significant computational savings while maintain- ing nearly identical network performance in most practical applications. Our main theoretical result (Theorem 4.1) pro- vides a principled explanation for the universal effectiveness of low-rank compression in preserving model performance.
The Swish (SiLU, sigmoid weighted linear unit) activation function 1 inherits the advantages of ReLU while alleviating the above problems, since Swish is an infinitely differen- tiable function with shape similar to that of ReLU and nonzero derivative almost everywhere. Empirical stud- ies across various tasks and architectures show that Swish neural networks generally perform better than ReLU neu- ral networks and are seldom significantly inferior to other popular activation functions such as ELU, GELU, Swish, and Mish(Eger et al., 2018; Dubey et al., 2022). However, works on UATs related Swish neural networks are scarce.
In this paper, we develop the theoretical foundation for low- rank compression from the perspective of approximation theory by answering the question of whether a low-rank neural network can serve as a good approximator for a wide range of functions. To be specific, we consider how Swish neural networks with a sufficiently narrow linear layer without bias between each of two adjacent nonlinear layers, called low-rank Swish networks, can approximate any function from the Hölder ball cs."((o, 11d) where R4, is the smooth parameter and R R. is the radius of the ball. For any f E C 1]d), divide [0, 1] into Md hypercubes where M E N,, then approximate f by a sum-product combination of Taylor expansions and approximate bump functions 2 at all grid points of [0, 1]d where an approximate bump function at a point refers to scalar function whose absolute value is small when the
Network compression aims decrease the computational and memory costs of neural networks by compressing their sizes. Common network compression methods can be clas- sified into four categories: pruning (Dong et al., 2017), low- 2020),
rank compression (Idelbayev & Carreira-Perpiieee quantization (Jacob et al., 2018), and knowledge distilla- tion (Hinton et al., 2015). Except low-rank compression, the remaining categories of methods are all underpinned by UAT' to some extent. For pruning. Yarotsky(2017), Petersen & Voigtlaender(211)) and Bolcskei et (201 show that sparse neural networks, which can be viewed as the results of pruming culle-connected networks, could also
'Strictly speaking, Swish x + x(1 + +e - proposed by
propossdd
Ramachandran et al. (2018) and SiLU x + x=1+e=111 -1)]+----1 proposed by Elfwing (2018) are slightly different. This paper ignored the difference by defaulting 3 to 1.
'Here we adopt the term "approximate bump function" to dis- tinguish from "bump function" (also called "test function") which refers to infinitely differentiable function with compact support.
2

---

Approximation to Smooth Functions by Low-Rank Swish Networks
input is away from the point. Then we construct Taylor polynomials and approximate bump functions using neural networks respectively, multiply them together, and sum up,
Liu & Chen(2024) proved that deep ReLU networks with width d + 1 can achieve the optimal approximation rate where is the input dimension. De et al.(2021) wrote survey on UAT's of ReLU networks.
Our main contributions are as follows:
Deep neural networks activated by popular ReL-U-like functions, including ELU (Clevert et al., 2016), GELU (Hendryeks & Gimpel, 2016), Swish (Ramachandran et al., 2018), and Mish (Misra, 2020), have attained great empirical success in diverse range of real-world applications (Ken- ton & Toutanova, 2019; Bochkovskiy et al., 2020), inspiring theoretical exploration of networks activated by them. Ohn & Kim(2019) achieved an approximation theorem appropri- ate for deep neural networks activated by a wide range of functions. Although their result encompassed Swish neural networks, we demonstrate in Corollary 4.2 that the same approximation error can be achieved with a shallower depth and smaller upper bound of absolute values of parame- ters. Zhang et al. (2024) showed that ReLUU networks can be approximated by networks with commonly used activation functions, at the cost of only increasing the depth and width of the networks by small constant multiple.
We derive an upper bound of error for approximating any function from the Hölder ball C ((0, 11() by us- ing low-rank Swish networks and provide the required depth, width of linear hidden layers, width of nonlinear layers, upper bound of number of nonzero parameters, and upper bound of absolute values of parameters.
Our constructive approximation guarantees that the width of linear hidden layers is no more than one-third of the width of nonlinear layers, indicating the quan- tity of multiplication operations occurred in all hidden layers except the first one could be reduced by at least one-third compared with a network with the same depth and width of nonlinear layers but without linear hidden layers.
2. Related Works
Besides classical feedforward networks, there are some novel studies the approximation capabilities of mod- ern network architectures. Shen et (l.222))) derived a asymptotitiic approximation error bound for deep convo- lutional neural networks in Sobolev space. Yun et (2020) and Zahcer et al. (2020) showed transformers are universal approximators of sequence sequence-to-ssquenn to-sequence functions. Lin & Jegelka(2018) 1)) studied the universal approximation property of residual networks activated by ReLUU.
2.1. Universal Approximation
The research on UATs began with one-hidden-layer neural networks. Hornik et (1989) proved that me-hidden-layer neural networks activated by an arbitrary squashing func- tion are capable of approximating any measurable function on compact set to any small error measured in the sup norm. In the same year, a similar result was published by Cybenko(199)) (1989). Hornik et (199) improved Hornik et al989''))'s 's result by replacing the sup norm with the first- order Sobolev norm. Barron(1993] further specified that the approximation rate of one- me--iddee--layy neural networks for specific class of functions is of the order ((2) where n. represents the number of hidden nodes.
2.2. Approximation Theory Foundations for Network Compression
Network compression methods are divided into four major categories: pruning, low-rank compression, quantization, and knowledge distillation. Apart from low-rank compres- sion, all the others have evidence from the universal approx- imation theory to support their rationality.
With the development of computational technology, the training and deployment of deep neural networks have be- come possible. A series of works have demonstrated that for certain functions, if approximated by shallow networks, the required width is far greater than that needed when approxi- mated by deep networks (Eldan & Shamir, 2016; Telgarsky, 2016; Safran& & Shamir, 2017; Rolnick & Tegmark, 2018).
Pruning methods downsize a network by eliminating either unimportant parameters (LeCun al., 1989; Dong al., 2017) or (groups of) neurons (Xia et al., 2022; Ko et al., 2023). The former is generally referred to as unstructured pruning, while the latter is known as structured pruning. For unstructured pruning, Yarotsky(2017), Petersen & Voigt- laender(2018), and Boleskei et al.99 showed sparse neural networks are enough to achieve the optimal approx- imation rate for smooth functions with OLL) nonzero parameters where L represents the depth and H represents the width. The UAT foundation of structured pruning is presented together with knowledge distillation later.
In the last decade, works on approximation theories of deep ReLU networks account for a large proportion. Yarot- sky(2017) first demonstrated how to approximate general smooth functions using deep ReLUU networks. He proved that ReLU networks with depth ((og({t) and number of nonzero parameters (s A lo)( can approximate any function from the unit ball of Sobolev space Wm ([0, 1]4)
within e. Yarotsky(2018) studied approximations of continu- ous functions on compact domains by deep ReLUU networks.
Quantization methods are designed to cut down on both
3

---

we Approximation to Smooth Functions by Low-Rank Swish Networks
computational load and storage requirements by employing reduced number of bits encode parameters(Jacob et al.,
maxi,j lat.yl For a function / 2 - R. |l supxe.2 [f())). Note that for function f 2 Rm with integer m > 1, (1) := max; [z)) because f(x) is a vector. For a vector or matrix, || llo denotes its total number of nonzero elements. Finally, we denote the combinatorial number for all m, n E N by
2018). Petersen & Voigtlaender(2015 8) demonstrated that networks with parameters encoded using (log2 )) bits are capable of approximating piecewise smooth functions with error E. Similarly, Gühring & Raslan (2021) established the identical quantization condition for approximating smooth functions by networks with general smooth activation func-
m
m!! nl(mm-n)
m2n20, 0,
tions.
0,
otherwise.
Knowledge distillation utilizes the outputs of large net- work as labels to train small network, then replaces the
3.2. Low-Rank Swish Network
large network with the small one to achieve compression (Hinton et al., 2015; Mirzadeh et al., 2020; Kim et al., 2022). In terms of compression, both knowledge distillation and structured pruning employ a network that is narrower and/or shallower to substitute for the original network. Research efforts on the upper and lower bounds of network sizes for rapid approximation rates can offer theoretically guaran- teed structural design for compressed networks (Shen et al., 2022b; Hon& & Yang, 2022; Liu & Chen, 2024).
Drawing upon established works in low-rank compression (including but not limited to Denil et al.(2013), Sainath et (2033,, and Idelbayev & Carreina-Perpinnn22225 formally define low-rank network as:
Definition 3.1 (Low-rank Network). Let d, o E N+ be the input and output dimensions and p nonlinear activation function. For a vector input E I2" p(x) == (p(z)), P(x2), ..... p(....))))))) T. Let L be the depth, i.c. the number of nonlinear layers, H. be the width of nonlinear layers and H; be the width of i-th linear hidden layer such that
+ R be the
3. Preliminaries
2H < H
(1)
3.1. Notations
that is called the low-rank condition. A low-rank Swish network nn 22 C Rddd - R with depth L, width of
We denote the set of real numbers by R, the set of positive
real numbers by R2,,, the set of natural numbers by N, and
nonlinear layers H. width of linear hidden layers H, number of nonzero parameters S. and maximum absolute value of parameters B is a function defined by
N- by N_. For n E N, we denote the set {0, 1,...,71 n} by [x] and [n] {0} by [m]++ If n = O, then [+]+ is 0.
For x E IR xx]] denotes the smallest integer greater than or
equal to x and [x] denotes the largest integer less than or
nn(x)) == 4. 41-11 4(z) == V:p W, bi) (i [4]+)
o 1/ o (z) + b
(2)
equal x.
(3)
Vectors are denoted by bold lowercase letters, for example := (#1, 22, Matrices are denoted by bold uppercase letters, for example W E &Raxa is a matrix with m rows and n columns whose element at i-th row and j-th column is wiy. A d-dimensional multi-index is a vector in Nd. For multi-index a and a vector 2r, we denote |ax| == Dd 04 ad == II Oi and aa :- III ,
where W1 E Hxd W; E R for i E [L] - (1),
E Rd is a d-dimensional vector.
V, E R for E [L [L- 11+, VI E ROXH b, E R for E [T]] and b E R such that
S= Et + 1IV[ll) + Σ libillo-
2. B = max/max,e[4)+ Willoo max [ max,e[I+1]. [[bylloe).
Let 2' and be two sets. The notation f: 2 + V de- notes the function of with domain 2 and co-domain V. For an univariate function f: 2 C R + R. its n-th derivative is denoted by f("). If < 3. we also use f' f"" and fd to denote 1tst, 2-nd, and 3-rd derivatives respectively. For a multivariate function f: 2 C Id + R and multi-index ar we denote
Figure illustrates the difference between the classical feed- forward network and the low-rank network. Here we explain why the "low-rank" network achieves low-rank compres- sion. Let's denote the output of the i-th nonlinear layer by 2, E RR and the output of the i + 1-th nonlinear layer by Zif1 E "R In a low-rank network, by Definition 3.1, we have
glal
(4) where the weight W # and the weight V. " For classical feedforward network with the same depth and width of nonlinear layers,
Zi41 p( W i+1 Viz, + bi+1)
8a
go 802 ... .o.
and the a--order partial derivative of f by f. The meaning of sup norm vector 2r, || == max; [rs]. For matrix A, || A |loc :=
||.s varies with its input. For a
1 = p (Wi+12 wit1 z + b(t1)
(5)
4

---

Approximation to Smooth Functions by Low-Rank Swish Networks
where the weight i+1 E R The low-rank condition (1) ensures that the number of elements in W4+1 and Vi is no more than the number of elements W i.e.
2 3, and 2 1, there exists low-rank Swish network
nn: : [0, 1] - R. with depth
max {
Rk [ogg dl
+ +1] + 1,
2HH < H²:
(6)
suggesting the memory required for storing weight matrices compressed. And the condition (1) also naturally implies that the quantity of multiplication operations to calculate from 2; to 2,+1 in the low-rank network is no more than that in the classical feedforward network, i.e.
width of nonlinear layers
d (a-1)) (--1)
d + -
K +
d ( d-1 1) + 6(M + 1)4,
width of linear hidden layers
HH2 HH² < H²:
d (d+1)
d (
d + K -3) 3) (d+x- ( 2 + 1)4
suggesting the computational cost is compressed.
At the end, the Swish activation function IR defined by
R is
upper bound of absolute values of parameters
p(z)
(8)
1ee=
d
A low-rank network activated by Swish is called low-rank Swish network.
max
(3M + 2)T,
max SK
Vi
.222
0ii
3.3. Hölder Function
and upper bound of number of nonzero parameters
Definition 3.2 (Hölder Space). Let d E N+. 2' E Rd and
3 E +. There exist & E N and 0 < y < 1 such that For a function f X + R. its Hölder norm is
C1 c2 + 112
K = defined by
such that
|nn(() f(xx)
liflles == max
sup jad|S
|aa flloo
(M + 1)d
(12)
(9)
+ cc + + Te
| f(x)) 80 f(y))
X2
sup sup |a |= s
| - y y
for all a E [0, where eı, 022 C32 c4, and es are positive constants depending only on d, K, and R.
And the Hölder space 8 (10, 1 is defined as the set
[ + R Ifflee < co}
(10)
An extended version of Theorem 4.1 is presented in Ap- pendix A as Theorem A.24. In Theorem A.24 we pro- vide the exact formulas for upper bounds of the number of nonzero parameters and the approximation error. Next, show way to set the network size in Corollary 4.2 to ensure that the approximation error can be arbitrarily small.
equipped with Hölder norm || llce.
We call functions in C ([0, 11(t) Hölder functions. When 0 < 3 < 1 (i.e. = 0), we call them Hölder continuous functions. When 3 > (i.e. & E N,), we call them Hölder smooth functions. Next we define the Hölder ball with radius/ R.
Corollary 4.2. Let B and R+. For all 0 < E S 3c4, there exists a low-rank Swish network nn [0, 1]d E with depth (1)) width of nonlinear layers O(E $). width of linear hidden layers O(E -3) maximum absolute value of 2id parameters O(s * ), and number of nonzero parameters
Definition 3.3 (Hölder Ball). Let d E N4, 2 ERC R E Ry, and 3 = + 7,K E N,7 (0, 1]. The Hölder ball (I, 1]d)) is defined by
O(E #) such that
2 2 - R ||llea < R}.
(11)
[f(x)) nn(x)) $
Var
1j (13)
4. Approximation Theorem for Low-Rank Swish Neural Networks
The proof ideas of Theorem 4.1 is showed in section 5. And the rigorous proofs of Theorem 4.1 and Corollary 4.2 are showed in Appendix A.
Theorem 4.1. Let 3 E R4, 84,8=A K K E.t.KENNEEE (0, and R E RT. For all I E (3,8(10, M N+, X
5

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Remark 4.3 (Comparison with (Ohn & Kim, 2019)). Ohn & Kim(2019) demonstrated in their Theorem 1 that for any continuous piecewise linear or locally quadratic function there exists feedforward neural network activated by it with depth (log 4), width O(=**), +), number of nonzero paramters O $ 1og(" )), and maximum absolute value of parameters O 41244))
y
h4
h5
can approximate any functions from the Hölder ball ((,, 1)4) within é. Because the Swish is a locally quadratic function, their result holds for Swish networks. Compared with Corollary 4.2, we only requires a constant depth to achieve an approximation error within 6. Moreover, our growth rates of the number of nonzero parameters and the maximum absolute value of parameters with respect to are better than those in Theorem 1 in (Ohn & Kim, 2019).
h
h2
hg
X
X2
X5
X4
Remark 4.4 (Low-rank compression). In Theorem 4.1, we notice that, when B 2 (i.c. 2 2). the width of linear hidden layers is always no more than one- noe-third of the width of nonlinear layers, since
Figure 2. An illustration of a hierarchical composite function. T1. #2, æz, and x are input variables. h1 = h (2). hv (2(21, 23), has = ha(r)). h4 h4 ha), hs = h1s ha)) and y = y(h4, hs). Though the input dimension the hicrarchical composite function is 4, the input dimensions of its component functions do not exceed
d d+1
d K -3
d R 22 d-1
3
+
+ +(M 1)**)
d
d -
d+1
d dd+n=2
+4 d + k k-
2
+4
+ +66M 1)"
of dimensionality, which poses difficulties in approximat- ing nigh-dimensional functions. Here we briefly introduce class of high-dimensional functions, called hierarchical composite functions, which are universal in reality and can be approximated without being affected by the curse of di- mensionality. A hierarchical composite function, as shown in Figure 2, is composed of multiple layers of functions, and each component function is of low input dimension. is obvious that the network size required to approximate hierarchical composite function is directly related the input dimension of each component function and has no direct relation to the input dimension of the hierarchical composite function, because we can construct networks to approximate component functions respectively, then com- bine them into one network (Schmidt Hicber, 2020; Kohler & Langer, 2021).
d d-1
d
d-1
d++1
d 2
(d+x-1)
+
d-1
d-1
d+1
d
d d+1 d d-1)
d-
For a low-rank network with depth I., width of linear hidden layers H, and width of nonlinear layers H. excluding the first and the last layers, evaluating it at one point requires (L-1) HHH2 + HH') multiplication operations. However, for a classical feedforward network with the same depth and width of nonlinear layers, excluding the first and the last layers, evaluating it at one point requires (L - 1)H multiplication operations. When H. > 3H, the low-rank network can guarantee that the quantity of multiplication operations is reduced by at least one-third compared to the classical feedforward network, since
5. Proof Ideas
(L (L- 1) H) < (L- 1
The proof of Theorem 4.1 can be segmented into four steps:
*HH² + H²H 2
Step 1: approximating any Hölder function I by a sum-product combination of Taylor polynomials and approximate hump functions
H3
H 2 H.
+ 9
3
Remark 4.5 (Curse of dimensionality). In the realm of neu- ral network approximation theory, the curse of dimensional- ity refers to the phenomenon that as the input dimension d goes to infinity, the network size required to achieve a given approximation error grows fast or the approximation error grows fast when the network size is fixed. Corollary 4.2 implies that our approximation result suffers from the curse
Let M N4. We divide [0, 1]d into congruent hypercubes with side length 1/M, then get (M+1)) grid points. For any I E C 1]d) and any m E [M]4, its k-order Taylor expansion at the grid point m/M M is denoted by
PM. (m:)
Σ
80h ((m/M)
(x:
m
(14)
ad!
M
SR
6

---

Approximation to Smooth Functions by Low-Rank Swish Networks
where a No stands for multi-index.
Note that for two multi-indexes we say V 2 iff v3 2 ap for all i. Lemma 5.3 shows P% could be represented as linear combination of monomials x with |a] S Next we construct all monomials x" in network and Taylor polynomials at all grid points by linear combinations of x whose coefficients determined by Lemma 5.3.
Lemma 5.1. Let 3 E RR, 3 = K Y,K E N E (0, 1]. and R E R.. For all I E ((0, ((0,))), M E N,, m E [M]], and x E [0,1]]
[f(x)) PM (x)) S
K6 + d - d 1
R
m
3 (15)
M
First we show Swish network of depth and width can approximate the square function.
Lemma 5.1 shows I'm a good approximator around m/M, but the error cannot be well controlled when x is far away from m/ m/M. Next we introduce a technique proposed by (Gühring & Raslan, 2021) to deal with this problem.
Lemma 5.4. Let X > O. Then for all R. there exist & between O and X and & between 0 and - such that
Let T E IR4. Define om. an approximate bump function at the grid point m/M. by
2x² (p
+ +p( 2 +-(-)) (1) 11 5
(5) + (() x²
x
12
X2
(20)
4/ (x) == - ((((T(2 + 2)) - p(T(x + 1))
1
and
7
- -((T(x- - (()] + +(T((( 2)))
(16)
2x² (p 2
+ +0(-5)
x2
x
(21)
12x²
d
8 (x) = II NTT (3M 21
mi)
(17)
M/
Together with the polarization identity, we show a Swish network of depth and width 4 can approximate the multi- plication function.
The graph of o'mm looks like a bump at the grid point m/M- /..
Lemma A.7 guarantees that l (x) is bounded when xx-
Lemma 5.5. Let O. For all x. E IR,
m/M] 2 1/M and the bound goes to zero as r increases. By intuition, on can preserve the value of Ps. when æ is near m. M/M and eliminate the influence of Pn, when æ is far from m/M. Thus, for every grid point, we use the product of the Taylor polynomial and the approximate bump function at this point to approximate I around this point, then sum up the products at all grid points to approximate f on [0, 1]d.
2x
p
x ++
+ p
I + y
y1
2A
2X
2X
(22)
x - y
2X ))
6x
y2
+ yd
xy
S
12X2
8
Next we show a Swish network of depth and width can mimic the identity function exactly.
Lemma 5.2. Let B > 0,3 = E N,7 E (0, and Re E R4. For all f E (I. M E N.. 2 1 and
x E [0, 1j.
Lemma 5.6. For all x E R.
-2
p(x) p(-x) 1+e 1 e:= = r.
(23)
f(x)
Σ
Ps (z)
mE[M]a
(2 1100) )d= -1
Next we briefly describe how to construct monomials and Taylor polynomials in a network as depicted in Figure in Appendix A. The detailed construction is presented in the proof of Lemma A. According to Lemma 5.6, 5.4 and 5.5, we use nonlinear layer followed by linear layer to construct all 1st- and 2nd-order monomials from input vari- ables I1, I2,... Fd. Then we utilize another linear layer of width (M + 1)d linked to the previous nonlinear layer to construct the first two orders of Taylor polynomials at all (M+1)² grid points. The weights and biases are determined by Lemma 5.6, 5.4, 5.5, and 5.3. We concatenate these two linear layers in parallel as one which follows behind the nonlinear layer. Next, following the similar way, we con- struct all 2nd-, 3rd-, and 4th-order monomials from 1st- and 2ndeorder monomials via a nonlinear layer followed by a linear layer. And we use another nonlinear layer followed by linear layer to preserve the first two orders of all Taylor
S <6e R
2/10/11 1xx
(18)
[K+ [k+d- d - 1) K
d-1 10--))22el,
3'M-9
|12211001
6(M + Te (N: 1)
12211ee ((20)1'1l00))* )-
dd-1)
Step 2: approximating (P%) (MM by a low-rank Swish network P
Lemma 5.3. Let a Rd and ba R for all a E Nd with |ac| < K. For all x E Rdd
Σ la (x= a== 10/SSN
Σ xa
by
II
1/4 ai
(- ag pi-a
|a|
S
2aa
i=1
(19)
7

---

Approximation to Smooth Functions by Low-Rank Swish Networks
polynomials by Lemma 5.6, then connect this linear layer to the previous nonlinear layer which constructs 3rd- and 4th-order monomials to approximate the first four orders of all Taylor polynomials. Then we concatenate these two nonlinear layers and two linear layers respectively as one nonlinear layer followed by one linear layer. In the follow- ing steps, letting the initial value of be 2, we repeat the process until (Pm) M is completely constructed:
Table 1. Cross-validation results for classical feedforward net- works and low-rank networks on various classification (top) and regression (bottom) datasets. L represents the depth (i.e. the num- ber of nonlinear layers) of both networks and H. represents the width nonlinear layers of both networks.
ACC(%)
DATASET
L
H.
I-statistic
classical
low-rank
1. using a nonlinear layer followed by a linear layer to construct 2nd-, (27 1tth, and (2l + 2) th--orrrrr mono- mials from 2nd-, (21- 1)th- and (2()th-order mono- mials:
Iris
4
20
95.3 - 4.3
94.7 + 5.0
0.36
Rice
2
35
92.7 = 1.9
92.6 + 2.0
1.00
BankMarketing
2
188
68.9
+ 15.3
71.1 + 15.4
-2.01
Adult
2
540
85.8
0.3
85.8 0.3
-0.47
RMSE
DATASET
L
H
I-statistic
2. using another nonlinear layer followed by linear layer to preserve the first 2/ orders of (Pm) mm]jee then adding connections to the nonlinear layer which con- structs (21 + 1) and (2! + 2)th-order monomials to approximate the first 21 + 2 orders of )mmmm
classical
low-rank
RealEsttaee Abalone
4
30
.078 + .021
.077 1: .020
1.29
3. 50
.077 = .022
.077 + .022
-0.44
WineQuality BikeSharing
4. 78
.123 = .009
.123 - .009
1.21
4
.100
.036
.070 = .024
3.90
3. concatenating these two nonlinear layers and two linear layers in parallel respectively as one nonlinear layer followed by one linear layer;
P and G. By Lemma 5.5, we construct a nonlinear layer with width 4(M + 1)d and a subsequent linear layer with width 1 to multiply the output dimensions of P and G cor- responding to the same grid point respectively, and then sum them up. We denote the final network by nn which ap- proximates Eme[Mie Ps error and size of nm is showed in Theorem 4.1 and Theorem A.24.
4. letting / := +++2 2 and constructing the next nonlinear and linear layers by step 1 to 4 until (P% me is completely constructed.
(x))))) (z))5, (z)). The approximation
We denote the network constructed above by P: [0, 1jd - RM++))* The approximation error and network size is shown in Lemma A.17.
6. Experiments
Step 3: approximating (6 (6m)mejaje by a low-rank Swish network G
Our Theorem 4.1 shows that for classical feedforward Swish network with appropriate size, compressing each of its weight matrix of size H. x H to the product of two small matrices of size H x 3 and H.. 3 x H will not result in a loss of approximation ability. Here we conduct experiments to verify that the ratio 1/3 is safe.
is obvious that e (3M m. can be exactly con- structed by nonlinear layer followed by linear layer. Then, to construct o the key is to construct the product of d variables. For convience, we suppose d = 2 where q E N and denote (3MM(z. - M )) as z;. We approximate the mapping (2,,22..... Zd) + (2122, using a nonlinear layer followed by a linear layer according to Lemma 5.5. By applying the above way q times iter- atively, we get II 24. i.c. om. For all m E [M] we construct ohm, in parallel. We denote the network constructed above by G [0, 1]d + R(M+1)* The approximation error and network size is shown in Lemma A.22.
H
We choose eight popular UCI datasets, four of which are used for classification tasks and four for regression tasks, For each dataset, convert each category feature several dummy features, then scale all features to [0, 1]. For regres- sion datasets, we also scale the targets to jo, 1). Table 2 in Appendix B records the basic information for these datasets. Then, for each dataset, we employ grid search with 10-fold cross-validation to identify the optimal depth and width for the classical feedforward Swish network. The candidate set for the depth consists of {2, 3, 4), and for the width, it is (4d, 5d, 6d), where d represents the input dimension. Subsequently, we conduct 10-fold cross-validation to evalu- ate the classical feedforward Swish network of the optimal depth and width and the low-rank Swish network whose depth and width of nonlinear layers are the same as those of the classical feedforward Swish network and width of
Step 4: approximating Eme[M]a th by the inner product of P and g
Based on the constructive approximation before, we have that network P approximates (Px) m and network g approximates (mmm)_[M Considering that the depths of P. and G may be different, we construct several nonlin- ear and linear layers according to Lemma 5.6 to align their depths. And we still denote the two aligned networks by
8

---

Approximation to Smooth Functions by Low-Rank Swish Networks
linear hidden layers is one-third of that of nonlinear layers. In addition, perform dependent t-tests for paired samples on the cross-validation results.
References
Barron, A. R. Universal Approximation Bounds for Super- positions of Sigmoidal Function. IEEE Transactions on Information theory, 39(3):930 9405 1993.
The results of t-tests in Table indicate that on all datasets, classical feedforward Swish networks do not significantly outperform low-rank Swish networks. Conversely, on the BikeSna dataset, the root mean square enror (RMSE) of the classical feedforward Swish network is significantly higher than that of the low-rank Swish network. The exper- imental results indicate that the compression ratio of 1/3 suggested by our Theorem 4.1 is reliable.
Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y. M. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.19334 2020.
Boleskei, H., Grohs, P., Kutyniok, G., and Petersen, P. Opti- mal Approximation with Sparsely Connected Deep Neu- ral Networks. SIAM Journal on Mathematics of Data Science, 1(1):8-45, 2019.
7. Conclusion
Chen, X. and White, H. Improved Rates and Asymptotic Normality for Nonparametric Neural Network Estimators. IEEE Transactions on Information Theory, 45(2):682- 691, 1999.
In this paper, we establish the theoretical foundation for low-rank compression from the perspective of universal approximation theory. Specifically, we prove that for any Hölder function, there exists a Swish network with narrow linear hidden layers sandwiched between adjacent nonlinear layers, which can approximate the Hölder function within a given small error. Through our constructive approximation, we find that the width of the linear hidden layers is at most one-third of that of the nonlinear layers. This leads sig- nificant reduction: the number of multiplication operations occurring in all hidden layers except the first one can be decreased by at least one-third compared with a classical feedforward network having the same depth and width of nonlinear layers. Extensive experiments have confirmed the reliability of our theoretical result. This research not only enriches the theoretical understanding of low-rank compres- sion but also holds great potential for practical applications where computational efficiency is crucial.
Clevert, D., Unterthiner, T., and Hochreiter, S. Fast and Accurate Deep Network Learning by Exponential Lin- ear Units (ELUs). In 4th International Conference on Learning Representations, 2016.
Cybenko, G. Approximation by Superpositions of a Sig- moidal Function. Mathematics of Control, Signals and Systems, 2(4):303-314, 1989.
Denil, M., Shakibi, B., Dinh, L.., Ranzato, M., and de Freitas, N. Predicting Parameters in Deep Learning. In 27th Annual Conference on Neural Information Processing Systems, pp. 2148-2156, 2013.
Vore, R., Hanin, B., and Petrova, G. Neural Network Approximation. Acta Numerica, 30:327-444, 2021.
Dong, X., Chen, S., and Pan, S. J. Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon. In Advances in Neural Information Processing Systems, pp. 4857-4867 2017.
Acknowledgements
Jingyuan Wang acknowledges the financial support of Na-
tional Natural Science Foundation of China (No. 72222022, 72171013). Hongjun Li acknowledges the financial sup- port of National Natural Science Foundation of China (No. 72342032). Ke Tang acknowledges the financial support of National Natural Science Foundation of China (No.
Dubey, S. R., Singh, S. K., and Chaudhuri, B. B. Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark. Neurocomputing, 503:92-108, 2022.
Eger, S., Youssef, P., and Gurevych, I. Is it Time to Swish?
72192802, 72342008).
Comparing Deep Learning Activation Functions Across NLP tasks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4415 4424, 2018
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Eldan, R. and Shamir, O. The Power of Depth for Feedfor- ward Neural Networks. In Conference on learning theory, pp. 907-940. PMLR, 2016.
Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-Weighted Linear Units for Neural Network Function Approxima- tion in Reinforcement Learning. Neural Nenworks, 107:
3-11, 2018.
9

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Farrell, M. H., Liang, T., and Misra, S. Deep Neural Net- works for Estimation and Inference. Econometrica, 89 (1):881-23,, 2021.
conference on artificial intelligence, volume 37, pp. 4365- 4373, 2023.
Kenton, J. D. M.-W. C. and Toutanova, L. K. BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, volume 1, pp. 2, 2019.
Graves, A., Mohamed, A., and Hinton, G. E. Speech Recog- nition with Deep Recurrent Neural Networks. In IEEE International Conference on Acoustics, Speech and Sig- nal Processing, pp- 6645-6649 IEEE, 2013.
Gühring. I. and Raslan, M. Approximation Rates for Neural Networks with Encodable Weights in Smoothness Spaces. Neural Networks, 134:107-130, 2021.
Kim, J., Park, J.-H., Lee, M., Mok, W.-L., Choi, J.-Y, Y., and Lee, S. Tutoring Helps Students Learn Better: Improving Knowledge Distillation for Bert with Tutor Network. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pp. 7371 2022.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
Hinton, G. E., Vinyals, O., and Dean, Distilling the Knowledge in a Neural Network. CoRR, abs/1503.02531, 2015.
Ko, J., Park, S., Kim, Y., Ahn, S., Chang, D.-S., Ahn, E.,, and Yun, S.-Y. NASH: A Simple Unified Framework of Struc- tured Pruning for Accelerating Encoder-Decoder Lan- guage Models. In Findings of the Association for Com- putational Linguistics: EMNLP 2023, pp. 6076-6093, 2023.
Hon, S. and Yang, H. Simultaneous Neural Network Ap- proximation for Smooth Functions. Neural Networks, 154:152-164 2022.
Horel, F. and Giesecke, K. Significance Tests for Neural Networks. Journal of Machine Learning Research, 21 (22 2020.
Kohler, M. and Langer, S. On the Rate of Convergence of Fully Connected Deep Neural Network Regression Estimates. The Annals of Statistics, 49(4):2231-22 -249,, 2021.
Hornik, K., Stinchcombe, M., and White, H. Multilayer Feedforward Networks are Universal Approximators. Neural networks, 2(5):359-366, 1989.
LeCun, Y., Denker, J. S., and Solla, S. A. Optimal Brain Damage. In Advances in Neural Information Processing Systems, pp. 598-5988-9 9055600511 1989.
Hornik, K., Stinchcombe, M., and White. H. Universal Approximation of an Unknown Mapping and its Deriva- tives using Multilayer Feedforward Networks. Neural networks, 3(5):551-560, 1990.
Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. Multi- layer Feedforward Networks with a Nonpolynomial Acti- vation Function can Approximate Any Function. Neural networks, 6(6):861-867, 1993.
Idelbayev, Y. and Carreira-Perpinan, M. Á. Low-Rank Com- pression of Neural Nets: Learning the Rank of Each Layer. In 2020 IEEE/CVF Conference on Computer Vi- sion and Patern Recognition, pp. 8046-8056, 2020.
Li, Q., Lin, T., and Shen, Z. Deep neural network approxi- mation of invariant functions through dynamical systems. Journal of Machine Learning Research, 25(278):1-57, 2024a.
Jacob, B., Kligys, S., Chen, B., Zhu, M.. Tang, M.. Howard, A., Adam, H., and Kalenichenko, D. Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Omi Inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704 2713, 2018
Li, Z., Shao, B., Shou, L., Gong, M., Li, G., and Jiang, D. WIERT: Web Information Extraction via Render Tree. In Thirty-Seventh AAAI Conference on Artificial Intelligence, pp. 3316-133733 2023.
Ji, J., Zhang, W., Wang, J., and Huang, C. Seeing the Unseen: Learning Basis Confounder Representations for Robust Traffic Prediction.
Li, Z., Tucker, R., Snavely, N., and Holynski, A. Genera- tive image dynamics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24142-24153, 2024b.
Proceedings of the 31th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2025.
Lin, H. and Jegelka, S. Resnet with One-neuron Hidden Layers is a Universal Approximator. In Advances in neural information processing systems, pp. 6172-6181, 2018.
Jiang, J., Han, C., Zhao, W. X., and Wang, J. Pdformer: Propagation delay-aware dynamic long-range transformer for traffic flow prediction. In Proceedings of the AAAI
10

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Liu, C. and Chen, M. ReLU Network with Width d O(1) Can Achieve Optimal Approximation Rate. In Forty-first International Conference Machine Learning, 2024.
Schmidt-Hieber, A. J. Nonparametric Regression using
Deep Neural Networks with ReLU Activation Function.
The Annals of statistics, 48(4):1875-1897, 2020.
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved Knowledge Distillation via Teacher Assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp- 5191-5198, 2020.
Shen, G., Jiao, Y., Lin, Y., and Huang, J. Approximation with CNNs in Sobolev Space: with Applications to Clas- sification. In Advances in Neural Information Processing Systems, 2022a.
Shen, Z. Yang, H., and Zhang, S. Optimal Approximation Rate of ReLU Networks in terms of Width and Depth. Journal de Mathématiques Pures et Appliquées, 157:101- 135, 2022b.
Misra, D. Mish: A Self Regularized Non-Monotonic Activa- tion Function. In 3st British Machine Vision Conference 2020, 2020.
Ohn, I. and Kim, Y. Smooth function approximation by deep neural networks with general activation functions. Entropy, 21(7):627, 2019.
Shi, H., Tian, Q. Wang. J., and Cheng, J. LibEpidemic: An Open-source Framework for Modeling Infectious Dis- ease with Bigdata. In Proceedings of the 31st ACM In- ternational Conference on Information and Knowledge Management, pp. 4980-4984, 2022.
Opschoor, J. A., Schwab, C., and Zech, J. Exponential relu dnn expression of holomorphic maps in high dimension. Constructive Approximation, 55(1)5537-582 2022.
Shintani, M. and Linton, O. Nonparametric Neural Network Estimation of Lyapunov Exponents and Direct Test for Chaos. Journal of Econometrics, 120(1):1-33) 2004.
Petersen, P. and Voigtlaender, F. Optimal Approximation of Piccewise Smooth Functions Using Deep ReLU Neural Networks. Neural Networks, 108:296-3333 330, 2018
Shyu, E. and Caswell, H. Calculating Second Derivatives of Population Growth Rates for Ecology and Evolution. Methods in Ecology and Evolution, 5(5):473-482, 2014.
Ramachandran, P., Zoph, B., and Le, Q. V. Searching for Activation Functions. In 6th International Conference Learning Representations, 2018.
Siegel, J. W. Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces. Journal of Machine Learning Research, 24(357):1-52, 2023.
Ren, H., Wang, J., Zhao, W. X., and Wu, N. RAPT: Pre- training of Awae- Transformer for Learning Robust Healthcare Representation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3503-3511,
Telgarsky, M. Benefits of Depth in Neural Networks. In Conference on Learning Theory, pp. 1517-1539, 2016.
Ren, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T. Al- most Unsupervised Text to Speech and Automatic Speech Recognition. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5410- 5419, 2019.
Tolstikhin, I. O., Houlsby. N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J.. Steiner, A., Keysers, D.,, Uszkoreit, J., Lucic, M., and Dosovitskiy, A. MLP- Mixer: An all-MLP Architecture for Vision. In Advances in Neural Information Processing Systems, pp. 24261- 24272, 2021.
Rolnick, D. and Tegmark, M. The Power of Deeper Net- works for Expressing Natural Functions. In International Conference on Learning Representations, 2018
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. LLaMA: Open and Efficient Foundation Language Models. CoRR, abs/2302. 2023.
Safran, I. and Shamir, O. Depth-Width Tradeoffss in Ap- proximating Natural Functions with Neural Networks. In International Conference on Machine Learning, pp- 2979-2987, 2017.
Wang, J., Ji, J., Jiang, Z., and Sun, L. Traffic Flow Prediction based on Spatiotemporal Potential Energy Fields. IEEE Transactions on Knowledge and Data Engineering, 35 (9)9073-90877, 2022a.
Sainath, T. Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B. Low-Rank Matrix Factorization for Deep Neural Network Training with High-Dimensional Output Targets. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6655-6659, 2013.
Wang, O. and Werning, I. Dynamic Oligopoly and Price Stickiness. American Economic Review, 1 112(8):2815 2849, 2022.

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Wang, Z., Pan, Z.,, Chen, S., Ji, S., Yi, X., Zhang, J., Wang, J., Gong, Z., Li, T., and Zheng, Y. Shortening Passengers' Travel Time: A dynamic metro train scheduling approach using deep reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 3555):5282-5295 2022b.
Xia, M., Zhong, Z., and Chen, D. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408, 2022.
Yarotsky, D. Error bounds for approximations with deep relu networks. Neural networks, 14:103-114, 2017.
Yarotsky, D. Optimal Approximation of Continuous Func- tions by very Deep ReLU Networks. In Conference on Learning Theory, pp. 639-649, 2018.
Yarotsky, D. and Zhevnerchuk, A. The Phase Diagram of Approximation Rates for Deep Neural Networks. Advances in Neural Information Processing Systems. pp. 13005-13015, 2020.
Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., and Kumar, S. Are Transformers Universal Approximators of Ceateenceto-SSqqenne Functions? 8th International Conference on Learning Representations, 2020.
Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Al- berti, C., Ontañón, S., Pham, P., Ravula, A., Wang. Q., Yang, L., and Ahmed, A. Big Bird: Transformers for Longer Sequences. In Advances in Neural Information Processing Systems, 2020.
Zhang, S., Lu, J., and Zhao, H. Deep Network Approxi- mation: Beyond ReLU Diverse Activation Functions. Journal of Machine Learning Research, 1--35):1-39, 2024.
Zhong, Q., Mueller, J., and Wang, J.-L. Deep Learning for the Partially Linear Cox Model. The Annals of Statistics, 2022. 2022.
1348)1334
12

---

Approximation to Smooth Functions by Low-Rank Swish Networks
A. Technical Proofs
A.I. Approximatng f E ((0, 11 by EmE[Ar] P T
First we prove Lemma 5.1 which shows the approximation error a Taylor polynomial at a grid point.
Proof of Lemma 5.1. By Taylor expansion theorem, there exists fm E [0, 1] for all E [M] such that Vx E [0, 1]a
[f(x)
Pm (x)1
Σ
001
m
+
Em
m
(xr
-m/Maaa
0
m
(x
m/M)
M
M
ad!
M
ad!
8a
f
m
+
5m
x
m
-
ja
m M
M
M
ad!
a
<
R
5m
m
1(x
m\
(because f E ((.,(((011)))
(24)
M
m M
<
R
x
na x
d - 1)
m72
R
d 1
M
Next we show the boundedness of PM, which is used to prove Lemma 5.2 latter.
Lemma A.1 (Boundedness of P% For all x E [0, 10,
K 1 t -- d-1
m 3 M 00
PK (a:) S
R.
(25)
Proof of Lemma A.I. For all x E [0, 11 by Lemma 5.1,
Ph (x)) S P (zx) f(z)) + lf(x))
+ d 1
(26)
d-1 1 æ
R.
M
00
Next we show some important properties oh which are used the proof Lemma A. 17. Lemma A.2. Vr 2 1, (p'xx) - S 3.
Proof of Lemma A.2. Vx 0,
1te + IC
(d))-11
(1 + c e=
-2- 20
(1 e e=²
5 6 x + re x + 2m
< 3re x
Lemma A.3. Vr S 11, 1,)()))) S -3re.
13

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Proof of Lemma A.3. Vx < 1,
+ 6 +xe
le((z)
(1 + 1
+ E - + xe -
e=2x 2x
5 2x + e² ze""
1 -z0e".
Lemma A.4 (Boundedness of m. Let E R. For all Rd
1 (a)) z (211)'11=0))*
(27)
Proof of Lemma A.4. For all x E Rd, by Lagrange's Mean Value Theorem, there exist Ex, Si E [1,2] (i = 1,2, ..................... d) such
that
1 (a:) = II R
33M
m1 mi
M )
(28)
II o' (3MT 1=1
(u::
+ EST)
-p' (3Mr (x:
m;
M 617
M
$ (2112'l1bo) a
Lemma A.5 (Locality of o part I). Let r 2 1. 1f M my 2 1
M for some j E (1,2....,dd)
(0) S
Gre
(29)
Proof of Lemma A.5. By Lagrange's Mean Value Theorem, there exist &is Si E [1,2] (i = 1.2..... such that
Imm (x))
II i=1
2it 3M (I:
m)) mr))
II 0' (3MT (m: mi
+ G.T
- P (3MT (zi m4) M
- -siT)
M
(30)
< (2]1e' (0ol"ll>))) ) po (3MT [zzg
my
+ EST) - pl (3MT [z) - my
- -GiT)
M
M
< 12=) d-1 |0' (3MT (Ig
my
- 1 -1|+| - pp (3MT [jj
- ()7r))
+ {TT
M
M
Because I2
my M
2y and 2 1, we have
my M
m2 M
3MT
Tj
+ EGT 3MT
37 -
(31)
Together with the fact that re x decreases monotonically on [1, +o0), it follows
T (x) < (211p'll )l0) )d (1e' (3MT (xz
my
1 + |||1- p¹ (3MT (zg - my
M (TT)
+ +
M
< (211p' 100) )211 (3MT [x.
mj
{gT)
-33M ++5))
M
(32)
3 (3MT [x)
my
(57)
(3MM7 (xs 71 )-(r))
(by Lemma A.2)
M
< <211p/ |10) d-1 Gre
14

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Lemma A.6 (Locality of on part II). Let 1f my S 3/4 for some j (1,
1 (x)) - (2)1p'11=0) 6
(33)
Proof of Lemma A.6. By Lagrange's Mean Value Theorem, there exist Sis Si E [1 (i = 1,,,...... .,2, such that
d
1 (x)]
II
2/7
3M
(I;
m.
)
M
1=1
I p² 3MMT (x, m4
+ EST)
pp' (3MT (2, m4
-GoT)
M
M
(34)
(2]1e'll [000) p0 (3MTT (xg
my
+ EST)
pp (3MT [zj - my
-M -G,T)
(2)1)')l>0)) 1-1 (10 (3MT (Iz
my
+ STT
-1 1
-1 + 1-p- p² (3MT Zjj
m
<5T
Because
my M
S Nf and r 2 1, we have
3MT
[z
my M
CGT S 3MT (1z
M2 M
+ Egt S -37 + EnT - S -1.
(35)
Together with the fact that -re" increases monotonically on (-oc, - -1]
J it follows
1 (x)] S (211p'11=0) d-
(p' 3MM (Iz
my
+ ExT
- 1
-1|+ 1|+ - D (3MT (zz
my
(5T)
M
M
(211011l)))
3 33MT (xz
my
+ SST)
M
(36)
-3 -3Mr
833
m2
(57) <3M-(a)- 70 (2)-c11 )-(r)
(by Lemma A.3)
M
s (211p'1l>0) )d11 Gre
Then the following Lemma A.7 follows directly from Lemma A.5 and Lemma A.6.
Lemma A.7 (Locality of 6T.) Let 1. If x3
my M
as for some f1,2,... .).
10% S (21le' 11x)) d- 6re
(37)
Lemma A.8 (Partition of unity property of Φ.)). For all a E [0. 1|d.
(2)1 )dod'llo)dd - 2/ loc
Σ om
Gre
(38)
mc[M]]
Proof of Lemma A.8. Let r 2 1. For all x E [0, 1]d, by Lagrange's mean value theorem, there exist & G
[1,2]
15

---

Approximation to Smooth Functions by Low-Rank Swish Networks
...,...; d) such that
Σ
Pm (2))
mE [M]a
E mE[M]
II
33M (0+ m,
M m4
)
3M [z
=,=0
P33MT(
M + - P(3M77(; m. ))+ + p(3MT( M) - p(3MT(a 3/ -2))
Σ
m,=0
P3MTI, + 2T) - p(3MTx, + -M33MTra 1) - 7) + p(3MT((x - -1 2T)
(d' (3MTT,+ p (33MT(I - - G(T)
1=1
By the inequality that
21 + 2132
im1
1m 1
2m
(39)
< 11 ril + | x2
T2
and the fact that 3MTx; + ExT 2 + 2 1 and 3MT(2; - 1) - SiT S - -1 1......, it follows that
Σ (x))
mEIMM
1 P'3MTax + -1
+ - - - (1 p333Mrr + (2T) + (3M 1) - (2T) + +..
p3(MMT + - - 1) -()) 1 p'33Mra EaT) p3M33(3ra 1) CaT)
d=1
< 6re-T. -1 211p 11 + +... (21lo(1)))))
(by Lemma A.2 and A.3)
(2 lxx) -1
p/ 1000 -1
At the end of this section, we prove Lemma 5.2, that is, Emelary Pr approximates f E C3.R 1]9).
16

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Proof of Lemma 5.2. For all E [0, 1j4.
[(x) - E Pm (x) (x)
mE[M]
$ f(x)
Σ f(z))
Σ f(x)
Σ
Pm (x) (x)
mE|M
me(
meiM
(2|1p/ 100
6re
[
Σ (f(x)) PM (z)) om (2t)
(by Lemma A.8)
p/ 6o
mEiM
(2110/1 -1
6Te
R +
Σ (f(z) Pm (2)) 8im (x)
2|1p'11a
mE[M]
For the second term,
Σ mE(M]
(f(x) - Pm (x)) on (x)
Σ
(f(x) Pm (x)) (x)
+
Σ
(f(x)) - Pm (az) S (2))
mEIMI" ||0 32 S.
mEIMI"
| 37 11
K + -
RM-
Σ (x))
+
( 1x+d-1) +
R
Σ mE(M)
on (x)
(by Lemma 5.1)
d 1
d-1
ME
||0 7E 10
| 77 loo T
+ d - 1
[ (x+ d
1-1 RM R3M-83222110/ 1100)//0110000 +
d-1 R ((M 11 2nd) (211 - (by Lemma A.7)
K + d - 1) R(2110'11>)) + 6(M + re 1)0 ( 1) d 1
< 3d M
d-1 1(221011100)--1
A.2. Approximating (P% m by Low-Rank Swish Network P
We first prove Lemma 5.4 which shows how to approximate the square function with Swish network of depth 1 and width 2 and Lemma 5.5 which shows how to approximate the multiplication function with Swish network of depth 1 and width 4.
Proof of Lemma 5.4. For all E R, by Taylor expansion theorem, there exist & between O and and Ç between 0 and such that
I = p(0)) + (0())
p"(0)
2
TH (0)
3
(44 (E)
4
X +
(40)
21
3!
X
4!
X
and
F p(0) p'(0) (0
22
(C)
4
p
(41)
2!
X2
3
X
This, together with p(0) and p²(0) = 2.2 implies that
(())(E) + (C)
x 2xa (p (
+ + x²
(42)
12
X2
17

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Then, by the fact that |lc S 1, follows
2xa 0
I
2 )
=²²
211
100
x
+
(43)
12A2
12X2
Proof of Lemma 5.5. By Lemma 5.4, there exist & between 0 and <1 between 0 and
2t
&2
between 0 and 2-2
and (2 between 0 and
-n 2x
such that
2x
I I +
2 +
I -- y
x - y
0
-
xy
21
2A
2A
2x
pl (E1) + (()
I x+D y
(E2) (2) + p()) (C2)
I
12X2
12X2
2
2110 00
( ( y/
+
2 211p 0c
I
(44)
12X²
2
12X²
4
x y
12X
2
x²
6a 6x²y1 1
+ y
12X²
8
For convenience, we denote
id(x) == p(x) - p(-x),
sq(x)= = 2X² (p (())
())+))))) (-1)),
+
and
mult(x) 8q
( I + 2
21² (o
( I +
y/ 2x
sq
p
2 p
2X 0
2X
Obviously, id can be implemented by network of depth 1, width 2, number of nonzero parameters 4, and maximum absolute value of parameters 1, sq can be implemented by network of depth 1, width 2, number of nonzero parameters 4, and maximum absolute value of parameters max (2X² }). and mult can be implemented by network of depth 1, width 4, number nonzero parameters 12, and maximum absolute value of parameters max(2X² 2A }-
Next we want to construct monomials by stacking id, 8q, and mult. To prepare for the subsequent approximation error analysis, we show some conclusions about the output ranges sq and mult.
Lemma A.9 (The output range of sq). Let > For all x E IR,
0 < sq(x) < 2?
(45)
Proof of Lemma A.9. (I), We first prove 0 < sq(x) for all x E R. The derivative of sq is
se (x) = 21 p'
4 / (-
1 lte + fe
+ e - fea
22
(1 + e x
(1 + ex
2r # + X E -
21
(1 (1tex) ex
18

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Let g()) = xxx + e 1. For all x E I, the sign of sz)(x) is consistent with that of g(x), because
0. When
(
< 0, g(z) < x -1 1 < e - 1 0. when >> 0, g(x) > e 1 > eº - 1 = 0. Therefore, sq is monotonically decreasing on (-oc, 0) and increasing on (0, +o0). It follows that for all x E R, sq(z) 2 sq(0) = 0.
(2). Next we prove sq(x) S x²2 for all x E R. The derivative of x²² - =s((z) is
d dr (x² =s(()) = 2x - 212. e + _ - (1 + e
2x + 2xe 3x
2xe 2x +
(1+ei)²
Let h(x)= ;- 2x + 2xe 3- 2 + 2X. Then the derivative of h is
4x 2e
= =2 2 2e xx4 +
e x -2 xx
A
and the 2nd derivative is
1²(x) = & 25 X ex
When x < 0, (x) < O; when x > 0, (z) > 0. Therefore, h' is monotonically decreasing on (-oo, 0) and increasing
on (0, +.x). Then for all x E R, H'(x) 2 10( = 0. It follows that h is monotonically increasing on R. By the fact that
1(0)= = 0, we have h(x) S 0 when x < 0 and h((x)= 2 0 when x > O. It implies that x² sq(xx) is monotonically decreasing
on (-o0, 0) and increasing on (0, +oo). Finally, we have x² - =s(x)) N 0 sq( = 0 for all x E R.
Lemma A.10 (The output range of mult. part D). Let >> 0. For all x,y 2 0, 0 S mult(1, v) S ay.
Proof of Lemma 10 From the proof of Lemma A.9, we know that both sq(x) and x²2 sq(x)) are monotonically increasing on (0, +oo). Therefore, for all x, y O, when y 2 O, since x + y 2 y 0,
+ 1
I y/
x + y
80
8Q
- mull(x, y) = 89
89
2 0,
2
2
2
2
x + y
x + y
σ y
y
2
- 89
2
2
2
2
2
2
x + y
I V
x + y 2
y
sq
8q
xy + mult((z, v) S xzy
2
2
2
and when x- y S 0, since x + y 2 y
x
220 0.
+ V 2
y
y x
sq
3q
sq
8q
2 0 = mult(z, y) 2 0,
2
2
T + y
I +
-
x
I
I y
80
80
=
88
2
2
2
2
2
2
2
2
x + y
x y/
x +
y
J
y
sq
8q
<
= xy = mult(1, y) S zzy.
2
2
2
2
Similar to the proof of Lemma A. it is easy to prove the following three lemmas.
Lemma A.11 (The output range of mult. part II). Let X > O. For all x < 0 and 2 0, xy S mull(z, v) S 0.
Lemma A.12 (The output range of mult. part III). Let X > 0. For all x. S 0, 0 < mult(s, v) S ay.
Lemma A.13 (The output range of mult. part IV). Let > O. For all x 2 and y S 0. xy S mult(s, v) S
19

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Combining Lemma A0 A.1I, A.12, and A.13, have:
Lemma A.14 (The output range of mult). Ler O. For all x. y R. |malt((r y)1 S |ay]-
Next we define a series of functions Ma where a N by stacking sq and mult. These functions will be implemented by hidden neurons of P that is network outputting Taylor polynomials all grid points. For any No. the function Ma is defined by:
1. when |o| (a))) ) x=:;
2. when | = 2 and
a = 1 such that a 2a', M(()) == (Ma
3. when |x| = 2 and
3 3/o| = |a"| = such that = a' + a" and a' + a", M(x) == (x)
4. when [ 4 and
3 a'l 2 such that a 2a', M(x) MM (z)):
5. when [ = 4 and
a = |a"l = 2 such that a = + a" and a' + a" M(xx) ==
6. when |a| 2 5 3 and 3|o/ 2, o |a| laa"| such that a' + M(x) == mull(MMa (x)).
Next show the upper bound of Ma on [-1, 1] using Lemma A.9 and A.14.
Lemma A.15 (The upper bound of [Mal)). Let and a E Nd. For all [_,,
(M
(46)
Proof of Lemma A.15. Here prove it by mathematical induction. When a 1,
[M (a)) = |x 1 1.
When [x| = 2 and 3|o'| = such that a - 2a', by Lemma A.9,
[Ma [sg(Marr (z)) & (Mar(x))² $.
When [x| = 2 and
3|a'| = |a"l such that a = a' a" and a' 7 a", by Lemma A.14,
[Ma(x)] |mmult(Ma (x) Ma (z)) < Ma (x) Mar (x) S
When a and 3/01| 2, |a"l ||ax - laa" such that a' a", by Lemma A.14,
[Ma())] |mult(MMar (x), Mar (x)) (a) Mar <
When ax| = 4 and 3|0²| = 2 such that = 22a' by Lemma A.14,
MM (x)) = [sg[Mas (x)) I (Mar(x)) s 1.
When [x| = 4 and 3|o'| = |a"l = such that a = a' + a" and a' + a", by Lemma A.14,
IMa (x:) = (mull((Ma/ (x) (x)) [Mx (w) Ma S
When a 2 5 and 3|0'| 2, aa =|ox| |aaa| such that a a' + a", by Lemma A. 14 and induction,
MM (x:) (ma (Mar (x) Ma) < 1.
The following lemma shows the enror of Ma( to approximate a measured by sup norm on [-1,
Lemma A.16. Let A and a E Nd. Then for all E [-1,1]²
Ma(x) x 1 1
(47)
12X²
20

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Proof of Lemma A.16. Here prove it by mathematical induction. When ao] 1,
[Ma - = xa = O. When a and 3|a'| such that 2a'. by Lemma 5.4,
(M x² = (g((Mr(a))) _x2xx |sg(xx 2,20x 1s x
1
S
12X²
12x²
When ax| 2 and 3|//| =
= I that = a' a" and a' & a", by Lemma 5.5,
[Ma(z) σ Imult(Ma-(zz) (a), (x)) - xa xa" |mult( x²²) xa xx"1
nda' + 6a x2a" w
12X2
8
12X²
When ac| = 3 and 3|0'| = 2, a01 |a| - oa" such that a = a' + a", by Lemma 5.5 and A.15,
Ma(xx) - 1
Imult(Mar Ma (x)) x x0a1
S |mult(Marr (Z) (z)) - Mar(x) (eo2x)Maall Ma)( (Mar(x)) MMa (x) - xa'1 +|/0011 (|Mar((x) xa
1
1
+0
12A²
12x
2
12X2
When a = 4 and 3'''| = 2 such that a = 2a', by Lemma 5.4 and A.15,
Maaxx) 1
< lag(Mar (x)) (Mar ((Mar 20
+ +|Mar (Mar((r) - a + | (MMar x
1212
s 12X²*
When ax| 4 and 3|0'| la"l 2 such that = a' + a" and a' + a", by Lemma 5.5 and A.15,
Ma(x)) - x
mull((Mor Ma) (x)) x x
s Imult(Mar Ma) Mar(x)) Man(z)) (ar(xx) (M) (Mrr()) (Ma( - xa'++ ||/20/ (x) - xa
S 12A2
+
12x2
12X²
3
12A2
When ax| 2 5 and 3/a'l 2, 1ala""1[0a =la|- - lla" such that a'' a", by Lemma 5.5 and A.15 and induction,
(a(x)) - x
Im (ar),
(M),Maa Mar (x)) nd x==
S Imult(Mar (2t (x))) (M.(x)) Ma)((x)) + (M) (Ma (x:) - a |||001 -|Mar xa
1
|o'| -1
la" -1
+
12A2
12X²
12X2
a -1
12X2
21

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Next prove Lemma 5.3 which can provide the coefficients of monomials in Taylor polynomial a point.
Proof of Lemma 5.3. The term æa in the expansion of Eje Sx (a a) can only come from terms (æ: a) with 2 a and |o| S K. The coefficient of the term xº in the expansion of (ær a)" is Il= (c:) (-a;)<<-a By summing
up coefficients from all terms (2 - a)" satisfying 2 a and | I K, we obtain that the coefficient of x is
Σ v2a by II== (-a))) MI
(xr*) 40
1000 100
A
(x** A/0
(1r1 14
M
3
A
(x*
(x 1e..
(x*
X
first two orders of Pa) )
first four orders (F" )
first six orders of (P* hanur
Figure 3. An example constructive approximation for Taylor polynomials (Pr) (Pm)m by network P where x = 8. "A'" stands for input neuron, "O" stands for nonlinear neuron, and "O" stands for lincar neuron. The neurons divided into several groups by the dashed ellipses. Except for the input neurons, the number of neuron marks does not represent the actual number of neurons. (c"))))==) refers to all monomials of order and (P% me(Mja refers to Taylor polynomials of order all grid points.
Lemma A.17 (Neural networks approximate (P Let 3 all f E ((o, 1jd), M E N,, and À 2 2- 3, letting (P%, (Pm)mE(M (m/M m E [M]*], there exists a low-rank Swish network P
7,8 N, (0, and R RR. For m be kth-order Taylor polynomials of J at all grid points
[-1, 1] + R(M+1) with depth
K
(48)
2
width of nonlinear layers
dd11 d-1
d (d+A k - 2
d - (
(d+k- 11+k-1)
2 +
+ 4
+ 1)4
(49)
-1
width of linear hidden layers
d+1
d d
+ dd 2)
)) + (M 1)
+
d - (
(50)
d-1
d -1
upper bound of absolute values of parameters
d
max
2X max la|sk
Vi
21
(51)
v!
v2a M
1=1
22

---

Approximation to Smooth Functions by Low-Rank Swish Networks
and upper bound of number of nonzero parameters
K
d
12
d (d+1-1) (M + 1)dd
2d 2d 4
K
d [d+111)
(52)
2
d-1
d d-
2
d d-1
1=2
such that
(1))
(P(x) (P) mENN))l
Σ
(lal-1) 1) Σ
R
(
(53)
12X 25 a Sx
|
for all x E [-1,
Proof of Lemma A. 17. Let f E C(O. 119), M E N,, and X 2 2
If 3 I 1, then 0. For any m E [M] P. (x)) (mm Then we directly build a linear layer the output layer without any connection to the input layer and with bias (f(m/M))
We call the above network P. Obviously, P(x)= = (P% (z)) me[M]] for all [-1, and P is of depth (i.e. no hidden layer), number of nonzero parameters no more than (M + 1)² and maximum absolute value of parameters no more than R.
If B 1, we construct network layer by layer. In the following proof, we do not specifically mention the slight differences in the construction method when is small or odd, but the way to handle these is quite naive.
Step 1: constructing the first nonlinear and linear layers
By Lemma 5.6, for each input variable Ii, we construct two Swish neurons followed by one linear neuron to exactly preserve the value of vi. We arrange neurons for preserving all x, in parallel and thus obtain a nonlinear layer of width 2d followed by a linear layer of width d. The total number of nonzero parameters of these two layers is 4d and the maximum absolute value of parameters of them is
Meanwhile we approximate each square term 2/ with two Swish neurons followed by one linear neuron by Lemma 5.4 and each cross term 2,J, with four Swish neurons followed by one linear neuron by Lemma 5.5. By arranging neurons for approximating all x² and z,Zyy. we obtain a nonlinear layer of width no more than 4 (d followed by linear layer of width no more than (d-1) because the number of all 2nd-order monomials is (d+)). The total number of nonzero parameters of these two layers is no more than 22(+1) and the maximum absolute value of parameters of them is 2X² since À 2 2- implies 2X2 > 1 2 - It is easy know that given input x E [-1, 1]d the outputs of the linear layer equal to Ma (x)
(d-1)
with a 2. So by Lemma A.16 the approximation error for 2nd-order monomials is bounded by 12x
Next we construct a linear layer of width (M + 1)d to approximate the first two orders of (P% (Pm)melMy For each [M]d the linear neuron, approximating the first two orders of Pk links to two Swish neurons preserving xa 8" ((m//M
with weights Σ 12a
M II (*) m. ) and - Σ v2a
fmm(m/M) w!
I[ (2)) my yiia for all |a| = 1,
ml
M
two Swish neurons approximating the square term xa with weights 2X Σ v2a
8" f(m//M II (u)) (-
m: ) M
w!
and 2X² Σ E2za P
f(m/m II (2) y for all |al = 2 satisfying a = 2a' where |a'| = 1, and
M!
v2a four Swish neurons approximating the cross term xa with weights 2x²Evze
8" fim/M)
I (c)) (-
m.
w!
M
| Sk
2X Σ u2a
8" f(m/M)
I (o)) (- m. )
-2X Σ u2a
8" f(m/al) M) v!
I (M)
my au
and
wl
M
M
-21 Σ uva
8" ((mm)) I (e)) (- my ju: for all |a|| = satisfying a = a' + a where |a'| = |a"l =
v!
and a' A a'. In addition, to construct the Oth-order items, we add the bias term Ejr|sh 8" f(m/M) [- mi M ) for the
wl
linear neuron approximating the first two orders of Ps for all [M]] Therefore the total number of nonzero parameters
of this linear layer is no more than (M + 1)d 2d + ( + +1) and the absolute values of parameters are no more than
2X² max Σ U2za R II (c)) By Lemma A.16 and 5.3, we notice that our construction to the Oth- and 1st-order
S
23

---

Approximation to Smooth Functions by Low-Rank Swish Networks
terms is errorless, so the total approximation error to the first two orders of (Pm) memme only comes from the 2nd-order terms which is no more than T2X 22 u2a
R 1= (x))
Finally, we concatenate two nonlinear layers and three linear layers respectively, obtaining nonlinear layer of width no more than 2d + 4 (d=1) followed by a linear layer of width no more than d+ (d+1) + (M + 1)9. The total number of nonzero
parameters of these two layers is more than 4d + 12(2=1)
(M + 1)dt 2d + ( + and the absolute values of
them are no more than max { 2x²; 2X max 52 Euza II (3) The approximation error for the first two orders |
of ( (P mm by the outputs of the last (M + 1)d linear neurons is no more than 1 12X²
Eial=2 E Euza R [NiSK
v! IT (+)
Step 2: constructing the second nonlinear and linear layers
The method to construct the second nonlinear and linear layers is similar to the above.
We use a nonlinear layer of width 2(++)) followed by a linear layer of width (d=1) to preserve the approximate 2nd-order monomials constructed before. Mcanwhile we multiply the 1st-order monomials with approximate 2nd-order monomials to construct approximate 3rd-order monomials using a nonlinear layer of width 4(2-1) followed by a linear layer of width (Z=1). In the meantime, we square approximate 2nd-order monomials and multiply different approximate 2nd-order monomials to approximate all 4th-order monomials using nonlinear layer of width no more than 4 (d=3) followed by a linear layer of width no more than (d-i)- Simultaneously we employ a nonlinear layer of width 2(M + 1) followed by a linear layer of width (M + 1)d to preserve the approximation of first two orders of (PM) m then connect the linear layer with the previous two nonlinear layers used to approximate 3rd- and 4th-order monomials to approximate the first four orders of (Pr))vrr mm(M)
Finally we concatenate four nonlinear layers and four linear layers respectively, obtaining a nonlinear layer of width no more than 2((+)) + 4(5t)) + 4(2+t) + (M + 1)d followed by linear layer of width no more than (d=1) + ( (d-1) + (M + 1)8. The total number of nonzero parameters of these two layers is no more than 1(d-1) + (2() + 12 (2-)) + (M + 1)dd (1+ ((=)) + ((+))) and the absolute values of them are no more than
max 2X max $4 Σ vza v! Il (*) } Mu
The approximation error for the first four orders of (Pr)mette by
the outputs of the last (M + 1)d linear neurons is no more than 12X² Eeslals - -1) E>a
w! I (*)
MMsK
Step 3: constructing the (I + 1)th nonlinear and linear layers by induction
Let E N+ with / 2 2. Now suppose that we can directly obtain approximate 2rd-, (2l 1)th, and (2l (2))th-order
monomials and the first 2/ orders of (P%, meEM from the last linear layer. The last nonlinear layer is of width no
more than 2( + 4( ((421- + 4 (d422-1) + 2(M + 1)d and the last linear layer is of width no more than (d+1) +
d-1
(1-19) + (d+211) + (M 1)dd The total number of nonzero parameters of them is no more than ( 12 d-1 +
24
12( 1) + (M + d)d(( (1 1 (1 22 + 4 d-1 and the absolute values of parameters of them are no more than
max 2X²
2X max 1 521 Σ
R v>a PISK
li (xt) The approximation error for the first 2/ orders of (P%) Pimmm(Ma
by the outputs of the last (M + 1)d linear neurons is no more than 12A E2s al 1)
R D!
d (*).
Then, following the above way, we use a nonlinear layer of width 2 followed by a linear layer of width (2+1) to preserve the approximate 2nd-order monomials. Meanwhile we multiply approximate 2nd-order monomials with approximate (2l 1)th-order monomials to construct approximate (21 1)th-order monomials using a nonlinear layer of width ) followed by linear layer of width (d+21) and multiply approximate 2nd-order monomials with approximate (2r)eh-rrder monomials to construct approximate (21 + 2h-h-rrr monomials using a nonlinear layer of width 4 ((d+22+1) followed by a linear layer of width (4+21+1). Simultaneously we employ a nonlinear layer of width 2(M + 1)d followed by a linear layer of width (M + 1)d to preserve the approximation of first (2l) orders of (Pm) me(ara then connect the linear
24

---

Approximation to Smooth Functions by Low-Rank Swish Networks
layer with the previous nonlinear layers used to approximate (2/ 1)th- and (2l + 2)th-order monomials approximate the first 27 orders of (m. m)mmje]]
Finally we concatenate these four nonlinear layers and four linear layers in parallel respectively, obtaining nonlinear layer of width no more than 2(d=1) + 4 (d=1) + 4 ((-=)))))) + (M + 1)d followed by linear layer of width no more than (d+1) ( + (d+21+1)
d-1 + (M + 1)4. The total number of nonzero parameters of these two layers no more
than (2=1) + 12 d-1 + 12("*211) + (M + 1)ah 4 (2=1)) + + a(<=211)) and the absolute values of them are
no more than max 2x² 2x max 1 Σxz J1 II (
The approximation error for the first 27 +
M
orders of (Pm)me melssa by the outputs of the last (M + 1)d linear neurons is no more than 12X Eaglalsa42(laa
1) vZa 5 III (.)) M
Through the above process, we finally construct a network of depth grid points, (Px (m with error no more than
[
called P. to approximate Taylor polynomials at all
R
Σ (la) - 1) Σ
12A2
v
a6
25/a/sk
D2a SK
i=1
Considering that there is no need construct 2nd-, [2 1)th- and (2 th-order monomials the last linear layer, the maximum width of nonlinear layers is no more than 2(d-1) + d-1 + a("+==)) + 2(M + 1)d, the maximum width of linear layers no more than (d-1) + (d+x-3)
( d-1
(d+k-2) + (M 1)d the absolute values of parameters are no
more than max
2x max SS
Σ U2a
R
I (4) 2x and the number of nonzero parameters is no more than
o!
MISK
4d + 4 ([
(++1) 12 (dti-)) + (M 1)dh (1 2d + ([ -1) 1) - 4 4 (())))).
A.3. Approximating (eM)aa by Low-Rank Swish Network g
We first define series of functions prod, where r E N4 by stacking mult, then analyze the error of approximating (#1,...,z,,) + H. x; using prod,. When constructing me in the proof of Lemma A.22, we can that prod, is implemented by hidden neurons of network G for E [d]+. For all N+, the function prod IR" - IR is defined by
1. when r = 1. prodi (x) ;=
2. when r 2 2, 29+1
.......,,2..)) = mult(prodz. (21,..., .2, prod, (#2v41....,,8) ..,,) where g E N and 2
Lemma A.18 (Boundedness of prod,). Let N4. For all x E R",
(prod,(:))
mll.
(54)
Proof of Lemma A.18. We prove this lemma by induction. When 1, for all RR
|prod (æ)l |a]].
Assume that Va E RR., (prod,(x))) II [2s] holds for all S Then, letting g E N satisfying <r +++1 15 20+1 for
25

---

Approximation to Smooth Functions by Low-Rank Swish Networks
all IR++1
ppood- (a) |muul(prodzd (Z1...., x29) prod, (F (I. 2r-1))
pprod2 (11,... (12-)))2200) prod,+ 1 1-20 (120+13-: Fr+11
(by Lemma A. A41
22
++
(by induction)
24 +1
[
Lemma A.19. Letr E N4 and X Rt, For all [-1,1]],
prod, (x)
II
T2
r
(55)
12x²
Proof of Lemma A.19. We prove it by induction. When r for all x [-1,
pprod (2) 0.
(56)
Assume that Vo [-1, 1]", Iprod, (a:) II. x.1 12xt holds for all S Then, letting q E N satisfying
<r+15
20+1, for all E R by Lemma 5.5 and A8,8,
++
prod, +1 (x)
II x;
(mull(proddd (z)...... x24) prod,+1-20
prodd (T22 Fr441)
prod2 (I), (1,...,,22))
r+
prodzs (#1...., x20) prod,41-20 (Z20
2r xr+1)
i=1
2
+
prodzo (1 (II,..., .......,,2........... 2244
+
(21,
x22
Fi
12X²
1=1
im 24
29
++ 1
24
+
(by induction)
12X²
12X²
12X²
r r-1
12X2
Next we introduce several lemmas which are helpful to explain how to approximate (Ø mmm)me using a network. Lemma A.20. For all n E N
2 2n.
(57)
Proof of. We prove it by induction. When n = 1, = 1. Yn E N_, if 2d 2 2n, then
21 = 2. 2 2- 2n 2(n + n) 2 2(n + 1).
Lemma A.21. For all d, M. E N4 and q E N, if 2 29, then
d
(M + 1)d² 2 (M 1)20
(58)
2a
26

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Proof of Lemma A.21. A.20,
There exists & 2 and p E N with
29 such that d k29 +p. If 0, then by Lemma
d
[d'
= k S 2 S (M + 1)
(M + 1) 1)2"
S (M + 1) (M + +
29
29
If p > 0, then by Lemma A.20,
+ S 2k 22. 2-1 22( + +))2(k-11 & (M + 1)2"(k-1)+p
29
[d]
(M + 1)20
I (M + +) = (M+
+ +1/
2a
Lemma A.22 (Neural networks approximates ( mE(M))) For all M E N+. A 2 2-3, and 5 2 1, there exists a low-rank Swish network G [-1,, - (M++1) with depth
Doga d) + 1,
(59)
width of nonlinear layers
4(M - 1)",
(60)
width of linear layers
(M + 1)
(61)
upper bound of absolute values of parameters
max MaM + 2)r, 2x²),
(62)
and upper bound of number of nonzero parameters
(12 flog, d) + 8) + 1)d,
(63)
such that
d-1
[[(()) (m (1)))m)())llll <
(64)
12X2
for all [-1, 1j4.
Proof of Lemma A.22. For any E (1,
.......d...... (1.2.....de)
we list 1/7 (3M (s; - (x)-)))) for m; E {0, [,..,..,
1
7
3M
(P(3MT2 - 2T) - p(3MTT; + p(3MT. - -) - 27)),,
M
1 (P(3MTT 7) - pP33TT -r) pP33MT - 47) + p(3MTx - 5T)),
3M
)
M
2
3M
1
(P 47) p(3Mrr; -5) - p(3Mrx; 77) + p(3MMz; - -87)),
M
M M-11
1
3M
(M(3MT (-3M + +5)) +-33MT,,+(33)) + 4)r)
M
P(3MTI + (-3M + 2)r) p(3MT2 (-3M + 1)7)),
3M
M
(0(3MTT: (-3M + - + +(33M
-1)7
M
p(3MTo; (-3M 1)7) + p(3 +0-0::4+3300 (-3M-2)7) 2)7)).
27

---

Approximation to Smooth Functions by Low-Rank Swish Networks
As we can see, exactly construct & (3,((z; )))) )) for all [4]+ and m, E [M]. only need a nonlinear layer of width (2M 4)d followed by linear layer of width (M 1)d, where (2M + 4)d 4(M + 1)d I 4(M 1) and (M + 1)d v (M + 1)df by Lemma A.21 with q O. The number of nonzero parameters of these two layers is no more than 2(2M + 4)d + 4(M + 1)d I 8(M + 1)d I 8(M + 1) (by Lemma A.21 with q = 0) and the absolute values of parameters of them are no more than (3M + 2)7 since 2 implies (3M + 2) 2 1/T.
Let q E N. There exist E and E with S < 29 such that d = k:29 + p. Assume that there is a network of depth q+ 1, width of nonlinear layers 4(M + 1)². width of linear layers (M + 1) upper bound of number of nonzero parameters (12g + (M 1)d, and upper bound of absolute values of parameters max (3M + 2)r, 2X²) outputting
prodzs (vT (3M (z)
m1 M ))
18 (3M (2/420-1 m1+20-1
M
approximate
29-1
II di (3M (zs+z- mitte
))
M
for all j (1,29 1,2 + 1..... (k: 1) + 1} and m [M]] if > 0 (i.e. d2 29) and
prod, (vi (3M (xa-pt1 Md md-p+t) ,..., .b" 3M
m. M
Idd
approximate
II t5 (3M ratt p+1
Mi++-
M
for all m E [M] if 0.
Then when of = + 1, there exist E N and p' with 0 S p' < 29' such that k229' + +p/. We build nonlinear
layer of width 4(M + 1)2d of 1 I 4(M + 1)d and subsequent linear layer of width (M 1)2*
[ (M + 1)d upon
2
the network to approximately multiply the outputs of it using mult. Thus the new network outputs
prodye (3*(3M(
(3M (zz
m, m)))
)) M
10 (3M (19422-1
mJ+25' -1 1)
M
to approximate
20' -1
II c=
TT 3M (jjt -
mj+t)
within 20 12x²
(by Lemma A.19) for all j E (1,20' + 1,2 2d + 1,.... (K' - 1) 20' + and m E [M] if > 0 (i.e.
d > 29') and
prody (N7 (3M
(a-p'+1- md-w+1
M 1) ,..,. ,0 (3M (ra-
ma
m/
M ))
to approximate
II 37 33M (satt
Ma++ M
=-pp++1
within 1' 12x² (by Lemma A.19) for all m e [M] if 0. Conbining the newly built two layers, the new network is of depth
o + 1, width of nonlinear layer 4(M 1)d, width of linear layer (M + 1)d, upper bound of number of nonzero parameters (12g' + (M + 1)dd and upper bound of absolute values of parameters (3 2)r, 2X²). And the approximation enorr is within d-1
12X
We stop the above construction process after the iteration with [log, dl because at this point, (6TH ImEMMa is just approximately constructed.
28

---

Approximation to Smooth Functions by Low-Rank Swish Networks
A.4. Approximating Eme(M mm o by the Inner Product of P and G
We denote the output dimension of P to approximate I'm by Pm and the output dimension of g approximate om by Im. Here introduce some inequalities about Pm and Gm.
Lemma A.23 (Boundedness of Pm and 9m). Let M E N+, T 2 1, and À 2 2- For all a E [0, and m E [M]
1/ 1)
[Pm(()) S 1
Σ 25/ax
(|al- - 1) Σ
16 d 1) +
d - R+R
(65)
6
!!
Sx
22 P
and
9m (2)) S d-1 1
+ (2110(11=0)*
(66)
6
Proof A.23. To show the first inequality, by Lemma A.17 and A.
(Pm(2:) S iPm (2r) Pm (a:) + Pm (a:)
112 )
Σ
((a) - 1) Σ
R
( (x+d-1
( d - 1
R+R
12X²
d-1
SS
2za M
R
+ ( + d - 1)
6 2 SS
11) Σ
a.:
d d
R + R.
iml
MISK
To show the second inequality, by Lemma A.22 and A.4,
[mm())] $ (m))
6 (x) + lmmm()
d 1
+ +2111e'11 )²x
12x2
d 1-1 1
+ (211
6/2
Theorem A.24 (Neural Networks Approximates f E CB [0. 1. Let 3 E
E N. E (0, 1]. and
R E R4. For all f E CB.R 1]0, M E N+, x 2 2 3, and 1, there exists low-rank Swish network nn [0, 1] + R
with depth
max { ; llog dl + +1]+1 +1,
K
(67)
width of nonlinear layers
2 (d+1) dd1)
dd+k k - 2) d -
+ 4 (dd=) - - + +6(M+ + +1
(d+x-1)
+ +4
(68)
width of linear hidden layers
dd1) dd+ d-11 ("+=-3) + d --1 1 2)
d d-1) +
)+
-
d dd*=- - 2)
+ +2(M + +1)
(69)
upper bound of number of nonzero parameters
K 4
d 1
/d+l11 1)
dd+
- 12
d-1 +
2
d-
1=2
(70)
(M + 1)ad
14+16[logg d + 2d + 8 nR
+4 + (
2
1-2
29

---

Approximation to Smooth Functions by Low-Rank Swish Networks
and upper bound of absolute values of parameters
max
(3M + 2)T, 2x max |als
Σ
R
2x²
(71)
04i)
v2a MMS
1=1
such that
(nn(x:) - f(x)
(M+
C7 077+60903 + C2 C.C.+ +c - (M +1
1) Te CC
12x²
(M + 11d
C1 6 C
C2Cs C(dd++((d-11) +M- (M (M+ (Cs + C7
12x
for all x E [0, 1j3, where
()))
C
Σ 6
(1al-1) -
Σ
R
(1 (1:))
+ (n+
( 1-+ 1-1
d-1 ))
R + R,
Sx Sx
d-1
C2 = 6/22
+ (2110'1100)
d
C3 = E
(|al 1)
Σ
n!!
25k
v2a
i=1
P
K 1
m
:==
R
x
+
R,
d 1
M
(21101) 1|x))
-1
C3 = 6R
2||p'l 1000
K d 1
C. = 3d
R(211p'll
d-1
K (d+d-1 d-1
C7
1(211'111=))-11
Proof of Theorem A.24. For all f f C3. 119). MEN,, and 2 1, by Lemma A.17 and A.22, there exist network P and 9 approximating ME and (om)me(M)) respectively. Considering that the depths P and G may be not identical, we construct several nonlinear and linear layers of width 2(M + 1)d and (M + 1)d, which mimic the identity function by Lemma 5.6, upon the shallow one to align their depths. Note that adding these layers does not change the output, width, and upper bound of absolute values of parameters of the shallow network, but its number of nonzero parameters increases
4 (max
{ {[=] [log, dl + 1] min {[x]
[logz dl + +1] (M + +11 s 4 ( + +[ogg dl + 1) (M + 1)d.
Define function nn: [0, 1jd R. the inner product of P and G, i.e.
nn(w))
Σ mult (Pm(z)) (x)) mc[M
(72)
The function nn can be implemented by adding a nonlinear layer of width 4(M + 1)dd and a subsequent linear layer of width 1 upon depth-aligned P and 9 to approximately multiply Pm and Pm for all m E [M]] and sup them up. The number of additional nonzero parameters 12
12(M M + 1)d.
30

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Next analyze the approximation enror: for all E [0, 1j3.
(nn(a)) - (x))
< Σ
mult (m)()), - Σ (m(())m((2))
+
E
(m()))
E
Pm( +
meiMae
mE|MEa
mElMiee
mEIM[
Σ M
(x) f(x)
For the first term, by Lemma 5.5 and A.23,
(M + 1)d
(2) + 6P? (x (x) (m (x) + (m)
Σ
mult (P (m)))
Σ (m)
12X²
M
me(M]"
(M + 1)d
C 60 + C₁
12X²
For the second term, by Lemma A.17, A.22, A.1, and A.23,
Σ
(m)())0m(1))
Σ
PG (2)) (a))
ME(M
ME[M]
Σ (m)
Σ
(x) mm)))(())
+ Σ pa (m)
Σ
Pa (x) (x)
mE(M]"
mE(M]
mE(M]
mE[as]
Σ [m(())]
Prr (a)) (x)) + Σ
PM (a)) 19 5m (a))
mE[M]
mE[M]e
(M 1)d
C2Os+ (M+
Ca(d-1) ((d--). 1).
12X²
12X2
For the third term, by Lemma 5.2,
Σ
Ps (x) f(x)
S <Te + M (M + Te C7.
Proof of Theorem 4.1. Theorem A.24 directly implies Theorem 4.1 by setting
d (d+1
dd1-1) 1)
= 4
+ 12
d dd
d-1
1=2
= 24 + 16]log, d] + + 8 N67
+
+
(d+1-1) 1)
2
d-1
1=2
+++0003++0
C2Cs + Cald-1)
C3
12
4 = Co,
Cs
65 2d + C7.
31

---

Approximation to Smooth Functions by Low-Rank Swish Networks
Lemma A.25, For all r E IR,
2 Te
(73)
= 11 -1 Let g((r) 0. then the solution is 2 2. When 2m g' (T) 2 0; when S In 2, g'(r) S O. Therefore g(T) decreases monotonically on (-o0, 2 In 2].
Proof of Lemma A.25. Let g(T) c 7, then the derivative g'(r)=
increases monotonically on [2 2, +o0), and thus takes the minimum value g(2 2) = 2 - 2 2n 2 2 0. It follows that
e - 0 = e 2 2
Proof of Corollary 4.2. For all 0 3c1, letting
0 (E b)
M ==
3 c4
2- v
C3 E } 0 --))
À max
t == max
2ln
38
d+1
c4 C5E 3gd ) , 1 0
#
In
1
then
M 2 M23 3 11,
A A 2
7 T 1.
Therefore, by Theorem 4.1, because
(M + 1) 0 (c-#)
the width of nonlinear layers, the width of linear hidden layers, and the number of nonzero parameters are O( $); because
x² x²= (5-
and
(3M +2) - - 02 (m)
the maximum absolute value of parameters is O(s $ ); because
3 C
+2
(M + 1)ad
3d3 c4
-
C3
$ C3
C3
X2
X2
3 +d+1
244
3'
C3C
-B
88 8 S <as
3 33
5
3
3 d
Cs (M + 11 S cs(M + 1)de
s cs
c
S
(by Lemma A. 252
3
+d+1
c
Cg5
3
the approximation error is no more than c.
32

---

Approximation to Smooth Functions by Low-Rank Swish Networks
B. Supplementary Material for Experiments
Table Basic information about UCI datasets used in experiments. The number of features for each dataset refers to the total number of features after converting categorical features into dummy features. For example, the categorical feature "race" the dataset "Adult" takes values in the set ("White", "Asian-Pare Pac-Islander", "Amer-Indian-Eskimo "Other", "Black") and we replace this feature with five dummy features taking values {0, 1}.
DATASET UCI ID
# OBSERVATIONS
FEATURES
TASK TYPE
Iris
53
150
4
Classification
Rice
545
3810
7
Classification
BankMarketing
222
45211
47
Classification
Adult
48842
108
Classification
RealEstate
477
414
6
Regression
Abalone
4177
10
Regression
WineQuality
186
6497
13
Regression
SikeSn
275
17379
12
Regression
33


Reviews:
[{"review_id": "IT4phT2v5b", "reviewer": "Reviewer_uBR2", "summary": "This paper studies the question of whether networks with low rank matrices and Swish activations can approximate a class of Holder-continuous and smooth functions. The authors show that the number of parameters and operations can be reduced by 1/3 to still obtain the same approximation rates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We will try our best to relieve your concerns. \n\nQ1: Problem on the curse of dimensionality.\n\nA1: If you want to approximate any function in a Hölder ball, then theoretically it is impossible to escape the curse of dimensionality[1, 2]. In Remark 4.5 of Section 4, we introduce a novel function class that is prevalent in practical applications, and when approximating functions within this class, one can avoid the curse of dimensionality.\n\nQ2: Problem on the compression rate.\n\nA2: As mentioned in the Introduction, to our knowledge, there is no work providing a theorical explanation to answer why low-rank compression does not hurt performance of neural networks. We are the first one to develop the theoretical foundation for low-rank compression from the perspective of approximation theory by answering the question of whether a neural network processed by low-rank compression can serve as a good approximator for a wide range of functions. Though our compression rate is not very significant, we believe that our publication will inspire more important theories.\n\nQ3: Problem on the choice of activation function\n\nA3: From the perspective of technical proofs, we approximated the identity function precisely by leveraging the characteristics of swish function (Lemma 5.6). From the perspective of foundational contributions to future research, as highlighted in the Introduction, the infinite differentiability of Swish neural networks creates unique opportunities to extend our approximation theorems to higher-order Sobolev spaces.\n\nThank you once again. If there are still any doubts, we sincerely hope that you will kindly offer your comments.\n\n\n[1]. Schmidt-Hieber, A. J. Nonparametric Regression using Deep Neural Networks with ReLU Activation Function. The Annals of statistics, 48(4):1875–1897, 2020.\n\n[2]. Bauer, B. and Kohler, M. On deep learning as a remedy for the curse of dimensionality in nonparametric regression. The Annals of statistics, 47(4):2261 - 2285, 2019.", "claims_and_evidence": "The theoretical claims are supported by proofs. While I have not checked the proofs thoroughly, the logical structure of the proofs seems to be correct. The empirical evaluations in section 6 demonstrate comparable performance between low rank networks and full rank networks, however these are on small scale datasets and with small networks. Nevertheless the literature contains substantial empirical support for the natural emergence of low-rank structures within neural network layers. Since the main contribution of the paper is the theoretical results, I will not dwell on the empirical evaluation.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I did not check the correctness of the proofs in detail. However the results and proofs in the paper seem similar to other results on approximation with neural networks so I trust them.", "experimental_designs_or_analyses": "Some more details about the training (choice of regularization, optimizer, other hyperparameters) could be provided.", "relation_to_broader_scientific_literature": "The approximation results of the paper are in line with other neural network approximation results like [1] (for Sobolev spaces, not Holder).\n\n\n[1] Hrushikesh N Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural computation, 8(1):164–177, 1996", "essential_references_not_discussed": "I am not sure.", "other_strengths_and_weaknesses": "Weaknesses: The results do not seem to escape the curse of dimensionality. One hope of using low rank layers is to achieve more efficient approximation rates. This does not seem to be the case here.\n\nThe improvement in terms of number of parameters/computational steps is only a constant 2/3 factor.\n\nThe role played by the Swish activation is unclear - why is that a necessary component of the network architecture?\n\nOverall, while the paper proves that networks with Swish activations and low rank layers can approximate Holder functions at the same rate, the reasons to prefer these Low rank swish networks are not very compelling. \n\nThe amount of savings that can typically be achieved in practice is much more than 33%, so it is not clear the current results can explain how to achieve better efficiency.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Is there an advantage to using Low rank swish networks over full rank ReLU networks in terms of approximation? Or is your goal to show that Low rank swish networks achieve comparable rates of approximation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wAvGr45TEo", "reviewer": "Reviewer_j7PK", "summary": "This paper investigates low-rank compression techniques for neural networks by strategically inserting narrow linear layers between adjacent nonlinear layers. The authors theoretically demonstrate that low-rank Swish networks with a fixed depth can approximate any function within a Hölder ball Cβ,R([0,1]^d) to an arbitrarily small error. Notably, the theoretical analysis establishes bounds on the minimal width required for these inserted linear layers, achieving substantial reductions in computational cost without significant accuracy loss.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We will try our best to relieve your concerns.\n\nQ1: Complexity and Readability of Proof\n\nA1: To establish a novel theoretical foundation for low-rank compression, we develop a rigorous mathematical framework comprising multiple interlocking proofs that, while intricate for researchers new to neural network approximation theory, are indispensable for upholding formal rigor. To enhance accessibility, we distill the conceptual underpinnings of our constructive approximation framework in a dedicated subsection at the conclusion of the Introduction (preceding the contributions summary). In Section 5, we adopt a modular approach to present the complete derivation: Theorem 4.1 is decomposed into four self-contained modules, each addressing distinct aspects of the proof architecture while highlighting critical technical innovations and their interrelationships.\n\nQ2: Lack of Runtime Performance Analysis\n\nA2: The purpose of this paper is not to prove how much low-rank compression, as a network compression method, can reduce neural network inference time. Low-rank compression has been proven in many literatures to be an efficient, easy-to-implement, and hardware-friendly network compression method that significantly reduces computational cost. As emphasized in the Introduction, our work aims to develop a theoretical foundation for low-rank compression from the perspective of approximation theory. Our Theorem 4.1 provides an upper bound of approximation error for representing any function in the Hölder ball $\\mathcal{C}^{\\beta,R}([0,1]^d)$ using low-rank Swish networks. Consequently, our experiments also focus on error analysis. Furthermore, Section 4 presents theoretical calculations confirming that the low-rank architectures in Theorem 4.1 reduce the number of multiplication operations—a more scientifically rigorous metric than runtime, as runtime is highly susceptible to environmental variability.\n\n\nThank you once again. If there are still any doubts, we sincerely hope that you will kindly offer your comments.", "claims_and_evidence": "I can understand the claims in Section3, but not much in Section 4 and 5.", "methods_and_evaluation_criteria": "The selecting specific datasets and models in the evaluation is unclear. Clarification on the choice and potential applicability to more prevalent and larger-scale models would significantly strengthen the practical impact of the method.", "theoretical_claims": "The theory and proofs in section 4 and 5 are hard to follow for me.", "experimental_designs_or_analyses": "Please see the evaluation part for my confusion of model and dataset selection.", "relation_to_broader_scientific_literature": "This paper offers valuable theoretical insights into structured low-rank compression strategies.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\nRigorous Mathematical Proofs: The paper provides extensive mathematical proofs, rigorously demonstrating the proposed theorem, significantly contributing to the theoretical understanding of low-rank compression.\n\nClear Compression Theory: The authors articulate a clear theoretical basis for the achievable compression ratio and computational reductions, providing a solid analytical foundation.\n\nInnovative Use of Hölder Ball: Employing the Hölder ball in the analysis introduces a novel perspective, enriching the theoretical contributions of the paper.\n\nWeaknesses:\n\nComplexity and Readability of Proof: The mathematical proofs, while rigorous, are challenging to follow due to the absence of a clearly structured narrative to guide readers effectively.\n\nLack of Runtime Performance Analysis: The paper lacks empirical data on real-world runtime reductions. Providing measurements of actual runtime improvements could significantly validate the practical utility of the methods.", "other_comments_or_suggestions": "See strengths and weaknesses.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TksjIdj8ml", "reviewer": "Reviewer_US5n", "summary": "This paper discusses the universal approximation property of row-rank MLPs using the Swish activation function.\n\n## update after rebuttal\nAlthough the authors provide some explanation regarding the activation function and the number of parameters, my concerns are not fully addressed. Therefore, I will maintain my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We will try our best to relieve your concerns.\n\nQ1: The proof technique is highly standard, and it is difficult to consider it a significant mathematical advancement.\n\nA1: While our mathematical proof follows a well-established framework, we conducted a well-designed constructive approximation which guarantees that the width of linear hidden layers is no more than one-third of the width of nonlinear layers. To our knowledge, though many works demonstrated that low-rank compression performs in a variety of applications empirically, we are the first to develop the theoretical foundation for low-rank compression from the perspective of approximation theory by answering the question of whether a neural network processed by low-rank compression can serve as a good approximator for a wide range of functions. We believe that mathematics is a tool for reaching truth and what matters is the conclusion derived from the mathematical tools.\n\nQ2: The choice of activation function is too restrictive.\n\nA2: From the perspective of technical proofs, we approximated the identity function precisely by leveraging the characteristics of swish function (Lemma 5.6). From the perspective of foundational contributions to future research, as highlighted in the Introduction, the infinite differentiability of Swish neural networks creates unique opportunities to extend our approximation theorems to higher-order Sobolev spaces.\n\n\nQ3: I do not quite understand the significance of the linear hidden layer having a smaller width than the nonlinear layer.\n\nA3: As mentioned in the Introduction, to our knowledge, there is no work providing a theorical explanation to answer why low-rank compression does not hurt performance of neural networks. We are the first one to develop the theoretical foundation for low-rank compression from the perspective of approximation theory by answering the question of whether a neural network processed by low-rank compression can serve as a good approximator for a wide range of functions. Though our compression rate is not very significant, we believe that our publication will inspire more important theories.\n\n\nQ4: The authors conducted a quantitative approximation comparison only with Ohn & Kim (2019), but they should also compare their results with more recent studies for a more comprehensive evaluation.\n\nA4: While there are many articles discussing the approximation capabilities of neural networks, most focus on ReLU neural networks. Additionally, most studies target function classes of continuous functions, Lipschitz functions, and Hölder continuous functions. Many works also do not provide simultaneously the required depth, width, upper bound of absolute values of parameters, and upper bound of number of nonzero parameters for approximation. Considering all the above points, we selected Ohn & Kim (2019) for comparison.\n\n\nQ5: it does not explicitly specify the source of the definition of low-rank MLP when introducing it in Section 3.2.\n\nA5: Up to now, there is no mathematically rigorous and universally accepted definition of low-rank networks. After all, even the strict mathematical definition of neural networks themselves lacks a consensus—how can we expect one for low-rank networks? Our Definition 3.1 builds upon extensive prior research on low-rank compression, proposing a definition aligned with the majority of scholars' understanding. This definition captures the essence of low-rank compression: reducing computational cost by decomposing large matrices (or convolutional kernels) into products of smaller matrices (or smaller convolutional kernels). We will supplement additional references to demonstrate that this definition aligns with the network architectures proposed in these studies:\n\n[1]. Denil, Misha, et al. \"Predicting parameters in deep learning.\" Advances in neural information processing systems 26 (2013).\n\n[2]. Idelbayev, Yerlan, and Miguel A. Carreira-Perpinán. \"Low-rank compression of neural nets: Learning the rank of each layer.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[3]. Sainath, Tara N., et al. \"Low-rank matrix factorization for deep neural network training with high-dimensional output targets.\" 2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 2013.\n\n\nThank you once again. If there are still any doubts, we sincerely hope that you will kindly offer your comments.", "claims_and_evidence": "The authors provide the number of parameters required for the approximation used in Theorem 4.1, along with a bound. While I have not rigorously checked every detail of the proof, I find no issues with its overall structure.", "methods_and_evaluation_criteria": "The authors approximated the function using a commonly employed method: partitioning the space and connecting each partition with a bump function.", "theoretical_claims": "The main structure of the proof follows a natural progression, and I do not anticipate any major issues.", "experimental_designs_or_analyses": ".", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "The proof technique is highly standard, and it is difficult to consider it a significant mathematical advancement.\n\nThe choice of activation function is too restrictive. Why is the activation function in the paper restricted to Swish? As I understand it, the properties of Swish are used in Lemmas 5.4–5.6 to approximate $xy$, $x^2$, and $x$, but these approximations can also be achieved using many other activation functions. Additionally, approximating bump functions should also be feasible with alternative activations.\n\nI do not quite understand the significance of the linear hidden layer having a smaller width than the nonlinear layer. These values merely compare the number of parameters within the same network, whereas a meaningful comparison should be made between different networks.\n\nThe authors conducted a quantitative approximation comparison only with Ohn & Kim (2019), but they should also compare their results with more recent studies for a more comprehensive evaluation.", "other_comments_or_suggestions": "The authors mention that low-rank approximation is used as one of four methods for compressing neural networks. The paper does cite prior works related to low-rank approximation, but it does not explicitly specify the source of the definition of low-rank MLP when introducing it in Section 3.2. Clearly stating where this definition originates from would enhance clarity and strengthen the paper’s argument.", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1izRoiFnBR", "reviewer": "Reviewer_r7Et", "summary": "The paper investigates whether deep neural networks that have been compressed using low‐rank factorization can still approximate smooth functions as accurately as uncompressed networks from a universal approximation theory perspective. In low‐rank compression, a weight matrix in a network layer is replaced by the product of two smaller matrices. This paper proposes a specific construction: between each pair of standard nonlinear (Swish) layers, a narrow linear layer without bias is inserted. This “bottleneck” enforces a low-rank structure on the transformation. The authors show, via a constructive proof, that with this constraint the network can approximate any function from the Holder space arbitrarily well. The construction uses classical tools from approximation theory. The domain is partitioned into a grid, and at each grid point the function is locally approximated by its Taylor polynomial. To control the approximation error away from the expansion point, the Taylor polynomials are “localized” using smooth bump functions that nearly vanish outside a small region. Both the Taylor polynomials and bump functions are then implemented using Swish networks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We will try our best to relieve your concerns.\n\nQ1: The paper \"Approximation by Superpositions of a Sigmoidal Function\" should be cited.\n\nA1: Thank you for the reminder. We will add this groundbreaking work in our paper.\n\n\nQ2: I am not convinced that universal approximation theory can be used to demonstrate the benefits of low-rank factorization. Moreover, the empirical results appear to be based on relatively simple tasks.\n\nA2: Properly speaking, the aim of our work is not to demonstrate the benefits of low-rank compression. Low-rank compression, as a class of efficient and hardware-friendly network compression methods, can reduce the computation cost significantly, that is well examined in vast applications. However, as mentioned in our introduction: except low-rank compression, the remaining categories of methods are all underpinned by UATs to some extent. Hence, we aim to explain why low-rank networks can still perform comparably to full-rank ones from the perspective of approximation theory. Through our well-designed constructive approximation, we prove that the width of linear hidden layers needs only be no more than one-third of the width of nonlinear layers, implying that low-rank compression within this threshold does not impair the approximation capability of neural networks. By integrating our approximation results with sieve estimation theory, it is easy to derive the convergence rate of low-rank network estimates and limiting distributions for plug-in estimates based on these low-rank approximations.\n\nRegarding the simplicity of the experiments, our consideration is that the paper has already rigorously proven from the perspective of approximation theory that low-rank compression of weight matrices does not affect the approximation rate. The purpose of the experiments is to demonstrate the reliability of our theoretical proof. Just as one has used the axioms of Euclidean geometry to prove that the base angles of any isosceles triangle are equal, there is no subsequent need to draw numerous different isosceles triangles and measure their base angles for verification. According to academic conventions in neural network approximation theory, the vast majority of papers—including the work “Approximation by Superpositions of a Sigmoidal Function”—do not include any experimental validation.\n\n\nThank you once again. If there are still any doubts, we sincerely hope that you will kindly offer your comments.", "claims_and_evidence": "The theoretical claims are well-supported.", "methods_and_evaluation_criteria": "For empirical evaluation, the authors conduct grid searches over network depth and width on eight UCI datasets. They compare the performance of a classical fully connected Swish network with a low-rank version, which uses narrow linear layers of one-third the width. The evaluation follows standard criteria: accuracy for classification and RMSE for regression. The use of 10-fold cross-validation and statistical tests adds rigor to their experimental validation. However, the tasks remain somewhat simplistic.", "theoretical_claims": "The proofs have been checked loosely and appear to be correct.", "experimental_designs_or_analyses": "See above for my concerns regarding the experiments.", "relation_to_broader_scientific_literature": "It is difficult to connect this paper to the broader scientific literature. I believe that universal approximation theory offers little practical insight and fails to convincingly demonstrate the advantage of the proposed low-rank structure.", "essential_references_not_discussed": "The paper **\"Approximation by Superpositions of a Sigmoidal Function\"** should be cited. Published in 1989, it is one of the earliest works establishing universal approximation results for sigmoid networks.", "other_strengths_and_weaknesses": "See below.", "other_comments_or_suggestions": "I am not convinced that universal approximation theory can be used to demonstrate the benefits of low-rank factorization. Moreover, the empirical results appear to be based on relatively simple tasks. At this stage, I find the paper unsuitable for acceptance.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines whether networks using low-rank matrices and Swish activations can effectively approximate Holder-continuous and smooth functions, assuming a certain scaling of the width of the hidden layers. The question is interesting and relevant to neural network theory (in particular, approximation theory). Some reviewers doubted relevance of approximation-theoretic analysis in the context of low-rank learning, while others found that the proof appears to follow fairly standard steps. Perhaps an interesting point in such a context is whether low-rank compression manages to mitigate the curse of dimensionality, to which paper shows that in some classes this is indeed possible (Sec. 4).
Decision: Accept (poster)