TaylorNet: A Taylor-Driven Generic Neural Architecture
Authors: Hongjue Zhao
Yizhuo Chen
Dachun Sun
Yingdong Hu
Kaizhao Liang
Yanbing Mao
Lui Sha
Huajie Shao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Taylor Neural Networks
Image Classification
Physics Guided Machine Learning
Dynamical Systems

Abstract: Physics-informed machine learning (PIML) aims to incorporate physics knowledge into deep neural networks (DNNs) to improve the model generalization. However, existing methods in PIML are either designed for specific problems or hard to interpret the results using black-box DNNs. In this work, we propose Taylor Neural Network (TaylorNet), a generic neural architecture that parameterizes Taylor polynomials using DNNs without using non-linear activation functions. The key challenges of developing TaylorNet lie in: (i) mitigating the curse of dimensionality caused by higher-order terms, and (ii) improving the stability of model training. To overcome these challenges, we first adopt Tucker decomposition to decompose the higher-order derivatives in Taylor expansion parameterized by DNNs into low-rank tensors. Then we propose a novel reducible TaylorNet to further reduce the computational complexity by removing more redundant parameters in the hidden layers. In order to improve training accuracy and stability, we develop a new Taylor initialization method. Finally, the proposed models are evaluated on a broad spectrum of applications, including image classification, natural language processing (NLP), and dynamical systems. The results demonstrate that our proposed Taylor-Mixer, which replaces MLP and activation layers in the MLP-Mixer with Taylor layer, can achieve comparable accuracy on image classification, and similarly on sentiment analysis in NLP, while significantly reducing the number of model parameters. More importantly, our method can interpret some dynamical systems with Taylor polynomials. Meanwhile, the results demonstrate that our Taylor initialization can significantly improve classification accuracy compared to Xavier and Kaiming initialization.

Under review as a conference paper at ICLR 2023
TAYLORNET: A TAYLOR-DRIVEN GENERIC NEURAL ARCHITECTURE
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we propose the Taylor Neural Network (TaylorNet), a generic neural architecture that parameterizes Taylor polynomials using DNNs without non con-linear activation functions. The key challenges of developing TaylorNet lie in: (i) mitigat- ing the curse of dimensionality caused by higher-order terms, and (ii) improving the stability of model training. To overcome these challenges, we first adopt Tucker decomposition to decompose the higher-order derivatives in Taylor expan- sion parameterized by DNNs into low-rank tensors. Then we propose novel reducible TaylorNet to further reduce the computational complexity by removing more redundant parameters in the hidden layers. In order to improve training accuracy and stability, we develop a new Taylor initialization method. Finally, the proposed models evaluated image classification, natural language processing (NLP), and dynamical systems. The results demonstrate that our proposed Taylor-Mixer, which replaces MLP and activation layers in the MIP-Mixer with Taylor layer, can achieve comparable accuracy on image classification, and similarly on sentiment analysis in NLP, while significantly reducing the number of model parameters. More importantly, our method can explicitly learn and interpret some dynamical systems with Taylor polynomials. Meanwhile, the results demonstrate that our Taylor initialization can significantly improve classification accuracy compared to Xavier and Kaiming initialization.
broad spectrum of applications, including
INTRODUCTION
This paper proposes generic neural architecture, called TaylorNet, that parameterizes Taylor polyno- mials using deep neural networks. It can be employed to a variety of application domains, including image classification, dynamical systems, and natural language processing (NLP). Importantly, the proposed method does not use non-linear activation functions, thus promoting interpretability of DNNs in some applications, such dynamical systems.
This work is motivated by a growing popularity of toyyscc--guded machine learning (ML) (Jia et al. 2021 Daw al. (017). which integrates physical priors into neural networks. Thus, it endows neural networks with the ability to generalize to new domains better. As a result, physics guided ML has been widely applied a variety of areas, such as dynamical systems (Cranmer al. 2020
Lusch et al. 22018] Greydanus et al. 2019), quantum mechanics (Schütt et al., 2017). and climate changes (Kashinath al. [202]] Pathak al. 2022). However, existing methods based on DNNs
either tailored to solve certain specific problems, such as PDEs (Li et al. 2020 Raissi et al. 2017)
al.) 2019 Lusch al. 2018 Wang et al.) 2019), or hard to explain the results. Hence, the question is, can we develop a generic interpretable neural architecture that can be used in a wide range of machine learning domains?
and dynamics prediction (Greydanus
In this paper, we develop a novel Taylor-driven neural architecture, called TaylorNet, that parameter- izes Taylor polynomials using DNNs without non-linear activation functions, as shown in Fig. [ The proposed TaylorNet is able to generalize to wide spectrum of ML tasks, ranging from computer vi- sion and dynamical systems to NLP. However, developing TaylorNet poses two main challenges. First, the computational complexity of Taylor polynomial parameterized by DNNs grows exponentially as the polynomial order increases. Second, its higher-ordet terms often lead to training instability. To deal with these challenges, we first adopt Tucker decomposition to decompose the higher-order

---

Under review as a conference paper at ICLR 2023
crder
yd-4
Au
2dd order
P
O2
(8) Taylor Layer
(b) TaylorNet
Figure 1:
(a) Architecture of Taylor Layer of order using Tucker decomposition; (b) TaylorNet consists of N Taylor layers of order 2.
tensors in TaylorNet into set of low-rank tensors Malik & Becker 2018 Kolda & Bader, 2009).
To further reduce its computational complexity, we propose a reducible TaylorNet that removes more redundant learnable parameters in the hidden layers. In order to show the generalization of our architecture, we propose Taylor- Mixer that uses Taylor layers in place of both the MILP layers and activation functions in the MLP-Mixer (Tolstikhin et al. (021). Then new Taylor initialization method is developed to improve the stability and accuracy of the model.
Finally, we evaluate the proposed Taylor-Mixer and TaylorNet in variety of applications, including image classification, dynamical system, and NLP. The results show that our Taylor-Mixer can achieve comparable accuracy to the MLP-Mixe: image classification while exhibiting considerable reduction of model parameters. The proposed TaylorNet also explicitly learns and interprets the dynamics of two classic physical systems with polynomials. Besides, our method can also be applied to NLP. The evaluation results on sentiment analysis demonstrate a competitive accuracy to the recently proposed pNL.P- Rixerr (Fusco et al. 2022). Meanwhile, our Taylor initialization can reach an accuracy that is over 10% higher than the Xavier and Kaiming initialization methods for the proposed TaylorNet (Gorrt & Bengio 2010: He al. 2015).
In summary, our contributions include: 1) we design the TaylorNet, a novel neural architecture without activation functions that can generalize propose a reducible TaylorNet that remarkably reduces the number of learning parameters, 3) new Taylor initialization method is proposed to stabilize model training, 4) we develop Taylor-Mixer that replaces both the MLP layers and activation functions in MLP-Mixer with Taylor layers, which can achieve comparable accuracy to the MLP-Mixer on image classification and sentiment analysis, and 5) Our approach can explicitly learn and explain some dynamical systems with polynomials, making way for interpretable ML.
broad spectrum of ML domains, we then
2 RELATED WORK
Polynomial Neural Networks. We summarize the significant difference between the proposed TaylorNet and existing Polynomial Neural Networks. Earlier work Nikolaev & Iba (2006) mainly adopted Group Method of Data Handling learn partial descriptors in polynomial neural networks. Then follow-up pi-sigma network Shin & Ghosh (1991) was developed to filter the input features using predefined basis. These methods, however, fail to scale to high-dimensional data Chrysos et al. 2020). Recently, researchers designed II-Nets Chrysos et al. 2020 2022) that parameterizes the polynomial functions using deep neural networks and tensor decomposition. However, the performance of II-Nets will degrade in larger networks. In addition, II-Net can be viewed as a special case of TaylorNet at expansion point 0 since its adopted CP-decomposition a special case of Tucker decomposition. Furthermore, we develop novel Taylor initialization to improve the training stability while 11-Net does not. Our initialization method is different from the initialization paradigm for Tensorial Convolutional Neural Networks Pan et al. (2022).
Related Work on Taylor Series. Some recent studies developed Taylor- Taylor-based neural networks. For example, TaylorSwiftNett (ooutteetoouuryyarrri et al. (2021) was developed for time-series prediction, but it is not applicable to high- nighddimessional classification. Recently, Mao et Mao et al. (2022) developed Phy-Taylor to learn the physical dynamics based partial physics knowledge, but it suffers from
2

---

Under review as a conference paper at ICLR 2023
the curse of dimensionality. Different from these methods, it can be used in wide spectrum of application domains without using activation functions. Moreover, can interpret some dynamical systems using Taylor polynomials.
Learning Dynamics. Some researchers developed physics-based DNNs to learn the dynamics of physical systems. For example, Luseh et al. designed the Koopman operator (Geneva & Zabaras) 2022 Lusch et al. (2188) that maps the non-linear dynamics into linear Koopman representation space for predicting the future states of dynamical systems. Recent studies proposed Hamiltonian and Lagrangian neural networks (Cranmer et al. 2020) that strictly follow conservation laws. However, these methods are designed for specific problems, and they are hard apply other domains, such as computer vision and natural language processing (NLP).
Tensor Decomposition. Tensor decomposition (Kolda & Bader 2009) aims represent high- dimensional tensors using multilinear operations over the factor matrices or tensors. Some popular tensor decomposition methods include CP decomposition (Carroll & Chang 1970), Tucker decom- position (Tucker) [966). tensor train (TT) decomposition (Oscledets [01 and tensor ring (TR) decomposition (Zhao et al. 2016). Thanks to their ability to reduce computational complexity without breaking out data structure, tensor decomposition techniques are increasingly being widely used in machine learning applications (Wu et al., 2020 Pan et al. 2022 Qiu et al. 2021]. Inspired by prior work, adopt tensor decomposition to deal with the curse of dimensionality in our TaylorNet.
PRELIMINARIES
Notations. We summarize the main notations used throughout this paper in Appendix A Regarding mathematical symbols, scalars are denoted by normal letters, e.g. and y. while vectors denoted by lowercase boldface letters, e.g. æ. In addition, matrices are denoted by uppercase boldface letters, e.g. X, while tensors are denoted by calligraphic letters, e.g. X' and W.
Taylor Polynomial. Taylor polynomial is able to approximate non-linear smooth functions given an arbitrary compact Hausdorff space according to weierseessss theorem (Stone 1948). According to Taylor's is theorem (Thomas et al. 2005). for vector-valued multivariate function f Rd + R. its Taylor polynomial at a point æ
Bo can be expressed as
f(x) ~
A
(I)
Da;
where x Rd. vo E RRd, Ar, = Zj 25,0, j 1, ,,... d. can also written as the following tensor form (Chrysos et al. 2020).
f(x) = f(wn) +
Wil II x Ax
(2)
where X. denotes mode-m vector product, Ar 2n and Wilk E Ro xd are scaled higher-order derivatives of f at æ = To- However, the problem of Taylor polynomial is that its computational complexity grows exponentially as the order increases, making hard to be applied to high-dimensional data.
Tucker Decomposition. Tucker decomposition aims decompose tensor into a small core tensor multiplied by a set of matrices along the corresponding mode (Tucker [966] Kolda & Bader 2009). In essence, Tucker decomposition can be viewed as a higher-order principal component analysis. Let X' be N N-way tensors, then its Tucker decomposition is given by
X X1 A (1) X2
A (N)
(3)
where 9 is a core tensor and (m) (n = 1..... N) are the factor matrices. According to mode-n unfolding (Kolda & Bader 2009). Eq. can be expressed as the following matricized form:
(m)) A (m) (..)
(A (N @ & A (a+1)
(n-1) .... AT)
(4)
where X and G(v)) are matrices that mean the mode-n matricization of the tensor X and g and & denotes Kronecker product.
3

---

Under review as a conference paper at ICLR 2023
4 PROPOSED METHOD
In this section, we first introduce a lightweight Taylor Neural Network using Tucker decomposition. As extension, propose reducible TaylorNet further improve the computational efficiency by removing redundant trainable parameters in the middle layers. In order to stabilize the model training process and improve accuracy, novel Taylor initialization method developed in this work. Moreover, we present the connection between TaylorNet and some other existing neural networks.
4.1 TAYLOR NEURAL NETWORKS
Since mapping function f in Eq. 2is is unknown and needs to be learned by deep neural networks, we cannot calculate the derivatives Wa(M) of directly. To deal with this issue, this work adopts deep neural networks to parameterize the Taylor polynomial in Eq. 2 where f(zo) and Wylh) (k ....., N) are learnable parameters during model training. However, the computational complexity of tensor Wil grows exponentially, (dd),, as the polynomial order k increases. To overcome this issue, Tucker decomposition is adopted in this work. According to Eq. the scaled derivatives Wylk) can be decomposed into
W glu) Ok X2 T..... X Iak g X1 O II
Xs+1Iks j=1
(5)
where glk) Ro IT XPiaAAs is the core tensor, Iaj E pddrrinkks (j I....,k) and Ox RoXrout are imput and output factor matrices, respectively. Here we use Fin.k.i and Tout, to represent the Tucker ranks corresponding to the j-th input and output dimension in the k-th-order term the Taylor polynomial.
Substituting Eq.
5 into 2
the k-th term of Taylor polynomial can be written
W A,A== W
II
glt)
X1 O
[ xi+ Iss
(ii
1x++ Ax"
(6)
Based commutative law and associative property in mode-n product (Kolda & Bader 2009), Eq. can be reformulated as
W kk
II A X1 Oz
II
X5+1 AA Iws)
E TR.
(7)
Please refer the detailed transformation in Appendix.
B
However, to our knowledge, the current deep learning frameworks (e.g, Pytorch and TensorFlow) do not support batch-size-based mode-n product in Eq. 7 Fortunately, since the result of Eq. is a vector, according to mode-n unfolding as illustrated in Eqs. and we can convert it into matricized form as follows.
Voll 1+1
(aGG:((Ttte 2, ORGi (IMAA)) .... (I7
Finally, substituting the above Eq. 8 into Taylor polynomial in Eq. resulting in lightweight N-th-order Taylor layer as follows.
f(w) 3 Σ O.G. [ (I @ ... (I1Aw))
(9)
where == f(x)). Ox, Gt. and ITT (k= N; j == 1..... k) are learnable parameters.
After that, we can stack Taylor layers with a N-th order expansion to construct a new neural network, referred as the TaylorNet. According to Eq. above, the output of the 1-th layer in our TaylorNet with a W-th-order expansion can be written as
y 10 ( )
(1) {[(2))
(1 y (-1) }
(10)
4

---

Under review as a conference paper at ICLR 2023
where y Rdd is the output of the (-th layer and y y A Rdd da)) d. Here d is the input dimension of the I-th layer. In addition, 300) E Idd 0 ( E Bdd++ ousk G E
R Reoa xTT " iink., 1 E R x inkks (A 1 N:j ,,..., k) are leamable parameters of the 1-th Tucker Taylor layer.
Fig. shows the overall framework of the proposed TaylorNet. In this paper, we use the Taylor layer with second order expansion, since it is able to mitigate the overfitting problem and also reduce the number trainable parameters in DNNs.
The computational complexity of the k-th-order term in Taylor layer decreases from (od*) Of [5= Fin.k.j (inkk.. o'out,k dd Ed== I lin,k.j), where d and denole the dimension of the input and the output. When and d much larger than the rank of tensor in Tucker decomposition, the number of parameters will be reduced by orders of magnitude.
Remark 4.1.
4.2 REDUCIBLE TAYLORNET AND TAYLOR-MIXER
Reducible TaylorNet. We further propose reducible TaylorNet, called R- R-TaylorNet, to reduce the number of trainable parameters of TaylorNet. The basic idea is to use a single matrix as the new trainable parameter to replace the original product of the current layer's output factor matrix and the next layer's s input factor matrix, namely, compositing two consecutively multiplying parameter matrices O and Ikj (L+1) into single parameter matrix. Below, will theoretically derive the R- R-TaylorNet.
According to the block multiplication of matrices, Eq. 10 can be rewritten
y = -3 [ 0
o
(11)
G
3 z N1
(0) where =
(11) T - E ki we call it hidden features of the (-th layer in this work.
In order to further simplify the above equation, we define the following notations:
(
R
oulk
G 10)
(
211
h (x10
!
G(
222
20
E R ER 0 laut,
(12)
NN
G
(
2N1
(1-1) Then 2hj in the (1 1)-th hidden layer can be expressed by the following recursive form
((+1)
= (I
(000 + o (21). :...:...., =NNN
Zkj
(13)
((+1) ( + (IetT)
0 (210)..,,>NN)
where & = 1...., N. and k. k.
In order to reduce some intermediate parameters in DNNs, we introduce new lower-dimensional matrices (vectors) to replace the product of some matrices in the above Eq. 13
Namely, vkj
(1)
(T(411 3 and T!())
(I(1+) ( By doing so, simplify Eq. 13 as
(++1) (1) + T h (=19
(2 NN NN k= j 1.....
(14)
(1)) where o/ E R m
and T(() E r mn N ous, are the new parameters in DNNs.

---

Under review as a conference paper at ICLR 2023
Finally, use Eq. 14 above to implement a L-layer reducible TaylorNet. The feedforward method of a single layer is summarized in Algorithm I in Appendix C
Remark 4.2. In R-TaylorNet, the computational complexity of calculating hidden features
F11 ...... NN from # ".1....,,N.. in the 1-th layer can be reduced by N (dd+t)) out,
12+1)
(1=1)
(t)
((+1) Ej= (1+1)
-
(E
(1)
out.m (Ej= ((t))))) compared to the original TaylorNet.
im.k.
Taylor-Mixer. Building the R-TaylorNet, also propose new Taylor-Mixer Mixer that replaces both the MLP layers and non-linear activation functions in the MLP-Mixer Tolstikhin et al.) (2021) with Taylor layer. The resulting Taylor-Mixer can be applied image classification and natural language processing (NLP).
4.3 TAYLOR INITIALIZATION
We also develop robust Taylor initialization method to mitigate the training instability caused by higher-order terms. For simplicity, we omit superscript (() unless otherwise specified. Following Xavier Glorot & Bengio (2010) and Kaiming initialization He et al.
(013)), we assume that: 1) the input elements (variables) of the 1-th layer, denoted by y follow independent zero-mean Gaussian distribution. 2) the weights in O2, Gk Ikj (k = 1...,, W
I,..., k) are initialized independently with zero mean. B is initialized O. Then we have the following proposition. Proposition 4.1. The variance of input and output variables of the 1-th layer satisfies:
(o (0))² = Σ (0,
N
II Imm...,O...
(d 2)!! 111
[(d+ 2k - 2k (
TTk
(15)
where !! denotes double factorial, ()) and (a ) denote the variance of y and y (1-1), And oo & and o7,, denote the variance of the weights in Og, Gy, and Jsj (I: 1,..., N; j 1..... ......k,, respectively.
Following the prior works He et al. (2015); Glorot Bengio 2010), we should enforce (o(0)²²
(oy 1) for stabilizing the model training. Plus, we would also like to ensure that all intermediate features inside the Taylor layer have similar variance as (a To satisfy these requirements, the variance of weights should be initialized to:
1
(d 11-21 (d-2)!!
00.4 A
2k
OG.
Fout,k
11 Tinkjj
(d + 2 2)!!
N s.t.
x==
(16)
where A is coefficient that can be used to scale the importance of the k-th-order term. Please refer to the theoretical analysis of Proposition 4.1 in Appendix
Similarly, we have also developed an initialization method for Reducible TaylorNet, please refer to Appendix Foror more details.
4.4 CONNECTIONS TO EXISTING MODELS
We present the connection between TaylorNet and some existing neural networks. According to Eq. 3 one-layer TaylorNet of order 1 is linear function, f(x) = f(xg) + J 2o), where J the Jacobian matrix of f (x) at & Zg. Thus, the TaylorNet of order can be viewed as fully connected layers in deep neural networks. The second order term in a Taylor layer can be expressed as [(1)[( xo) X (x xo)], where H the scaled second-order derivative tensor of f fx) at can be viewed the pixel-level attention, analogous to
Zo. The Kronecker product of
the token-level attention in Transformer Vaswani et al. (2017); Dosovitskiy et al. (2020). Finally, our TaylorNet adopts higher-order terms compensate for the residual errors, shown in Fig. (b).
which shares the similar philosophy as a residual (or highway) network (rrvastaaa et al. (2015 He
al. (2016).

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
We first verify the effectiveness of the proposed Taylor initialization method. Then we evaluate the performance of the proposed methods in three different applications, including image classification, explainable dynamical systems, and NLP. The detailed model configurations and hyperparameter settings are presented in Appendix
5.1 TAYLOR INITIALIZATION
First of all, we compare our Taylor initialization with two commonly used initialization approaches: Xavier and Kaiming initialization. In this task, we conduct experiments
0.8
CIFAR10 using four-layer Taylor-Mixer (34,4M parameters) as described in Section 4.2 Fig. illustrates the comparison re- sults of different methods using random seeds. We can observe that our Taylor initialization can signif- icantly increase the classification accuracy by over 10% compared the next best approach. Xavier initialization. The primary reason why both Xavier and Kaiming initialization do not perform well that they fail to ensure the same variance for input and output at each layer.
Accuracy
0.6
Xavior
talming
25000
Training steps
Figure 2: Accuracy comparison of different initialization methods using 3 random seeds.
5.2 EVALUATION ON IMAGE CLASSIFICATION
We evaluate the performance of our proposed Taylor-Mixer on image classification. We compare it with the MLP-Mixer Tolstikhin al. (2021). which achieve competitive results image classification benchmarks. In our experiment, we choose the point of expansion at æn = 0 for Taylor-Mixer since the input data are normalized. Following the similar method in MLP-Mixer, we pre-train our model on ILSVRC2012 ImageNet that contains 1.3M training examples and Ik classes, and then data augmentation techniques, including RandAugment Cubuk et al. (2020), mixup Zhang et al. (2018), and stochastic depth Huang af. (2016). addition, we use the same fine-tur tne-ttning strategy as MLP-Mixer. The detailed parameter settings are introduced in Appendix D.1 Table I performance comparison our Taylor-Mixe and the baselines under different model sizes. We can that the proposed Taylor-Mixer performs better than Il-nets and achieves comparable accuracy to the MLP-Mixer with fewer model parameters on both CIFAR10 10 and CIFAR 100. In particular, the parameters of our Base model be reduced by about 42% compared to the MLP-Mixer. Therefore, we can conclude that our Taylor- Mixer outperforms the MLP-Mixer.
on CIFARIO and CIFAR datasets. Also, this work adopts the same
shows the
Table 1: Performance comparison for different methods using 5 random seeds. Here Small/16 and Base/32 mean the patch size is 16 16 and 32 x 32, respectively. We can observe that Taylor-Mixer has slightly higher accuracy but fewer parameters than the MLP-Mixer. In particular, our Base model exhibits a significant reduction in model parameters.
Small/16
Basc/32
Models CIFARIO
CIFAR 100 (%)
Parameters (M)
CIFARIO (%)
CIFAR 100 (%)
Parameters (M)
MLx-Mixer
93.21 10.08
F4.08
18
0.
0.230-16.25
$9.6
II-nets
NA
NA
NA
88. 12-00.22
67 83:33000032
37
Mixler-Mixer
93.68
74.63
17.2
94.97-10.33
79.00-10.40
34.4
5.3 EVALUATION ON DYNAMICAL SYSTEMS
Next, we apply our TaylorNet predict and interpret the dynamics of physical systems. We evaluate on two dynamical systems, Duffing equation and High-dimensional non-linear flow attractor. To train model, we use the 20 trajectories generated from 20 different initial conditions as the validation data. The time span of each trajectory is
generate 100 trajectories by randomly choosing 100 initial conditions. Then
0, 0.01, 0.02. ..., 10 with sampling time, 0.01. Thus, can
7

---

Under review as a conference paper at ICLR 2023
15
1.5
adeint
odeint
1.0
1.0
0.5
0.5
-1.0
-1.0
115
-1.0
-0.5
0.0
0.5
1.0
-1.5
-1.0
-0.5
0.0
0.5
3.0
1.5
Xg
X1
(a) TaylorNet
(b) MLP
Figure 3: Trajectory prediction of different methods on Duffing Equation.
odeint
odeint
orNet
1.0
100
0.8
10
1.0
0.5
0.5
-10
41
-0.5
0.5
0.0
0.5
1.0
1.0
1.0
(a) TaylorNet
MLP
Figure 4: Trajectory prediction of different methods on Non-linear Fluid Flow.
convert continuous dynamical system into a discrete dynamical system, #k+1 = f(xx). Next, we use regression technique to predict the next state using TaylorNet. We compare our approach with two methods: ODE solver (ground truth), called odeint, from SciPy package and 3-layer MLP.
Dufling equation. We first adopt TaylorNet to predict the dynamics of Duffing equation, given by
42
= x
(17)
22 = I1 x
In our experiment, we choose 11 (0), (0) E [-1,
Fig. illustrates the trajectory prediction of different methods on Duffing dynamics using random initial condition. can observe from it that our TaylorNet can attain very good trajectory prediction and its error is 1.492 10-? compared the ODE solver, odeint, from SciPy package. It thus significantly outperforms the MLP whose error is about 0.3514. More importantly, since our method does not activation functions, it has the ability to explicitly learn the dynamical systems in the following Eq. 18] After comparing to the original Duffing equation, we can see that our predicted model is very close the ground-truth model in Eq 17
d R 1.001
(18)
d2 1.001 1.001x0.
High-dimennional non-linear flow attractor. We then apply our method to predict the dynamics of non-linear fluid flow. According to Noack al. 2003). the dynamical system can be described by
8

---

learn Under review as a conference paper at ICLR 2023
the following low-dimensional model.
11 j.m w22 + A31
82 w + p ArzI3
(19)
33 1(23 - oं z2).
= A A= -0.1, and Following the prior work Lusch et al. (0018), choose μ 0.1, - = T1 (0) (0) E [-1.1, Zy
(0) E
[0, 2.42] in the experiment.
Fig. shows the trajectory predictions of flow attractor using different methods. We can observe that our approach can accurately predict the trajectory as the ODE solver, odeint (ground truth). In addition, the error of our method is about 3.361 10 which is three orders of magnitude smaller than that of the MLP (4.447 x 10-3). Finally, we leverage our TaylorNet to reconstruct the dynamical system in the following. We can see the predicted model very close to the ground truth in the above Eq. [9| Based on these two examples, we can conclude that our TaylorNet is able to explicitly and interpret the dynamics of some physical systems with polynomials.
1 R 0.095x1 1.003zz 0.100.z f = 1.0022; + 0.095zz + 0.100Zz243,
(20)
2 R 9.513 + 9.521x} + 9.521x2.
5.4 EVALUATION ON NLP
Finally, we also explore our method in NLP applications. In this work, we use sentiment analysis on IMDB Maas et al. (201T)
running example. Similar to image classification, we propose a new Tay that replaces all the MLP layers, other than those in the attention mechanism, in the recently developed PNLP-Mixer Fusco al. (2022). The parameter settings are described in Appendix [.3] Table 2
illustrates the performance comparison of different methods, and our Taylor-NLP attains comparable accuracy to the PNL.PMixer XL but with much fewer parameters.
Importantly, our Taylor-NLP does not use activation functions.
Table Performance comparison of the proposed Taylor-NLP and pNLP-Mise using IMDB dataset. Our results are averaged from 3 random seeds
Model
Accuracy
F1
Parameters
PNI--Mixxer XS
80.95
80.95
1.2M
PNLP-Mixer Base
81.46
81.38
2.0M
PNLP-Mixer XL
82.15
82.66
4.9M
Taylor-NLP
82.98 0.17
82.88 0.26
908K
6 CONCLUSION
This paper developed a Taylor-driven generic neural architecture, called TaylorNet, that is able to naturally introduce inductive bias to deep neural networks (DNNs). Different from classical DNNs, our TaylorNet adopted higher-order terms to replace the conventional non-linear activation functions. More specifically, we first proposed a lightweight Taylor Neural Network (TaylorNet) based on Tucker decomposition. As extension, redundant parameters in hidden layers to improve computational efficiency. Then we proposed a new Taylor-Mixer that replaces both the MLP layers and activation functions in the MI.--Mixer with Taylor layers. In order to improve the model performance, a novel Taylor initialization approach was proposed. Evaluation results illustrated that the proposed method can achieve comparable accuracy to the baselines on image classification and sentiment analysis in NLP. In particular, our approach can significantly reduce the number of desired model parameters on image classification. Importantly, our approach could explicitly leam and interpret some dynamical systems with polynomials, making way for explainable MI..
also developed reducible TaylorNet that remove
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
J Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via an n-way generalization of "eckart-young" decomposition. Psychometrika, 35(3):283-319 1970.
Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng. and Stefanos Zafeiriou. P-nets: Deep polynomial neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7325-7335, 2020.
Grigorios G Chrysos, Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks. In European Conference Computer Vision, pp. 692-712221 192-716. Springer, 2022.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Arka Daw, Anuj Karpaine, William Watkins, Jordan Read, and Vipin Kumar. Physics-guided neural networks (pgnn): An application lake temperature modeling. arXiv preprint arXiv: 1710:1710..1433 14431, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020.
Francesco Fusco. Damian Pascual, and Peter Staar. pnlp-mixer: an efficient all-mlp architecture for language. arXiv preprint arXiv:2202.043500 2022.
Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. Neural Networks, 146:272-289, 2022.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 646-661. Springer, 2016.
Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan Read, Jacob A Zwart, Michael Steinbach, and Vipin Kumar. Physics-gee Physics-guided guided machine learning for scientific discovery: An application in simulating lake temperature profiles. ACM/MMS Transactions on Data Science, 2(3):1-26, 2021.
K Kashinath, M Mustafa, A Albert, JL Wu, C Jiang, S Esmaeilzadeh, K Azizzadeneshelli R Wang, A Chattopadhyay. A Singh, et al. Physics- informed machine learning: case studies for weather and climate modelling. Philosophical Transactions of the Royal Society A, 379(2194):2020099, 2021.
10

---

Under review as a conference paper at ICLR 2023
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprini wrXiv:2010008893 2020.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):1-10, 2018.
Andrew Maas, Raymond E. Daly, Peter T Pham, Dan Huang, Andrew Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Osman Asif Malik and Stephen Becker. Low-rank tucker decomposition of large tensors using tensorsketch. Advances in neural information processing systems, 31, 2018.
Yanbing Mao, Lui Sha, Huajie Shao, Yuliang Gu, Qixin Wang, and Tarek Abdelzaher. Phy-taylor: Physics-moddl-basse deep neural networks. https://urxxvvorg//b///22 13511, 2022.
Nikolay Nikolaev and Hitoshi Iba. Adaptive learning of polynomial networks: genetic programming, backpropagation and Bayesian methods. Springer Science & Business Media, 2006.
Bernd R Noack, Konstantin Afanasiev, MAREK MORZYŇSKI, Gilead Tadmor, and Frank Thiele. A hierarchy of low-dimensional models for the transient and post-transient cylinder wake. Journal of Fluid Mechanics, 497:335-300 363, 2003.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):
2295 2317, 2011.
Yu Pan, Zeyong Su, Ao Liu, Wang Jingquan, Nannan Li, and Zenglin Xu. unified weight initialization paradigm for tensorial convolutional neural networks. In International Conference Machine Learning, pp. 17238-17257. PMLR, 2022.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Arizzadenesheli, al. Fourcast- net: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.112: 2022.
Mohammad Saber Pourheydari, Mohsen Fayyaz, Emad Bahrami. Mehdi Noroozi, and Juergen Gall. Taylor swift: Taylor driven temporal modeling for swift future frame prediction. arXiv preprint arXiv:2/ 10.22/111111 14392, 2021.
Hejia Qiu, Chao Li, Ying Weng, Zhun Sun, Xingyu He, and Qibin Zhao. On the memory mechanism of tensor-power recurrent models. In International Conference on Arlificial Intelligence and Statistics, pp. 3682-3690. PMLR, 2021.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 17Xiv:171..1500 2017.
Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature communications, 8(1):1-8, 2017.
Yoan Shin and Joydeep Ghosh. The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation. In JJCNN-91-Seate international joint conference on neural networks, volume pp. 13-18. IEEE, 1991.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXivv1505.00388, 2015.
Marshall H Stone. The generalized weierstrass approximation theorem. Mathematics Magazine, 21 (5):237-254, 1948.
11

---

Under review as a conference paper at ICLR 2023
George Brinton Thomas, Maurice D Weir, Joel Hass, and Frank R Giordano. Thomas' calculus. Addison- Weslly, 2005.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272, 2021.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3): 279-311, 1966.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Rui Wang, Adrian Albert, Karthik Kashinath, and Mustafa Mustafa. Towards physics-informe informed deep learning for spatiotemporal modeling of turbulent flows. In AGU Fall Meeting Abstracts, volume 2019, pp. IN32B-13, 13, 2019.
Bijiao Wu, Dingheng Wang, Guangshe Zhao, Lei Deng, and Guoqi Li. Hybrid tensor decomposition in neural network compression. Neural Networks, 132:309-320, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decomposi-
tion. arXiv preprint arXiv: 05535, 2016
12

---

Under review as a conference paper at ICLR 2023
A NOTATIONS
We summarize the main notations used throughout the paper in the following table.
Table 3: Summary of notations.
Notation
Dimension(s)
Definition
X
Kronecker product, mode-n matrix product, mode vector product
term Taylor polynomial
Taylor polynomial
1
N
the layer number in Reducible TaylorNet
24
Tucker ranks corresponds the j-4h input dimension in the 6-th order in TaylorNet
Fout.k
Tucker ranks corresponds the outpot dimension i-th order TaylorNet
Azr
12
Inpu: of TaylhrrNeuReducchle TaylorNet
yor f(x)
R
Output of TaylkrrNe/Rccuuiblee TaylorNet
+ = (44) T y0-11
Pre-G hidden features I-th layer in TaylorNet
N (I:I..... WW
Post-G hidden features of 1-th layer in TaylorNet
((ms)
R
Learnable vector parameter
glo)
gen Th Xhaad
Learablle core tensor of TaylorNet
G 1d g/
mode-1 matricization of g!))
Rexhnnad
Learnable input factoor matrices of TaylorNet
O₂
Learnable ourput factor matrices TaylorNet
New learnable vector parameters in Reducible TaylorNet
New matrix parameters Reducible TaylorNet
( 10 1-1 2o of
N
Initialization variance for " " ( G(O, 11
A
N
Initialization coeflicient for of.,
13

---

Under review as a conference paper at ICLR 2023
B PROPERTIES OF TENSOR MODE-N PRODUCT
Lemma B.1. For mode-n matrix product, satisfies the commutative law (Kolda & Bader 2009
X X A X B = X X B X A,
(21)
which means that the order of multiplication irrelevant when comes to different modes series of mode matrix product.
Lemma B.2. For mode-n matrix product, it satisfies the following associative property Koldaa & Bader 2009
X n A Xn B X X (BA).
(22)
Proof. Based on Lemma. B.1 the k-th term of Taylor expansion in Eq. can be rewritten as
W
x x,Ax = G x1 Oc
X x,+1 Ax"
(23)
Based on Lemma. B.2 the above Eq. 23 can be reformulated as
W
A, glh) X1 O
II x,+1 (Ax lns) E 12"
(24)
Proof finished.
C FEEDFORWARD METHOD FOR REDUCIBLE TAYLORNET
We summarize the feedforward method for Reducible TaylorNet below.
Algorithm 1: Feedforward Method for Reducible TaylorNet
Input :A E IRd
Output :y E R
Initialize (1)
U) _ (() (()), ( (!) kj
/xfrom the 1-st layer to the L-th layer forl=1, 1,... do /*from the 1-st order to the N-th order for N
for j = I.....ke do
if = then
2bj kj
(2)
Ax
else
(-1)
(1-1)
(1-11)
= (1-))
2 Uez
end
end
end
end
(L) g ( (L)
,..,,
D MODEL CONFIGURATIONS AND PARAMETER SETTINGS
In this section, we present the detailed model configurations and parameter settings for the following four different tasks.
14

---

Under review as a conference paper at ICLR 2023
D.1 EXPERIMENTAL DETAILS FOR IMAGE CLASSIHICATION
For image classification, our Taylor-Mixer is built on the existing MLP-mixer Tolstikhin et al. (2021). Thus we follow the experimental settings the MLP--iixr for pre-training and fine-tuning, unless stated otherwise.
Pre-training. We pre-train all the models at resolution 224 using linear learning rate warm-up and cosine learning rale decay. We set the batch-size to 1024 for Base and Small model due to GPU memory capacity limitation in our servers. Since the input data are normalized, we choose Taylor expansion point 2u 0 in model. The detailed model configurations and parameters settings are presented in Table 4 Our Taxlor-Mixer is set to 2 and 4 layers in our experiments. to 110 and 140, respectively. We
The Tucker rank of input and output factor matrices describe the rule of thumb for choosing the ranks as follows. For a N-th order Taylor layer with input and output rank Fin.k.j and k the effective width of this layer is approximately min Ok 1j=1 Ek=1 Fou Therefore, in order to achieve larger width with fixed num- ber of parameters, we should set Σk=1 E Jmink.j = Σk=1 Fout,k- Then we can scale rin.k. and k together to adjust the number of parameters of the model.
Table Configurations of Taylor- architectures different model scales: Small and Base.
Specification
Small/16
Base/32
Number of layers
2
4
Rank input factor matrices
110
110
Rank of output factor matrices
140
140
Patch resolution P P
16 x 16
32 x 32
Hidden size
512
768
Sequence length
196
49
Dimension for channel-mixing D.
2048
3072
Dimension for token-mixing D,
256
384
Parameters (M)
17.2
34.4
Initialization Å1 A2
0.99, 0.01
0.99, 0.01
Fine-tuning. For a fair comparison, we follow the experimental settings in the MLP-Mixer work. We use momentum SGD optimizer and cosine learning rate scheduler with a linear warm-up. The batch size of fine-tur -uninu is set to 512. We also use gradient clipping at global norm 1. In addition, we do not use dropout, the same MLP-Mixer.
D.2 EXPERIMENTAL DETAILS FOR DYNAMICAL SYSTEMS
In this experiment, we leverage lny-l TaylorNet based on Tucker decomposition. The rank of each dimension in the core tensor is set 16. In addition, the batch size is set to 128. We use Adam optimizer with learning rate 0.001.
D.3 EXPERIMENTAL DETAILS FOR NLPP
For sentiment analysis in NLP, we follow the same experimental setup in the PNL.-Mixer Fusco et al. (2022) unless otherwise stated. Following pNLP-Mixer, we set the batch size and hidden size to 256 and 256 respectively. We use Adam optimizer with learning rate 10 Different from PNLP-Mixer, the length of input tokens is set to 512. we use BERT embeddings for a token by averaging the embeddings of its subword units. order to make the number parameters similar to that of pixerMixer, choose 2-layers Taylor-NLP with the rank of 30 and 50 for the input and output matrices respectively. We also use dropout of 0.5 and weight decay of 0.01 to mitigate overfit problem. Initialization λ1, are 0.99, 0.01.
15

---

Under review as a conference paper at ICLR 2023
E ANALYSIS ON TAYLOR INITIALIZATION IN PROPOSITION 4.1
In this section, offer the theoretical analysis Taylor initialization in Proposition 4.1 First, introduce two lemmas used to decompose the variance of the output variables.
Lemma E.1. Suppose that (i) w1 is independent to W2, and 22, (ii) is independent to and B2, and (iii) B[wi] E[wz] 0, then have
Cov [wi W22r2] 0
(25)
Proof.
Cov(wix1, W2222 = E W W2222
E 1/ [[02222
E [w EE ueg [R [2)
I3 Eu [ B[
(
(26)
O,
completing the proof.
Lemma E.2. [[]] and wy independent wi and zi, for all j, # J. then we have
Var
Σ
W = Σ
Var[w]]E[x
(27)
Proof.
Var
Σ w
Σ
Var Fu
ΣΣ JP
2Corfw,,, myzy
(28)
According to Lemma
we can eliminate the second term above. Thus we have
Var
0,33
Varlw,at
E R|w [ [a
Σ E i [[r
Varlws [[{]]
(29)
completing the proof.
Next, introduce conjecture which will be used in main proof.
Conjecture F.1. For random vector x. following standard multivariate Gaussian distribution and for arbitrary ls S d, we have
(d + 2k -2)!!
Σ
E Blar,, a ... 2 (d 2)!!
(30)
11=1.i==1,....
where i ii denote the indices of xx, d is the dimension of the input and k is the order of the Taylor series expansion.
We first offer proof of this conjecture for small l (k: = 1,2, because we often choose lower-order Taylor expansion for each layer TaylorNet considering the computational cost.
3, 4) using enumeration below. This is
Proof. We use the property of Unit Gaussian distribution that E[ = (p 1)!!
(31)
where follows Unit Gaussian distribution and p positive integer.
16

---

Under review as a conference paper at ICLR 2023
For h 1, Eq. 30 can be rewritten
Ble = d
B
(32)
For 2, Eq. 30 can be rewritten and proved
Σ
E
a dx d(d 1) x d(d+ (33)
i 1,10=1
For = 3, Eq.
30 can be rewritten and proved as
El
d
d
C2.
Σ E|a x
Σ
Ela at si
f1
d x 15 + d(d - 1) 3 + - d(dd ( x
d(d () +4)
(34)
For k = 4, Eq.
30 can be rewritten and proved as
Σ
d
Ci
d
Σ
Ela + +.
Σ
El 2
+
Σ
Bla x
2
$1,8277i11
$1,82
C
B|x,, -
Σ
E[x,, x. xt, x.
in
d x 105 + 1
9+
d(d -- x3
d(d + ())
(35)
completing the proof.
Based on the above proof for small k; we can extrapolate Conjecture E. to all t S d. We have empirically validated that the this conjecture still holds for d E 1, 1..... 64, k < d using computer simulation. Nevertheless, we are still attempting to prove it thoroughly for our future work.
Based on the above lemmas and Conjecture E.1 offer the proof of Proposition 4. below.
Proof. We first define two hidden features in the Taylor layer 2n Var((hs)s] denotes the variance of hg. and
(In y
[In T ())) and h = G.ăz. Let oh
Ps.k = [[ {] denotes the second order origin moment of žμ.
We can first derive the relationship between the variance of ( and hg. According to Eq. 10 in TaylorNet, we have (1) B + Σn k= Oghi. And it can be decomposed into
Tout.
y B,
Σ
Σ
(k)aa(ha)
(36)
17

---

Under review as a conference paper at ICLR 2023
Therefore, according to Lemma E.2
we can derive the variance of y
Tout,
(o
B(sn))
(37)
1=1
Given that [[(h))]] = 2x] O, the above Eq.
37 can be further simplified as
N
(of2)²
2
oou R0
(38)
Next, we establish the relationship between o and Ma,k- We can decompose h2 &R into the following formula.
Fin.ke1
Fin.kekk
(ha)s =
Σ
Σ
(itk)
(39)
Therefore, according to Lemma E.2
we can derive of
o k
II
(40)
==1
Next, we establish the relationship between Vs and (o ( We can decompose ãة = [Iv y(-11) & ... &
[11 ((1-)) into
Σ Ian Lene (y (1)( ... Σ (RRa)Sn (y
(2x)ixx..xt x.xx
(41)
Therefore we can derive Ve,k as
V=,k = (Zx ) x
(42)
(arssi (1)))
Iisk (uklien ( (1)-1))))
E
E
2
(In J (3) 11 (2- >
(Kk) (1kn)) (1))
2x
51
d
E [IAR)
juctu ... (IaI) ] (1))) ... (--)
Ju
d
Σ
(1-)) (1-)
o T.k
J1
jn.
0T.k
E
(11))
(1)) }3
(43)
J1
dn
According to Conjecture
we can further simplify the above Eq. 42
Vs 2k (d + 2k 2)!!
(d - 2)!! (a (1-11) 2k
(44)
Combining Equation 38
40 and
we have
(0)))² =
Σ (..
N
II
k
(d 2h: 2)!!
(d - 2)!!
T
2h
(1
24
(45)
completing the proof.
18

---

Under review as a conference paper at ICLR 2023
Initialization. In order to stabilize the model training of TaylorNet, should choose oo & o such that ( (o Namely, any combinations of of G of satisfying the following equation is viable choice for initialization.
(o
II IIm.t.aa..
( ( (d- 22!!
-
2h:
2)!!
1,k
2k
=1
(46)
In addition to the requirement of ) ( - another desirable property is that the second
order moments of intermediate features h&s 2x should also be equal to the variance of the input.
Namely, we would also like to ensure that o = 12 = (og (o) Consequently, we should choose the initialization
1
(d 10-22) 2)
60 = >
GG
2k
Vout,
[T Kin.k.j
(d + 2k: 2)!!
s.t.
A =1
(47)
F ANALYSIS ON REDUCIBLE TAYLOR INITIALIZATION
In this section, we will elaborate the initialization method for Reducible TaylorNet (R- (R-TaylorNet). We keep using the notations in Section 4.2) 4.3 and Since the original input and output variable y((-1),yl (D) are omitted in reduced TaylorNet, will alternatively examine the relationship between the variance of (t-1) and h() Recall that (0) is defined in 4.2 and
h
(0)
[ G
211
G
(
(1) def
h
def
22
21
(48)
h
G N
NN
8 ... @
(0) 2N
Similar to the analysis in Section 4.3 we assume that 1) all elements in (1-1) follow independent zero-mean Gaussian distribution, 2) the weights T2, G5 are initialized independently with zero mean. 3) Ug is initialized to 0.
First, we can establish the relationship between (o )2 and 22 in the same way as described in Section E which can be written
(o h
II Tin.koo
(49)
Next, we establish the relationship between uaA and (o (11))) In reducible TaylorNet, 2n is calculated as
2kj = Ukj Tkyh (1-1)
21 2k1 & Zkk
(50)
Below, we introduce more fine-grained block multiplication notation of Ti (1-1)
h
h 11-)
T Tryh (1-1)
(1-1)
Thya
TkjNl]
(51)
h (1-1)
19

---

Under review as a conference paper at ICLR 2023
Given that Uy is initialized to O, can decompose the above matrix multiplication into
Tout.
N
Tuu.1
Σ
(Trkm)iss (/ 1
(Trimdis (hnh")) (52)
When choosing the initialization variance for the original TaylorNer as shown Eq. 16 we can set different d to scale the importance of k-th-order term. Similarly. in Reducible TaylorNet, we would also like scale the variance of Tkjm for different m. Namely, let of of.km. denote the variance of Tk Tajm and of be the standard variance for Taj. then they should satisfy 77 T. = A
Now we focus on 2-order Reducible TaylorNet. We can derive V2, for ts = 1, as
Fian Σ
P..1 =
((Tm))a ()
(f (1)))))))) )11))
Fout.1 (A1 + do) 111
(53)
E
Σ
((T2n Ji (h) )s (Tai2)ins (h2 (1))))
31
Fun.
(T221 (i (11111))5 + (Trza)ig (112 (122-11)
(An + 222 (1))))))))
(A + >2(11 (=())-))))
20/2 1111)33 2A + 23) r2n
(54)
Initialization. Using the same methodology in Section we need to choose ,7,,, o such that (oh,))² == (0 (1 On the other hand, we would also like to ensure that Vs.A = (oh 1-11) Hence, and 54 2-order Reducible TaylorNet in our paper should use the following
according to Eq. 49 53 initialization
1
c 11 Kin,k.j
o7
T7.
1out,
(2 (A} + X2) + )
s.t. A + 1
(55)
20


Reviews:
[{"review_id": "dmy1kMSzXXf", "reviewer": "Reviewer_LGk5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new architecture that relies on polynomial expansions. The idea is to use polynomial layers, where each layer expresses a polynomial expansion of order $k$ (where $k=2$ in practice). The new architecture uses a Tucker decomposition to reduce the learnable parameters. The authors also introduce a variant called reducible-TaylorNet, where the parameters are further reduced using a re-parametrization. The architecture is validated in small image datasets (CIFAR10/100), two dynamical systems and a small scale NLP task. ", "summary_of_the_review": "The idea of using polynomial expansions is interesting, while the use of the Tucker decomposition to reduce the parameters can be a useful tool. However, the paper currently has several flaws as identified above, making it hard for the reader to appreciate the contribution. At the same time, the experiments are weak. Nevertheless, if the revised version addresses all of the concerns, I am willing to reconsider my rating. ", "strength_and_weaknesses": " * [+] Improving the generalization in models that non-black box DNNs is important. \n\n * [+] Using the Tucker decomposition in a smart way in Mixer-like architectures is new to me, even if the Tucker decomposition and polynomial expansions have been used in the past in vision and deep learning approaches. \n\n * [-] Several technical parts are unclear, while the paper requires proofreading to make it more understandable.   \n\n * [-] The empirical evaluation is weak. The MLP-Mixer considered as a baseline is evaluated both on Imagenet, and VTA-1k which includes 19 datasets. Similarly, $\\Pi$-nets that are considered another baseline include more experiments in classification, generation, or mesh representation learning. \n\n * [-] The literature review on both Tucker decomposition and its use in deep learning and the polynomial nets could be significantly updated. (see below)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DjQtpJFvRRQ", "reviewer": "Reviewer_D6Yc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a new type of neural layer, which is inspired by the well-known Taylor expansion. In particular, for each layer, the learning model is represented by a Takyor summation within two orders. To address the high dimensionality of the Hassien tensor, the low-rank Tucker decomposition is taken into account. This paper also discussed suitable ways for the initialization of the new layers. In the experiments, this paper considered various tasks, including image recognition, identification of dynamical systems, and NLP.", "summary_of_the_review": "I agree that the polynomial learning model (such as the proposed TaylorNet) would be a good candidate as a universal approximator with potentially better interpretability than the existing deep neural networks. This paper clearly pointed out these potentials and also mentioned the main challenges, such as the dimension explosion and the stability. However, all these challenges are still not resolved well so far in this paper (I cannot believe that only using a good initialization can guarantee the stability of the model in a deep regime).\n", "strength_and_weaknesses": "++Strength\n\nThe new model can achieve comparable performance to some deep neural networks on many tasks.\n\n— Weakness\n\nThe basic idea of the TaylorNet is not new. It is strongly close to the old-school polynomial network (not overviewed in the related works), which was widely studied 30 years ago. A good book on this point can be found in (Nikolaev and Hitoshi, 2006). More recently, similar ideas are discussed in the tensor community, such as (Qiu et al., 202 and Rabusseau et al., 2019). In these works, they discussed multi-linear functions as the learning model. The proposed initialization trick is also not novel. A similar idea has been discussed by Pan et al. (2022), which is also not cited in the paper.\n\n*Nikolaev, Nikolay, and Hitoshi Iba. Adaptive learning of polynomial networks: genetic programming, backpropagation and Bayesian methods. Springer Science & Business Media, 2006.*\n\n*Qiu, Hejia, et al. \"On the Memory Mechanism of Tensor-Power Recurrent Models.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.*\n\n*Rabusseau, Guillaume, Tianyu Li, and Doina Precup. \"Connecting weighted automata and recurrent neural networks through spectral learning.\" The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.*\n\n*Pan, Yu, et al. \"A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks.\" International Conference on Machine Learning. PMLR, 2022.*", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8BUsysibHz", "reviewer": "Reviewer_9i73", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a neural architecture called TaylorNet. It is designed to mimic a taylor-series approximation of a multivariate function. The network itself is expressed as a composite function of Taylor layers, where each layer is a 2nd order taylor expansion. A key characteristic of the model is that it is free of non-linear activation functions, and the paper also proposes a customized weight initialization method. Experiments are performed on three tasks: image classification (CIFAR-10 and CIFAR-100), dynamical systems (Duffing Equation, Non-Linear Fluid Flow), and sentiment analysis (IMDB).", "summary_of_the_review": "\nOverall, the paper presents an interesting architecture based on the idea of Taylor approximation of multivariate functions. However, a majority of the conceptual contributions in this paper already exist in the literature. From a practical perspective, the contribution of TaylorNet over existing architectures is unclear.\n\nPost Rebuttal Update:\n\nI read the revised paper, comments from other reviewers, and the author's responses. The rebuttal addressed some concerns, especially related to experimental comparisons to $\\Pi$-Nets and SINDy. I increased the rating for the paper, but there are still some outstanding concerns.", "strength_and_weaknesses": "Strengths:\n1. The custom Taylor initialization is interesting, especially in the context of activation free networks.\n2. The concept of activation free networks is interesting, although its practical utility is unclear.\n3. Utility of TaylorNet is evaluated across a diverse set of tasks, image classification, sentiment analysis and dynamical systems.\n\nWeaknesses:\n1. The motivation for the work is not clear, especially given the existence of $\\Pi$-Nets. As far as I know, $\\Pi$-Nets are very similar, one can use 2nd order approximation in $\\Pi$-Nets and there is a version of it without activation functions as well.\n2. The motivation for networks without activation functions is also not very clear. TaylorNet is a composite function of TaylorBlocks. While TaylorBlocks themselves are interpretable, TaylorNet itself is too complex to interpret. The motivation is even less clear in tasks like image classification. \n3. Experimental comparisons to important baselines are missing. For example, $\\Pi$-Nets [1] in the case of image classification, SINDy [2] and its variants in the case of system identification for dynamical systems.\n4. Objectively the results are middling at best. There are a plethora of alternative CNN based architectures that achieve similar or better performance on CIFAR datasets compared to the results presented here. The advantage presented by TaylorNet over such models is unclear.\n\nOther Comments:\n- The claim about interpretability is too strong. The dynamical systems example is too simplistic.\n- In the experiment in 5.1, the conclusion that Taylor initialization is better than Xavier or Kaiming initialization is drawn from a simple two-layer network. Can you comment on whether this is expected to hold over larger/deeper networks as well?\n\n[1] $\\Pi$-nets: Deep polynomial neural networks, CVPR 2020\n[2] Discovering governing equations from data by sparse identification of nonlinear dynamical systems, PNAS 2016", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject