Highly Parallel Deep Ensemble Learning
Authors: Xiao-Yang Liu
Zeliang Zhang
Xiaodong Wang
Venue: NeurIPS 2022 Submitted (2022)
Keywords: parallel
deep ensemble learning
spectral tensor

Abstract: In this paper, we propose a novel highly parallel deep ensemble learning, which leads to highly compact and  parallel deep neural networks. The main idea is to first represent the data in tensor form, apply a linear transform along certain dimension and split the transformed data into different independent spectral data sets; then the matrix product in conventional neural networks is replaced by tensor product, which in effect imposes certain transformed-induced structure on the original weight matrices, e.g., a block-circulant structure.  The key feature of the proposed spectral tensor network is that it consists of parallel branches with each branch being an independent neural network trained using one spectral subset of the training data. Besides, the joint data/model parallel amiable for GPU implementation. The outputs of the parallel branches, which are trained on different independent spectral, are combined for ensemble learning to produce an overall network with substantially stronger generalization capability than that of those parallel branches. Moreover, benefiting from the reducing size of inputs,  the proposed spectral tensor network exhibits an inherent network compression, and as a result, reduction in computation complexity, which leads to the acceleration of training process.  The high parallelism from the massive independent operations of the parallel spectral subnetworks enable a further acceleration  in training and inference process. We evaluate the proposed spectral tensor networks on the MNIST, CIFAR-10 and ImageNet data sets, to highlight that they simultaneously achieve network compression, reduction in computation and parallel speedup.

Highly Parallel Deep Ensemble Learning
Anonymous Author(s) Affiliation Address email
Abstract
In this paper, we propose novel highly parallel deep ensemble learning, which leads to highly compact and parallel deep neural networks. The main idea is to first represent the data in tensor form, apply linear transform along certain dimension and split the transformed data into different independent spectral data sets; then the matrix product in conventional neural networks is replaced by tensor product, which in effect imposes certain transformeddinduced structure on the original weight matrices, e.g., a block-circulant structure. The key feature of the proposed spectral tensor network is that it consists of parallel branches with each branch being an independent neural network trained using one spectral subset of the training data. Besides, the joint data/model parallel amiable for GPU implementation. The outputs of the parallel branches, which are trained on different independent spectral, are combined for ensemble learning to produce an overall network with substantially stronger generalization capability than that of those parallel branches. Moreover, benefiting from the reducing size of inputs, the proposed spectral tensor network exhibits an inherent network compression, and as a result, reduction in computation complexity, which leads to the acceleration of training process. The high parallelism from the massive independent operations of the parallel spectral subnetworks enable a further acceleration in training and inference process. We evaluate the proposed spectral tensor networks on the MNIST, CIFAR-10 and ImageNet data sets, highlight that they simultaneously achieve network compression, reduction in computation and parallel speedup.
2
10
11
13
14
15
17
19
21
Introduction
23
Deep neural networks (DNNs) have made impressive successes many applications, such as computer vision 2 4 online game 6 natural language processing []]] au- tonomous driving I
24
25
However, DNNs are memory- which are two major challenges for wider adoption, e.g., in Internet of Things (IoT) applications
13 and robotics [14
26
and computation-intensin
27
[17]. Modern DNNs may have billions of parameters that consume exces- sive amount of memory and usually require long training time. For example, AlexNet [18] consists
28
29 of three fulne-con layers and five convolutional layers, containing 60 million parameters and consuming about 250 MB of memory and about 40 hours for training on ImageNet data set [19]
30
31
To mitigate the impact of memory and computation intensive, there are two types of techniques most investigated by researchers, namely model compression and parallel training. The model compression methods, which represents the conventional neural layers in DNNs by compact layers,
32
33
34
such as the strucred linear layers, tensor layers, etc., are one of the most efficient methods to reduce memory consumption of DNNs. Besides, the proposed parallel training methods can be generally categorized into data parallelism and model parallelism methods. It should be noted that tensor layer provides potential parallelism for efficient GPU computing as well the reduction of the
35
38
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Table I: Different tensor layers. An N2 x N2 weight matrix organized into a d-th order tensor with the size of each dimension n. i... N n². and is the tensor rank.
Methods
Model size
Inference time
Fully connected layer
O(N4)
O(N4)
Low-rank matrix [29]
O(N*r)
O(N*r)
CP tensor layer [31]
O(nr)
O(nr)
Tucker tensor layer 32
O(nr +r²)
O(wr)
HT tensor layer 32
O(dn'r + ra)
O(dnv
Tensor-train layer
29
O(dnr²)
N2d
38
memory consumption. Thus, it naturally motivates us to employ tensor layer to achieve both the model compression and parallel computing.
39
40
In this paper, we propose unified approach that simultaneously achieves both model compression and parallel learning without communication overhead. The key technique is novel spectral tensor layer that enables joint data/model parallll implementation of DNN as follows: 1) The training data set split into multiple orthogonal spectral sets; 2) The neural network is split into parallel branches with each branch being a conventional neural network, that are trained asynchronously and independently on the corresponding spectral sets; 3) The outputs of the parallel branches are finally combined to yield an overall neural network with substantially stronger generalization capability than that of those parallel branches.
41
42
43
45
46
47
48
The remainder of this paper is organized as follows. Section 2 presents an brief overview of the 49 study on the model compression and parallel computing. Section presents multiple spectral tensor layers, including fully connected layers, convolution layers and recurrent layers. Section 4 presents the experimental results and we conclude this paper in Section 5.
50
51
Related Works
52
53 Network compression: Consider a linear layer that is a central building block of modern DNNs, where an input æ E N (c.g., an N x N image) is transformed by weight matrix W E N**?² D. i.e.,
54
Wæ E R The memory size for storing W is O(N*) and the computational complexity of the matrix-vector product also O(N*), both are very demanding for smartphones, robots, and embedded devices.
56
58
23 showed that over 95% of the parameters are redundant, thus the network can be greatly compressed. The structured linear layer imposes certain structures on W. including circulant 22
Many works 20
22
59
60
22
24), prrculantbblck 22 26), and Teoplitz-like structures (2 (7 [3)
61 For example, for a circulant weight matrix W [21 22 [27 24), the memory size becomes O
62 and the computational complexity becomes O( log N). at the expense a slight drop in the
inference performance.
64
On the other hand, the tensor layer exploits different types of low-rank tensor representations of For example, the weight matrix W E N= has 66 rank r < N2, such that W = AB, A E R xr, B E RrXN² then the memory size becomes O(222r), and the computational complexity becomes O(2 Table 1 summarizes the model size and computational complexity for various tensor representation schemes.
65 the weight matrix W
28
29
30
31
32
87
68
Parallel training: Existing works on parallel machine learning take advantage of either data- parallelism 333 34 35 or moodel-pralleeiin [36] 37
70
38 or both [39]. In the data-parallel approach, the data are distributed among multiple processors that apply the same model [3]. The processors
71
72 periodically exchange the outputs and gradients. In the model-paralle approach, exact copies of the
whole data set processed by multiple processors that operate in parallel different parts of the
74 same model [3). There are frequent aggregation of model parts and distribution of gradients 40).
From the above discussion, we see that the computation associated with a DNN can be speed up via 76 two separate ways: one is through model compression so that reduced number of weight parameters leads to reduced number of multiplications and additions; and the other is through parallel computing,

---

Input
Hidden Layers
Output
Subnetwork Family
Fully Connected
Spectral Tensor loyer
Convolutional
W
Spectrol Tensor loyer
:
Spectral Dataset
Ensemble
Spectral Tensor loyer
Subnetworks
Figure I: The framework highly parallel deep ensemble learning.
where the speedup due to distributing the computation among parallel processors, at the expense of 79
78
communication overhead.
3
Highly Parallel Spectral Tensor Networks
80
81
3.1
Overview
82
As shown in [ a batch of data is firstly pre- pre-processed to the spectral dataset and splitted into Q independent subsets. Then, Q different subnetwork, namely spectral tensor layer, works Q independent joint spectral dataset to output own result. Last, the final result is the ensemble result from Q subnetworks. Tne subnetwork family includes the fully connected spectral tensor layer, 86 convolutional spectral tensor layer and recurrent spectral tensor layer. The feature of joint data and model parallel makes it suitable for computing GPUs. The operations in the forward pass, such as t- t-rruuuct further provides potential high parallelism for acceleration on GPUs.
84
88
3.2 Notations and Basic Tensor Operations
Scalars, vectors, matrices and tensors are denoted by lowercase, boldface lowercase, boldface capital, and calligraphic letters, e.g.. E IR, a E I A TRaixM A E IR1 xns, respectively. We use 92 A(:, ;, k), A(:,J, :), :(i,;;;) to denote the frontal, lateral, and horizontal slices.
90
91
93
Given an invertible discrete linear transform C RT - C3 let L and its inverse --1 be taken along the third-dimension of third-order tensors. That is, for A E XR2Xn, A = L(A) E C X Xns,
94
with A(i.j,.) :) C(A(i, ) 1, 1,... ni, 1,..,, 112. And for Ā CD) Xng x L- (A).
85
with A(i :) -- (A :)), i 1,... R 1...., n2. The general spectral tensor product
96
42 43 is detined as
C - A L- (((A) A L(B)),
(1)
where A denotes the frontal-slice wise multiplication, i.c., for Ā E C xns B E C" Xmgxna A A B, then C(:..,A k) = A(:, k) B k), & 1,.., The t-product is special case 45
98
99
100
of I where the transform
the DFT
101
For a tensor A E T Xn2x we define the following operations. bcire(A) E
R TR organizes
102
its ng frontal slices into block-circulant matrix
Ali,;,
A(a,;naa)
A(;,;2)
Al;,;,
1)
A(;;;3)
bciro((A)
(2)
:
A()
A(;,,n3 1) A(;,,1)
Further, unfold(.) is defined as
unfold(.4) = [A(;,;1) ... A(t.ina) k
3

---

104
and fold(-) organizes it back to A, such that
fold(unfold(A)) = .A.
(3)
Given A R xns and B TR"" X the t-product 45 can be expressed as follows
A *2 = (bld( unfold(B)) E R X
(4)
The vec(-) operation maps matrix in R" xaa into a vector in R" while vee v-)= is the inverse mapping.
106
107
3.3
Fully Connected Spectral Tensor Layer
109
conventional --layer fully connected network takes m input vectors each of size &g x and represents them as a matrix X E 186 For example, if each input vector represents a color image of size x 3, then % 3n Each input vector will be classified one of the L classes.
110
The network parameters at the j-th layer include a weight matrix W E [R and an offset RC xm and the forward pass can be represented as
112
113 B
b
b
X = (W'. + j N 1,
(5)
where X E xm and (-) an element-wis activation function, e-g.. linear, sigmoid, ReLU, 115 and softmax. The last, i.e, N-th, layer produces the output Y R 4xx corresponding to the m input
114
116
vectors, where
X W X-N
(6)
Y= Y=f(X f(xx).
(7)
117
and the output function f(X) operates on the columns of X, i.e., (() maps X(; 8) to an output score vector (t, RR representing the probabilities that the s-th input data vector x belongs to different classes. For example f(.) can be a softmax.
119
120
For our proposed fully connected spectral tensor network, the input data vectors are organized as tensor A x For the n X n x 3 color image example, we can set Q 3n and to n, namely each image is lateral slice of X Using the weight tensor WA E R x and offset
121
122
tensor B R&G xmxx a fully connected tensor layer corresponding to (5) and becomes
123
X e(V
x Bd) j I,... N
(8)
N
W
N-1
(9)
where X E x the spectral tensor product is given in (I. and the censor-activatio function
125 o(-) under transform L is defined by applying the conventional element-wise activation function (-))
126
in the spectral domain, i.e.,
e(x) - (a((2)))
(10)
127 A salient feature of the proposed spectral tensor network is the fully parallel implementation. Specifi- cally, for X' T denote X E cexxxQ as the transform of X along the third dimension, i.e.,
128
Xi(i, 8, (i 8, :)), i 1,.... f, 1,... m. Denote further X,, (;,;,g). Then according
129
130 to can be split into Q branches of matrix computations
X
W
X X + BB
(11)
N- = 1,..... Q
(12)
q
where X E xm B =
E C6 xm and W E x6
131
9 has low tubal-rank [45] [46] such that W 133 C Dj, where Ci E xrxx, Dí E Crxt,-1 x and & min Lo lo, L. Correspondingly, the
132 We further assume that the weight tensor Wi in
134 weight matrix of each branch has low-rank structure, i.e.
W C D 1,... Q
(13)

---

where Cj E C and D E Crxt, 1, Then, 11--12 12 become
135
2 q X j I.... N,
(14)
q = o C Z B¹) j = ,,,, -1, 1,
(15)
X ON Z
(16)
where Z E Cr .9=1.....
136
137
Therefore, an N-layer fully connected spectral tensor network in 8 is split into 2N-layer network, such that each layer in (TT) is now implemented by two sub-layers, namely a linear layer
138
(1) and nonlinear layer (1). while the N-th layer in (12) is implemented by two linear sub-layers, namely (14) and (T6). There are multiple parallel matrix multiplications with the same size and along
140
the same dimension in Q subnetworks. Therefore, we employ the batch matrix multiplication using GPUs to accelerate the computations.
142
143
Finally, we specify the network output. At each branch g, the output function f(-) is applied to the last layer output X,N, in (7, i.e.
YU = f(x;),,) 9 = 1,...,Q. Q.
(17)
145
Finally the network output is the weighted sum of the outputs of the Q branches, i.e..
Q
Y=
W Yg s.t. Wq 20.000 0,
Σ
w, =
(18)
9=1
The loss function can be cross-entropy function as follows:
I. Loss = Σ Σ (c) = 1) In s)).
(19)
a=1
Once the spectral tensor network in Fig. is trained, for inference, given new data sample x E R we first matricize it into X E Rfxx and then take transform along each row to obtain X. We input
147
148
149
the q-th column of X, i.e., X (t. q), to the q-th sub-network and obtain the output y/ 1,.... Q.
The final output is then y Σg Wgyg.
150
151
3.4
Convolutional Spectral Tensor Layer
152
A convolutional neural network 1 47
takes input images each of dimension Ho x Wo x Co. i.e.,
153
each image has a size Ho x Wo and the number of channels Cp, and represents them as a fourth-order tensor X E TR. Ho CCx The input to the j-th layer is Xi-1 E R 1x,
154
which is
processed by convolutional kernel W E x xC, and an offset B E #Hf CC
yield Y " xW, XC, xm with H" H - Hz + 1, W W;-1 W + where
156
Cy-1 W/ -1-1
wh,,w,c,)) = Σ Σ
Σ
Xi i, w (i, d, c) + B (h, u. w.,,)),
(20)
= H", w 1, W" c l, Cy.
157
To introduce the convolutional spectral tensor network, we represent 20 as matrix product form 158 that is similar to 5
Y WT +
+ B¹,
(21)
159 where YJ E W xm Wi e R W, and X -- W
180 are formed from YJ, W and Xi-1 In particular,
Y (t, 8) vec(unfold (Y (1)):,8))):
(22)
X - (;, 8) = (x-,,, 8 = L,..,
5

---

Assume that C2 = nB for j = O,..., N, then similar to
8
21
leads to convolutional tensor
162
layer
y W W
(23)
where V E R'WWOX W E W 1 E' E B
N E BB/ and
163
184
Consider the case when L is DFT, according to 44. 23
be written 21 where Y' = unf)ld()"), X = )nfold(2('-), B = unfold(B)) and W = beire ( E
186
R - has block-circulant structure, namely B x B blocks organized in
circulant form and each block has size HHWW"n x Hj-1 -i- Recall that W (1 is derived
168
from the convolutional kernel W E RR "R""""" xxBBBBB in (20). following a linear mapping that is
169
consistent with (22). Therefore, the block-circulant structure of W in (1) implies a block-circulant structure of each matrix W :.) in 20
170
171
172
Similar to the fully connected case in Section 3.3 the proposed convolutional spectral tensor network also features a fully parallel implementation. Specifically, for X' E 1g#Wnxxx denote X' =
173
((X) E CCWwxxxxx as the transform of X' along the third dimension. Denote X, (;;;;;
174
175
Then, (23) can be split into parallel branches matrix computations as follows
Yg W. X + By, b I...,,,
(24)
where Y E WffWWjnnxn nxm W C " W and X C 1nxm, To fully
177
utilize the parallelism B branches, we employ batch matrix multiplication on GPUs to accelerate (24).
178
179
convert
24 back to the convolutional form in 20 using an inverse mapping of (22) as follows
((,,)) = fold(vee -1 (t,(s))))
(25)
X (1):),8) fold(vee (X (3,)))); 1,... m.
180 For the b-th branch in 24 the input is - E C 1xW, 1xnxm, the output feature map is
Yo E CHf'sW xnxm and the kernel weight is W2 E CHfXWjxxxX Then for b 1,..., ..., B, 24
182 can be rewritten as
Yi,, w. c,:)
X ii C,d,)) WW (j(h, (26)
2= ==0
h =
W", c= 1....,n. n.
183
Similar to the fully connected tensor networks in Section 3.3 we apply the activation function in the spectral domain as follows
184
Z (Y C xW" xnxm
(27)
185
Then, the pooling operation is performed at the j-th layer of each branch, resulting in the output E C X xnxm
X
186
At the last layer, the output function f(-) is applied to XN as in 22
188
3.5 Recurrent Spectral Tensor Layer
189
To present the recurrent spectral tensor layer, We take the same case in Section 3.3 Different from the fully connected layer, the network parameters of the recurrent layer at the j-th layer
190
include weight matrix W - W e x W E -6-1xe and offset
192
B
RC xm and the forward pass can be represented as
Hi (W x W H B'), x = o2 w H²), j 1, N,t = 1, 1,,,...,
(28)

---

193 where X E R is the input matrix at the j-t time step, and Y, X."
194 For the introduced recurrent spectral tensor layer, the forward can be computed
Hi- (W x-- W H.- - BB),
(29)
X = e2 Hy), j = ,,.,, N, = 1,2,... ...,T, T,
where X E R
195
Furthermore, the implementation of Q branches for computations can be written as
Hy. e ( X27 W HiT BB), q,1 22 ( Hl,,): j = l,.,, N, L = 1, 1.2...2... T,
(30)
q.t E C..xm is the imput for the q-th subnetwork at the t-th time step, likewise for H.,-. There are 2Q parallel matrix multiplications with the same size and along the same dimension, thus we use the batch matrix multiplication on GPUs to accelerate (30)
where X²
197
g.f*
198
199
Performance Evaluation
200
We first describe the experimental settings, then present the results on the MNIST, CIFAR-10 and ImageNet data sets.
202
203
4.1
Data Sets and Performance Metries
204 We verify the performance of the proposed spectral tensor networks on the following three widely used data sets: MNIST [48) contains grayscale images of handwritten digits. Each image has
205
206
28 X 28 pixels. The training set has 60, 000 images and the testing set has 10, 000 images. 2)
207
CIFAR- 10 49 contains 60, 000 color images in 10 classes, where each image has size 32 32 x 3.3)
208
ImageNet-Ie IK [19]: It contains 12, 000, 000 training images and 50, 000 testing images with size of 224 x 224 x 3, labeled with the presence or absence of 1000 object categories that do not overlap
209
210
with each other.
We are interested in the following performance metrics:1) Compression ratio: the ratio of the conventional network size to the spectral tensor network size, which is the also the total reduction in computation due to the reduced number of non-zero network weights; Parallel speedup: the ratio of the training time of a conventional network to that of the spectral tensor network, due to the fully parallel training of all sub-networks; 3) Convergence: the loss value versus the training iterations: 4) Accuracy: the percentage of correctly estimated labels. Both the training and testing processes are executed on DGX-2 server (50) that has two 64 core AMD CPUs, NVIDIA A100 GPUs and
212
213
214
215
216
217
218
TB of memory. The operating system is Ubuntu 20.04 with CUDA 10.1. We use PyTorch [5]] to implement neural networks.
219
220
We summarize the compression ratio, the reduction of computation, and the parallel speedup in Table 2 They are theoretical upper bounds, while their actual values depend on data sets and implementations. For the compression ratio and reduction in computation, each fully connected network/ / convolutional network has two columns: the right one corresponds to select-the-best weighting, and the left one to other weighting methods.
222
223
224
225
4.2
Evaluation of Fully Connected Spectral Tensor Networks
For comparison, we consider conventional fully connected network (FC) [1] the INN [44), and the fully connected spectral tensor network (FC-tensor) in Section 3 All three methods use the ReLU activation function as a (-)) ()) in the hidden layers, the softmax function as the output function f(-) in the last layer, and the cross-entropy loss function in (19.. We use = 8 layers in each method and the DCT transform in tNN and the proposed FC-tensor method. The learning rate set to be 0.01, 231 the batch size was set to be 64, and we used the Adam optimizer [52].
227
229
For the MNIST data set, the conventional FC method has 28, 6 -.. -..=- = and L = 10. Both the tNN method and our FC-tensor method have n = 28, Q = 28, bg = = l 28, and
233
L =
10. Our FC-tensor method has 8. For the CIFAR-10 10 data set, the following parameters
7

---

Training Loss on MNIST
Training Loss on CIFAR-10
0.5
3.45
Subnet-1
0.45
FC
0.4
Supret-8
0.4
Sutret-1
Submeb
0.55
0.35
Subnet-19
0.3
0.2
0.15
0.1
0.1
0.05
0.05
0
Iterations
Iterations
Figure 2: Training loss of fully connected networks on the MNIST data set (left) and CIFAR-10 data set (right).
Table 2:
Upper bounds for model compression and parallel speedup. For fully connected networks, denotes the input size of each sub-network, convolutional networks, there are B branches.
rank value, and there are Q branches. For
Fully Connected
Convolutional (ID)
Compression ratio Reduction in computation Parallel speedup
(nn /2r), O(nQ
O(B), O(B))
(nQ/22r) (n (
O(B), O(B))
235
different: n = 32, = f- = -..- = 1, 124, and to :.. = 32. Therefore, our methods achieve compression ratio of 49x and 64x for the two data sets, respectively.
237
The training loss over iterations is shown Fig. 2 with the left one for the MNIST datas set and the right one for the CIFAR-10 data set. Our scheme converges faster than INN and FC, while the training process more stable than FC. The possible reason is that the FC-tensor has much less
239
parameters that more stable model can be learned from the same amount of data sample The
241
loss values our sub-networks are lower than both tNN and FC.
Training Loss ImageNet
12
AloxNat
1
10
NoxNatt Subnet
NlesNet-Subret-4
CycleMLP
Table 3: MNIST and CIAFR-10 data sets.
DycleMLP-Subreed 1ubbeet--
Methods
MNIST
CIFAR-10 10
CycleMLP-S P
FC
III
98.71%
59.19%
DycleML P-Steet--2
CycleMLP-S 4-S
INN
44
97.59%
44.50%
FC-tensor (average)
97.43%
47.24%
FC-tensor (weighted sum)
98.02%
48.13%
FC-tensor (geometric)
99.01%
48.33%
0
50
100
Iteratinss
200
250
300
Figure 3: Training loss on ImageNet-IK data set.
In Table report accuracy results on both MNIST and CIAFR-10 data sets. Among the four schemes for weighting the sub-networks. the geometric weighting gives the best performance. For the MNIST data set, all three methods achieve relative high accuracy, i.e., 97%, while our FC-tensor method reaches 98.36%. For the CIFAR- 10 data set, all three methods achieve relative
243
245
246 low accuracy, i.c., below 60%. This is consistent with the known fact that fully connected layers are
Note that use the same number of layers and the same batch size.

---

Table 3: Results on the ImageNet-lk 1K data set.
Methods
Accuracy
Size
Training Time
AlexNet [8]
63.44%
244 MB
40.8 h
Alexeeet-ppectral (average)
61.26%
130 MB
31.9 h
AlexNet-spectral (weighted sum)
58.01%
130 MB
31.9 h
Alextet-specttal (geometric)
62.26%
130 MB
31.9 h
AlexNet-spectral (select-the-best)
56.45%
32.5 MB
31.9 h
CycleMLP [53]
83.23%
103 MB
93.6
CycleMM.P-spectrall (average)
78.80%
76 MB
60.4 h
CycleMI.P-specrrll (weighted sum)
77.54%
76 MB
60.4
CycleMI.P-spectral (geometric)
83.20%
76 MB
60.4 h
CycleMLP-spectral (select-the-best)
72.45%
19 MB
60.4 h
not enough for the classification task CIFAR-10. Note that both tNN and FC-tensor achieve lower accuracy than the FC method.
248
249
4.3
Evaluation on ImageNet Data set
The ImageNet data set [19] is split into B 4 spectral subsets, where each image is organized into a tensor of size 56 x 56 x 3 x and then processed into a spectrum tensor using DCT transform. Note that the three RGB channels
251
processed independently.
253 Our proposed spectral tensor methods have the same structure in Fig. || where each branch is replaced by either AlexNet 18 or CycleMLP. We use the DCT transform in our spectral methods. We set the learning rate 0.01 and the batch size 128. We follow the standard practice in the community by reporting the top-1 accuracy on the testing set.
255
257
For the ImageNet data set, the training loss over training iterations shown in Fig. 3. Our spectral sub-networks have similar loss curve to their original networks. In Table 3 we report the accuracy, model size, and training time. For the AlextNet structure, our spectral network achieves 1.88x model compression and 1.28x speedup in training time, at the cost of an accuracy drop of 1.18%. For the
258
259
260
261
CycleMLP structure, our spectral network achieves 1.36x model compression and 1.55x speedup in
262
training time, at the cost of an accuracy drop of 0.03%.
Conclusions
263
254
In this paper, we have proposed a spectral tensor form of deep neural networks that is inherently com- pressive and allows communication-free parallel/distributee implementations. The data is organized into tensors and linear transform is applied along certain dimension, resulting in different spectral subsets. The overall network consists of parallel branches of networks, each independently performs training and inference on a spectral data subset. We tested the proposed spectral networks, including fully connected, convolutional, AlexNet, and CycleMI.P, on the MNIST, CIFAR-10 and ImageNet data sets, and results show that they can achieve relatively high accuracy with substantial network compression, computation reduction, and parallel speedup, compared with conventional networks. Limited by the pages, We do not provide experiments using the recurrent spectral tensor layer.
285
286
267
268
269
271
273
For future works, we would like explore ensemble-style approach model soup (54) that takes average over multiple trained models and achieves state-of-the-art performance on the ImageNet data
274
275
276 References
277
[1] Goodfellow, Y. Bengio, and A. Courville, Deep learning, MIT Press, 2016.
[2] Yang Liu, Peng Sun, Nickolas Wergeles, and Yi Shang. "A survey and performance evaluation of deep learning methods for small object detection," Expert Systems with Applications, vol. 172, pp. 114602, 2021.
279

---

281
[3] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. "Dynamic head: Unifying object detection heads with attentions." " in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7373-7382.
282
284
[4] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al., "Sparse r-cnn: End-to-end object detection with learnable proposals." in Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2021, pp. 14454-14463.
285
287
288
[5] Xiangjun Wang, Junxiao Song, Penghui Peng Peng, Zhenkun Tang, Wei Zhang, Weimin Li, Xiongjun Pi, Jujie He, Chao Gao, et al., "Scc: an efficient deep reinforcement learning agent mastering the game of starcraft ii," in International Conference on Machine Leurning. PMLR, 2021, pp. 10905-10915.
290
[6] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu, "Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning," " in International Conference Machine Learning. PMLR, 2021, pp. 12491-125000
293
[7] Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar, "Coach- player multi-agent reinforcement learning for dynamic team composition," in International Conference Machine Learning. PMLR, 2021, pp. 2860-6870.
296
297
88] Zhiqi Huang, Fenglin Liu, Xian Wu, Shen Ge, Helin Wang. Wei Fan, and Yuexian Zou, "Audio- oriented multimodal machine comprehension via dynamic inter-and intra-modality attention," in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, pp. 15098-13100
299
300
[9] Tianyang Zhao, Zhao Yan, Yunbo Cao, and Zhoujun Li, "Asking effective and diverse questions: a machine reading comprehension based framework for joint entity-relation extraction," " in P Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 2021, pp. 3948-3954.
302
303
304
[10] Cong Sun, Zhihao Yang. Lei Wang, Yin Zhang, Hongfei Lin, and Jian Wang. "Biomedical 305
306
named entity recognition using bert in the machine reading comprehension framework," Journal of Biomedical Informatics, vol. 1 118, pp. 103799, 2021.
307
308
[1]] Aditya Prakash, Kashyap Chitta, and Andreas Geiger, "Multi-modal fusion transformer for end-to-end autonomous driving." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7077-7087.
309
310
311
[12] Yingfeng Cai, Tianyu Luan, Hongbo Gao, Hai Wang, Long Chen, Yicheng Li, Miguel Angel Sotelo, and Zhixiong Li, "Yolov4-5d: An effective and efficient object detector for autonomous driving," " IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-13, 2021.
312
313
314
[13] Sudeep Fadadu, Shreyash Pandey. Darshan Hegde, Yi Shi, Fang-Chieh Chou, Nemanja Djuric, and Carlos Vallespi-Gonzalee "Multi-view fusion of sensor data for improved perception and prediction in autonomous driving," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp.
315
316
317
318
[14] Huu-Thiet Nguyen, Chien Chern Cheah, and Kar-Ann Toh, "An analytic layer-wise deep learning framework with applications to robotics," Automatica, vol. 135, pp. 110007, 2022.
319
320
[15] Radouan Ait Mouha et al., "Deep learning for robotics," Journal of Data Analysis and Information Processing, vol. 9, no. 02, pp. 63, 2021.
321
322
[16] Yinong Chen and Gennaro De Luca, "Technologies supporting artificial intelligence and robotics application development," Journal of Artificial Intelligence and Technology, vol. 1, no.
323
324
1. pp. 1-8, 2021.
325
[17] G. Menghani, "Efficient deep learning: survey on making deep learning models smaller, faster, and better," arXiv preprint (XXiv2210608900 2021.
326
327
[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," Advances in Neural Information Processing Systems, vol. 25, pp. 1097-1105, 2012.
328
10

---

330
[19] J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, "ImageNet: A lcrle-scale hierarchical image database." IEEE CVPR. Icee, 2009, pp. 248-255.
331
332
[00] M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. De Freitas, "Predicting parameters in deep learning," in Advances in Neural Information Processing Systems, 2013, pp. 2148 2156.
333
334
[11] V. Sindhwani, T. Sainath, and S. Kumar, "Structured transforms for small-footprint deep learning," in Advances in Neural Information Processing Systems, 2015, pp. 3088-3096.
335
[22] Y. Cheng, F. Yu, R. S. Feris, S. Kumar, A. Choudhary, and S.-F. Chang, "An exploration of parameter redundancy in deep networks with circulant projections," in IEEE International Conference on Computer Vision, 2015, pp. 2857-2865.
337
338
[23] M. Moczulski, M. Denil, J. Appleyard, N. De Freitas, Z. Wang, M. Zoghi, F. Hutter, D. Matheson, and S. Reed, "ACDC: A structured efficient linear layer," in ICLR, 2015, vol. 55.
340
341
[24] A. Prabhu, A. Farhadi, and M. Rastegari, "Butterfly transform: An efficient fft based neural architecture design," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12024-12033.
343
344
[25] C. Ding, S. Liao, Y. Wang, Z. Li, N. Liu, Y. Zhuo, C. Wang, X. Qian, Y. Bai, and G. Yuan, "Cirenn: accelerating and compressing deep neural networks using blrck-cireulant weight matrices." in Annual IEEE/ACM International Symposium on Microarchitecture, 2017, pp. 395-408.
346
[26] S. Liao and B. Yuan, "Circconv: A structured convolution with low complexity," in AAAI Conference on Artificial Intelligence, 2019, vol. 33. pp. 4287-4294.
349
350
[27] B. Gong, Jou, F. Yu, and S.-F. Chang, "Tamp: A library for compact deep neural networks with structured matrices," in ACM International Conference on Multimedia, 2016, pp. 1206- 1209.
351
352
353
[28] T. N Sainath, B. Kingsbury, V. Sindhwani. E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets," in IEEE ICASSP, 2013, pp. 6655-6659.
354
355
356
[29] A. Novikov, D. Podoprikhin, A. Osokin, and D.P. Vetrov, "Tensorizing neural networks," in Advances in Neural Information Processing Systems, 2015, pp. 442-450.
357
358
[30] W. Wang, Y. Sun, B. Eriksson, W. Wang, and V. Aggarwal, "Wide compression: Tensor ring nets," in IEEE CVPR, 2018, pp. 9329-9338.
359
360
[31] V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky, "Speeding-up convolutional neural networks using fine-tuned cp-decompositionn." ICLR, 2015.
361
362
[32] M. Yin, S. Liao, X.-Y. Liu, X. Wang, and B. Yuan, "Towards extremely compact rnns for video recognition with fully decomposed hierarchical tucker structure," in IEEE CVPR, 2021, pp. 1 12085-12094.
364
365
[33] J. Verbracken, M. Wolting, J. Katzy, J. Kloppenburg, T. Verbelen, and J. S. Rellermeyer, "A survey distributed machine learning." ACM Computing Surveys (CSUR), vol. 53, no. 2, pp. 1-33, 2020.
367
368
[34] Vipul Gupta, Dhruv Choudhary, Peter Tang, Xiaohan Wei, Xing Wang. Yuzhen Huang, Arun Kejariwal, Kannan Ranchandran, and Michael W Mahoney, "Training recommender systems at scale: Communication-effiiaae model and data parallelism," " in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp.
370
372
[35] Xiangyu Ye, Zhiquan Lai, Shengwei Li, Lei Cai, Ding Sun, Linbo Qiao, and Dongsheng Li. "Hippie: A data- parallaaalle pipeline approach to improve nemory-eeficieen and scalability for large dnn training." in 50th International Conference on Parallel Processing, 2021, pp. 1-10.
373
[36] A. L. Gaunt, M. A. Johnson, M. Riechert, D. Tarlow, R. Tomioka, D. Vytiniotis, and S. Webster, "AMPNet: Asynchronous model-parallel training for dynamic neural networks," arXiv preprint 17Xiv:1705.09786. 2017.
376
11

---

378 [37] An Xu, Zhouyuan Huo, and Heng Huang, "On the acceleration of deep learning model parallelism with staleness," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2088-20977
379
(38) Kabir Nagrecha, "Madel-parallel model selection for deep learning systems," in Proceedings of the 2021 International Conference on Management of Data, 2021, pp. 2929-2931.
382
383
[39] E. P. Xing, Q. Ho, P. Xie, and D. Wei, "Strategies and principles distributed machine learning on big data," Engineering, vol. 2, no. 2, pp. 179-195, 2016.
[40] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., "TensorFlow: A system for large- scale machine learning," in USENIX Symposium Operating Systems Design and Implementation, 2016, pp. 265-283.
386
[41] H. Kernfeld, M. Kilmer, and S. Aeron, "Tensor- tensor products with invertible linear transforms," Linear Algebra and its Applications, vol. 485, pp. 545-570, 2015.
389
380
[42] M. E. Kilmer, L. Horesh, H. Avron, and E. Newman, "Tensor-tensor algebra for optimal representation and compression of multiway data," Proceedings of the National Academy of Sciences, vol. 118, no. 28, 2021.
392
393
[43] X. Liu and X. Wang, "Fourth-ordes tensors with multidimensional discrete transforms," " arxiv preprint arXiv: arXiv: 170.001576, pp. 1-37, 2017.
394
395
[44] E. Newman, L. Horesh, H. Avron, and M. Kilmer, "Stable tensor neural networks for rapid deep learning," arXiv preprint 1xXiv:881.06550 2018.
396
387
[45] M.E. Kilmer and C.D Martin, "Factorization strategies for third-order tensors," Linear Algebra and Applications, vol. 435, no. 3, pp. 641-658, 2011.
396
389
[46] M.E. Kilmer, K. Braman, N. Hao, and R.C. Hoover, "Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging," SIAM Journal on Matrix Analysis and Applications, vol. 34, no. I, pp. 148-172, 2013.
401
402
[47] Z. Li, F. Liu, W. Yang, Peng, and J. Zhou, "A survey convolutional neural networks: analysis, applications, and prospects," IEEE Transactions Neural Networks and Learning Systems, 2021.
404
405
[48] L. Deng, "The MNIST database of handwritten digit images for machine learning research," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141-142, 2012.
[49] A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," Master's thesis, University Tront, 2009.
408
409
[50] J. Choquette et al., "NVIDIA A100 tensor core GPU: Performance and innovation," IEEE Micro, vol. 41, no. 2. pp. 29-35, 2021.
410
411
[51] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, and et al, "PyTorch: An imperative style. performaree deep learning library." in Advances in Neural Information Processing Systems, 2019, pp. 8026-8037.
412
413
414
[52] D.P Kingma and Ba, "Adam: A method for stochastic optimization," ICLR, 2015.
415
[53] Chen, Xie, C. Ge, Liang, and P. Luo, "CycleMLP: A MLP-like architecture for dense prediction," ICLR, 2022.
[54] Mitchell Wortsman, Gabriel Ilharco, and et al, "Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing inference time," preprint Niv:2203.005882 2022.
418
12

---

Checklist
420
421
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
423
contributions and scope? [Yes]
424
(b) Did you descrihe the limitations of your work? [No]
425
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
427
428
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
430
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
435
mental results (either in the supplemental material or as a URL)? [Yes] See Section 4.1.
435
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4,2 and 4.3.
436
437
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We do not supply experiments of recurrent spectral tensor layer.
438
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1.
441
442
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
443
(a) If your work uses existing assets, did you eite the creators? [Yes]
444
(b) Did you mention the license of the assets? [Yes]
445
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See Section 4.3.
447
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] All the consent is open-source.
448
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We do not use any personally identifiable information or offensive content.
450
452
5. If you used crowdsourcing or conducted research with human subjects...
459
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
455
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
456
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
458
13


Reviews:
[{"review_id": "syVPaW_9wjK", "reviewer": "Reviewer_thRQ", "summary": "This paper introduces a parallelization of DNN from a tensor perspective. The data is split into their spectral subsets and is passed into parallel subnetworks and finally, the results are based on an ensemble of the subnetworks. Specifically, the author provides detailed spectral counterparts for fully connected layer, convolution layer and recurrent layers. The tensorized structure naturally enables network compression at the same time.", "questions": "1. how to make sure that the divided subsets are independent and orthogonal to each other.\n\n2. What is the performance (accuracy, runtime, compression) compared to the sota architectures in the benchmark in the experiment session.\n\n3. Is there any way to demonstrate or analyze the generalization ability of proposed structures compared to original structures?", "limitations": "Limitations not discussed. ", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall idea is interesting and novel to some extent. By separating data into spectral subsets, the compression limit is extended as well as the level of parallelization. The experimental supports the claim that applying the technic speeds up the training and the inference.\n\n\nThe overall structure is well-organized. However, there may be space for improvement in conveying the motivation, ideas, and justifications.\n\nIn the introduction part, the author mentions that the proposed method has no communication cost. It may be better if the author could support such claims by comparing them with other parallelization ideas which have such costs. \n\nFor the main idea, the author mentions the independence of each subset and the orthogonality among each subset. However, how to obtain such subsets and a principal way to select the desired dimension (Q) is not very clear and well-justified. \n\nMy main concern with the work, besides the justification, is how significant and general it is. \n\nIn the experiment session, the author compares performance and runtime on FC, tNN, AlexNet and CycleMLP, but no state-of-the-art architectures on the benchmarks. \n\nAlso there is a lack of runtime comparison on other parallel DNN baselines. \n\nAs for the recurrent settings, the attention mechanism also achieves a certain level of parallelization, which may be worth comparing as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HeaCe6D9f2W", "reviewer": "Reviewer_JvnU", "summary": "The paper introduces a method to train DNN models using parallel ensemble learning. The key idea is to build networks with parallel branches and update each branch independently using split, independent data sets. The evaluation shows that such a method can lead to reduced computation and model size on MNIST, CIFAR-10, and ImageNet. ", "questions": "Please describe the relationship between the proposed method with respect to existing parallelism strategies for model training and MoE models.\n\nWhy choose AlexNet for the evaluation? It would strengthen the work a lot if it is performed on more standard MLPerf benchmarks, such as ResNet or BERT models. ", "limitations": "No, there is almost no discussion on the limitations of the proposed method. It would be better if the authors could discuss the rationale behind the chosen workloads and also the generalizability of the proposed method. ", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Interesting idea to combine parallel ensemble learning and compression. \n- The paper obtains preliminary promising results on training speedups and model sizes. \n\n\nWeaknesses:\n\n- The proposed method in this paper is highly related to parallel training and compression but appears to miss a lot of related work in both areas. On the training side, there are many existing works on reducing the memory consumption of DNN training, such as [1-3]. On the compression side, the proposed method looks closely related to mixture-of-experts (MoE) where a model has multiple parallel expert branches and each branch is sparsely activated during training, but it misses citing any MoE related papers such as MoE[4], Gshard[5], SwitchTransformer[6]. Overall, parallel ensemble learning is an interesting idea and could lead to something quite impactful. However, given its current status of not positioning it well with many existing works on parallel training and sparse training, it is hard to evaluate the significance and novelty of the proposed method. \n\nThere are some mismatched statements. For example, in line 27, the paper says modern DNNs have billions of parameters, but then the examples given are AlexNet, which has only 60 million parameters and is one the earliest network far from representing billion-scale models. \n\n[1] Rajbandari et al. \"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models\",  https://arxiv.org/pdf/1910.02054.pdf\n[2] Shoeybi et al. \"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism\", https://arxiv.org/abs/1909.08053\n[3] Huang et al. \"GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism\", https://arxiv.org/abs/1811.06965\n[4] Shazeer et al. \"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer\"\n[5] Lepikhin et al. \"GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding\"\n[6] Fedus et al. \"Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "F55Png4fp9s", "reviewer": "Reviewer_evpK", "summary": "The authors proposed a spectral tensor decomposition of a DNN into independent, parallel and computationally cheaper subnets that process corresponding spectrally decomposed data, resulting in speedup in training; ensembling of these subnets yielded reasonable generalization performance. \n", "questions": "- Though a recurrent spectral tensor layer was derived, it was not demonstrated with experiments.  \n- The example DNNs experimented are not representative or practically relevant. \n- Do spectral tensor transformers exist?  \n- It seems decomposition of data with orthogonal bases is not unique.  Nor is the number of components and the grouping thereof.  How does one choose optimal or reasonable decomposition?  \n- With a certain data decomposition, task-relevant information in different components naturally vary.  Corresponding subnet experts might need to have different capacities to realize maximal efficiency?  Or how does one equalize component importance, or do spectral pruning?\n- Fig. 2, why was the FC training numerically unstable, with a non-monotonic training loss curve? \n\n", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] Novel idea\n\n[-] Inadequate supporting experiments\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CJKH_Foss1j", "reviewer": "Reviewer_sYxS", "summary": "This work proposes a tensorial representation of data, and linear transformation and split into multiple spectral subsets. An independent neural network is trained on each of these subsets at training and combined through an ensembling approach. This reduces the computational complexity due to smaller data size per network and the parallel training of each subnetwork.", "questions": "* In related works, it would be helpful if the proposed approach is contrasted against other ensembling techniques that learn sparse subnetworks [1, 2]. Also, discussion on other data subsetting approaches []  will provide a good context.\n\n* For the fully connected spectral tensor layer, is it possible to set Q < 3n? if so, has the sensitivity to this been evaluated? Similarly, is the effect of the choice of B for the convolutional spectral tensor layer evaluated?\n\n* It is mentioned that the performance metrics of Compression ratio, Parallel Speedup, Convergence, and Accuracy are considered, but these metrics are not explicitly shown in table 3 and 4\n* How are the neural network hyperparmeters chosen for MNIST and CIFAR-10 FC-tensor method?\n* The performance of tNN and FC-tensor seems to be poor in CIFAR-10 compared to a vanilla FC, why is this?\n* Why not use a convolutional spectral tensor layer for CIFAR10.\n* Is the O(Q) and O(B) speedup seen during training? \n* A discussion on the limitations of the approach will be helpful. Can this be applied for residual networks?\n* Minor typos in multiple places \n   * Strucred linear layers (pg 1)\n   * CIAFR-10 used multiple times (pg 8)\n\n", "limitations": "the limitations and potential negative societal impact of this work is not provided ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths**\n\n* Able to train an ensemble of subnetworks on a subset of data in a highly parallel fashion is an efficient way to learn generalizable and efficient neural network model.\n* This can lead to a paradigm shift in the way models are learned and can enable the scaling of approach to larger data and network sizes\n\n**Weaknesses**\nThe (novel) contribution of the paper and contrast with literature is not clear. The tensor layer idea seems to be already proposed in the literature. If the parallel implementation and ensembling the network outputs is the contribution, it should be made clear", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject