Dynamic Sparse Network for Time Series Classification: Learning What to “See”
Authors: Qiao Xiao
Boqian Wu
Yu Zhang
Shiwei Liu
Mykola Pechenizkiy
Elena Mocanu
Decebal Constantin Mocanu
Venue: NeurIPS 2022 Accept (2022)
Keywords: time series classification
dynamic sparse training
adaptive receptive field

Abstract: The receptive field (RF), which determines the region of time series to be “seen” and used, is critical to improve the performance for time series classification (TSC). However, the variation of signal scales across and within time series data, makes it challenging to decide on proper RF sizes for TSC. In this paper, we propose a dynamic sparse network (DSN) with sparse connections for TSC, which can learn to cover various RF without cumbersome hyper-parameters tuning. The kernels in each sparse layer are sparse and can be explored under the constraint regions by dynamic sparse training, which makes it possible to reduce the resource cost. The experimental results show that the proposed DSN model can achieve state-of-art performance on both univariate and multivariate TSC datasets with less than 50% computational cost compared with recent baseline methods, opening the path towards more accurate resource-aware methods for time series analyses. Our code is publicly available at: https://github.com/QiaoXiao7282/DSN.

Dynamic Sparse Network for Time Series Classification: Learning What to "See"
Qiao Xiao!.", Boqian Wu?.,, Yu Zhang 3.4 Shiwei Liu Mykola Pechenizkiy',
Elena Mocanu', Decebal Constantin Mocanu 1,2
Eindhoven University of Technology. 2 University of Twente
3 Southern University of Science and Technology
Peng Cheng Laboratory, University of Texas at Austin xqao, www.wpeekcccceeeeee.. @tue. nl
{b. wu, wuu,e.mocan mocanu, , wow.mnue@uuttttomoo wettettt.wee...eeeeeeeeeeeeeeeet -nl
yu. zhang ust@gmail com
shiwei lww@austin utexas.ed. edu
Abstract
The receptive field (RF), which determines the region of time series be "seen" and used, is critical to improve the performance for time series classification (TSC). However, the variation of signal scales across and within time series data, makes it challenging to decide on proper RF sizes for TSC. In this paper, we propose a dynamic sparse network (DSN) with sparse connections for TSC, which can learn to cover various RF without cumbersome hyper-paramett kernels in each sparse layer are sparse and can be explored under the constraint regions by dynamic sparse training, which makes it possible to reduce the resource cost. The experimental results show that the proposed DSN model can achieve state-of-art performance on both univariate and multivariate TSC datasets with less than 50% computational cost compared with recent baseline methods, opening the path towards more accurate resource-awee aware methods for time series analyses. Our code is publicly available at: https: : //github. com/ 1iaaxX1a77222//SS
tuning. The
Introduction
Time series classification (TSC) as an important research topic in the data mining communities 14] has wide range of applications from health monitoring [24 [6], and public security (58), to grid energy (41), and remote sensing [43]. In the last decade, neural networks, especially deep neural
networks, have achieved competitive or even better performance than traditional TSC approaches
(e.g. DTW 46), BOSS 47). COTE [I] in many cases [56 14]
However, how to discover and exploit the various scaled signals hidden in time series (TS) is still a significant challenge for TSC tasks [52 36, [2]] One main reason lics several variances, like sampling rate and record length, which natural during time-series collection 47
31 5
Furthermore, the amplitude offset, warping and occlusion of data points are unavoidable (see Figure T). So determining the optimal scales for feature extraction is difficult but significant in the TSC task. One main solution is to cover as many receptive fields (RF) scales as possible, so it does not ignore
any useful signals from the time series inputs [52 g 25
Grounding on the motivation that multiple kernels in combination can capture discriminative patterns
despite complex warping, Fawaz et al. [15] tried to ensemble convolutional neural networks with large number and variety of kernels. Tang et al. (53) have stacked several OS-CNN blocks, each of
"quual contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

which consisted of a list of kernel sizes to cover all scales of RF and achieve better performance on several TSC tasks. In Multi-scale Convolutional Neural Network (MCNN) 4. a grid search was applied to find suitable kernel sizes. Without elaborate setting and fine-grid search, Dempster et al. 7 transformed time series by using random convolutional kernels and then trained time series classifier. Even though the use of kernels with various sizes can help to extract latent hierarchical features of multiple resolutions, it tends to cause a huge computational problem and overfits the datasets when limited amount of training data is available [25]. To address this problem, dilated convolution [59] has been used to reduce the computational cost while keeping the receptive field size. Franceschi et al. [17] presented an unsupervised learning model with convolutional kernels for time series feature transformation, in which the dilation factors of kernels increased exponentially layer by layer. To date, there is still an unresolved challenge with hyperparameter selections (e.g. kernel sizes and dilated factors) to capture patterns at different scales. This raises a foundational question: Can we design scalable method for TSC that achieves a trade-off between computation and performance without cumbersome hyperparameter tuning?
inpul
input
W W
k-1
k-3
co conatenntio
(1) (2) (3)
Standard CNN Layer
Dynamic Sparse CNNer
(1)
(?)
(a)
(b)
Figure 1: (a) Toy example illustrates the various time scales characteristic of TS data. Data in (1), (2), and (3) are sampled from three different TS datasets. The sequences with the same color represent the same class. We can find that several sources of variances like phase, warping and offset are common in TS data. (b) The comparison between other methods [53 25 (shown in bl) and our proposed DSN (shown in b2) for TSC. To cover various RF, DSN does not need to be configured with series of kernel sizes k.
Inspired by sparse neural network models which can match the performance of dense counterparts with fewer connections 40 35 33 60), we propose dynamic sparse network (DSN) that is composed of CNN layers with large but dynamic sparse kernel for TSC tasks, which can automatically learn sparse connections in terms of covering various RF. The proposed DSN is trained by dynamic sparse training (DST) strategy, which makes it possible to reduce the computational cost (e.g. dooting-poiit operations (FLOPs)) for training and inference well. Different from the traditional dynamic sparse training methods, which explore the connections in a layer-wise manner [40] 33 [3 and can hardly cover smaller RF, we propose a more fine-grained sparse training strategy for TSC. Specifically, the CNN kernels in each layer are split into several groups, each of which is sparse and can be explored under a constraint region during sparse training. In this way, each layer can extract features in more scalable and diverse way, and then by cascading, the model can cover more various scales of RF. To sum up, our main contributions are:
We propose a novel dynamic sparse CNN layer with various effective neighbour RF, which can learn to "see" selectively and adaptively for more accurate TSC.
By stacking the above dynamic sparse CNN layers, we introduce novel DSN model natively for TSC, which is trained by DST to reduce computational and memory cost.
To bridge the gap between resource awareness and accuracy, we propose more fine-grained training strategy for DSN, which can more easily cover diverse effective RF for TSC.
We perform extensive empirical evaluation on both univariate and multivariate TSC tasks, showing that DSN achieves superior performance in terms of accuracy and resource cost.
2

---

2 Related Work
2.1 Time Series Classification
In the last decade, the success of deep learning encourages researchers to explore and extend its
application on TSC tasks [28 14 56 29]. For univariate TSC, deep learning-basee models attempt directly transform raw time series data into low-dimensional feature representation via
convolutional or recurrent networks [25] 45 42 For multivariate TSC, LSTM and attention layers
often stacked with CNN layers to extract complementary features from TS data [26] 62)
Recently, many works have attempted extract features in a more scalable manner as TS data is unavoidably composed of signals on various scales 47 5 52]. One main solution is to configure suitable kernels of various sizes such that they increase the probability of capturing the proper scales Rocket- based methods 7 88 9 aim to use random kernels with several sizes and dilation factors to cover diverse RF for TSC. Different from these works, our proposed dynamic sparse CNN layer can adaptively learn the various RF with deformable dilation factors and trade-off both computation and performance without cumbersome hyperparameter tuning.
[22 53 25
2.2 Sparse Training
Recently, the lottery ticket hypothesis has shown that it is possible to train a sparse sub-netee net-network to match the performance of its dense counterpart with less computational cost (18). Rather than iteratively pruning from a dense network [99 63 18,, recent works try to find an initial mask with one-shot pruning based gradient information during training [30 [55] After pruning, the topology of the neural network will be fixed during training. However, this kind of models can hardly match the accuracy achieved by their dense counterparts (38 [5 51).
Introduced new training paradigm before the lottery ticket hypothesis, DST starts from a sparse network and allows the sparse connectivity to be evolved dynamically with fixed number of parameters throughout training 44 13 35 60 [0]. Nowadays, DST attracts increasing attentions from other research fields such as reinforcement learning 49 and continual learning [48], due to its
potential of outperforming dense neural networks training 40 60 35). Different from the traditional
DST methods, our proposed DSN is trained with fine-grained sparse training strategy rather than traditional layer-wise manner to capture more diverse RF for TSC.
2.3 Adaptive Receptive Field
The RF, which can be adaptively changed during the training, has proven to be effective in many domains [54] 44 57 The adaptive RF can usually be captured by learning the optimal kernel sizes or the masks of kernels during the course of training (50 20 57 However, neither the kernels nor the masks are sparse, which may cause the huge computational problem when larger RF is needed. Different from these methods, our dynamic sparse CNN layers in DSN can be trained with DST, which can learn to capture variable RF with deformable dilation factors. What's more, the kernels always sparse during training and inference, which makes possible to reduce the computational cost.
3 The Proposed Model
3.1 Problem Definition
Definition 1. (Time Series Classification (TSC)). Given a TS instance X (X,,...... "ins...,,,).. E Raxm. where m denotes the number of variates and n denotes the number of time steps, TSC aims to accurately predict the class label y E (I....,c) from e classes. When mn equals 1, TSC univariate and otherwise it is multivariate.
Definition 2. (Time Series (TS) Training Set). A training set D {(X() (0)))))).....) ..... (X(M), (()) consists of N time series instances, where X E Raxm could either be a uni- variate or multivariate time series instance with its corresponding label (f)) (l....,c).
3

---

Note that, in our case, all instances have the same number of time steps in TS dataset. Without the loss of generality, given a training set, we aim to train a CNN classifier with adaptive and various RF with low resource cost (e.g. memory and computation) for the TSC task.
3.2 Dynamic Sparse CNN Layer with Adaptive Receptive Field
straightforward strategy to cover various RF is apply multi-sized kernels in each CNN layer, but it has several limitations. Firstly, TS instances from different TS datasets do not have the same length and cycles P in most cases, making difficult even with prior knowledge. Secondly, obtaining a large receptive field commonly requires large kernel or stacking more layers, which introduces more parameters and thus increases storage and computational costs.
fixed kernel configuration for all the datasets
To tackle those challenges, the proposed dynamic sparse CNN layer possesses large but sparse kernels, which are learnable to capture adaptive RF. Specifically, given the input feature map x' E RR xhx in the Ith layer (h equals I for univariate TSC and c-1 is the number of input channels) and kernel weights O' E = x x1xk (k is the kernel size and et is the number of output channels), the convolution with stride and padding in our proposed dynamic sparse CNN layer is formulated as
0₂-
Σ (T'(0))ss e 0%,)) 2/ DEiSA-1NE2
(1)
where Oj R denotes the output feature representation the j-th output channel, Z denotes {0, 1]---xaxxxx is an indicator function, I' I'OO")..
the set of integers, r'()) Ra-1xaxIxx
indicates activated weights for 0%,, E Ixi which is the kernel in the (i. j-th channel, O denotes the element-wise product, and denotes the convolution operator. The indicator function I'(.), which learned during the training of the proposed DSN, satisfies that ()llo S (1 S) where 0 S S < is the sparsity ratio, ll-llo denotes the Lo norm and N; = cJ-1 X CJ X x k. When S > O. the kernel is sparse, and we can use a large in the dynamic sparse CNN layer to obtain a large RF with reduced computational cost.
Remark. (Effective Neighbour Receptive Field (eNRF) size in the dynamic sparse CNN layer).
The receptive field is defined as the region in the input that the feature of CNN model is looking at. We defined the Neighbour Receptive Field (NRF) as the region considered by each feature in the successive layer. Specifically, the NRF size is equivalent to the kernel size in the standard CNN layer (Considering the case that dilation equals to 1). Differently, the NRF size of the proposed dynamic sparse CNN layer is smaller than the kernel size, when the first or last weight in kernel is not activated, e.g., Di E {L...., 9--1 and E {l...., ,.), such that ((O)),,,1 = 0 or ) O. As illustrated in Figure 2 we defined the eNRF size fi, of the kernel 0h,,, as the distance between the first activated weight and last activated weight in the 1-th CNN layer as
max((nd(,s) min(Ind).) 5) +1, if Ind, # 1 otherwise
Pi :=s=11 0.
where Indi,, denotes the set of indices corresponding to non-zero weights in the kernel O..y. Obviously we have 0 S fl,- S k.
Cj.
The eNRF size set of 1-th dynamic sparse CNN layer is denoted by RB()), which satisfies that 0 S min(F()) and max(F()) < k. Take the case in the Figure 2 as an simple example for 1-th layer, then F(t) {0, 1, 2, 3, 4, 5, 6). Each dynamic sparse CNN layer is likely to cover various eNRF sizes ranging from 1 to k. When global information is expected, I'( could ac- tivate weights more dispersed to get a larger eNRF and exploit the input features selectively. To capture the local context, smaller eNRF is expected, thus the activated weights tend to be
f=6
f 5
f=1
==4
f ==
f =
Ci
f /6
=1
f f=3
Figure 2: Ilustrationn for the eNRF size. The weights with blue color are activated.
"The term 'cycle' means that the data exhibit rising and falling that not of fixed frequency.

---

concentrated. By taking k = 5 as an example, I' ((0),,, may be [1. O, 1, 0, 1] for global context, while it may be [0, 1, 1, 0] for local context. In this way, the eNRF be adaptively adjusted.
3.3 Architecture in DSN
The proposed DSN model consists of three sparse CNN modules, each of which composes a dynamic sparse CNN layer and 1 x CNN layer. Following the stacked sparse CNN modules, there is an additional dynamic sparse CNN layer, two adaptive average pooling layers, and one x 1 convolution layer acting the classifier in the DSN model. The overall architecture is shown in Figure
Sporss CAN Modtle
Spense CAN Medule
Spereo CAN Medule
Dynante Spare CAN Layer
Dynamic Soarse CNN Lager
Comrtet
Pot
POLL
Comvint
Sparse CNN Modulo
Figure 3: The proposed DSN model (left), which includes several sparse CNN modules (right) followed by dynamic sparse CNN layer, two adaptive average pooling layers, and one convolution layer.
The set eNRF sizes ((1) in the /th sparse CNN module equal to that the dynamic sparse CNN layer, since the eNRF size of 1 convolution constantly equal to sü satisfies that 0 min(g(1) and max(3()) < k. Then, the set of eNRF sizes for the three successively stacked sparse CNN modules can be described by RF as
RF mmxx(( s ( s(3) 2) (0)) e s f1, (1,
(3)
According to Eq. 3). we can see that stacking multiple sparse CNN modules can increase the size of RF linearly, where /th dynamic sparse CNN layer increases by the size of g(), For simplicity, the kernel size in each dynamic sparse CNN layer is consistently set to & in our study. Then, IF satisfies that max(RF) 3k 2 and S (RFF................... Thus, large i in sparse CNN modules can increase the range of the eNRF sizes for stacked sparse CNN modules.
group N
Cl
group
group
Figure 4: An illustration of the dynamic sparse CNN layer. In dynamic sparse CNN layer, the kernels are split into N groups, each which is with sparse connections learned by dynamic sparse training across specific constraint regions. The connections with color indicate non-zero values.
3.4 Dynamic Sparse Training for DSN Model
In this section, present the training strategy to discover the weights required to be activated in order to ensure well well-performing RF. That is, we need to study how to update the indicator function (-) during the training of the proposed DSN.

---

We sparsely train the proposed DSN model from scratch to keep kernel sparse follow- ing the main idea of DST methods [40]. By
Algorithm Dynamic Sparse Training for DSN Model Input:
Dataset D, Network fe, Sparsity ratio Exploration regions in layer I: (Ri,.., Exploration schedule: T, A7, a; fatrayy
design, during the training phase, the to- tal number of activated weights must not exceed [1 - 5). However, we observe
1: 0, < initialize activated weights in O using S and R
2: for t = to T do
that the tiny eNRF is hard to be captured by
3: Sample batch Bg ~ D
DST methods with layer-wise exploration
4:
I, -+== Liss, L (foo (344) (
manner, namely activated weights are dis- covered layer by layer, especially when the sparsity ratio S is undersized (more
5:
if (t mod AT) == 0 then
6:
for each layer / 1 to L do
7:
for each group i to N do
analysis in Section 4.5) Motivated by this observation, the kernels in each dynamic
8:
u = n (R)) (t;a, (1 (1- S)
9:
Trnae ArgTopK (- 10 u)
sparse CNN layer are divided into different
10:
Iggoo RandomK (R. (1.
groups, whose corresponding exploration
11:
I')) + Update I'()) using Igrnne and grow
regions are of different sizes as shown in Figure Contrary the DST methods, in
12:
0 < Update activated weights ('(e)) o o
13:
end for
DSN, the exploration of activated weights
14:
end for
separately performed in different ker-
15:
else
nel groups which is more fine-grained 16:
8= 8 VVo.
strategy. Specifically, the kernel weights O' E TR- x1x xx, in the ith layer are split into N groups along the output chan-
17:
end if
18:
end for
nel, that is, o'...., 6'v- E R 1x x x1x* with corresponding exploration regions Ri...., R'y- The exploration region R. in the ith group of the Ith layer is defined as the first 1x NN" positions of each kemel in this group. By taking k = 6 and N = 3 for example, activated weights in the first group only within the first two positions of each kernel in 0), while they are within the whole kernel for the last group (as shown in Figure 4 In this way, the various eNRF can be covered, and exploration space can be reduced (more details can be found in Appendix I to improve exploration efficiency,
Given the weight exploration regions R₁.-. ....... R2x and the sparsity ratio S, we train the proposed DSN model as shown in Algorithm [T] The activated weights are explored within the exploration regions and updated every A, iterations. The fraction of updated weight decays over time according the function faccay (t: a. T), which follows the cosine annealing [1] as
(1 1 cos
fdecay (t: a. T)
(4)
2
T
where a is the initial fraction of activated weights updated, t is the current training iteration, and T is the number of training iterations. Thus, during the tth iteration, the number of updated activated weights in the ith group of the Ith layer is n2()) (t: a; T) (1 S), where n n(R)) is the number of weights be explored in region R² During the update of activated weights, we firstly prune the activated weights determined by ArgTopK (- (-1o) [0][,u) and then randomly grow new activated weights by RandomK (R) 0', u), where ArgTopK larger values in vector v. RandomK(y, u) outputs the indices of random u elements in vector v, and R/ of denotes the weights within R! but except 0¹.
u) gives the indices of top elements with
The exploration of activated weights is straightforward. Firstly, pruning weights with small mag- nitudes is intuitive, because the contribution of weights with smaller magnitudes is insignificant or even negligible. Considering the pruning recoverability, we randomly regrow new weights with the same number as the pruned weights to achieve better activated weight exploration. In this way, the exploration of weights is dynamic and of plasticity, compared with the methods of pruning weights before and after training (more explanations about pruning and dynamic sparse training can be found in 22
32
4 Experiments
In this section, we evaluate the resource cost and accuracy of the proposed DSN model with recent baseline methods on both univariate and multivariate TSC data.

---

4.1 Datasets and Baselines
The details of each kind of dataset are as follows.
Univariate TS dataset from UCR 85 archive [3 This archive consists of 85 univariate TS datasets, which are collected from various domains (e.g. health monitoring and remote sensing), have distinguishable characteristics, and various levels of complexity. The number of instances in the training set varies from 16 8926, while their time step resolutions range from 24 to 2709.
The EEG2 dataset contains 1200 instances of 2 categories and 64 variates. The Human Activity Recognition (HAR) dataset consists of 10,299 instances with 6 categories, and the number of variates is 9. The Daily Sport dataset contains 9, 9,120 instances of 19 categories, and its number of variates is 45. The processing of these datasets is the same in [26].
Three multivariate TS datasets from UCI [12]:
Univariate TSC baselines: The performance of the following three baseline methods are reported, OS-CNN (5). InceptionTime [25]. and ResNet([56), they widely used for univariate TSC. Multivariate TSC baselines: Similar with [62], the following three multivariate baselines are selected: OS-CNN, TapNet [62), MLSTM-FCN [26]. To avoid an unfair comparison, we let outside the ensemble methods. However, based on our best knowledge, our performance analysis accounts for most of tate-of-the-an methods.
4.2 Experimental Settings and Implementation Details
For optimization, we use the Adam optimizer with an initial learning rate of 3 10-4, cosine decayed to 10-4. Our model is trained for 1,000 epochs with mini-batch size of 16. As in [25] (6), the best model, which corresponds to the minimum training loss, used to evaluate performance over the testing sets. Inspired by (25). the kernel sizes & in each dynamic sparse CNN layer are set to 39, The number kernel groups in Algorithm i set to 3, which helps to cover the small, medium, and large eNRF. Each setting is repeated five times, and the average results are reported.
4.3 Results and Analysis
We show the performance on both univariate and multivariate TSC benchmarks in Table 2 For univariate TSC, we can see that our method outperforms the baseline methods in most cases with smaller number of parameters (e.g. Params P) and less computation cost (e.g. FLOPs) on UCR 85 archive datasets. For multivariate TSC, our proposed DSN method achieves better performance on EEG2 and HAR datasets, while for Daily Sport, MLSTM-FCN was with 0.46% more accurate. is worth highlighting that the resource cost (i.c. Params and FLOPs) of our proposed DSN is much smaller than that of bascline methods. Due to the page limitation, the detailed results, including test accuracy and resource cost each dataset of UCR 85 archive, can be found in Appendix C More results on UCR 128 (6) and UEA 30 archive 2 can be found in Appendix B.9
Table 1: Pairwise comparison of test accuracy (%) and mean resource cost (i.e. Params (K) and FLOPs (M)J) of our method and other univariate TS baseline methods on UCR 85 archive datasets. Note that all the accuracies in comparison rounded two decimal places.
Archive
Methods
Baseline wins
DSN wins
Tic
Params
FLOPs
ResNet
29
53
3
479.20
803.38
InceptionTime
40
40
389.34
325.49
UCR 85 archive
OS-CNN DSN (ours)
33
49
3
262.14
200.12
126.22
104.88
4.4 Sensitivity Analysis
Effect of Sparsity Ratio: From Table we can find out how the sparsity ratio of dynamic sparse CNN layer affects the final test accuracy and the resource cost in multivariate TS datasets. Here,
The amount of non-zero parameters in the model in 49
27
30

---

Table 2: Test accuracy (ACC(%)) and resource cost (i.e. Params (K) and FL.OPs (M)L of different models on three multivariate TS datasets from UCI.
Daily Sport
EEG2
HAR Params
Methods
ACC
Params
FLOPs
ACC
Params
FLOPs
ACC
FLOPs
TapNet
99.56
1885.58
346.97
85.30
1369.21
780.19
93.33
1275.78
310.61
MLSTM-FCN
99.65
324.77
70.59
91.00
342.51
161.53
96.71
284.97
63.38
OS-CNN
99.30
317.05
78.01
93.03
320.34
162.04
96.37
280.39
70.98
DSN (ours)
99.19
163.68
40.48
99.10
162.14
82.31
96.82
160.13
40.64
we analyze the trade-off between test accuracy and the resource cost under various sparsity ratios (i.e. [50%, 80%, 90%]) and dense DSN model. Note that the dense model defined exactly as our DSN model with dense connections along entire kernels. We can see that it is not that the more parameters
model, the better performance it will achieve. This is mainly because more parameters will lead to overfitting and the proper sparse ratio will easily cover the desired receptive field (more demonstrations can be seen in Appendix B.I). Taking the EEG2 data set as an example, with the decrease of sparse rate, the performance set has more expectations for features from small receptive fields (the validation can be found in Appendix B.2). According the experiments, a 80% sparsity ratio reflects good trade-off between accuracy and resource cost, which is the default setting for the dynamic sparse CNN layer in our DSN model.
drops sharply. We hypothesise that this data
Table 3: Test accuracy (ACC(%)) and resource cost (i.e. Params (K) and FLOPS (M)J) of the proposed DSN model with different sparsity ratios on three multivariate TS datasets from UCI.
Daily Sport
EEG2
HAR
Methods
ACC
Params
FLOPs
ACC
Params
FLOPs
ACC
Params
FLOPs
Dense
99.19
1059.65
262.10
89.60
1058.11
536.17
96.06
1056.10
267.57
S 50%
99.28
370.44
91.63
96.23
368.90
187.05
96.62
366.89
93.01
S 80%
99.19
163.68
40.48
99.10
162.14
82.31
96.82
160.13
40.64
S 90%
99.25
94.76
23.44
98.90
93.22
46.64
96.75
91.21
23.18
Effeet of Architecture: To show how the architecture of DSN affects the final performance, we adopt the critical difference diagram for detailed comparison, which is usually used for evaluation in TSC [52 225). From Figure 5 can find that the DSN model with 4 dynamic sparse CNN layers and 177 output channels in each layer performs consistently better than the other structures. The proposed model with more output channels seems to be crucial for preserving the capacity of the network, while requires more parameters and FLOPs compared to other counterparts. This highlights an interesting off between accuracy and computational efficiency. According to the experiments, the DSN model with 4 dynamic sparse CNN layers and 141 output channels in each layer is the trade-off between the performance and resource cost, which is the default setting for our DSN model.
DSN(d3-c141) 32235
2.6765
DSN(d4-c177)
DSN(d5-c141) 31000
2.9794
DSN(d4-c141)
DSN(d4-c105) 30706
Figure 5: The critical difference diagram shows the average rank of the proposed DSN model with different architectures in UCR 85 archive. DSN(di-cj) represents the DSN model with dynamic sparse CNN layers and j output channels in each layer. The smaller average rank corresponds to the better performance of the model.
Effect of Kernel Group: As shown Figure training DSN without kernel group hampers the capture of small eNRF, which can not guarantee satisfactory performance on the datasets such as FEG2 expecting more local information. With the decrease sparsity ratio S, the phenomenon that large RF occupies the majority becomes more severe (more demonstrations see in Appendix B.I.

---

which increases the performance gap between training with group and without it, as shown in Table 4 Grouping the kernels can achieve various RF coverage and adorable accuracy for all datasets.
1st laver
2nd layer
3rd laver
eNC 5285
(A)
(b)
(c)
Figure 6: The distribution of eNRF sizes in different sparse CNN layers between DSN models training with and without grouping HAR dataset. The number on y-axis corresponds to the frequency of eNRF sizes.
4.5 Ablation Study
Effeet of Dynamic Sparse Training: Furthermore, we study the effect of the dynamic sparse training in our DSN method. We conduct ablation studies to compare DSN with its static variant (i.e. DSN which fixes its topology during training after acti- vated weights initialization. In other words, the indicator function (-() in Eq. (1) will not update during the training process. According to the re- sults shown in Figure
Table 4: Average accuracies (%) of the proposed DSN model affected by kernel grouping.
Sparsity ratio
Methods
Daily Sport
EEG2
HAR
DSN Group
99.27
97.87
96.53
S 60%
DSN w/o Group
99.23
96.77
96.87
DSN w Group DSN w/o Group
99.20
98,70
97.14
8 70%
99.25
97.27
96.52
DSN w Group DSN w/o Group
99.19
99.10
96.82
S 80%
99.28
98.77
96.79
7 (a), can find that the DSN model mostly outperforms the DSN UCR 85 archive, indicating that what is learned by dynamic sparse training is the suitable activated weights together with their values.
300
100
10c
FquM
Seal
10
90-
IIN Is better here
DSN ls better here
nex 1rl bettor hore
TEX better here
OS-CMS is better here
40-
Wdr: 70 Loss: 11
Win: Loso: 33
30
20
20-
Los: 30
20
DRN
ISSN
OS-CINN
Oel
(c)
Figure 7: Accuracy plotting showing how the proposed DSN model is affected by the dynamic sparse training UCR 85 archive.
4.6 Case Study other DST Method
Likewise, the effectiveness of the pro- posed DSN method also analyzed in the presence of other DST methods, such RigL [13. Rather than grow- ing weights randomly (as in the line of Algorithm []. RigL grows them with the highest magnitude gradients. From Figure 7 (c), we can see that growing weights according to gradi- ents (i.e. DSN can outperforms
(4)
Figure 8: The activated weights in different kernel groups.
the OS-CNN method in most cases on

---

UCR 85 archive. Furthermore. shown in Figure 7 (b), we can find that DSN grad has similar performance as growing weights randomly (i.e. DSN rand). Note that, rand the main setting of the proposed method in this paper with the reason that we don't t need to compute the full gradients at any moment time. In summary, the proposed DSN with fine- fine-grained sparse strategy can be easily combined with the existing DST techniques, showing the potential for further improvements by merging it with other advanced DST methods in the future.
4.7 Visualization of Activated Weights
Figure 8 shows the normalized activated weights of the DSN model after sparse training on HAR dataset. (a), (b) and (c) correspond to the activated weights of the three kernel groups in the first dynamic sparse CNN layer, and each row of the figure represents one kernel. We can see that the weights in each kernel are sparse, and can be activated within the constraint region defined in Section 3.4
Conclusions
This paper proposes dynamic sparse network (DSN) cover diverse effective neighbour receptive field (eNRF) for time series classification (TSC) without the cumbersome The proposed DSN can achieve about 2x reduction in memory and computational cost, while achieving superior performance in terms of accuracy on both univariate and multivariate TSC datasets. Furthermore, DSN explores the activated weights in more fine-grained strategy, which can be easily merged with existing DST techniques, showing a potential for further improvements when combined with advanced technology. DSN provides feasible solution to bridge the gap between resource awareness and various eNRF coverage for TSC, hoping to inspire other researchers in different fields. is worthwhile extending DSN to other research domains (e.g. time series forecasting and computer vision). Our DSN model has not yet been executed on real edge devices, and the performance in terms of accuracy and memory footprint on edge devices is not investigated. In the future, we hope to extend our work other fields and decentralisc training edge devices.
tuning.
Acknowledgements
This work has been partly funded by the NWO Perspectief project MEGAMIND. We thank everyone who worked on making the datasets available and their contributions to the TSC community.
References
[1] Anthony Bagnall, Jason Lines, Jon Hills, and Aaron Bostrom. Time-series classification with cote: the collective of transformation-bassee ensembles. IEEE Transactions on Knowledge and Duta Engineering, 2709):2522-2535 2015.
[2] Anthony J. Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn. James Large, Aaron Bostrom, Paul Southam, and Eamonn J. Keogh. The UEA multivariate time series classification archive, 2018. CoRR, Abs/1811.00075 2018.
[3] Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, and Gustavo Batista. The uer time series classification archive, July 2015. www. ucr . edu/ htt////aamonntt//e series data/
[4] Zhicheng Cui, Wenlin Chen, and Yixin Chen. Multi-scale convolutional neural networks for time series classification. arXiv preprint arXiv:1603.06995 2016.
[5] Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Famonn Keogh. The uer time series archive. IEEE/CAA Journal of Automatico Sinica, 1293:1393 2019.
[6] Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdul- lah Mueen, Gustavo Batista, and HeX The uer time series classification archive, October 2018
https::/www. cs. ucr edu/ hatp///eamon//ttt/ ttttttimeeerrrr aet____eeeeee__e___e______ data__aa88
[7] Angus Dempster, François Petitjean, and Geoffrey Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery.
34(5):1444-1495) 2020.
10

---

[8] Angus Dempster, François Petitjean, and Geoffrey I. Webb. ROCKET, exceptionally fast and accurate time series classification using random convolutional kernels. Data Min. Knowl. Discov. 34(5):1454-1495. 2020.
[9] Angus Dempster, Daniel F. Schmidt, and Geoffrey I. Webb. Minirocket: A very fast (almost) deterministic transform for time series classification, In KDD '2: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 248-22 257. 2021.
[10] Tim Dettmers and Luke 7ettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv: 2019.
[11] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. CoRR, htss/1977.04844 2019.
[12] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[13] Utku Evei, Trevor Gale, Jacoh Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Proceedings of the 37th International Conference on Machine Learning, volume 19, pages 2943-2952. PMIRR 2020.
[14] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: a review. Data Min. Knowl, Discov,, (((4):17-963)) 2019,
[15] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep neural network ensembles for time series classification. 2019 International Joint Conference on Neural Networks (ICNN), pages 1-6, IEEE, 2019.
[16] Germain Forestier. François Petitjean, Pavel Senin, Fabien Despinoy, Arnaud Huaulmé, Hassan Ismail Fawaz, Jonathan Weber, Lhassane Idoumghar, Pierre-Alain Muller, and Pierre Jannin. Surgical motion analysis using discriminative interpretable patterus. Artificial intelligence in medicine, 91:3-11, 2018.
[17] Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
[18] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
[19] Sharath Girish, Shishira R. Maiya, Kamal Gupta, Hao Chen, Larry S. Davis, and Abhinav Shrivastava. The lottery ticket hypothesis for object recognition. In IEEE Conference Computer Vision and Pottern Recognition, pages 762-771, 2021.
[20] Shizhong Han, Zibo Meng, Zhiyuan Li, James O' O'Reilly, Jie Cai, Xiaofeng Wang, and Yan Tong. Optimizing filter size in convolutional neural networks for facial action unit recognition. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 5070 5078, 2018
[21] Jon Hills, Jason Lines, Edgaras Baranauskas, James Mapp, and Anthony J. Bagnall. Classification of time series by shapelet transformation. Data Min. Knowl. Discov., 28(4):851-88 1811 2014.
[22] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22(1), jan 2021.
[23] Seth H Huang, Xu Lingjic, and Jiang Congwei. Residual attention net for superior cross-domain time sequence modeling. arXiv preprint arXiv:2001.00 04077, 2020.
[24] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, François Petitjean, Lhassane Idoumghar, and Pierre-Ala Alain Muller. Automatic alignment of surgical videos using kinematic data. In Conference on Artificial Intelligence in Medicine in Europe, pages 104-113. Springer, 2019.
[25] Hassan Ismail Fawaz, Beajamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idounghar, Pierre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 1436))193611962 2020.
[26] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford. Multivariate Istm-fens for time series classification. Neural Networks, 116:237-245 2019.
[27] Aditya Kusupati, Vivek Ramanujan, Raghav Somami, Mitchell Wortsman, Pratcek Jain, Sham M. Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020. volume 119, pages 5544-5555. 2020.
[28] Martin Längkvist, Lars Karlsson, and Amy Loutfi. A review of unsupervised feature learning and deep learning for series modeling. Pattern Recognition Letters, 42:11-24 2014.
[29] Dongha Lee, Seonghyeon Lee, and Hwanjo Yu. Leamable dynamic temporal pooling for time series classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8288 8296, 2021.
11

---

[30] Namhoon Lee. Thalaiyasingam Ajanthan, and Philip H. S. Tomr. Snip: single-shot network pruning based connection sensitivity. In 7th International Conference on Learning Reprexentations. ICLR 2019, 2019.
[31] Jason Lines, Luke M Davis, Jon Hills, and Anthony Bagnall. A shapelet transform for time series classification. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 289-291 2012.
[32] Shiwei Liu, Spurse Neural Network Training with In-Time Peter-rrrumeteizziionn PhD thesis, Mathematics and Computer Science, April 2022. Proefschrift.
[33] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin. Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang. and Decehal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34, 2021.
[34] Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei. and Mykola Pechenizkiy. Selfish sparse RNN training. In Marina Meila and Tong Zhang, editors. Proceedings the 38th International Conference on Machine Learning. ICML, volume 139, pages 6893-6904. PMLR, 2021.
[35] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parammtterization? in-time vee-paraaettrizzttionn in sparse training. In International Conference on Machine Learning, pages 6989-7000. PMLR, 2021.
[36] Benjamin Lucas, Ahmed Shifaz, Charlotte Pelletier, Lachlan O'Neill, Nayyar Abbas Zaidi, Bart Goethals, François Petitjean, and Geoffrey I. Webb. Proximity forest: an effective and scalable distance- classifier for time series. Data Min, Knowl. Discov, 33(3):607-635, 2019.
[37] Matthew Middlehurst, James Large, Michael Flynn, Jason Lines, Aaron Bostrom, and Anthony Bagnall. Hive-cote 2.0: a new meta ensemble for time series classification. Machine Learning, 110(11)33211332 -3243. 2021.
[38] Decebal Constantin Mocanu, Elena Mocanu, Phuong H Nguyen, Madcleine Gibeseu, and Antonio Liotta. A topological insight into restricted boltzmann machines. Machine Learning, 104(2):243-270, 2016.
[39] Decebal Constantin Mocanu, Elena Mocanu, Tiago Pinto, Selima Curci, Phuong H. Nguyen, Madeleine Gibescu, Damien Ernst, and Zita A. Vale. Sparse training theory for scalable and efficient agents. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, page 34-38, 38, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems.
[40] Deccbal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):1-12, 2018.
[41] Elena Mocanu, Decebal Constantin Mocanu, Phuong H Nguyen, Antonio Liotta, Michael E Webber, Madeleine Gibescu, and Johannes G Slootweg. On-line building energy optimization using deep reinforce- ment learning. IEEF Transactions on Smart Grid, 10(4):3698-3708, 2018.
[42] Kewei Ouyang, Yi Hou, Shilin Zhou, and Ye Zhang. Convolutional neural network with an elastic matching mechanism for time series classification. Algorithms, 14(7):192, 2021.
[43] Charlotte Pelletier. Geoffrey Webb, and François Petitjean. Temporal convolutional neural network for the classification of satellite image time series. Remote Sensing, 1 (55:5223, 2019.
[44] Silvia L. Pintea, Nergis Tomen, Stanley F. Goes, Marco Loog, and Jan C. van Gemert. Resolution learning in deep convolutional networks using scale-space theory. IEEE Trans. Image Process., 10:8332-8835.. 2021.
[45] Elahe Rahimian, Soheil Zabihi, Seyed Farokh Atashzar, Amir Asif, and Arash Mohammadi. Xceptiontime: Independent time-window xceptiontime architecture for hand gesture classification. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1304-1308, 2020.
[46] Thanawin Rakthanmanon, Bilson Campana, Abdullah Mueen, Gustavo Batista. Brandon Westover, Qiang Zhu, Jesin Zakaria, and Eamonn Keogh. Scarching and mining trillions of time series subsequenees under dynamic time warping. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 262-270,220 2012.
[47] Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 2506):1505-155 1530, 2015.
[48] Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu, Peter F. Latham, and Yee Tch. Powerpropa- gation: A sparsity inducing weight reparamcterisation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, pages 28889-28903, 2021.
[49] Ghada Sokar. Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peterk Stone. Dy- namic sparse training for deep reinforcement learning. In 31st International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence, 2022.
12

---

[50] Domen Tabemik, Matej Kristan, and Ales Leonardis. Spatially-adaptive filter units for compact and efficient deep neural networks. Int. J. Comput. Vis., 12868):0099-2667, 2020.
[51] Hidenori Tanaka, Daniel Kunin, Daniel L. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 2020.
[52] Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Michael Blumenstein, and Jing Jiang. Omni-scale cans: simple and effective kernel size configuration for time series classification. In International Conference Learning Representctions, 2022.
[53] Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Jing Jiang, and Michael Blumenstein. Rethinking Id-enn for time series classification: A stronger baseline. arXiv preprint 20o::2002.0061,,,
[54] Nergis Tomen, Silvia-L.aur Laura Pintea, and Jan van Gemert. Deep continuous networks. In Proceedings of the 38th International Conference on Machine Learning, ICMI, 2021, volume 139, pages 10324-10335, 2021.
[55] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
[56] Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 Internationaal Joint Conference on Neural Networks (IJCNN), pages 1578-1585, IEEE, 2017.
[57] Zhitong Xiong, Yuan Yuan, Nianhui Guo, and Qi Wang. Variational context-deformable convnets for indoor scene parsing. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, pages 3991-9000 Computer Vision Foundation / IEEE, 2020.
[58] Fei Yi, Zhiwen Yu, Fuzhen Zhuang, Xiao Zhang, and Hui Xiong. An integrated model for crime prediction using temporal and spatial factors. In 2018 IEEE International Conference on Data Mining (ICDM), pages 1386 1391. IEEE, 2018.
[59] Fisher and Vladlen Koltun. Multi-scale context aggregatiin by dilated convolutions. In International Conference on Learning Representations (ICLR), May 2016.
[60] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Syxtems, 34, 2021.
[61] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Fickhoff. based based framework for multivariate time series representation learning. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2114-2124, 2021.
transformer-
[62] Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tiere Tien Lu. Tapuet: Multivariate time series classification with attentional prototypical network. Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 6845-6852, 2020.
[63] Zhenyu Zhang. Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data more. In Proceedings of the 38th International Conference on Machine Learning, volume 139. pages 12380-12390. PMLR, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section[5]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.2
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasin new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "gedBMBCsO97", "reviewer": "Reviewer_3Hph", "summary": "The authors proposed a dynamic sparse network (DSN) with sparse connection to better determine the appropriate receptive field (RF) of time series classification (TSC) tasks. The contribution included alleviating hyper-parameter tuning while maintaining low resource costs. They pointed out the lack of balance between resources and performance without tortuous manual hyperparameter tunings. Where the authors’ DSN model would utilize their designed dynamic sparse CNN layers on resulting effective RF balance with TSC modeling. The experiment was done on real-world time-series datasets and their method showed better accuracy performance as well as lower resource cost", "questions": "1. Did they use the same architecture for all Ablation studies, even though each dataset innately carried different characteristics? Is that a fair comparison?\n\n2. In Section 3.4, the authors would need to clarify their weight pruning with more details and insights. Did the weight decay operation train concurrently with the weight activation? \n\n3.   RF was not properly explained properly in the abstract or introduction. “Hidden signals that can be ‘seen’” was not licitly discussed. What is meant by this?\n\n4. The authors could be more specific about their claim of “more fine-grained training strategy” that achieving “diverse effective RF”, so that the readers could better grasp why theirs would be better. \n\n5. The authors could spend a bit more details how the resource awareness would scale up.  \n\n6. It would be better for the authors to complete their experiments to have more intuitive illustrations of their model’s impact.\n", "limitations": "The authors have discussed the limitations but only in brief.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has a number of strengths. The authors incorporated dynamic sparse training into RF, which is interesting and could guarantee the region attended to have high information.  Although keeping the kernel sparse during learning was difficult, the authors presented promising results through experiments, the solution could be quite significant for time series work which tends to lag other areas of ML data (images, notes, etc.).\n\nThe paper is well written and presented. The authors provided a careful literature review in illustrating the limitations of prior works. The authors’ architecture analysis of both theirs and other works are thorough\n\nThe paper has good experimental results. The authors seemed to heuristically apply weight pruning exploration but it was effective, and  provided seemingly comprehensive analysis as they applied 3 different datasets in experiments.\n\nHowever there are a number of concerns and questions raised about the significance, originality, and technical contribution. The authors would need additional clarification about the implementation of each dataset. Ablation study was done on kernel group or sparse training, but the authors might want to consider taking other components out of the model- Such as sparse CNN modules- and seeing how much each component leads to improvements, to understand the specific contributions of the work. Since they claimed overall contribution was finding the important region in time series data, the authors would need to consider comparing transformer-based works since those would be stronger competitors in sequential data sets (currently none) and attention mechanisms that highlight regions of importance.\n\nAdditionally, the authors need clarification on the space of problems solved. Are there bounds on the time-series types of classification problems addressed by this work?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V-rovi5U0Lx", "reviewer": "Reviewer_skdR", "summary": "The variation of signal scales across and within time series data, makes it challenging to decide proper RF sizes for time series classification. This paper proposes a dynamic sparse network with sparse connections for time series classification, which can learn to cover various receptive field without cumbersome hyper-parameters tuning. The kernels in each sparse layer are sparse and can be explored under the constraint regions by dynamic sparse training, which makes it possible to reduce the resource cost. The experiment results show that the model can achieve state-of-art performance on both univariate and multivariate time series classification datasets. ", "questions": "1. Whether there are simpler ways to achieve adaptive receptive fields?\n\n2. The significance of experimental results.\n\n3. Are the baselines strong enough?\n\n4. A naive question: can the authors elaborate on how the convolution operators perform between a \\in R^k and x^l_i \\in R^{hw}? It is not straightforward to me somehow. Why O_j \\in R^{hw}?", "limitations": "Yes", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper proposes a method that can learn to cover various receptive field without cumbersome hyper-parmeters tuning.\n\nThe experiments show that the method achieves state-of-the-art performance on several time series classification datasets.\n\nThe paper is well-written and well-structured.\n\nWeakness:\n\nThis paper proposes a method that can cover various receptive fields. I wonder whether there are simpler ways to achieve this, e.g. combining the prediction of several models with different receptive fields?\n\nThe improvements seem very marginal e.g. on HAR. The authors may need to run the experiments multiple times and report both mean and standard deviation.\n\nI wonder whether the baselines are strong enough? How about comparing to ViT models that has full receptive fields?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U6sCu5e0i9R", "reviewer": "Reviewer_Ni8e", "summary": "The paper proposes DSN, a dynamic sparse network for time-series classification. The goal of DSN is to avoid hyper-parameter tuning and operate with sparse kernels to improve the resources and computational cost. Experiments over univariate and multivariate time-series datasets show that DSN can reduce in half the cost vs. strong recent baselines while achieving state-of-the-art accuracy performance.", "questions": "The problem is important. However, there are many concerns with this work:\n\nW1. The paper needs substantial effort to improve its presentation.\n\nRF terminology is not properly introduced. The connection to time-series analysis specifically is never discussed. \n\nThere is not enough emphasis on preliminaries and background. Existing compression and sparse DNN solutions are properly introduced. Heavy re-writing is needed\n\nW2. Lack of novelty and technical depth\n\nIt's like the first time a sparse NN architecture is applied to the particular time-series classification models, but it's unclear what is novel in this work. This is relevant to the previous comment about properly introducing the 3-4 areas in that space that focus on reducing resources/computational costs. Sparse connection is an emerging area. The paper reads mainly as an adaptation of these solutions and applied to a new domain but the technical depth is not clear, unfortunately.\n\nW3. Missing baselines \n\nHow out of the shelf methods that focus on reducing DNN parameters perform on this task. The paper only present how DNS performs vs. baselines that do not consider this problem so for sure there is an improvement. There are multiple works in that area from reducing from FP32 to FP16, from operating with only a few fourier coefficients [a], etc. Multiple baselines in that area are needed to prove how significantly better is DSN\n\n[a] Dziedzic, Adam, et al. \"Band-limited training and inference for convolutional neural networks.\" International Conference on Machine Learning. PMLR, 2019.\n\nW4. UCR contains 128 datasets now and not 85.\n\nThere are 30 multivariate time-series datasets in timeseriesclassification.com - All results have to be updated with the recent benchmarks in this area.\n\n", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1. Reducing the costs of DNN is a timely and important problem\n\nS2. Extensive experimentation using many public datasets\n\nWeaknesses:\n\nW1. The paper needs substantial effort to improve its presentation.\n\nW2. Lack of novelty and technical depth\n\nW3. Missing baselines ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept