Taking ROCKET on an efficiency mission: A distributed solution for fast and accurate multivariate time series classification
Authors: Leonardos Pantiskas
Kees Verstoep
Mark Hoogendoorn
Henri Bal
Venue: ICLR 2022 Submitted (2022)
Keywords: distribution
time series
classification
multivariate
wavelet
scattering
feature selection
scaling

Abstract: Nowadays, with the rising number of sensors in sectors such as healthcare and industry, the problem of multivariate time series classification (MTSC) is getting increasingly relevant and is a prime target for machine and deep learning solutions. Their expanding adoption in real-world environments is causing a shift in focus from the pursuit of ever higher prediction accuracy with complex models towards practical, deployable solutions that balance accuracy and parameters such as prediction speed. An MTSC solution that has attracted attention recently is ROCKET, based on random convolutional kernels, both because of its very fast training process and its state-of-the-art accuracy. However, the large number of features it utilizes may be detrimental to inference time. Examining its theoretical background and limitations enables us to address potential drawbacks and present LightWaveS: a distributed solution for accurate MTSC, which is fast both during training and inference. Specifically, utilizing a wavelet scattering transformation of the time series and distributed feature selection, we manage to create a solution which employs just 2,5% of the ROCKET features, while achieving accuracy comparable to recent deep learning solutions. LightWaveS also scales well with more nodes and large numbers of channels. In addition, it can give interpretability into the nature of an MTSC problem and allows for tuning based on expert opinion. We present three versions of our algorithm and their results on training time, accuracy, inference speedup and scalability. We show that we achieve speedup ranging from 8x to 30x compared to ROCKET during inference on an edge device, on datasets with comparable accuracy.

Under review as a conference paper at ICLR 2022
TAKING ROCKET ON AN EFFICIENCY MISSION: A DISTRIBUTED SOLUTION FOR FAST AND ACCURATE MULTIVARIATE TIME SERIES CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Nowadays, with the rising number of sensors in sectors such healthcare and industry, the problem of multivariate time series classification (MTSC) is getting increasingly relevant and is a prime target for machine and deep learning solu- tions. Their expanding adoption in real-world environments is causing a shift in focus from the pursuit of ever higher prediction accuracy with complex models to- wards practical, deployable solutions that balance accuracy and parameters such prediction speed. An MTSC solution that has attracted attention recently is ROCKET, based on random convolutional kernels, both because of its very fast training process and its state-of-the-arr accuracy. However, the large number of features it utilizes may be detrimental to inference time. Examining its theoretical background and limitations enables us to address potential drawbacks and present LightWaveS: distributed solution for accurate MTSC, which is fast both during training and inference. Specifically, utilizing a wavelet scattering transformation of the time series and distributed feature selection, we manage to create solu- tion which employs just 2.5% of the ROCKET features, while achieving accuracy comparable to recent deep learning solutions. LightWaveS also scales well with more nodes and large numbers of channels. In addition, it can significantly reduce MTSC problem by keeping only the
the input size and also provide insight most useful channels. We present three versions of our algorithm and their results training time, accuracy, inference speedup and scalability. We show that we achieve speedup ranging from 9x to 65x compared to ROCKET during inference on an edge device, on datasets with comparable accuracy.
INTRODUCTION
Time series classification is the task of characterizing a series of values observed in sequential mo- ments in time belonging to of two more categories, classes. There has been extensive work on univariate time series classification with machine and deep learning methods, as observed in surveys such |Bagnall et al.] (2017). In practice, noted in Ruiz et al. (2021), many prob- lems are described by more than one channels of information, turning the task into multivariate time series classification (MTSC). Some recent contributing factors have been the development of smaller and cheaper sensors for the measurement of various quantities, the general advancement of Internet of Things and the engagement with inherently more complex problems in sectors such as healthcare and industry, which benefit from additional information channels. In addition, the rise of edge computing and deployment of models such environments makes necessary to also con- sider the conditions and resources of the computational nodes on which those models will execute. This change in perspective makes it important to include, apart from accuracy, variables such as prediction speed or throughput in the evaluation of model's suitability for given task.
In the recent evaluation of advances in MTSC (Ruiz et al., 2021). one solution that shows good performance is ROCKET (Dempster et al., 2019). both in terms of accuracy and training time. ROCKET utilizes random convolutions to transform the time series channels and then extracts two features per convolution that are used with linear classifier. The evolution of ROCKET, MINIROCKET (Dempster et al.) 2020). is proposed as the new default variant of ROCKET by its authors and utilizes only feature per kernel (percentage of positive values), thereby halving

---

Under review as a conference paper at ICLR 2022
the features. It also utilizes other optimizations to speed up ROCKET in general. and uses mini- mally random approach with a given set of kernels. Another variant, MultiRocket (Tan et al. 2021), more features per convolution to achieve better accuracy at the expense of transtormation time. Both ROCKET and MINIROCKET have been developed for univariate time series. Although the authors of MINIROCKET characterize their provided repository code for multivariate time series as naive, it is the same algorithmic logic in the form of a ROCKET extension in the sktime (Löning al. 2019) repository that achieves the impressive performance mentioned above.
In (MINIROCKET, the authors encourage research into more sophisticated approaches for multi- variate time series. We can start by identifying potential points that can be improved. One point with (MINI)ROCKET that we try to address in our work is that the default number of features, although beneficial to accuracy, can highly redundant, and extreme for some datasets, leading to unnec- essarily high transformation and inference times during deployment. In addition, due to the fixed process and number of generated features, there is a large discrepancy the representation (features per channel) across datasets with different dimensions. Moreover, the indiscriminate inclusion of all channels the feature generation susceptible to uninformative series. We present experiments to illustrate the above points in section A.I Finally, the stochastic element, especially the random channel combination, although beneficial to accuracy, does not offer significant interpretability.
Based on this analysis, propose LightWaveS, framework for fast, distributed transformation of multivariate time series based on convolutional kernels, wavelet scattering and feature selection, for lightweight and accurate classification with linear classifiers. Our solution aims keep the suc- cessful aspects of the ROCKET model family, such as the short transformation time, the arbitrary convolutional kernel approach and the few descriptive features per kernel. On top of that, Light- WaveS adds well-studied signal theory, multi-node" distribution and smart feature selection to ad- dress the drawbacks identified above. The trade-off that we propose compared to (MINIJOCCET is clear: we take advantage of additional computational resources during training time to keep the short duration and also save time during inference. Our contributions with LightWaveS are:
We introduce the concept of wavelet scattering based arbitrary kernels
We achieve accuracy comparable to state-of-the-an art in the majority of the UEA datasets (Bagnall et al.] 2018), using only a fraction of the number of features used by (MINTROCKET
We distribute the framework to achieve training time comparable to (MINI)ROCKET on node, and shorter on multiple nodes
We achieve linear scaling of training time with the number of channels, tested in experi- ments with up to 900 channels
Depending on the dataset size, we achieve good speedup with additional nodes, tested in experiments with up 8 nodes
We achieve inference speedups ranging from 9x to 65x compared to ROCKET on an edge device in datasets with comparable accuracy
With this work, we take an efficiency-centrrc approach, focusing on the practicality and the inference speed of model that may be deployed on edge device, without necessarily trying to surpass the state-of-ther ahe in terms of accuracy.
2
RELATED WORK
2.1 MULTIVARIATE TIME SERIES CLASSIFICATION
Due to the recent rise in popularity of the deep learning field, there multitude of models that can be casily adapted to incorporate the additional dimension of MTSC (Ismail Fawaz et al.) 2019). A detailed evaluation of recent MTSC methods appears in Ruiz et al. (2021). Convolutions in particular, either 1-D) or with more dimensions, are very popular module when dealing with time series, as seen in models such as TapNet (Zhang et al. 2020), Inception Time Time (Fawaz et al., (020)
and OS-CNN (Tang al. 2021) among others. In these works, convolutions are combined in
With the term node refer computing node distributed (networked) system.
2

---

Under review as a conference paper at ICLR 2022
various architectures with other modules, such fully connected networks, and achieve impressive classification accuracy. The majority of those deep learning models takes a significant amount of time and memory train on GPU nodes, ranging from hours to even days, depending on the dataset size (Ruiz et al. (02T). In contrast, our method takes less than 20 minutes to transform, train and test all 30 UEA datasets on single CPU node with a linear classifier. If we distribute the solution on e.g. 8 CPU nodes, this time drops to under 4 minutes. An additional challenge is the interpretability of deep learning models. The usage of wavelets by LightWaveS and their method of application gives the potential to interpret the result based on signal theory and the input channel filtering helps to extract useful conclusions about a given time series problem.
2.2 FEATURE EXTRACTION AND SELECTION
Deep learning models implicitly extract features from the input along their first layers. In contrast, explicitly extracting features to use with classifiers has also been visited in multiple ways. We have already described how the *ROCKET family extracts features from the convolutions of random ker- nels with the input, an idea that has also been explored before, such as in Farahmand et al. (2017). Another example is tsfresh (Christ al.) 2018), which extracts a large number of predefined statisti- cal features from the time series and then through feature selection reduces them to the most useful. Similarly, catch22 (Lubba et al.) 2019) is a solution that uses only 22 predefined characteristics to transform time series, aiming for very fast transformation. A different approach is presented in WEASEL+MUSE (Schäfer & Leser 2018), which extracts features based on a bag-of-patterns ap- proach and selects the most useful based on x² test. LightWaveS, similarly to MINIROCKET, depends on an arbitrary set of convolutions and only 4 statistical features, extracted however from the coefficients of wavelet scattering. Due to the speed of the convolution operation and the sim- plicity of the features, we can achieve extremely fast training and inference times, while giving the model enough complexity to accommodate difficult datasets, where predefined statistical features on the raw time series values may not be descriptive enough.
2.3 DISTRIBUTED TRAINING
As the size of the deep learning models and the amount of input data increase, it is increasingly dif- ficult to perform training single node. For that reason, distribution of the training process across multiple nodes is becoming increasingly necessary (Mayer & Jacobsen| 2020). This distribution can be implemented with methods such data parallelism, where each node applies the same opera- tions on different parts of the input data, with frequent communication among the nodes to update the model. Distribution can also be applied on solutions that do not depend on deep learning, such as tsfresh mentioned above, which supports operation on a cluster through Dask (Dask Development Team 2016). LightWaveS is distributed using MPI in a data-parallel way, but in contrast to the com- munication heavy training of DL models, there is efficient and minimal communication between the worker nodes and the central coordinator. Specifically, the nodes only send a limited number of fea- ture scores and descriptions once during the execution, making communication trivial percentage of the whole process.
2.4 WAVELETS
Wavelets are localized waveforms (as seen in Figure Jon the right) and are well-studied method in signal processing that has been used extensively in the analysis of time series (Mallai 1999). There vast literature with methods and applications of wavelets on all types of problems, ranging from healthcare to audio analysis. and well-studied and developed families of wavelets suitable for spe- cific applications (Addison| [017 Merry & Steinbuch, 2005). There are also numerous approaches that combine wavelets with machine and deep learning methods, either as implicit or explicit feature extractors (Li et al. 2021 Wang et al. 2018). A seminal work is Bruna & Mallat] (2013), where the concept of a wavelet scattering network using Morlet wavelet is introduced, in combination with linear and support vector machine classifiers for hand-written digit classification and texture recognition. This method
constructed to be invariant translations of the input and stable to small deformations. It has also been recently used in combination with deep learning networks for specific applications (ooro& & Lee 2019 Jin & Duan 2020).
3

---

Under review as a conference paper at ICLR 2022
LightWaveS aims to combine the strong points of these works under single generalized framework, with a focus on efficiency. We aim to bridge the gap between ROCKET and the wavelet theory, and progress to the next logical step of wavelet scattering. We keep this approach lightweight, both in depth and paths of the scattering, so we can apply to time series channels on a massive scale in very short time. The arbitrary base set of wavelets can potentially be extended based on expert opinion, backed by the solid theory behind wavelets and their applications. making LightWavel suitable platform for experimentation on solutions for MTSC problems. Finally, the hierarchical feature filtering leads to the most relevant output features of the scattering coefficients being selected.
PROPOSED FRAMEWORK
3.1 PRELIMINARIES
3.1.1 MINIROCKET FUNDAMENTALS
(MINI)RCCKET is primarily based on the convolution operation, in which kernel & with size I (k E RR', bias 3 and dilation factor d is used to calculate a sliding dot product with a 1-D input R² of size L and produce an output æ', where each element is calculated (Dempster al. 2019:
x = Σ kj Litdoj + +
(1)
2=
MINIROCKET generates a large number (10000 by default) of those kernels. The weights are selected from an empirically chosen subset of 84 kernels of length 9 with weights in {2, -2.. This has been chosen to limit the possible weight combinations but is not unique for the purpose, and other lengths, different values or weights drawn from (O0 1) are equally effective (Dempster et al., 2020). The only thing that is important is that the kernel weights have sum 0. The bias drawn from the convolution output, and the dilation is drawn from the set ID = {[29] J), where e~ (4( m), with m such that the length of the dilated kernel does not exceed that of the input. Finally, half of the kernels use padding, which appends zeros either side of the input so that x' and æ have the same dimension. The feature extracted from each of the 10000 outputs is the percentage of positive values, and is used for the final classification.
3.1.2 WAVELET SCATTERING
Wavelet scattering the process of applying wavelet transforms in a cascading manner (Bruna & Mallat, 2013), combined with non-lincarities and pooling. The wavelet transform is method used to approximate a signal using a set of wavelets which originate from a "mother" wavelet (t), scaled by and shifted by b (Young 2012). Each such wavelet can be described as:
1
V. (1,
11
Intuitively, we can relate these wavelets to the convolution filters that we discussed above, with the dilation being the scale of the wavelets (how "narrow" or "wide" they are), and the shift parameter b corresponding to the starting point of the convolution on the input (i-d+j). This connection between convolutional networks and the scattering architecture has also been thoroughly explored in Mallat (016)). We can see example of wavelet convolution in Figure where the response is strong the points where the sliding wavelet "matches" the signal. In addition, the wavelet set created from mother wavelet is parallel to the way that MINIROCKET has a fixed set of kernel weights, for which different paddings and dilations are randomly selected, generating the child kernels.
In the wavelet scattering transform, each level X, the previous result is convoluted with each wavelet six,, (kernel) and a complex modulus operator is applied before propagating the result to the next level, such that:
UiA] JU[A 1] l
(3)
and the scattering coefficients that result from each level are
S(A|
U(A]
(4)
4

---

Under review as a conference paper at ICLR 2022
where ø is an averaging kernel. Since every level of the scattering transform there can be multiple candidate wavelets (kernels), there a geometric progression of potential paths, as can be seen in the graphic representation of the process in Figure
mmm /
Input Signal
4
UP41
U[4]]
UP41
Wavelet
42
4
mmy mmn w
Response
U[a
Upa3]
U[Aa]
UP22]
Up2
Up:3
Upa
Up:2
Figure 1: 2-level wavelet scattering (Bruna & Mallat 2013) and wavelet convolution example
Throughout the above intuitive description, we connected the concepts of MINIROCKET with the wavelet theory. In MINIROCKET the prototype kernels have zero mean, which corresponds to a desirable constraint of mother wavelets (Young) 2012). Under this new context, we can re- consider MINIROCKET as classifier based on convolutions with random child wavelets of an arbitrary set of mother wavelets. In addition, although not being wavelet scattering, MINIROCKET be intuitively mapped on the first level, with its kernel set equivalent the first level set, (Vo, Jo with 8 drawn from DD as we said above. Since each kernel is slid across the input, we can think of b as taking all discrete values from linput Ikernel for each candi- date wavelet, and the convolution output (response) as being the combined response of all shifted wavelets, portrayed in Figure [] However, MINIROCKET extracts the features before the appli- cation of any modulus operator, so it does not satisfy the rest of the wavelet scattering transform requirements.
3.2 ALGORITHM
Based on the above observations, we improve the approach and reach the crux of LightWaveS: Lightweight Wavelet Scattering based on random wavelets. The term lightweight refers both to the system optimizations of the framework that make it fast, such as the distribution, as well as the fact that only reduced set of wavelet scattering paths are computed. Although it established that not all paths need to be considered in scattering network (Bruna & Mallat 2013), since we are aiming for fast training and inference, we take this notion to its limit. We compute all coefficients for the given kernels and dilations for the first level, but consider only path per first level output for the second level. In this way, we limit significantly the memory and computation time required for the extracted features, while accepting the trade-off of losing some descriptive coefficients.
The choice of the kernels to be applied in the second layer is not immediately clear. We can consider heuristics such selecting kernels whose first level coefficients gave features with high correlation to the classification task. However, after experimenting with such heuristics, and even with up to four second layer paths, we found that the final classification results were not better than the simple approach of applying the same kernel and dilation, concept intuitively and loosely similar to the process of discrete wavelet transform (Sundararajan, 2016). Thus, we end up computing only the paths shown in bold in Figure [ which means that the same kernel is applied twice consecutively. In addition, we limit the depth to two layers, as a good balance point between computational speed and informative coefficients, observed in Bruna & Mallat (2013).
The main steps of LightWaveS are the following: across the channel dimension. For larger datasets, the nodes receive only sample of the total training examples, in order to speed up computation. This sample selection is the only source of randomness in the algorithm. Then, on each node, the kernels are generated, which for our purpose are based on the same subset of 84 kernels that MINIROCKET uses. We limit randomness even more, each kerne is dilated with all dilations in the set (20 2 21......., 25... We also completely
Initially, the dataset is split among the nodes
5

---

Under review as a conference paper at ICLR 2022
remove bias, and the padding is common to all kernels so that the output dimension is equal to the input.
All those kernels applied each of the imput channels according to the wavelet scattering process described above. The generated kernels have no complex part, so the non-lin non-lincarity modulus operator is equal to the absolute value of the convolution output. We also downsample the [[11] result by a factor of 2 before propagating it, while keeping the same scale for the kernel, so it operates in lower frequencies. This is according to the insight in Bruna & Mallat (2013) that the most useful paths are frequency decreasing. As far as the pooling is concerned, since we want to limit the number of coefficients that will be used as features, we use max and min pooling, instead of the averaging kernel. In this way. although we accept the loss of more information, we have fewer features and we also avoid two additional convolutions per path. The features that are extracted from each scattering path are 8: the first four are the max and min value of U[\1] and [7)) We selected those as the most straightforward non-linearities that can be quickly calculated during the convolution process. The other four are the percentage of positive values (ppv) and normalized longer sequence of positive values (Is) in ['{11] and U[A2] before the application of the modulus operator. We keep these features based MINIROCKET and MultiROCKET respectively, since they have proven to be useful during classification.
After the feature extraction, the first selection phase is performed each node in a supervised way using ANOVA. The main node then gathers the top scoring features and performs the final feature selection, using the minimum-redundang crementally selecting the feature with the highest score. This score is determined by its F-statistic (relevance the class), divided by its average correlation to the previously selected features (redun- dancy). In our case, we use the Pearson correlation coefficient as the correlation indicator. In both phases the feature selection methods are filter based, which are fast, classifier-indese innepiiir--ndeeedeeet and can be implemented to scale well with the number of features (Li et al. 2017). The design choices that are not guided or restricted by theory, such as the feature scoring function, are straightforward and focused on computational efficiency. We also performed empirical experimentation on develop- ment subset, although without exhaustive parameter tuning of the method.
maximun-relevance algorithm (Peng et al.] 2005). in-
4 EXPERIMENTS
4.1 DATASETS
We select as benchmark the UEA collection of multivariate datasets (Bagnall al. 2018), excluding Insect Wingreat,,, since due to its large size it presented issues when training ROCKET. The datasets described in detail in Ruiz al. (202T). Following the example of Dempster al. (2019), we selected 15 of the 30 UEA datasets to work on when developing the method, in order to draw generic conclusions and avoid overfitting the whole collection. In addition, we prepare and use five machinery related datasets:
MAFAULDA, from Machinery Fault Database (MAF) is a dataset with 8 sensor measure- ments on machine fault simulator, taken under normal conditions and five different fault types. We downsample the measurements so that the input length is 1000 steps and we split the dataset in train and test with ratio 85-15 %.
TURBOFAN (Saxena & Goebel 2008), is engine degradation simulation dataset col- lection, containing 4 datasets with operation simulations of engines that run until failure under different conditions, with measurements from 26 sensors. The goal is to predict the remaining useful life (RUL) of the engines. In order to turn the problem into binary classification, we prepare the dataset and the labels in a suitable way with the goal being classifying RUL as more or less than 20 operational cycles.
4.2 EXPERIMENTAL SETUP
All training experiments were run on nodes which have dual 8-core 2.4 GHz (Intel Haswell E5- 2630-v3) CPUs and 64GB of RAM. The inference experiments are executed on Jetson Xavier board which has an 8-core ARM CPU. We ran (MINIJROCKET on the 29 UEA datasets and the additional datasets using the default number of features (20 and 10 thousand respectively). As for

---

Under review as a conference paper at ICLR 2022
LightWaveS, we present three variants of the model, termed L1,L2 and L1L2. These versions refer to keeping the features only from the scattering level 1, level or both, although we consider the LIL2 version the default. We select 500 features the default variant, which has shown good balance between training time, inference speed and accuracy in the development set. We use a Ridge regression classifier from the Scikit-learn package (Pedregosa al. 2011) for all methods. We use a critical difference diagram to present the results, popular method of comparing classifier perfor- mance across multiple datasets (Ruiz al. 2021] [smail Fawaz et al. 2019). This shows the ranking of the classifiers and groups the ones that do not show statistical difference, based on pairwise com- parisons using Wilcoxon signed-rank test (Wilcoxon) [992) with Holm's 's alpha correction (Holm) 1979 Garcia & Herrera 2008). The grouped classifiers appear the diagram connected by thick horizontal line.
4.3 ACCURACY RESULTS
We present the performance of LightWaveS in terms of accuracy in comparison with (MINDROCKET, as well as other recent solutions, namely (M)OSS-CN) (Tang et al.] [202T) WEASEL+MUSE (Schäfer & Leser 2018). TapNet (Zhang et al. 2020) and Catch22 (Lubba et al., 2019). Apart from LightWaveS and (MINDROCKET, the rest of the accuracy metrics taken from the repositories of Tang et al. (2021 and Dhariyal et al. 2020 3 Due to missing values in those metrics, we manage to compare the methods on 28 of the 30 UEA datasets.
Accuracy
10
6
5
2
Catch22
ROCKET
LightWaveS-L2
Zein
4.000
MINIROCKET
TapNet
MOS-CNN
LightWeveS-LL
56007
OS-CNN
WEASEL MUSE
LighWWaveS-L112
Figure 2: Mean rank of LavetWaveS methods vs recent classifiers in terms of accuracy
In Figure 2 we can see the results. Although lower in rank, LightWaveS belongs in the same sta- tistical group with (MIN)ROCKET, which have the best accuracy. In addition, it is among the ranks of more complex DL methods, such as TapNet and MOS-CNN. Out of the three variants, LightWaveS-L2 performs the worst, without the benefit of faster execution that L1 has. We fo- cus on (MINDROCKET for a more detailed comparison, and also include the 5 additional datasets. Since the aim of LightWaveS is to approach their state-of-the-an accuracy with fewer features, not necessarily surpass it, we place the LightWaves results for all datasets in four accuracy bins com- pared (MINDROCKET: one for higher or equal accuracy, and three bins for lower accuracy, with difference less than 0,05, between 0.05 and 0.1 and more than 0.1 respectively. In Figure S we see the amount of datasets in each bin. For the majority of the datasets the accuracy stays in the first categories for all variants apart from 1.2, and the datasets with large accuracy deviation are few. In addition, we can see that just increasing the number of features to 1500 leads to improvement of the comparison results, especially in the case of MINIROCKET, showing the potential of the method to successfully tackle the harder datasets as well.
4.4 CHANNEL AND NODE SCALING OF TRAINING TIME
As a point of reference, ROCKET processes the whole UEA set (apart from InsectWingbeat) on single node in approximately 9 minutes, MINIROCKET in and LightWaveS (L.112) in 14. RMINDROCKET has complexity Okkrneels samples- tslength). Since LightWaveS uses one more convolution per kernel it adds a fixed factor of two the complexity, essentially keeping lin- ear scalability with these variables. Thus, we can focus on the scalability of the Light training time with the additional channel dimension and with the number of distribution nodes.
huteas://attbbb.ccom//nsssssaaaa http:://giithbbb.cco/mmmgg/////

---

Under review as a conference paper at ICLR 2022
20
20
(40 05)
[40.00
(8.05-0.11
(0.055..11)
15
12
10
10
L1
L2
L1L2 L1L2-1500
L1
L2
L1L2 L1L2-1500
Variant
Variant
(a) ROCKET
(b) MINIROCKET
Figure 3:
Distribution of datasets based on average accuracy difference between LightWaveS vari- ants and (MIN))ROCKTT
We select PEMS-SF from UEA which has 963 channels, and we use subsets of the total dataset train Ligh on 2 nodes, starting from 100 channels and incrementing by 100. We see in Figure 4a that all variants show (sub)linear scaling with the number of channels, which is expected since each additional channel results in fixed amount of additional convolutions, and there are no combinations or permutations of channels. For the node scaling, we also select FaceDetection, with 144 channels and we measure the training time across 1,2,4 and nodes. As we see Figure 4b) with PEMS-SF the gained training speedup is more close to idcal, since there are many channels for each node to work on and the distributed computation constitutes the majority of the execution time. In contrast that, with FaceDetection that has fewer channels, the computational cost starts being dominated by the operations on the main node (application of kernels, feature selection, final transformation), the gained speedup falls more quickly.
PEMS
Execution time factor
8
PEMS
PEMS L1L2
L1L2
FaceD L1
FacaD
Speedup
FoceD
2
700 800 900
Number of channels
Nodes
(a) Channel scalability
(b) Node scalability
Figure 4: Scalability of LightWaveS traing time with the number of (a) channels and (b) nodes
4.5 INFERENCE RESULTS
We manage to achieve significant speedup during inference due to the small amount of features, which heavily reduces the amount of convolutions required to transform a test sample. We perform a qualitative analysis, focusing on measuring the inference on the machinery datasets, since they could correspond to scenario of edge deployment of model for e.g. fault detection of machine. However, our conclusions hold for all datasets. We can see in Figure 5 the relation between the accuracy difference of the LightWaveS variants with (MINDROCKET and the speedup achieved. This figure frames clearly the trade-off that we propose: it is up to the end user to select their preferred balance of speedup and accuracy, in the cases where ideal case is available. The MF
8

---

Under review as a conference paper at ICLR 2022
dataset in the MINIROCKET comparison acts outlier in the graph due to its large length in combination with the optimizations of MINIROCKET. Our method of applying the kernels matches ROCKET's, s, but could also benefit from these optimizations, restoring the discrepancy.
Inference speedup (times)
50
TF2
40
TF3
TF3
TF4
15
30
10
10
-0.1
0.0
-0.2
Accuracy difference
Accuracy difference
(a) ROCKET
(b) MINIROCKET
Figure 5: Average inference speedup-accuracy difference scatterplot for the 5 machinery datasets, for each of the LightWaveS variants (note the different y axis scale)
DISCUSSIONS AND CONCLUSION
Summarizing, LightWaveS offers comparable to state-of-the-art accuracy on the datasets tested, with training time similar to (MINDROCCET and inference time 9x-65x faster on an edge device in the case of ROCKET and 1.3x 14x for MINIROCKET. Apart from the advantages a significant advantage of Wave is that it is based on wavelet theory, which has strong theo- retical base. A promising future direction is explore expert tuning of the framework, by preparing and including in the base set well tested wavelets, with different properties such as padding or di- lations depending on the case. Another benefit is that due the feature selection, LightWaves, can filter the channels to a subset of the originals. As seen in Table 5 this reduction ranged from 92%, with an average of 15% across all datasets and the larger datasets benefiting more. This has multiple advantages: it can give insights into which channels contain useful information for the free
have mentioned,
problem, leading to knowledge extraction. On edge devices, where resources are valuable, up incoming signal channels. Finally, LightWaveS can act as an initial fast channel filtering method that precedes another deep learning solution, reducing the training data required. LightWaveS can also benefit from, and is indeed orthogonal to, recent works on feature selection, since additional descriptive features can be used on the scattering coefficients, improving accuracy. Future work can include initial, more informed selection of the wavelets, that the wavelet scattering network can extend to more paths with wavelet combinations, as well as explicit, informed combinations of different channels.
REPRODUCIBILITY STATEMENT
The code of LigetWaveS, as well as the code used to train (MINT)OCCET,, run the inference exper- iments and preprocess the machinery datasets provided as supplementary material. All accuracy and time experiments were repeated multiple times (30-100), with different seeds and the average of metrics was taken for fairer representation of the results. The detailed metrics are provided in the Appendix. All datasets used publicly available. There has been effort to explicitly set the random seed for all methods that accept such a parameter, in order to enable reproducibility of the results as closely possible.
REFERENCES
Machinery Fault Database http://wwww222mmmmttrrrrfffrfnne 01. 01.htll.

---

Under review as a conference paper at ICLR 2022
Paul S Addison. The illustrated wavelet transform handbook: introductory theory and applications in science, engineering, medicine and finance. CRC press, 2017.
Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time ries classification bake off: A review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 31(3):6066660 660, 2017. doi: 10.1007/s 10618
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. arXiv: 181 1.00075 [es, stat), 2018.
Joan Bruna and S. Mallat. Invariant Scattering Convolution Networks. IEEE Transactions Pattern Analysis and Machine Intelligence,
2013. doi: 00.1099/PAAM..2022230
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W. Kempa-Liehr. Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh - A Python package). Neurocomputing, 2018. doi: 10.1016/j.neucom 2018.03..6..
Dask Development Team. Dask: Library for dynamic task scheduling, 2016. URL https:/ dask.
Angus Dempster, François Petitjean, and Geoffrey I. Webb. ROCKET: Exceptionally fast and ac- curate time series classification using random convolutional kemels. 1rX1v:1910.13511 [cs, stat], 2019.
Angus Dempster, Daniel F. Schmidt, and Geoffrey Webb. MINIROCKET: Very Fast (Almost) Deterministic Transform for Time Series Classification. arXiv:2012.08791 [cs, stat], 2020.
Bhaskar Dhariyal, Thach Nguyen, Severin Gsponer, and Georgiana Ifrim. An Examination of the Statc-of-the-Art for Multivariate Time Series Classification. In 2020 International Conference on Data Mining Workshops (ICDMW), pp. 243-250 Sorrento, Italy, 2020. IEEE. ISBN 978-1- 72819-012-9. doi: 10. 111BIICCDMM/1111.2220000.0
Amir-massoud Farahmand, Sepideh Pourazarm, and Daniel Nikovski. Random projection filter bank for time series data. In NIPS, pp. 6562-6572 2017.
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petit- jean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 34(6):193-11622 2020.
Salvador Garcia and Francisco Herrera. An extension on" statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons. Journal of machine learning research, 912),, 2008.
Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statis- fics, pp. 65-70, 1979.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: A review. Data Mining and Knowledge Discovery, 33(4):917-963 2019. doi: (00007//00018-01900000
Yang Jin and Yunling Duan. Wavelet scattering network-based machine learning for ground pene- trating radar imaging: Application in pipeline identification. Remote Sensing, 12(21)):36 2020.
Tianfu Li, Zhibin Zhao, Chuang Sun, Li Cheng, Xuefeng Chen, Ruqiang Yan, and Robert X. Gao. Wet An Interpretable Deep Neural Network for Industrial Intelligent Diagnosis. IEEE Transactions on Systems, Man, and Cybernetics: Systems, pp. 1-11, 2021. doi: 10.1 TTSMC.200030488550
Yun Li, Tao Li, and Huan Liu. Recent advances feature selection and its applications. Knowledge and Information Systems, 53(3):551-577, 2017. doi: 10.1007/s1011
10

---

Under review as a conference paper at ICLR 2022
Markus Löning, Anthony Bagnall, Sajaysurya Ganesh, Viktor Kazakov, Jason Lines, and Franz J. Király. Sktime: A Unified Interface for Machine Learning with Time Series. arXiv:190..0780 [cs, stat], 2019.
Carl H Lubba, Sarab S Sethi, Philip Knaute, Simon R Schultz, Ben D Fulcher, and Nick S Jones. catch22: Canonical time-series characteristics. Data Mining and Knowledge Discovery, 33(6): 1821-1852, 2019.
Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999.
Stéphane Mallat, Understanding deep convolutional networks. Philosophical Transactions of the Royal Society A: Mathematical,, Physical and Engineering Sciences, 374(2965):20150205 2016.
Ruben Mayer and Hans-Arno Jacobsen. Scalable deep learning on distributed infrastructures: Chal- lenges, techniques, and tools. ACM Computing Surveys (CSUR), 33(1):1-37,2022
RJE Merry and M Steinbuch. Wavelet theory and applications. literature study, Eindhoven university of technology, Department of mechanical engineering, Control systems technology group, 2005.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based mutual information criteria of max- duuedeeend,,,y, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence. 27):): 1236-1238, 2005.
Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall.
The great multivariate time series classification bake off: A review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 35(2))401-4499 2021.
doi: 10. 10.1007/s 111111/0/00111---1 00008000000007---55
A. Saxena and K. Goebel. Turbofan engine degradation simulation data set.
http///i.arc.nnaa.ggo hatpaa//woooooppoooooonooaaaar/e
2008
Patrick Schäfer and Ulf Leser. Multivariate Time Series Classification with WEASEL+MUSE.
arXiv:1711.11333 [cs], 2018.
Bedionita Soro and Chaewoo Lee. A wavelet scattering feature extraction approach for deep neural network based indoor fingerprinting localization. Sensors, 1998):7990, 2019.
D Sundararajan. Discrete wavelet transform: signal processing approach. John Wiley & Sons, 2016.
Chang Wei Tan, Angus Dempster, Christoph Bergmeir, and Geoffrey I. Webb. MultRooket: Effec- tive summary statistics for convolutional outputs in time series classification. arXiv:2102.00457 [es, stat), 2021.
Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Jing Jiang, and Michael Blumenstein. Rethinking ID-CNN for Time Series Classification: A Stronger Bascline. arXiv:2002.1006 [cs, stat), 2021.
Jingyuan Wang. Ze Wang, Jianfeng Li, and Junjie Wu. Multilevel Wavelet Decomposition Network for Interpretable Time Series Analysis. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1337-2446, London United Kingdom, 2018. ACM. ISBN 978- 1-4503-55525-4 doi: 10.1145/321981 9.3220060.
Frank Wilcoxon. Individual comparisons by ranking methods. Breakthroughs in statistics, pp. 196-202. Springer, 1992.
Randy K Young. Wavelet theory and its applications, volume 189, Springer Science & Business Media, 2012.
Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tien Lu. Tapnet: Multivariate time series classi fication with attentional prototypical network. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6845-688) 2020.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 (MINT)ROCKET ANALYSIS
Using the sktime repository version of the code, we can see Figure 6a that even keeping the top half features of MINIROCKET using unsophisticated chi-squared test does not result in statistically significant difference in the classifier performance (rather, slightly increases the rank). As for the channel inclusion, as extreme case, we can add noise channels to the UEA datasets (10% of the original channel number) and we see that both solutions are affected negatively when trained on them, belonging in statistically different group.
F1 Score
Score
2082
ROCKET, Noise
ROCKET MINIROCKET
MINIROCKET 3000 2032
2018
MINIROCKET,
T_CCCEET_11000 10000
MINIROCKET Noise
(a) Selection of fractions of the default number of fea- (b) Training on normal and noisy versions of the UEA tures
dataset
Figure 6: Mean ranks of (MINI)ROCKTT under different experiments, in terms of F1 score
A.2 DETAILED EXPERIMENT METRICS
Below the detailed metries accuracy, inference speedup and channel reduction.
12

---

Under review as a conference paper at ICLR 2022
Table I: Detailed accuracy metrics of LightWaveS-LLL2 and (MINT)ROCKET
Dataset
L1L.2
ROCKET
MINIROCKET
Mean
Std
Mean
Std
Mean
Std
KrticularWWorReeoopoe
0.997
0.000
0.993
0.000
0.993
0.002
AtrialFibrillation
0.267
0.000
0.067
0.000
0.124
0.023
BasieMotions
1.000
0.000
1.000
0.000
1.000
0.000
CharacterTrajectories
0.983
0.000
0.992
0.001
0.992
0.001
Cricket
0.931
0.000
1.000
0.000
0.986
0.000
DuckDuckGeese
0.440
0.000
0.505
0.033
0.696
0.027
ERing
0.959
0.000
0.986
0.003
0.981
0.003
Eigen Worms
0.962
0.000
0.901
0.008
0.954
0.007
Epilepsy
0.978
0.000
0.988
0.004
1.000
0.000
EthanolConcenttration
0.635
0.000
0.412
0.023
0.475
0.013
FaceDetection
0.614
0.009
0.638
0.008
0.616
0.008
FingerMovements
0.520
0.000
0.535
0.013
0.497
0.032
HandMoveemnntDireccion
0.270
0.000
0.493
0.031
0.380
0.027
Handwriting
0.371
0.000
0.585
0.004
0.510
0.006
Heartbeat
0.756
0.000
0.740
0.011
0.762
0.011
Japanese Vowels
0.922
0.000
0.966
0.002
0.987
0.004
LSST
0.382
0.039
0.639
0.003
0.652
0.004
Libras
0.900
0.000
0.906
0.004
0.922
0.009
MotorImagery
0.580
0.000
0.572
0.012
0.545
0.047
NATOPS
0.644
0.000
0.884
0.008
0.926
0.013
PEMS-SF
0.873
0.000
0.826
0.012
0.829
0.018
PenDigits
0.937
0.003
0.983
0.001
0.965
0.002
Phoneme
0.178
0.010
0.276
0.002
0.294
0.003
RacketSports
0.888
0.000
0.910
0.008
0.879
0.016
SelfRegulationnSCPI
0.761
0.000
0.849
0.005
0.917
0.006
SellRegulationSCP2
0.511
0.000
0.546
0.017
0.509
0.008
SpokenArabicDigiin
0.964
0.003
0.997
0.001
0.994
0.002
StandWalkJump
0.600
0.000
0.524
0.023
0.358
0.041
UWaveGestureLibrary
0.909
0.000
0.937
0.004
0.936
0.006
FD001
0.938
0.007
0.936
0.016
0.950
0.012
FD002
0.888
0.006
0.931
0.004
0.913
0.012
FD003
0.936
0.008
0.946
0.019
0.932
0.015
FD004
0.824
0.008
0.879
0.008
0.877
0.012
Mafaulda
0.762
0.012
0.922
0.008
0.900
0.009
A.3 NUMBER OF FEATURES SELECTION
the evolution of the accuracy with the increasing number of features for the LIL2 method, as measured in our randomly selected development subset of datasets. We see that a common pattern that the accuracy initially increases, and reaches plateau after a number of features (different for each dataset). Another pattern is the oscillation of accuracy around an initial value. The selection of 500 as the default number of features based on this graph, and is reasonable number which balances accuracy and method speed, as well as ensures that most development datasets are close to in the plateau region.
In Figure
13

---

Under review as a conference paper at ICLR 2022
Table 2: Detailed accuracy metries of LightWaveS-LI, L2 and LIL2-1500 variants
Dataset
L1
L2
L1L2-1500
Mcan
Std
Mcan
Std
Mean
Std
Articulary WordRecognition
0.997
0.000
0.980
0.000
0.993
0.000
AtrillFibilllation
0.267
0.000
0.067
0.000
0.200
0.000
BasioMotions
1.000
0.000
1.000
0.000
1.000
0.000
Character' CharacterTrraecctrries
0.980
0.000
0.983
0.000
0.987
0.000
Cricket
0.972
0.000
0.944
0.000
0.986
0,000
DuckDuckGeese
0.400
0.000
0.360
0.000
0.420
0.000
ERing
0.967
0.000
0.922
0.000
0.974
0.000
Worms
0.809
0.000
0.962
0.000
0.969
0.000
Epilepsy
0.986
0.000
0.978
0.000
0.993
0.000
EthanofConcentration
0.551
0.000
0.559
0.000
0.631
0.000
FaceDetection
0.622
0.008
0.504
0.008
0.618
0.007
FingerMovements
0,580
0.000
0.490
0.000
0.530
0.000
HandMoveemenDirecctoon
0.270
0.000
0.243
0.000
0.297
0.000
Handwriting
0.329
0.000
0.279
0.000
0.388
0.000
Heartbeat
0.766
0.000
0.761
0.000
0.737
0.000
Japanese Vowels
0.932
0.000
0.935
0.000
0.965
0,000
LSST
0.402
0.035
0.343
0.004
0.425
0.014
Libras
0.872
0.000
0.794
0.000
0.928
0.000
MotorImagery
0.490
0.000
0.520
0.000
0.520
0.000
NATOPS
0,644
0.000
0.633
0.000
0.622
0,000
PEMS-SF
0.896
0.000
0.861
0.000
0.879
0.000
PenDigits
0.943
0.003
0.922
0.002
0.958
0.001
Phoneme
0.179
0.008
0.143
0.018
0.222
0.004
RacketSports
0.862
0.000
0.763
0.000
0.888
0.000
SelfRegulattonSCCPI
0.744
0.000
0.696
0.000
0.761
0.000
SellRegulatiooSCCP2
0.539
0.000
0.522
0.000
0.533
0.000
SpokenArrabiDDigits
0,967
0.003
0.966
0.002
0.980
0.002
StandWalkJump
0.400
0.000
0.600
0.000
0.533
0.000
UWaveGestureLLiraan
0.900
0.000
0.738
0.000
0.931
0.000
FD001
0.958
0.010
0.941
0.005
0.944
0.006
FD002
0.877
0.007
0.907
0.004
0.927
0.003
FD003
0.965
0.009
0.935
0.010
0.956
0.008
FD004
0.796
0.008
0.867
0.006
0.876
0.006
Mafaulda
0.745
0.010
0.680
0.014
0.833
0.010
14

---

Under review as a conference paper at ICLR 2022
Table 3: Inference speedup of LightWaveS with fixed seed (0) compared to ROCKET
Dataset
Inference Speedup
L1L2
L1
L2
LIL2
1500
Mean
Std
Mean
Std
Mean
Std
Mean
Std
Articulary WordRecognition
33.3
4.4
23.8
2.5
24.5
3.0
14.4
1.4
AtrialFibrillation
36.3
3.9
19.1
1.7
19.0
1.5
11.7
0.8
BasicMotiion
28.5
5.4
19.3
2.4
21.4
3.7
12.0
0.8
CharacterTrajectorfe
28.1
4.6
19.9
1.4
20.6
2.7
13.0
0.6
Cricket
59.1
5.6
34.0
1.0
41.9
2.1
20.0
0.6
DuckDuckGeese
257.1
17.6
180.0
5.2
179.1
9.6
84.1
2.7
ERing
23.3
3.1
16.3
2.1
17.4
3.0
11.2
1.1
Eigen Worms
68.9
3.1
41.1
1.0
41.3
1.1
17.4
0.6
Epilepsy
29.1
2.8
19.6
26.0
3.0
0.7
EthanolConnennrattoo
44.8
2.9
28.3
1.2
37.3
1.6
16.1
0.7
FaceDetection
92.0
11.0
65.9
8.7
64.7
11.1
35.0
2.7
FingerMovements
39.2
5,4
27.2
4.4
28.1
3.9
16.3
1.6
HandMovementDirection
61.1
3.8
38.0
2.0
37.8
1.8
17.2
0.5
Handwriting
28.1
4.0
19.2
2.1
20.9
2.0
12.0
0.6
Heartbeat
67.3
5.2
42.3
2.5
44.6
1.8
17.6
0.7
Japanese Vowels
25.1
2.7
19.5
2.5
19.5
2.6
13.0
1.6
LSST
21.2
2.5
16.3
2.0
17.2
2.8
10.5
1.8
Libras
14.8
1.5
10.7
1.4
12.1
1.7
8.3
1.3
Motorlmagery
79.0
2.0
46.6
1.0
47.7
1.0
18.0
0.4
NATOPS
37.1
4.4
25.3
3.6
29.2
4.2
16.9
1.7
PEMS-SF
283.1
36.2
207.3
19.8
205.9
21.8
93.9
3.4
PenDigits
8.3
8.9
1.4
6.8
0.9
Phoneme
47.4
5.3
30.1
2.1
31.7
1.7
14.7
0.4
RacketSports
19.8
2.4
14.1
1.8
15.0
1.9
8.8
1.0
SeliRegulattonSCCPI
55.6
3.0
32.5
1.5
35,4
1.0
15.3
0.3
SellRegulationSCCP2
59.7
2.7
35.6
1.0
37.6
1.0
15.1
0.4
SpokenAArabicDigits
37.4
5.4
25.9
3.0
28.4
3.3
14.9
0.9
StakJmal
54.7
1.5
32.9
0.7
33.9
0.8
15.3
0.4
UWaveGestureLibrray
36.6
3.3
21.8
0.9
26.6
1.4
12.8
0.4
FD001
23.4
2.1
20.2
2.9
20.1
2.8
13.8
1.8
FD002
21.3
2.9
18.9
2.3
19.3
3.3
14.4
2.8
FD003
30.2
4.0
23.7
4.0
25.6
3.4
16.0
1.9
FD004
28.4
24.9
3.2
24.4
2.8
2.4
Mafaulda
55.2
8.1
36.4
4.6
37.6
6.1
16.2
1.4
15

---

Under review as a conference paper at ICLR 2022
Table 4: Inference speedup of LightWaveS with fixed seed (0) compared to MINIROCKET
Dataset
Inference Speedup
L1L2
L1
L2
LIL2
1500
Mean
Std
Mean
Std
Mean
Std
Mean
Std
Articulary WordRecognition
5,9
0.8
4.2
0.5
4.4
0.6
2.6
0.3
AtrialFibrillation
2.8
0.3
1.4
0.1
1.4
0.1
0.9
0.0
BasicMotiins
6.5
1.3
4.4
0.6
4.9
0.9
2.7
0.2
CharacterTraaectoorr
4.5
0.8
3.2
0.3
3.3
0.5
2.1
0.1
Cricket
2.3
0.2
1.3
0.1
1.6
0.1
0.8
0.0
DuckDuckGeese
232.9
15.6
163.1
5.2
162.2
8.1
76.2
2.3
ERing
6.7
1.0
4.7
0.6
5.0
0.8
3.2
0.3
Eigen Worms
1.6
0.1
0.9
0.0
0.9
0.0
0.4
0.0
Epilepsy
4.1
0.4
2.7
0.2
3.6
0.4
1.8
EthanolCooncenraation
1.5
0.1
0.9
0.0
1.3
0.0
0.5
0.0
FaceDetection
71.1
8.5
51.0
6.6
50.0
8.5
27.0
2.1
FingerMovements
19.9
2.7
13.8
2.3
14.3
2.0
8.3
0.8
HandMovementDirection
4.9
0.3
3.1
0.2
3.0
0.2
1.4
0.1
Handwriting
4.7
0.7
3.2
0.4
3.5
0.4
2.0
0.1
Heartbeat
13.5
1.1
8.5
0.5
8.9
0.4
3.5
0.1
Japanese Vowels
12.6
1.3
9.8
1.2
9.8
1.3
6.5
0.8
LSST
8.9
1.2
6.9
0.9
7.3
1.3
4.4
0.8
Libras
6.3
0.7
4.5
0.6
5.1
0.7
3.5
0.6
Motorlmagery
4.7
0.1
2.8
0.1
2.8
0.1
1.1
0.0
NATOPS
17.6
12.1
13.9
2.1
8.0
0.8
PEMS-SF
258.2
33.3
189.0
17.6
187.7
19.8
85.7
3.0
PenDigits
7.9
1.9
7.0
7.5
1.4
0.9
Phoneme
7,4
0.8
4.7
0.4
5.0
0.3
2.3
0.1
RacketSports
9.6
1.2
6.8
0.9
7.2
0.9
4.3
0.5
SeliRegulattonSCCPP
2.8
0.2
1.6
0.1
1.8
0.1
0.8
0.0
SellRegulationSCCP2
2.5
0.1
0.1
1.6
0.0
0.6
0.0
SpokenAArabicDigits
10.7
1.6
7.4
0.8
8.1
1.0
4.3
0.3
StakJmal
1.6
0.1
1.0
0.1
1.0
0.1
0.5
0.0
UWaveGestureLibrray
3,8
0.4
2.3
0.2
2.8
0.2
1.3
0.1
FD001
14.5
1.5
12.5
1.9
12.5
1.7
8.5
1.1
FD002
15.2
2.1
13.5
1.6
13.8
2.3
10.3
1.9
FD003
17.4
2.4
13.6
2.3
14.7
1.9
9.2
1.1
FD004
20.8
18.2
2.3
17.9
2.1
10.8
1.7
mafaulda
2.6
0.3
1.7
0.1
1.8
0.2
0.8
0.0
16

---

Under review as a conference paper at ICLR 2022
Table 5: Number of original dataset channels used by LightWaveS
Channels
Dataset
Channels
retained by method
L11.2
LI
L2
LIL2
1500
Mean
Std
Mean
Std
Mean
Std
Mean
Std
AWR
9
7
8
0
7
0
7
0
AtrialFib
2
2
2
0
BasicMotions
6
5
5
0
5
0
ChTrajectories
3
3
3
0
Cricket
6
6
4
0
6
0
0
DuckGeese
1345
102
124
98
0
174
0
ERing
4
4
0
EigenWorms Worms
6
6
6
6
0
Epilepsy
3
3
0
EthanoIC
3
3
3
0
3
0
0
FaceDetection
144
106.05
6.92
134.38
2.48
117.78
6.89
142.87
1.42
FingerMov
28
28
28
O
28
0
28
0
HMD
10
10
10
10
0
Handwriting
3
3
3
0
3
0
3
0
Hcartbeat
61
46
51
O
42
0
58
0
Vowels
12
6
10
4
0
10
0
LSST
6
4.9
0.84
4.83
0.69
4.5
0.97
5.7
0.46
Libras
2
2
2
0
0
MotorImagery
64
64
64
0
64
0
64
NATOPS
24
6
6
0
PEMS-SE SF
963
113
209
0
132
0
300
PenDigits
2
2
2
0
Phoneme
11
11
11
11
O
11
packetSports
6
6
6
0
ScIIRegSCPI
6
6
6
6
0
0
Sel
7
7
O
0
SpArabicDigits
13
8.47
0.57
8.83
0.38
8
0.52
9.27
0.45
SWJump
4
4
4
0
0
UWaveGL
3
3
3
0
3
0
0
FD001
26
11.38
2.11
19.23
1.87
13.53
1.79
20.23
1.74
FD002
26
21.63
0.49
20.2
0.76
21.23
0.5
22.93
0.83
FD003
26
22.72
1.65
23.1
1.52
23.57
1.5
25.93
0.25
FD004
26
19.73
0.52
25.07
0.94
20.33
0.96
25.97
0.18
Mafaulda
8
8
8
8
0
0
17

---

Under review as a conference paper at ICLR 2022
Accurscy
06
BasicMotions
ERing
DigenWormss
Epilepay
EthanooCoocanttttoon
FaceDetectio
Finger Movements
HardhhovementDeedooo
Handwilling
LSST
Motorlmagery
NATOPS
SefTiegulatonSC22
SpokenArabicDlgits
200
400
600
800
1000
1200
1400
1600
Number of features
Figure 7: Evolution of accuracy with increasing number of features for LIL2 method
18


Reviews:
[{"review_id": "uKLzLZUP2Lf", "reviewer": "Reviewer_zJvV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\nThe paper studies an important research direction. \n\n## Weaknesses:\nThe presentation of the work is meandering and unfocused, some typos are present which negatively impact reading. \n\nMultivariate time series classification is a very well studied topic. The proposed method, LightWaveS, is a relatively minor extension of the ROCKET method. The overall contribution of this work may be incremental. Moreover many design choices seem rather arbitrary and lack either intuition or rigor (see i-iii). E.g. what allows the authors to only consider the paths in bold in Figure 1 and ignore all the others? It is clear that this reduces computational complexity but it is not described what are the modeling issues related with this arbitrary choice.\n\n**Major issues:**\n\n**i-** the distributed implementation, while being counted as a contribution, to me seems rather straightforward and is not described in much detail. If the distributed implementation is not straightforward I believe it is worth to precisely describe it in the appendix, which at this stage is not present, and highlight the specific and unique, to this work, design choices the authors made.\n\n**ii-** the connection with the scattering transform is not clear. The overall LightWaveS architecture is NOT a scattering network, nor an approximation in any way. The computational “trick” suggested by the authors (i.e. considering only the paths in bold in Figure 1) oversimplifies the proposed model. I believe it would be important to spend more words and perhaps perform an experimental evaluation (if no theory can be used) on this crucial aspect. \n\n**iii-** What is the intuition behind the use of ANOVA as a general feature selection criterion? Why has L1 regularization or any other learnt subset of features not been compared? The choice of using ANOVA seems rather arbitrary and does not help the reader in gaining any deeper intuition on the design choice.\n\n**iv-** Both features’ interpretability and incorporation of domain knowledge is explicitly cited in the abstract, I do not find experiments supporting this claim in the main text, these are briefly mentioned in the “Discussions and Conclusions” section. Please update the abstract or add a more in depth discussion (validated through proper experiments). \n\n\n**Minor:** \n\nWhat do authors precisely mean by: “there is efficient and minimal communication between the worker nodes and the central coordinator.”?\n\nFigure 1 should be improved in quality, moreover its font should be corrected to be consistent with the main text.\n\nThe L1 and L2 versions of the algorithm should be considered as ablated models of the L1L2 model rather than fully fledged models. \n\n**Experimental results:**\n- Give some intuitions on why the L2 model is worse than the L1 one (in all the experiments).\n- Missing appendix with extra details on datasets and “implementation” (e.g. distributed implementation and Ridge classifier).\n- The reader would expect the experiment in Figure 2.b repeated for LightWaveS to test the capability of features selection of ANOVA.\n- Add standard deviation on accuracy results. \n\n**Typos and notation:**\n- Bad notation in section 3.1.1.: “max” for the exponent used to sample the max length of dilation, and “b” is used both in the scattering transform and in Equation (1).\n- Section 3.2: Be consistent with common literature on scattering transform: use subscripts (e.g. \\lambda_1 in place of \\lambda 1).\n- In section 3.1.2, the following quotes:  ”child”. \n- Algorithm 1 does not add much to the exposition and is not very clear (missing quantities such as L and typo on x in place of \\times in the definition of the input \\mathbf{X}).\n- Part of the introduction is better suited for the related work section (too details which are hard to grasp for the reader a this time).\n- Improve the writing of section 4.5: the last part is hard to read and understand, while the first part is trivial. \n\n\nHere are some references which worth mentioning are:\n\n[1] (NIPS 2017) Random Projection Filter Bank for Time Series Data\n\n[2] (ICLR 2020) Classification-Based Anomaly Detection for General Data", "summary_of_the_paper": "This paper presents a distributed algorithm to solve Multivariate time series classification (MTSC). The proposed solution LightWaveS is based on the ROCKET algorithm. The advancements over ROCKET are: feature selection and distributed implementation. The authors show these advancements allow LightWaveS to outspeed both ROCKET and MINIROCKET (a faster variant of MINIROCKET) in inference.", "summary_of_the_review": "The presentation of the work is meandering and unfocused, some typos are present which negatively impact reading. \nMultivariate time series classification is a very well studied topic. The proposed method, LightWaveS, is a relatively minor extension of the ROCKET method. The overall contribution of this work may be incremental. Moreover many design choices seem rather arbitrary and lack either intuition or rigor (see i-iii). For these reasons I believe the manuscript does not meet ICLR's standards and should be rejected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "u8NmfoS24-L", "reviewer": "Reviewer_uVpJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths :\n \nLightWaveS offers comparable accuracy to the original ROCKET model, with training time similar to (MINI)ROCKET and inference time 8x-30x faster on an edge device. \n \nWeaknesses :\n \n1. The authors claim that they utilize a wavelet scattering transformation in the model as the main contribution. The wavelet is realized by using a dilation operation in convolutional kernels. There is only one sentence: “Intuitively, we can relate these ... wavelet ...” to illustrate the relationship between them, which is not convincing. The author should justify this important claim and provide theoretical explanations if any. \n \n2. The author claim that the proposed LightWaveS is much faster than (MINI)ROCKET. The reason is very simple: (MINI)ROCKET uses the default number of features (20 and 10 thousand), and LightWaveS only uses 500 features. Obviously, the computational cost of the LightWaveS should be significantly less than (MINI)ROCKET. However, the authors only conduct feature selection on their method, what if feature selection is also conducted on (MINI)ROCKET? \n\n3. The authors should conduct ablation studies on the features generated with wavelet scattering and the feature selection, and show which part contributes to the better accuracy/efficiency tradeoff more?\n \n4. The figures are difficult to understand, e.g., some notations are not explained, “Count” in Fig.4. what are numbers mean? (0.05, 0.05-0.1)\n", "summary_of_the_paper": "This paper presents a multivariate time series classification (MTSC) model, namely LightWaveS, that extends another MTSC framework ROCKET. In addition to inheriting the successful aspects of the ROCKET model family (accuracy and training time), it significantly improves inference time. To be specific, LightWaveS adds wavelet scattering, multi-node distribution, and \"smart feature selection\" to address the drawbacks of the ROCKET. Experiments on five machinery-related datasets show the efficacy of LightWaveS.", "summary_of_the_review": "While the results are promising compared to existing ROCKET models, the proposed solution is not well explained and the experimental results are not fully justified. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q583eLQRatp", "reviewer": "Reviewer_ooqH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides promising directions for improving the inference time which is critical for edge devices; however, there are a couple of points that needs to be addressed first. \n\n1) A major source of confusion in Algorithm1 is that non of the functions GetDataSlice(), GenerateKernels(), WaveletScatteringFeatures(), TopFeatureSelection(), GatherTopFeatures(), FeatureTransform(), and MRMRFeatureSelection() are defined and the descriptions within the manuscript are very high-level. It is recommended to provide more details about the functionality of these subroutines.\n\n2) During the inference, every evaluation of the same dataset may take slightly different runtime, as it is usually always the case in practice. Hence, in order to show the improvement in inference speedup, as in Figure 6, authors need to repeat the inference test several times and report the mean and standard deviation (SD) of the inference speedup instead of just a single number. Hence, the simulations for Figure 6 needs to be repeated and both the average and SD should be reported. If the standard deviation is very small, Figure 6 can be modified to only contain the mean values; however, in either case, the result should be based on multiple tests and not just a single inference test.", "summary_of_the_paper": "This paper tries to improve the inference time of deep learning models by proposing a distributed feature selection process based on wavelet scattering transformations of the time series. In particular, authors proposes a method called LightWaveS for a fast, distributed transformation of multivariate time series based on convolutional kernels, wavelet scattering and feature selection, for lightweight and accurate classification with linear classifiers. The end goal is to improve the inference time while maintaining the performance.", "summary_of_the_review": "See the comments provided above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DiSd69-v9cJ", "reviewer": "Reviewer_Vj84", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors present a multivariate time series classification (MTSC) pipeline that is an improvement over the (MINI)ROCKET algorithm. The authors use wavelets in a clever way that requires only 2.5% of ROCKETs features. The authors also report averaged performances, which is the right practice and generates robust results.\n\nI would say that the main weakness is that nothing is said about hyper-parameter searches. Thus, all the results are greatly dependent on whichever values were chosen. Hence, it is not possible to assess the generality of the presented results.", "summary_of_the_paper": "The authors present the LightWaveS pipeline, whose training is comparable to that of (MINI)ROCKET and an inference time 8x to 30x times faster. ", "summary_of_the_review": "The authors present an optimized pipeline with good results, whose generality is not possible to assess.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AVI83Ol-Y1E", "reviewer": "Reviewer_PviB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper proposes an algorithm that deals with an important problem.\n2. The experiments demonstrate the scalability and execution time benefits and show that the accuracy losses are not high.\n\nWeaknesses:\n1. Several aspects of the algorithm need further explanation:\n     a. Why was the same set of kernels used for both levels of the two-level wavelet scattering? This should be justified and some comparison should be done with the case where different features are chosen at each level.\n     b. Is algorithm 1 run at each level or at the top level? How are inputs distributed among the different nodes and how are outputs collected from the different nodes and merged? This needs explanation.\n2. In figure 2 (part of experiments), what do the numbers 2000, 3000, 5000, and 10000 represent? ", "summary_of_the_paper": "This paper works to improve upon the ROCKET algorithm for multivariate time series classification. In particular, the proposed algorithm, LightWaveS, added wavelet scattering, convolutional kernels, and feature selection to make the algorithm much faster and more scalable while preserving close to the same accuracy. The algorithm can also be distributed and the experiments demonstrate good execution time reduction with additional computational nodes.", "summary_of_the_review": "Overall, the paper does well at handling an important problem. However, there are some key details about the algorithm that need to be clarified and a key aspect of the experimental results need to be clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3X4v1fIQmLr", "reviewer": "Reviewer_DUKD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Distributed training and inference is a timely and important problem for time-series tasks\n2. Real-world datasets are used and the code is available\n3. The method is simple and seems effective\n\nWeakness\n\n1. The solution is mostly a nice engineering effort but is lacking in terms of methodological results\n2. The results are not impressive: there is a loss in accuracy\n\nDetails:\n\n1. The solution seems a combination of two existing methods. There are claims of the generality of this methodology but no proof of that. How does it apply to other kinds of features? how well it works. The \"random\" kernels of ROCKET help to apply the exact same framework, but I am not sure the same methodology would help combine features from catch22 for example or other approaches.\n\n2. How much is the benefit of your approach vs. using the Dask framework (or other parallelization frameworks) for MINIROCKET for example and perform inference by distributing model and fingering class per time series using a similar amount of resources. There are way simpler approaches to increase throughput once the model is built considering the significant training overhead your method is having.\n\n3. There is a significant loss in accuracy. Even though it's not statistically significant, we can observe a significant loss. It's not significant because here you evaluate such a small number of datasets so there is no enough evidence to show significant better or worse results either way. but going from 2nd best to 8th best or 9th best indicates some significant loss. (CD lines cover the entire area...)\n\n4. There is growing literature for DNNs once the model is trained to keep only a small subset of parameters/connections to significantly compress the models. Similar methodologies could be applied here as a lot simpler solutions (as post-processing steps once the models are built). Even though you make this connection with the work of Bruna and Mallat, there is a growing literature these days that is not mentioned or compared against that is critical for understanding if your path is the best/more meaningful than these more \"modern\" approaches.\n\nSee ideas from \"deep compression\" literature:\n\n Han, Song, Huizi Mao, and William J. Dally. \"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.\" arXiv preprint arXiv:1510.00149 (2015).", "summary_of_the_paper": "The paper proposes LightWaveS, a distributed methodology for multivariate time-series classification based on wavelet scatting transformation. The method utilizes a small percentage of the initial set of features while achieving comparable accuracy performance but with significant speedup. The method is evaluated on several publicly available multivariate time-series datasets.", "summary_of_the_review": "The method shows good speedup, however, there is a significant loss in accuracy too. DNN literature for compressing models is missing but seems relevant for this problem. The solution is not clear how it generalizes to other problems as claimed. No proof/experimentation is provided at least.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1XvRT1yAIlW", "reviewer": "Reviewer_u4XS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is largely well-written and provides a detailed analysis against several benchmarks. The idea of adding scattered wavelets to an existing model is interesting but it is not new as the authors have also cited related work in this domain. \n\nThe proposed method allows controlling the trade-off between complexity and accuracy when performing inference.\n\nThe presentation of results using the mean rank doesn't describe the accuracy of the model against the other models except for (Mini)Rocket. This is not an issue since (Mini)Rocket has the best mean rank, but it would have been very useful to see a full table of results; so that the reader can get a clear idea of the improvements or losses to accuracy/F1/etc they might expect.\nEspecially in some use cases, the trade-off between recall and precision will be very important. For example in an engine failure prediction, recall could have a much higher weight compared to precision. \n\nWhile the wavelet scattering is well described, the motivation behind using it, except improving the speed, is not very clear. And it is not also clear if with reduced features or reduce accuracy (some of) the existing models could also provide the same level of accuracy. \n\nA more intuitive explanation of the structure of the data that this method works on would have been very helpful. Real-world time-series data could be noisy, dynamic with deterministic or non-deterministic changes into its distribution and it would have been good to explain for what type of datasets and in what of the applications the proposed model could be useful.  \n", "summary_of_the_paper": "The paper describes a model for feature extraction and classifying multivariate time-series data. It proposes using scattered wavelet and multiple kernel analysis to extract features from multivariate time-series data. The model can run in a distributed manner and on multiple nodes. It uses a feature selection mechanism which also improves the interpretability of the method. \n\nThe work is built on the existing ROCKET and Mini-ROCKET models and instead of the kernel function used in ROCKET, it proposes using Wavelet Scattering. \n\nThe key contribution of the paper is modifying the ROCKET model and adding Wavelet Scattering and allowing a flexible number of kernels/features that can be extracted from each segment of the time-series data. This has contributed to improving the speed and scalability of the model compared to ROCKET. The work has also been extensively evaluated on the UEA benchmark datasets and two other separate datasets (one of these includes 5 sub-sets). ", "summary_of_the_review": "The paper is largely well-written and it is clear that the authors have put a lot of effort to implement and evaluate their work and benchmarking it again existing methods and have also tried it on new datasets. \n\nThe presentation of the results in terms of providing detailed Precision/Recall/F1/AUC would have been very helpful. Comparing the speed/complexity and/or performance of the other methods at the same level of accuracy would have provided a more balanced comparison. \n\nFigure 6 is hard to read and comprehend. It seems the mean difference is shown in Figure 6(b); but it is not clear what for example a -0.3 difference means. Does that mean 30% lower accuracy?  \n\nOverall, the idea of wavelet scattering as a feature extraction is interesting. However, as the authors have mentioned \"LightWaveS can act as an initial fast channel filtering method that precedes another deep learning solution\"; the latter could have been a good starting point to start the work and build a processing pipeline.\n\nIn terms of code and reusability, a set of python code have been provided but it would have been very helpful if dependencies or a read me or a notebook code accompanied the code to make it more readable/re-usable. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an algorithm called LightWaveS to improve the ROCKET (and mini-ROCKET) algorithm for multivariate time series classification, by using wavelet scattering instead of the kernel function. More than the usual number of reviewers were invited to provide independent reviews on the paper.

A concern was raised regarding the lack of hyperparameter search in the paper. The authors responded that this was intentional to avoid overfitting the solution to the tested datasets. This response is not convincing. Note that other important reasons to vary the hyperparameter values (as commonly adopted by ML researchers) are to study the sensitivity of the proposed method to hyperparameter settings and to perform more holistic performance comparison with other methods.

Other concerns on both novelty and significance have also been raised.

Although 2 of the 7 reviews show a weak support for acceptance, other reviewers have pointed out legitimate concerns that make this paper not ready for publication in ICLR in its current form. We appreciate the authors for clarifying some points in their responses and discussions and even including further results, but addressing all the concerns raised really needs a more substantial revision of the paper. We hope the comments and suggestions made by us can help the authors prepare a revised version that will be more ready for publication.
Decision: Reject