Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a distributed algorithm to solve Multivariate time series classification (MTSC). The proposed solution LightWaveS is based on the ROCKET algorithm. The advancements over ROCKET are: feature selection and distributed implementation. The authors show these advancements allow LightWaveS to outspeed both ROCKET and MINIROCKET (a faster variant of MINIROCKET) in inference.
main_review: ## Strengths:
The paper studies an important research direction. 

## Weaknesses:
The presentation of the work is meandering and unfocused, some typos are present which negatively impact reading. 

Multivariate time series classification is a very well studied topic. The proposed method, LightWaveS, is a relatively minor extension of the ROCKET method. The overall contribution of this work may be incremental. Moreover many design choices seem rather arbitrary and lack either intuition or rigor (see i-iii). E.g. what allows the authors to only consider the paths in bold in Figure 1 and ignore all the others? It is clear that this reduces computational complexity but it is not described what are the modeling issues related with this arbitrary choice.

**Major issues:**

**i-** the distributed implementation, while being counted as a contribution, to me seems rather straightforward and is not described in much detail. If the distributed implementation is not straightforward I believe it is worth to precisely describe it in the appendix, which at this stage is not present, and highlight the specific and unique, to this work, design choices the authors made.

**ii-** the connection with the scattering transform is not clear. The overall LightWaveS architecture is NOT a scattering network, nor an approximation in any way. The computational “trick” suggested by the authors (i.e. considering only the paths in bold in Figure 1) oversimplifies the proposed model. I believe it would be important to spend more words and perhaps perform an experimental evaluation (if no theory can be used) on this crucial aspect. 

**iii-** What is the intuition behind the use of ANOVA as a general feature selection criterion? Why has L1 regularization or any other learnt subset of features not been compared? The choice of using ANOVA seems rather arbitrary and does not help the reader in gaining any deeper intuition on the design choice.

**iv-** Both features’ interpretability and incorporation of domain knowledge is explicitly cited in the abstract, I do not find experiments supporting this claim in the main text, these are briefly mentioned in the “Discussions and Conclusions” section. Please update the abstract or add a more in depth discussion (validated through proper experiments). 


**Minor:** 

What do authors precisely mean by: “there is efficient and minimal communication between the worker nodes and the central coordinator.”?

Figure 1 should be improved in quality, moreover its font should be corrected to be consistent with the main text.

The L1 and L2 versions of the algorithm should be considered as ablated models of the L1L2 model rather than fully fledged models. 

**Experimental results:**
- Give some intuitions on why the L2 model is worse than the L1 one (in all the experiments).
- Missing appendix with extra details on datasets and “implementation” (e.g. distributed implementation and Ridge classifier).
- The reader would expect the experiment in Figure 2.b repeated for LightWaveS to test the capability of features selection of ANOVA.
- Add standard deviation on accuracy results. 

**Typos and notation:**
- Bad notation in section 3.1.1.: “max” for the exponent used to sample the max length of dilation, and “b” is used both in the scattering transform and in Equation (1).
- Section 3.2: Be consistent with common literature on scattering transform: use subscripts (e.g. \lambda_1 in place of \lambda 1).
- In section 3.1.2, the following quotes:  ”child”. 
- Algorithm 1 does not add much to the exposition and is not very clear (missing quantities such as L and typo on x in place of \times in the definition of the input \mathbf{X}).
- Part of the introduction is better suited for the related work section (too details which are hard to grasp for the reader a this time).
- Improve the writing of section 4.5: the last part is hard to read and understand, while the first part is trivial. 


Here are some references which worth mentioning are:

[1] (NIPS 2017) Random Projection Filter Bank for Time Series Data

[2] (ICLR 2020) Classification-Based Anomaly Detection for General Data
summary_of_the_review: The presentation of the work is meandering and unfocused, some typos are present which negatively impact reading. 
Multivariate time series classification is a very well studied topic. The proposed method, LightWaveS, is a relatively minor extension of the ROCKET method. The overall contribution of this work may be incremental. Moreover many design choices seem rather arbitrary and lack either intuition or rigor (see i-iii). For these reasons I believe the manuscript does not meet ICLR's standards and should be rejected.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a multivariate time series classification (MTSC) model, namely LightWaveS, that extends another MTSC framework ROCKET. In addition to inheriting the successful aspects of the ROCKET model family (accuracy and training time), it significantly improves inference time. To be specific, LightWaveS adds wavelet scattering, multi-node distribution, and "smart feature selection" to address the drawbacks of the ROCKET. Experiments on five machinery-related datasets show the efficacy of LightWaveS.
main_review: Strengths :
 
LightWaveS offers comparable accuracy to the original ROCKET model, with training time similar to (MINI)ROCKET and inference time 8x-30x faster on an edge device. 
 
Weaknesses :
 
1. The authors claim that they utilize a wavelet scattering transformation in the model as the main contribution. The wavelet is realized by using a dilation operation in convolutional kernels. There is only one sentence: “Intuitively, we can relate these ... wavelet ...” to illustrate the relationship between them, which is not convincing. The author should justify this important claim and provide theoretical explanations if any. 
 
2. The author claim that the proposed LightWaveS is much faster than (MINI)ROCKET. The reason is very simple: (MINI)ROCKET uses the default number of features (20 and 10 thousand), and LightWaveS only uses 500 features. Obviously, the computational cost of the LightWaveS should be significantly less than (MINI)ROCKET. However, the authors only conduct feature selection on their method, what if feature selection is also conducted on (MINI)ROCKET? 

3. The authors should conduct ablation studies on the features generated with wavelet scattering and the feature selection, and show which part contributes to the better accuracy/efficiency tradeoff more?
 
4. The figures are difficult to understand, e.g., some notations are not explained, “Count” in Fig.4. what are numbers mean? (0.05, 0.05-0.1)

summary_of_the_review: While the results are promising compared to existing ROCKET models, the proposed solution is not well explained and the experimental results are not fully justified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to improve the inference time of deep learning models by proposing a distributed feature selection process based on wavelet scattering transformations of the time series. In particular, authors proposes a method called LightWaveS for a fast, distributed transformation of multivariate time series based on convolutional kernels, wavelet scattering and feature selection, for lightweight and accurate classification with linear classifiers. The end goal is to improve the inference time while maintaining the performance.
main_review: The paper provides promising directions for improving the inference time which is critical for edge devices; however, there are a couple of points that needs to be addressed first. 

1) A major source of confusion in Algorithm1 is that non of the functions GetDataSlice(), GenerateKernels(), WaveletScatteringFeatures(), TopFeatureSelection(), GatherTopFeatures(), FeatureTransform(), and MRMRFeatureSelection() are defined and the descriptions within the manuscript are very high-level. It is recommended to provide more details about the functionality of these subroutines.

2) During the inference, every evaluation of the same dataset may take slightly different runtime, as it is usually always the case in practice. Hence, in order to show the improvement in inference speedup, as in Figure 6, authors need to repeat the inference test several times and report the mean and standard deviation (SD) of the inference speedup instead of just a single number. Hence, the simulations for Figure 6 needs to be repeated and both the average and SD should be reported. If the standard deviation is very small, Figure 6 can be modified to only contain the mean values; however, in either case, the result should be based on multiple tests and not just a single inference test.
summary_of_the_review: See the comments provided above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes a model for feature extraction and classifying multivariate time-series data. It proposes using scattered wavelet and multiple kernel analysis to extract features from multivariate time-series data. The model can run in a distributed manner and on multiple nodes. It uses a feature selection mechanism which also improves the interpretability of the method. 

The work is built on the existing ROCKET and Mini-ROCKET models and instead of the kernel function used in ROCKET, it proposes using Wavelet Scattering. 

The key contribution of the paper is modifying the ROCKET model and adding Wavelet Scattering and allowing a flexible number of kernels/features that can be extracted from each segment of the time-series data. This has contributed to improving the speed and scalability of the model compared to ROCKET. The work has also been extensively evaluated on the UEA benchmark datasets and two other separate datasets (one of these includes 5 sub-sets). 
main_review: The paper is largely well-written and provides a detailed analysis against several benchmarks. The idea of adding scattered wavelets to an existing model is interesting but it is not new as the authors have also cited related work in this domain. 

The proposed method allows controlling the trade-off between complexity and accuracy when performing inference.

The presentation of results using the mean rank doesn't describe the accuracy of the model against the other models except for (Mini)Rocket. This is not an issue since (Mini)Rocket has the best mean rank, but it would have been very useful to see a full table of results; so that the reader can get a clear idea of the improvements or losses to accuracy/F1/etc they might expect.
Especially in some use cases, the trade-off between recall and precision will be very important. For example in an engine failure prediction, recall could have a much higher weight compared to precision. 

While the wavelet scattering is well described, the motivation behind using it, except improving the speed, is not very clear. And it is not also clear if with reduced features or reduce accuracy (some of) the existing models could also provide the same level of accuracy. 

A more intuitive explanation of the structure of the data that this method works on would have been very helpful. Real-world time-series data could be noisy, dynamic with deterministic or non-deterministic changes into its distribution and it would have been good to explain for what type of datasets and in what of the applications the proposed model could be useful.  

summary_of_the_review: The paper is largely well-written and it is clear that the authors have put a lot of effort to implement and evaluate their work and benchmarking it again existing methods and have also tried it on new datasets. 

The presentation of the results in terms of providing detailed Precision/Recall/F1/AUC would have been very helpful. Comparing the speed/complexity and/or performance of the other methods at the same level of accuracy would have provided a more balanced comparison. 

Figure 6 is hard to read and comprehend. It seems the mean difference is shown in Figure 6(b); but it is not clear what for example a -0.3 difference means. Does that mean 30% lower accuracy?  

Overall, the idea of wavelet scattering as a feature extraction is interesting. However, as the authors have mentioned "LightWaveS can act as an initial fast channel filtering method that precedes another deep learning solution"; the latter could have been a good starting point to start the work and build a processing pipeline.

In terms of code and reusability, a set of python code have been provided but it would have been very helpful if dependencies or a read me or a notebook code accompanied the code to make it more readable/re-usable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper works to improve upon the ROCKET algorithm for multivariate time series classification. In particular, the proposed algorithm, LightWaveS, added wavelet scattering, convolutional kernels, and feature selection to make the algorithm much faster and more scalable while preserving close to the same accuracy. The algorithm can also be distributed and the experiments demonstrate good execution time reduction with additional computational nodes.
main_review: Strengths:
1. The paper proposes an algorithm that deals with an important problem.
2. The experiments demonstrate the scalability and execution time benefits and show that the accuracy losses are not high.

Weaknesses:
1. Several aspects of the algorithm need further explanation:
     a. Why was the same set of kernels used for both levels of the two-level wavelet scattering? This should be justified and some comparison should be done with the case where different features are chosen at each level.
     b. Is algorithm 1 run at each level or at the top level? How are inputs distributed among the different nodes and how are outputs collected from the different nodes and merged? This needs explanation.
2. In figure 2 (part of experiments), what do the numbers 2000, 3000, 5000, and 10000 represent? 
summary_of_the_review: Overall, the paper does well at handling an important problem. However, there are some key details about the algorithm that need to be clarified and a key aspect of the experimental results need to be clarified.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes LightWaveS, a distributed methodology for multivariate time-series classification based on wavelet scatting transformation. The method utilizes a small percentage of the initial set of features while achieving comparable accuracy performance but with significant speedup. The method is evaluated on several publicly available multivariate time-series datasets.
main_review: Strengths:

1. Distributed training and inference is a timely and important problem for time-series tasks
2. Real-world datasets are used and the code is available
3. The method is simple and seems effective

Weakness

1. The solution is mostly a nice engineering effort but is lacking in terms of methodological results
2. The results are not impressive: there is a loss in accuracy

Details:

1. The solution seems a combination of two existing methods. There are claims of the generality of this methodology but no proof of that. How does it apply to other kinds of features? how well it works. The "random" kernels of ROCKET help to apply the exact same framework, but I am not sure the same methodology would help combine features from catch22 for example or other approaches.

2. How much is the benefit of your approach vs. using the Dask framework (or other parallelization frameworks) for MINIROCKET for example and perform inference by distributing model and fingering class per time series using a similar amount of resources. There are way simpler approaches to increase throughput once the model is built considering the significant training overhead your method is having.

3. There is a significant loss in accuracy. Even though it's not statistically significant, we can observe a significant loss. It's not significant because here you evaluate such a small number of datasets so there is no enough evidence to show significant better or worse results either way. but going from 2nd best to 8th best or 9th best indicates some significant loss. (CD lines cover the entire area...)

4. There is growing literature for DNNs once the model is trained to keep only a small subset of parameters/connections to significantly compress the models. Similar methodologies could be applied here as a lot simpler solutions (as post-processing steps once the models are built). Even though you make this connection with the work of Bruna and Mallat, there is a growing literature these days that is not mentioned or compared against that is critical for understanding if your path is the best/more meaningful than these more "modern" approaches.

See ideas from "deep compression" literature:

 Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015).
summary_of_the_review: The method shows good speedup, however, there is a significant loss in accuracy too. DNN literature for compressing models is missing but seems relevant for this problem. The solution is not clear how it generalizes to other problems as claimed. No proof/experimentation is provided at least.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present the LightWaveS pipeline, whose training is comparable to that of (MINI)ROCKET and an inference time 8x to 30x times faster. 
main_review: The authors present a multivariate time series classification (MTSC) pipeline that is an improvement over the (MINI)ROCKET algorithm. The authors use wavelets in a clever way that requires only 2.5% of ROCKETs features. The authors also report averaged performances, which is the right practice and generates robust results.

I would say that the main weakness is that nothing is said about hyper-parameter searches. Thus, all the results are greatly dependent on whichever values were chosen. Hence, it is not possible to assess the generality of the presented results.
summary_of_the_review: The authors present an optimized pipeline with good results, whose generality is not possible to assess.
review_title: null

Meta-review: This paper proposes an algorithm called LightWaveS to improve the ROCKET (and mini-ROCKET) algorithm for multivariate time series classification, by using wavelet scattering instead of the kernel function. More than the usual number of reviewers were invited to provide independent reviews on the paper.

A concern was raised regarding the lack of hyperparameter search in the paper. The authors responded that this was intentional to avoid overfitting the solution to the tested datasets. This response is not convincing. Note that other important reasons to vary the hyperparameter values (as commonly adopted by ML researchers) are to study the sensitivity of the proposed method to hyperparameter settings and to perform more holistic performance comparison with other methods.

Other concerns on both novelty and significance have also been raised.

Although 2 of the 7 reviews show a weak support for acceptance, other reviewers have pointed out legitimate concerns that make this paper not ready for publication in ICLR in its current form. We appreciate the authors for clarifying some points in their responses and discussions and even including further results, but addressing all the concerns raised really needs a more substantial revision of the paper. We hope the comments and suggestions made by us can help the authors prepare a revised version that will be more ready for publication.
Decision: Reject