Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes DSN, a dynamic sparse network for time-series classification. The goal of DSN is to avoid hyper-parameter tuning and operate with sparse kernels to improve the resources and computational cost. Experiments over univariate and multivariate time-series datasets show that DSN can reduce in half the cost vs. strong recent baselines while achieving state-of-the-art accuracy performance.
strengths_and_weaknesses: Strengths:

S1. Reducing the costs of DNN is a timely and important problem

S2. Extensive experimentation using many public datasets

Weaknesses:

W1. The paper needs substantial effort to improve its presentation.

W2. Lack of novelty and technical depth

W3. Missing baselines 
questions: The problem is important. However, there are many concerns with this work:

W1. The paper needs substantial effort to improve its presentation.

RF terminology is not properly introduced. The connection to time-series analysis specifically is never discussed. 

There is not enough emphasis on preliminaries and background. Existing compression and sparse DNN solutions are properly introduced. Heavy re-writing is needed

W2. Lack of novelty and technical depth

It's like the first time a sparse NN architecture is applied to the particular time-series classification models, but it's unclear what is novel in this work. This is relevant to the previous comment about properly introducing the 3-4 areas in that space that focus on reducing resources/computational costs. Sparse connection is an emerging area. The paper reads mainly as an adaptation of these solutions and applied to a new domain but the technical depth is not clear, unfortunately.

W3. Missing baselines 

How out of the shelf methods that focus on reducing DNN parameters perform on this task. The paper only present how DNS performs vs. baselines that do not consider this problem so for sure there is an improvement. There are multiple works in that area from reducing from FP32 to FP16, from operating with only a few fourier coefficients [a], etc. Multiple baselines in that area are needed to prove how significantly better is DSN

[a] Dziedzic, Adam, et al. "Band-limited training and inference for convolutional neural networks." International Conference on Machine Learning. PMLR, 2019.

W4. UCR contains 128 datasets now and not 85.

There are 30 multivariate time-series datasets in timeseriesclassification.com - All results have to be updated with the recent benchmarks in this area.


limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a dynamic sparse network (DSN) with sparse connection to better determine the appropriate receptive field (RF) of time series classification (TSC) tasks. The contribution included alleviating hyper-parameter tuning while maintaining low resource costs. They pointed out the lack of balance between resources and performance without tortuous manual hyperparameter tunings. Where the authors’ DSN model would utilize their designed dynamic sparse CNN layers on resulting effective RF balance with TSC modeling. The experiment was done on real-world time-series datasets and their method showed better accuracy performance as well as lower resource cost
strengths_and_weaknesses: The paper has a number of strengths. The authors incorporated dynamic sparse training into RF, which is interesting and could guarantee the region attended to have high information.  Although keeping the kernel sparse during learning was difficult, the authors presented promising results through experiments, the solution could be quite significant for time series work which tends to lag other areas of ML data (images, notes, etc.).

The paper is well written and presented. The authors provided a careful literature review in illustrating the limitations of prior works. The authors’ architecture analysis of both theirs and other works are thorough

The paper has good experimental results. The authors seemed to heuristically apply weight pruning exploration but it was effective, and  provided seemingly comprehensive analysis as they applied 3 different datasets in experiments.

However there are a number of concerns and questions raised about the significance, originality, and technical contribution. The authors would need additional clarification about the implementation of each dataset. Ablation study was done on kernel group or sparse training, but the authors might want to consider taking other components out of the model- Such as sparse CNN modules- and seeing how much each component leads to improvements, to understand the specific contributions of the work. Since they claimed overall contribution was finding the important region in time series data, the authors would need to consider comparing transformer-based works since those would be stronger competitors in sequential data sets (currently none) and attention mechanisms that highlight regions of importance.

Additionally, the authors need clarification on the space of problems solved. Are there bounds on the time-series types of classification problems addressed by this work?


questions: 1. Did they use the same architecture for all Ablation studies, even though each dataset innately carried different characteristics? Is that a fair comparison?

2. In Section 3.4, the authors would need to clarify their weight pruning with more details and insights. Did the weight decay operation train concurrently with the weight activation? 

3.   RF was not properly explained properly in the abstract or introduction. “Hidden signals that can be ‘seen’” was not licitly discussed. What is meant by this?

4. The authors could be more specific about their claim of “more fine-grained training strategy” that achieving “diverse effective RF”, so that the readers could better grasp why theirs would be better. 

5. The authors could spend a bit more details how the resource awareness would scale up.  

6. It would be better for the authors to complete their experiments to have more intuitive illustrations of their model’s impact.

limitations: The authors have discussed the limitations but only in brief.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The variation of signal scales across and within time series data, makes it challenging to decide proper RF sizes for time series classification. This paper proposes a dynamic sparse network with sparse connections for time series classification, which can learn to cover various receptive field without cumbersome hyper-parameters tuning. The kernels in each sparse layer are sparse and can be explored under the constraint regions by dynamic sparse training, which makes it possible to reduce the resource cost. The experiment results show that the model can achieve state-of-art performance on both univariate and multivariate time series classification datasets. 
strengths_and_weaknesses: Strength:
The paper proposes a method that can learn to cover various receptive field without cumbersome hyper-parmeters tuning.

The experiments show that the method achieves state-of-the-art performance on several time series classification datasets.

The paper is well-written and well-structured.

Weakness:

This paper proposes a method that can cover various receptive fields. I wonder whether there are simpler ways to achieve this, e.g. combining the prediction of several models with different receptive fields?

The improvements seem very marginal e.g. on HAR. The authors may need to run the experiments multiple times and report both mean and standard deviation.

I wonder whether the baselines are strong enough? How about comparing to ViT models that has full receptive fields?


questions: 1. Whether there are simpler ways to achieve adaptive receptive fields?

2. The significance of experimental results.

3. Are the baselines strong enough?

4. A naive question: can the authors elaborate on how the convolution operators perform between a \in R^k and x^l_i \in R^{hw}? It is not straightforward to me somehow. Why O_j \in R^{hw}?
limitations: Yes
review_title: null

Meta-review: metareview: The paper presents a sparse network model for time series classification which dynamically determines receptive fields through the use of sparse kernels. The method matches state of the art performance, while reducing cost.

All the reviewers agreed that the paper solves an important problem and that the authors conducted extensive experiments. There was the question of novelty raised by reviewer Ni8e, although the authors included, in their response, a detailed description of how their work is different from existing methods. The authors also provided clarifications to the issues raised by the other reviewers, even adding new baselines, which were found convincing by reviewers 3Hph and skdR.
confidence: Certain
Decision: Accept