T-WaveNet: A Tree-Structured Wavelet Neural Network for Time Series Signal Analysis
Authors: Minhao LIU
Ailing Zeng
Qiuxia LAI
Ruiyuan Gao
Min Li
Jing Qin
Qiang Xu
Venue: ICLR 2022 Poster (2022)

Abstract: Time series signal analysis plays an essential role in many applications, e.g., activity recognition and healthcare monitoring.
Recently, features extracted with deep neural networks (DNNs) have shown to be more effective than conventional hand-crafted ones.
However, most existing solutions rely solely on the network to extract information carried in the raw signal, regardless of its inherent physical and statistical properties, leading to sub-optimal performance particularly under a limited amount of training data.
In this work, we propose a novel tree-structured wavelet neural network for time series signal analysis, namely \emph{T-WaveNet}, taking advantage of an inherent property of various types of signals, known as the \emph{dominant frequency range}. Specifically, with \emph{T-WaveNet}, we first conduct frequency spectrum energy analysis of the signals to get a set of dominant frequency subbands. Then, we construct a tree-structured network that iteratively decomposes the input signal into various frequency subbands with similar energies. Each node on the tree is built with an invertible neural network (INN) based wavelet transform unit. Such a disentangled representation learning method facilitates a more effective extraction of the discriminative features, as demonstrated with the comprehensive experiments on various real-life time series classification datasets. 

Published as a conference paper at ICLR 2022
T- WAVENET: A TREE-STRUCTURES WAVELET NEU- RAL NETWORK FOR TIME SERIES SIGNAL ANALYSIS
Min Li", Minhao LIUin,Ailng Zeng', Qiuxia LAT', Ruiyuan G
Jing Qin!, Qiang Xul
"Dpaarmeett of Computer Science & Engineering. The Chinese University of Hong Kong i School of Nursing, The Hong Kong Polytechnic University {mhhiu, qxu}écse. cuhk. edu. hk
ABSTRACT
Time series signal analysis plays essential role in many applications. e.g., activity recognition and healthcare monitoring. Recently, features extracted with deep neural networks (DNNs) have shown to be more effective than conventional hand-crafted ones. However, most existing solutions rely solely on the network extract information carried in the raw signal, regardless of its inherent physical and statistical properties, leading to sub-optimal performance particularly under a limited amount of training data.
In this work, we propose a novel tree-structured wavelet neural network for time series signal analysis, namely T-WaveNet, taking advantage of an inherent property of various types of signals, known as the dominant frequency range. Specifically, with Waveeeett, we first conduct frequency spectrum energy analysis the signals to get set of dominant frequency subbands. Then, we construct network that iteratively decomposes the input signal into various frequency sub- bands with similar energies. Each node on the tree is built with an invertible neural network (INN) based wavelet transform unit. Such disentangled representation learning method facilitates a more effective extraction of the discriminative fea- tures, as demonstrated with the comprehensive experiments on various real-life time series classification datasets.
stecstrrct
INTRODUCTION
Time-varying signal analysis plays a crucial role in various applications. For example, smartwatches utilize inertial signals for human activity logging (Khan al. 2019); brain-computer interfaces employ electroencephalographh (EEG) signals to identify user intentions (Zhang et al. 2020 Aut thasan et al. 2021); clinical diagnosis systems
surface clectromyography signals (sEMG) for
neuromuscula ar pathological analysis (Duan et al. 2015), and develop muscle-computer interfaces to control external devices (Pancholi et al. 2021).
The signals mentioned above are all typical time series data (i.e., a set of observations collected and ordered chronologically) and express information specific frequency ranges. Generally speaking, time series signal analysis consists of three steps: (i) data segmentation, wherein the continuous signals are partitioned into segments using fixed- or variable-sized windows; (ii) feature extraction, wherein various techniques are applied on each segment to extract distinguishing features; and (iii) downstream tasks, which generate the desired outputs for certain tasks (e.g., classification and anomaly detection) with the extracted features.
Among the three steps, feature extraction is usually the most critical one. Traditional feature extraction approaches for time series signals can be broadly categorized as statistical and structural methods (Lara et al. 2012). The former utilizes statistical measurements time- or frequency- domain to figure out discriminative features, where exemplar time-domain measurements are mean and variance (Kao et al. 2009 Vepakomma et al. 2015), and typical frequency-domat measurements include short-time Fourier transform (STFT) and discrete Wavelet transform (DWT) (Jiang & Yin 2013), Duan et al.) 2015)). On the other hand, structural methods aim at describing the morphological interrelationship among the data via polynomial and/or exponential analysis (Osszeski 2001)

---

Published as a conference paper at ICLR 2022
While the above hand-crafted features are efficient for some simple signal analysis tasks, their limited representation capability makes them incapable of dealing with complicated signals contaminated by
noise artifacts, and hence less competitive in real-world tasks (Chen et al.. [021). Recently, deep
neural networks (DNNs) have become the mainstream approaches for feature extraction in time series
signal analysis. Various DNN models, including CNN-based solutions (Xi al.. 2018 Lawhern
iet al. 2018 Lin et al. 2020; Amin et al. 2019), CNN-LSTM combined models (Wang ct al. 2020
Xu al. 2019 Yuki et al. 2018), and Transformer-basse techniques (Song et al. [021] Li et al.
2021). are proposed in the literature, achieving promising performance in many tasks.
Despite of the remarkable success of these deep models, most of them place heavy demands on a large
amount of labeled data. However, in many realdworld upplications, it is difficult and/or expensive to acquire sufficient labeled data to train these models. In addition, noise and artifacts that commonly exist in the time series signals make it even harder to obtain discriminative and robust representations. To the end, it may not fully reveal the superiority of deep models when solely relying on the network to directly extract information from raw signals. To solve this problem, some works attempt to use prior knowledge or hand-crafted features to guide the training of the deep learning models, aiming at extracting more effective features with limited amount of training data (Ito et al. 2018; Wei et al., 2019; Laput & Harrison| 2019). However, these methods either simply leverage prior knowledge as gates select features intuitively add some statistical measurements regularization terms to the losses, which fail to bring significant improvement. Moreover, designing tailored feature sets for different kinds of signals is tedious and time-consuming, which severely limits the generalization capability of such methods. In this regard, a promising way to acquire more discriminative features is to explore approaches deeply integrating the inherent properties of the signals into the training process of the deep learning models.
An inherent property termed as the dominant frequency range has been evidenced in many time series signals, which is a small subset of the frequency components that carries the primary information of the signal (Telgaskky] 2013). For instance, more than 95% of human body motion energy exists in the frequency components below 15 Hz. (Karantonis et al. (006); the informative frequency ranges in brain signals (EEG) have been discovered and named as (0.1-3 Hz) 0 (4-7 Hz), a (8-13H2), 8 (14-30 Hz) and y (31-100Hz) (CR & MP 2011). However, most existing deep learning models are unaware of this essential property, and do not fully consider the different roles of various frequency components, leading to sub-optimal solutions. Motivated by the above, we propose a novel tree-structured wavelet neural network, namely T- T-WaveNer, to extract more effective features from time-series signals by seamlessly and effectively integrating this property into a deep model. Different from previous models, the proposed T- T-WaveNet adaptively represents the dominant energy range of the input signal with more discriminative features, which can be naturally and easily generalized and applied to different kinds of time series signals. The main contributions of this work are threefold:
We perform frequency spectrum energy analysis for signal decomposition in 7- N-WaveNett wherein the frequency ranges with more energies are divided into finer-grained subbands and are thus represented with more dimensions compared with other low-energy frequency ranges in the feature vector, which facilitates effective learning of informative and discriminative features with a limited number of potentially noisy training samples.
To extract the features effectively, we propose a novel invertible neural network (INN) based wavelet transform as the tree node in the WeWereNer. Compared to fixed-baste wavelet Haar or lifting scheme-based wavelet, INN-based wavelet provides better representation capacity owing its entirely data-driven characteristic. To the best of our knowledge, this is the first attempt to model wavelet transform using INN
Finally, inspired by the self-attention mechanism in Transformer, we propose an instrumental feature fusion module which considers the feature dependencies across different frequency components, thus effectively enhances the robustness of the model by mitigating the impact of the heterogeneity exists in sensor signals recorded from different subjects
Extensive experiments on four popular sensor signal datasets, namely UCI-HAR for activity recog- nition, OPPORTUNITY for gesture recognition, BCICIV2a for intention recognition, and NinaPro DB1 for muscular movement recognition, show that our T-WaveNet consistently outperforms otate-of- the-art solutions.
2

---

Published as a conference paper at ICLR 2022
2
RELATED WORK
In this section, review related work on feature extractions for time series analysis and wavelet transform modeling using deep learning techniques.
2.1 FEATURE EXTRACTION FOR SIGNAL ANALYSIS
Existing feature extraction methods for time-varying signal analysis can be broadly classified into hand-crafted and deep learning-based methods, and the former can be further divided into statistical and structural approaches.
Hand-crafted statistical and structural approaches are widely used in early studies. For instance, Kao et al. (2009) applies statistical features such as the mean and the mean absolute deviation (MAD) for online activity detection from portable device. Duan et al. (2015) performs discrete wavelet transform on surface- Beaeicc--eeetttooogrrappoo (sEMG) signal for hand motion classification. While
relatively easy to calculate, these hand-crafted features are not effective for complicated tasks.
In recent years, DNNs have become the mainstream approaches for signal feature extraction. CNN- based models are widely used to extract local temporal correlations of time series data. For exam- ple, al. (2017) combines multiple CNN layers with different kernel sizes to obtain the temporal dependencies at various time scales. Lawhern et al. (2018). Lin et al.] (2020), and AAmin et al. (2019) utilize CNNs in BCI applications to establish end-to-end EEG decoding models and achieve promising performance. As CNN-based models are often insufficient for extracting long-term temporal features, (uki)) et al. (2018). and Chambers Yoder (2000) propose to combine CNNs with LSTMs extract both short- and long-term temporal features. Xu et al. (2019) bring out the Inception CNN structures to extract local temporal features at various time scales and utilize gated recurrent units (GRUs) to obtain global temporal representations. Recently, Transformer-basse methods utilize the self-attention mechanism to model the global temporal dependencies and shows superior performance various tasks. For example, (ong et al. (202 construct simple yet effective Transformer- -baseddde--aed model to obtain discriminative representations for EEG signal classification. Li al. (202)) design a two-stream convolution convolution-uugmente transformer to extract features for human activity recognition, which considers both time-over- over-channel and channel-over caennel-over-tinee time dependencies.
Ordóñez & Roggen (2016), Yuki
The above deep learning based solutions try to extract features directly from the raw signal, which ignore the unique characteristics of each type of signals (e.g., the frequency spectrum information) and usually become less effective when training on scarce signal data. Notably, some works try to alleviate this issue by guiding the DNNs with the traditional time-frequency features such that more effective deep representations can be learned. For instance, Ito et al. (0188) feed the spectrogram images constructed from the temporal features of the inertial signals into CNN models learn features
inter-modality features. Laput & Harrison (019) train a CNN on the time- fwweuueccc-sseeeeeeeeeeeeeeeeeee of the input sensor signal to obtain fine -naiine hand activity sensing system. Furthermore, (S et al. (2019) integrates the Short-Time Fourier Transform into the DNNs to directly learn frequency-doma features. However, the above solutions often need to design tailored features for different kinds of signals, which is tedious and time-consuming, and severely limits the application scenarios.
2.2 WAVELET TRANSFORM MODELING
Frequency-domait feature extraction methods such as Fourier or Wavelet transform are more prefer- able to dome-domain ones, since it is usually easier to extract discriminative features in the frequency domain than directly from the raw input signal. Compared with Fourier transform, which decomposes signal into fixed frequency components, Wavelet transform is shown to have excellent properties for transient signal analysis, thanks to its capability to analyze signals at different frequencies with various time resolutions. In this section, focus Wavelet transform.
Recently, integrating wavelet transform with deep learning techniques is shown to be effective in signal and image processing. Early attempts directly replace certain layers in the DNNs with traditional wavelet transform to reduce the number of parameters as well as improve the interpretabilityy. For instance, Williams & Li (2018) replace the max-pooling layer with wavelet pooling algorithm to address the overfitting problem. Fujieda et al. (2017) propose to replace the pooling and convolution layers with wavelet transform.
3

---

Published as a conference paper at ICLR 2022
Imput X
Anrphicke
Nott-llxdethee
(c) Frequency Operator
Proguncy
Subtands Energy Diaribation
Vrator: modil
Energy H/ Ampijdf
(a) Frequency Spectrum Energy Analysis
(b) T-WaveNet
(d) F Feature Fusion Medule
Figure 1: Given an input signal, we first perform (a) Frequency Spectrum Energy Analysis to decompose signal (fs) into multiple frequency subbands with comparable energy, where each subband contains at most one formant. Then, according to the above decomposition, we con- struct (b) T- T-WaveNet, tree-structured network, where each node is (c) Frequency Bisection Operator built with an INN-based wavelet transform. The operator outputs the high- (d) and low- frequency (c) components of the input if its binary gate value is "I", and otherwise bypass the input (ø, 15, P. and y have the same structure: Cony 1 l>>eLeeayRellue = 0.01) (roo)=Dropoutrrale 50%) ID(3, 1) 1)-Tanhh.). Considering the personalized heterogeneity of the input, we utilize (d) Feature Fusion Module to fuse the subband features } according to the feature dependen- cies across various frequency components. Finally, the enhanced feature vectors are supervised by Cross-Entropy loss for classification. See Section for more details.
The above methods typically utilize fixed wavelet basis (such as Haar (Haar 1910). which tend to be sub-optimal as it is less flexible and expressive for complex data. Recently, Rodriguez et al. (2020) propose to replace the fixed wavelet basis the lifting scheme with deep learnable modules to realize an adaptive wavelet transform unit for image classification, aiming at learning more discriminative frequency features from images.
In this work, to better handle the complex spatial-temporal time series data, we build novel deep wavelet transform unit with powerful representation capability, named frequency bisection operator. It decomposes the input signal into frequency subbands of various sizes and models the wavelet basis with an INN (Dinh et al., 2014). To the best of our knowledge, this is the first attempt to model the wavelet transform using INN. We detail the proposed solution in Section 3.2
METHODS
Instead of solely relying on the neural network for information extraction from raw signals, T- T-Waeeeer takes into consideration the uneven information distribution among different frequency ranges of signals and constructs tree-structured network for feature learning, where those frequency ranges with more energy are divided into finer-grained subbands and represented with more dimensions in the feature vector, leading to more powerful representations embedding more desired information extracted from time series signals.
An overview of the N-WaveNel is shown in Fig. | We first perform frequency spectrum energy analysis (Fig. T (a) obtain the set of signal frequency subbands with various formants and energy envelops. Then, we construct 7- WaveNet, a tree-structured network (Fig. (b)), which iteratively decomposes the input signal into frequency subbands of various sizes according to frequency spectrum energy analysis. Each node on the tree is a frequency bisection operator built with an INN-based wavelet transform unit (Fig. ] (c)). The operator is conditioned on binary gate. outputs the high-frequency and low-frequency components of the input if the gate value is "I", and otherwise bypass the input. In addition, different subjects may have some unique characteristics, resulting in distribution divergence. To deal with such individualized heterogeneity, after obtaining the set of features from each frequency subband, we utilize an effective feature fusion module to enhance the task-oriented subband features (Fig. (d)).
4

---

Published as a conference paper at ICLR 2022
3.1 FREQUENCY SPECTRUM ENERGY ANALYSIS
To quantitatively evaluate the information within the frequency range and perform the subband splitting accordingly for constructing the T- T-aveeeer, we perform frequency spectrum energy analysis (FSEA) obtain a set of frequency subbands with roughly equivalent energy. In this way, the informative frequency range would be divided into finer subbands and be represented with more feature dimensions, casing the representation learning of signal data.
FSEA consists of two phases: a foormatts- guiiaacce frequency band splitting phase to obtain the initial subbands set, and an energy-guidance advanced subband splitting to further balance the energy of each subband. In the first phase, given the input signal X E <R=,, we first calculate the spectrum with the Fourier transform on the entire signal (Fig. (a)) so as to obtain the overall frequency distribution of the signal. Next, we obtain the set of formants P = {fp}p. where fp is a local maximum (i.e., formant) of the envelope of spectrum"] Such formants represent the most direct source of the signal information. Then, we recursively bisect the frequency band until there is most one formant that falls in each frequency subband [f:. fyl. where fü and fj are the starting and ending frequency. respectively. All these subbands are collected in set Q. In the second phase, we calculate the energy of each subband in Q with Eq. (1) (Boashash, 2003) and further bisect the subband whose energy exceeds ( times of Emin- where Emin is the the minimum subband energy for subbands in Q, and Ç is scaling factor. This subband splitting phase further ensures that each resulted subband in the new set Q' contains similar amount of energy. Here, (mp(f) is the amplitude of the frequency f.
Energy - f/² Amp (f) (f)dg-
(1)
The FSEA, in principle, can be regarded perquisite for T- T-WaveNet construction. Based on Q' obtained from FSEA, we construct the proposed T WaWveeerr in two steps: (i) bottom-up up marking, and (ii) pruming. In the bottom-up marking process, given a full binary tree with some height (Fig. (b)), we locate each subband in Q' and set its binary gate as "0", which serves as the leaves. Next, we set the binary gates of all other nodes the path from the leaves to the root "1". In the pruning process, the nodes without gate settings are removed from the tree. The leaves of resulted sub-tree cover all the frequency subbands in Q'. With the above, FSEA divides the frequency ranges with more energy into finer-grained subbands.
3.2 FREQUENCY BISECTION OPERATOR
Each node in the 7- W-WaveNet is a frequency bisection operator built with an INN-based second- generation wavelet transform. The second-generation wavelets, also known Lifting Scheme theory (Sweldens) 1998), is a simple yet powerful approach to construct different wavelets such as Haar. The main idea is utilize the strong correlation among the neighboring samples in the signal to separate the low-frequency (approximation) and figh-frequency (details) subbands. The Lifting Scheme separates the input vector X (=10), [[]],.... r22k 1]), E N, into low- and high-frey subbands in three steps.
The signal is split into two non-overlapping partitions as shown in (2). The general partition method is dividing the signal to even part Xeven = xx]]],x[2].... r22 - 2)) and odd part Xodd = (x[1), 22] .... x [2k-- 1]). The spliting operater is:
Splitting.
Predictor. The two partition sets are distributed alternatively in the original signal. Based on the signal correlation, it is possible build a good predictor P to predict one partition from the other. One example to predict Xodd from X Xeven is shown in Eq. (3), where d (details) denotes the difference between X and the prediction set P(Xeven).
Updator. As shown in Eq. (4), the details d is further used to update the even part with an updator U to preserve some consistent characteristics of the original signal, such mean and higher moments. Here, e is also called "approximation".
(Xeven: = Splitting(X)..
(2) Xould P(Xeven). (3) c= Xeven + U(d). (4)
In traditional wavelet transform, the choice of the wavelet basis is important yet difficult, which would affect the analysis results significantly. Traditional wavelet construction framework with fixed
In case that the signal is too noisy, one can first conduct standard denoising approaches in the corresponding domains to minimize the impact of noise.
5

---

Published as a conference paper at ICLR 2022
coefficients () usually lacks of adaptability to extract more informative features from complex signals. signals.Therefoon Therefore, we take advantage of the INN (Dinh et al. 2014). a bijective transformation, to build the wavelet transform unit, which can effectively model the correlations between the inputs and the outputs with learnable structures. To combine the Lifting Scheme with INN, the improvements include: (i) adapting Eq. (3); (4) to an affine function, as shown in Eq. (5), (6), which improves
the transformation ability; (ii) learning the wavelet coefficients (c, 1. and 11) in Eq. (5), (6) using separate CNN modules.
d Xodd O exp( - p(Xeven), (5);
c= even 2 (x)( + (dd).
(6)
Here, cxp()(-)), exp(W(-)) stand for scale and p. 7 stand for translation. O is the element wise production. The cxp(-) is to introduce non-linearity into the function and is omitted in Fig. 1 (c). Note that, if the values and v are set to zero, Eq. (5), (6) will degenerate to the standard Litting Scheme equations (Eq. (3), (4)).
The proposed INN-based wavelet transform has two advantages. First, being an invertible architecture, INN avoids information loss during feature extraction and, hence, inherits the conventional wavelet transform property. Second, the wavelet basis is entirely data-driven, and learning high-dimensional coefficients can be regarded implementing various wavelet basis to the input simultancously according to the requirements of the downstream task, which is more effective than conventional fixed-basis wavelets.
3.3 FEATURE FUSION MODULE
There inevitably exists personalized heterogeneity in the signal data collected on different subjects. Thus, the features extracted from a certain frequency subband may contribute differently different subjects. To alleviate the effect of personalized heterogeneity and achieve better task performance, we build an efficient feature fusion module inspired by the well- Transformer (Vaswani iet 2017). The feature fusion module is built on the self-attention mechanism, which learns the personalized feature dependencies across different frequency subbands, and adaptively enhances the task-related subband features with higher weights during fusion, as illustrated in Fig. (d).
To calculate the weights of the features h E R=xm extracted from the frequency subband set Q', we first calculate the query q. keys k, and values v from subband features h:
q W,, b,, q E Rrxm; k Wgh bg, k E R v W.h + b., E RR
(7)
where W, E grxx and b. I' are the weight matrix and bias, respectively, features, n is the feature dimension of h, and r is the feature dimension of q. k and v. Then, we compute the attention weights for each value the compatibility score of the query with (q) Then the output of the feature fusion module calculated as the weighted sum of the values in (9)
is the number of
softmax (k" 4) pmxm
(8)
H = a TT H E R
(9)
The flattened attentive frequency subband representation Haaten E R8 (mrr is finally fed into a standard softmax classifier to produce the prediction results:
softmax aWHamme + b). D RC
(10)
where W E (x(r) and b E C are the weight matrix and bias, respectively, P. is the i-th element of p. which denotes the predicted probability for class = 1,.... C, and C. is the number of class.
3.4 Loss FUNCTION
Our loss function is shown in Eq. (11). The first term is the cross-entropy loss for classification, where C' denotes the number of classes, and y, is the binary ground- Besides this term, we also explore incorporating
additional term to regularize the wavelet decomposition during training. The second loss term ensures that, for each frequency bisection operator, the mean value of the decomposition output ej is close that of the input Tij- Here, M is the total number of operators in the tree-structured network, and À tunes the strength of the regularization.
C
M
L=
Σ yi log(p,)
Σ 115 c2ll2-
(11)

---

Published as a conference paper at ICLR 2022
Table
The four datasets used in our experiments.
Dalasets
classes
samples in timing window
Signal types
signals
Sampling rate(lla)
subjects
Data partiioning
Tram: Subjeci I.A. drili
Subject 2:A17L(1.2), drill;
OPPORTUNITY
Ineetial data
77
30
Subject 3:ADLI dril;
Subject 2:A12 Subjezt 3.DL)44
UCEHAR
Traiyy lest
BCKIVV2
400
BEG
22
250
Leave subyect-out
Training: Subjcets (1.7..
NinaPr DRI
52
150
surface EMG
10
100
Test: Subjects (2,5,10)
EXPERIMENTS
In this section, evaluate the performance of T- T-WaveNer four datasets, namely OPPORTUNITY (OPPOR) (Ordóñez & Roggen 2016), UCI-HAR (Davide et al. 2013), BCICIV2a (Brunner et al. 2008), and NinaPro DB1 (Manfredo al. 2012). A brief description of these datasets is listed in Table | More details on experimental settings, additional experimental results and discussions (c.g.. performance on other datasets, hyperparameter analysis, robustness, and computational time) are presented in the supplementary materials.
4.1 COMPARISON WITH STATE-OP-THE-ART METHODS
We compare WaveNett with otate-of-the-ar methods on various signal classification tasks and datasets, and the experimental results are shown in Tables 5
As can observed, the performances of previous state-of-the-an methods vary with different types of signals, while the proposed 7- WaveNet consistently achieves better performance across all these datasets. We attribute inherent dominant energy range property of signals into our deep model. In the following, we present detailed analysis of comparison on benchmarks of each task.
the fact that the proposed method seamlessly and effectively integrates the
Table 2: Performance comparison on UCI-
Table 3: Performance comparison on OPPOR dataset.
HAR dataset.
UCI-HAR
Methods
OPPOR
Accuracy
Fw
Methods
P'm
Fw
DeepConv LSTM
0.908
0,905
DeepConyLSTM
0.672
0.915
Res-LSTM
0.916
0,915
LSTM-S
0.698
0.912
CNN-LSTM
0.921
LSTM Ensembles
0.726
InnoHAR
0.945
ARN
0.903
STFNet
0.929
0,923
Res-LSTM
0.905
Harmonic
0.929
0.929
Harmonic
0.575
0.894
LSTM-CNN
0.958
FilterNtt
0.743
0.928
MI-CNN-GRU
0.962
0.961
7- T-Waeeee
0.763 0.011
0.931 1: 0.013
T-WaveNet
0.971 1: 0.012
0.974 1 0.009
Human activity recognition (UCI-HAR and OPPOR)
As can be observed from Table 2 7- WaveNet outperforms both the recent end-to-end deep mod- els (Harmonic (Hu et al. 2020) LTMM-NNN (Xia et al.) (020), MI-CNN-GRU (Dua et al. 2021) and hybrid model (STFNet (S et al. 2019)) on the UCI-HAR dataset. To be specific, compared with MI-CN-GRR, T-WaveNer yields 181 and 1.35% relative improvements in terms of Accuracy and Fw score, respectively. This is because, MI-CNN-GRU solely relies on combining the CNN and GRU model for feature extraction, without considering the essential information in the frequency domain. In contrast, STFNer integrates the STFT (Short-Time Fourier Transform) into the neural network to learn features directly in the frequency domain. However, it does not distinguish the roles of various frequency components. T- WaWeeeer decomposes the most informative frequency range into finer-grained subbands, which increases the representation capacity for feature learning under limited training samples. Similarly, for the OPPOR dataset in Table 3 compared with the popular CNN-LSTM structure used in FilterNet (Chambers & Yoder 2020 T- WaveNer clearly demonstrates better feature learning capability with a 2.66% improvement Fm score. As for, and 2.66% im- provement of F., for OPPOR. Such improvements clearly demonstrate the feature learning ability
7

---

Published as a conference paper at ICLR 2022
Moreover, 7- Waweeeeer can better capture the inter-class differences than existing solutions, because it learns the feature dependencies across different frequency subbands.
of the proposed method.
Brain intention recognition (BCICIV2a)
It challenging to learn effective features for brain intention recognition since EEG signals are usually quite noisy. As shown in Table 7- WaveNet achieves superior performance on BCICIV2a compared with standard CNN classifiers (e.g., CTCNN (Schirreiister et al., 2017) and EEGNet (Lawhern et al.] 2018)). The EEG-Image (Bashivan et a 2018) model tries to apply all sorts of hand- crafted features (i.e., spectral, spatial, and temporal features) into the CNN-RNN architecture, but it actually mislead the neural network in feature learning, resulting in poor performance. The graph neural tetwork--asee classifier NG-CRAM (Zhang et al.) 2020) is effective in modeling the spatial correlations among signals, but it also suffers from noises during message passing among neighbouring nodes. Finally, MIN2Ner (Autthasan et al. 202T utilizes auto-encoder to extract features from the input signals, but it may lose some essential information during this procedure. Also, the implementation of the Conv2D may restrict the model's ability in learning long-term temporal dependencies. In contrast, T-WaveNet concentrates on the features extracted from the dominant energy range with more discriminative power, and the proposed INN-based wavelet can avoid the information loss during feature extraction, thereby yielding more robust representations.
Table 4: Performance comparison on BCI-
Table 5: Performance comparison NinaPro dataset.
CIV2a dataset.
Methods
Accurecy
BCICIV2a
Methods
Accuracy
NinaPro
CTCNN
0,4767 + 0.1506
GengNet
0.778
EEGNet
0.5130 + 0,0518
MV-CNN
0.874
EEG Image
0.3270 + 0.0430
TCN
0.898
Cascade model
0.3183 + 0.0399
HuNet
0.870
Parallel model
0.3267 1 0.4499
WeiNet
0.850
NG-CRAM
0.601I 1 0.0996
XocptionTime
0.918
MIN2Net
0.6033 0.0924
DLPR
0.911
T-WaveNer
0.6301 0.0212
WoWeeerr
0.932 1 0.0103
Muscular movement recognition (NinaPro DB1)
As shown in Table 5 T- WaveNer achieves a relative 6.65% improvement over WeiNer (Wei et al. 2019) that trains CNN classifier hand-crafted features, and 1.52% improvement over Xcep- tionTime (Rahimian et al. 2020) that solely relies on time domain features. DIPRR (Pancholi et al., 202I) combines both the time- (peaks and zero-crossing) and frequency- (power spectral) domain features with CNNs and achieves remarkable performance for this task. However, this solution requires lots of domain knowledge and much efforts to design signal-specific features, which is challenging to be generalized to other signal types. In contrast, the proposed INN-based wavelet transform in T- WaveNet is adaptive for learning information from both time- and frequency-domain which facilitates to extract more discriminative features for muscular movement recognition.
Fm = 0.766
Fm = 0.749
Fm 0.735
(b)
(c)
Figure 2: Effectiveness of FSEA for tree structure configuration. The histograms below denote the energy preserved in the frequency subband of each leaf. (a) is the configuration obtained with FSEA (Section (..]). wherein the energy contained in each subband is similar. (b) and (c) are other two exemplar configurations with uneven energy distribution due to the coarser split of the dominant frequency range, all of which suffer from performance degradation (decreased Fm.). See Section 4.2
8

---

Published as a conference paper at ICLR 2022
4.2 ABLATION STUDY
Here, we study the impact of each component in T-WaveNer by performing a detailed ablation study on the challenging OPPOR dataset and the corresponding frequency spectrum is shown in Fig (a).
Impact of frequency spectrum energy analysis. To verify the effectiveness of FSEA, we train two other variants of the T-WaveNer with different feature vector distributions (Fig. 2 Fig. 2 (a) shows that the tree structure constructed with the dominant energy range concept outperforms other variants, indicating that dominant frequency bands represented with more dimensions in feature vectors indeed contain more discriminative information for classification. The impact of the two-phases subband splitting scheme is shown in the supplementary.
Impact of frequency bisection operator. To verify the effectiveness of our INN-based wavelet transform in the frequency bisection operator, we also experiment with two variants: (i) T-WaveNer- Haar, which replaces the INN-based wavelet transform with the traditional Haar wavelet basis; (ii) T-WaveNet-LS, which implements the deep version of the Lifting Scheme, in which the P and U in (3)(3)(4) (4) are realized using the same deep modules as o, 10 +p and y (Fig (c)). The results in Table show that our T- W-weveeet achieves 18.6% and 3.8% improvements on Fn, score in OPPOR dataset over T-WaveNet-Haur and T- T-WaveNel-LS, respectively. We attribute the improvements to the fact that our high-dimensional deep wavelet coefficients learned from data. Thus, INN-based wavelet is more adaptable to various signals compared with the fixed Haar wavelets, and has higher representation capacity than the deep Lifting Scheme. See the supplementary for more details.
Feature fusion module. To demonstrate the effectiveness of the proposed adaptive feature fusion module for handling the personalized heterogeneity of the signal data, we remove the fusion module from T-WaveNel and fuse the leaf features [h:] with equivalent weights (T- WaveNett-nouusion
Table 6: Results with structural variants. "- "Harr means the frequency bisection operator is replaced by the traditional Haar wavelet. " "-I.5" denotes the deep version of the Lifting Scheme. "-noFusion Fusion" means the feature fusion module is removed from T- WaveNet.
As shown in Table the performance of the resulted model decreases by 2%-3%, which indicates that the proposed feature fusion module learns the feature de- pendencies across various frequency compo- nents and enhances task-related features for classification. More details are presented in the supplementary.
Structural variants
Fm
Fw
7- T.WavNNet-Had
0,644 1 0.012
0.908 0,016
T-WaveNet-LS
0,735 : 0.008
0.926
0.007
T- T-WaveNet-noFusee
0.747 - 0.013
0.928 4 0,006
T- T-WaveNer
0.763 - 0.011
0.931 + 0.013
4.3 DISSCUSSION AND LIMITATION
Our approach is currently more suitable for the time-series data with some specific patterns in their frequency spectrum across subjects/cases. Biological signals often have such characteristics and in this regard, we extensively evaluate the proposed network on biological signals. However, if other time-series signals have similar properties, our network can also been applied to analyze them and achieves performance gains (see the supplementary for more experimental results). On the contrary, if the time-series data does not have this characteristic, especially when the frequency energy spectrum is time-variant (e.g., some financial data), the proposed network cannot achieve remarkable performance improvements.
CONCLUSION
We propose T-WaveNet, a novel tree-structured wavelet neural network for time series signal analysis, which exploits an inherent property of various types of signals, i.e., the dominant frequency range, to more efficiently extract informative representations from input signals than most existing solutions that depend only on the raw data. We first conduct frequency spectrum energy analysis of the signals to get a set of dominant frequency subbands. We then construct a tree-structured network with INN-based wavelet transform units to iteratively decomposes the input signal into various frequency subbands with similar energies. Finally, all the leaf node features are adaptively fused to mitigate the impact of the individualized heterogeneity of the signals. Experimental results on four datasets clearly show the superiority of T- WaveNeer over other of solutions for time series classification task. This work, in general, offers insights on how to leverage both data and empirical knowledge to drive a learning model towards more powerful representations for time series signals, which is of great significance for many practical applications with limited amount of training data.

---

Published as a conference paper at ICLR 2022
REFERENCES
S.U. Amin, M. Alsulaiman, M. Ghulam, M.A. Mekhtiche, and M.S. Hossain. Deep learning for eeg motor imagery classification based on multi-layer cnns feature fusion. Future Gener. Comput. Syst., 001:542-554, 2019.
P. Autthasan, R. Chaisaen, T. Sudhawiyangkul, P. Rangpong. S. Kiatthaweephong, N. Dilokthanakul, and Wilaiprasitpor. Min2net: End-to-end multi-task learning for subject-independen motor imagery eeg classification. ArXiv, abs/2102.03814, 2021.
Bashivan, I. Rish. M. Yeasin, and N. Codella. Learning representations from eeg with deep recureen--oonolltional neural networks. CoRR, abs/1511.064455 2018.
B. Boashash. Time-frequency signal analysis and processing. 2003.
C. Brunner, R. Leeb, G. Müller-Putz, A. Schlögl, and G. Pfurtscheller. Bei competition 2008 2008-graz data set a. Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz University of Technology, 16, 2008
R. D. Chambers and N. C. Yoder. Filternet: many-to-many deep learning architecture for time series classification. Sensors, 20(9):2498, 2020.
K. Chen, D. Zhang, L. Yao, B. Guo, Z. Yu, and Y. Liu. Deep learning for censor-based human activity recognition. ACM Computing Surveys (CSUR), 54:1 40, 2021.
H. CR and P. MP. An analysis of the effect of eeg frequency bands on the classification of motor imagery signals. International Journal of Biomedical Soft Computing and Human Sciences: the official journal of the Biomedical Fuzzy Systems Association, 16(1):121-126, 2011.
A. Davide, G. Alessandro, O. Luca, P. Xavier, and R. Jorge Luis. A public domain dataset for human activity recognition using smartphones. In Esann, volume 3, pp. 3, 2013.
L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear independent components estimation. arXiv preprint 14Xiv:1410.8511 2014.
N. Dua, N. Singh, and V. B. Semwal. Multi-input enn-gru based human activity recognition using wearable sensors. Computing, 103:1461-1477 2021.
F. Duan, L. Dai, W. Chang, Z. Chen, C. Zhu, and W. Li. semg-based identification of hand motion commands using wavelet neural network combined with discrete wavelet transform. IEEE Transactions on Industrial Electronics, 63(3):1923-1934, 2015.
S. Fujieda, Takayama, and Hachisuka. Wavelet convolutional neural networks for texture classification. arXiv preprint arXiv: 170.077977 2017.
Alfred Haar. Zur theorie der orthogonalen funktionensystemme. Mathematische Annalen, 69(3): 331-371, 1910.
Y. Hu, X. Q. Zhang, L. Xu, X. He, Z. Tian, W. She, and W. Liu. Harmonic loss function for sensor-based human activity recognition based on 1stm recurrent neural networks. IEEE Access, 135617-135627, 2020.
C. Ito, X. Cao, M. Shuzo, and E. Maeda. Application of enn for human activity recognition with fft spectrogram of acceleration and gyro sensors. Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium Pervasive and Ubiquitous Computing and Wearable Computers, 2018
Wenchao Jiang and Zhaozheng Yin. Human activity recognition using wearable sensors by deep con- volutional neural networks. Proceedings of the 23rd ACM international conference on Multimedia, 2015.
T. P. Kao, C. W. Lin, and J. S. Wang. Development of a portable activity detector for daily activity recognition. In 2009 IEEE International Symposium on Industrial Electronics, pp. 115-120. IEEÉ, 2009.
10

---

Published as a conference paper at ICLR 2022
Karantonis, D. M., M. R. Narayanan, M. Mathie, N. H. Lovell, and B. G. Celler. Implementation of real-time human movement classifier using a triaxial accelerometer for ambulatory monitoring. IEEE transactions information technology in biomedicine. 10(1):156-167, 2006.
I. Khan, S. Ali, and S. Khusro. Smartphone-based lifelogging: An investigation of data volume generation strength of smartphone sensors. In SimuTools, 2019.
Gierad Laput and Chris Harrison. Sensing fine-grained hand activity with smartwatches. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019.
Lara, D., Labrador, and M. A. A survey on human activity recognition using wearable sensors. IEEE communications surveys & tutorials, 15(3): 11111122999 2012.
Lawhern J. A. J. Solon, N. R. Waytowich, Gordon, S. M., C. P. Hung, and B. J. Lance. Eegnet: compact convolutional neural network for eeg-based brain-computer interfaces. Journal of neural engineering, 15(5):056013, 2018.
S. M. Lee, S. M. Yoon, and H. Cho. Human activity recognition from accelerometer data using convolutional neural network. 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). pp. 131-134, 2017.
B. Li, W. Cui, W. Wang, L. Zhang, Z. Chen, and M. Wu. Two-stream convolution augmented transformer for human activity recognition. In AAAI, 2021.
Lin, C. H. Chuang, Y. C. Hung, C. Fang, D. Wu, and Y. K. Wang. A driving performance forecasting system based on brain dynamic state analysis using 4-d convolutional neural networks. IEEE transactions cybernetics, PP, 2020.
A. Manfredo, G. Arjan, H. Simone, H. Anne-Gabrielle Mittaz, D. Olivier, V. Patrick, C. Claudio, C. Barbara, and M. Henning. Building the ninapro database: A resource for the biorobotics commu- nity. In IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp.
2012.
Robert Thomas Olszewski. Generalized feature extraction for structural pattern recognition in time-series data. Carnegie Mellon University. 2001.
Ordófiez and D. Roggen. Deep convolutional and Istm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1):115, 2016.
S. Pancholi, A. M. Joshi, and D. Joshi. A robust and accurate deep learning based pattern recognition framework for upper limb prosthesis using semg. 2021.
S. Rahimian, E.and Zabihi, S. F. Atashzar, A. Asif, and A. Mohammadi. Xceptiontime: Independent time-window xceptiontime architecture for hand gesture classification. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1308-1308, 2020.
Rodriguez, M. X. B., A. Gruson, L. Polania, S. Fujieda, F. Pricto, K. Takayama, and T. Hachisuka. Deep adaptive wavelet network. In The IEEE Winter Conference Applications of Computer Vision, pp. 3111-3119 2020.
Yao. S, Piao. A, Jiang. W, Zhao. Y, Shao. H, Liu. S, Liu. D, Li Wang. T, Hu.S, et al. Stfnets: Learning sensing signals from the time-frequency perspective with short-time fourier neural networks. In The World Wide Web Conference, pp. 2192-2202, 2019.
R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tanger- mann, and T. Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping. 38:5391 5420, 2017.
Y. Song, X. Jia, L. Yang, and L. Xic. Transformer-basee ppttial-tempora feature learning for ceg decoding, ArXiv, abs/2106.1 1170, 2021.
Wim Sweldens. The lifting scheme: A construction of second generation wavelets. SIAM journal on mathematical analysis, 29(2):511-544 1998.
11

---

Published as a conference paper at ICLR 2022
R. Telgarsky. Dominant frequency extraction. ArXiv, abs/1306.0113, 2013.
A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, J., Jones, L.., Gomez, A. N., and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Vepakomma, De, S. K. Das, and S. Bhansali. A-wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities. 2015 JEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN), pp. 1-6, 2015.
Yanwen Wang, Jiaxing Shen, and Yuanqing Zheng. Push the limit of acoustic gesture recognition. IEEE INFOCOM 2020 IEEE Conference on Computer Communications, pp- 506-5666775 2020.
W. Wei, Q. Dai, Y. Wong, Y. Hu, M. Kankanhalli, and W. Geng. Surface-electoomyoorrahyy gesture recognition by multi-view deep learning. IEEE Transactions on Biomedical Engineering. 66:2964-2973, 2019.
Williams and R. Li. Wavelet pooling convolutional neural networks. International Conference on Learning Representutions, 2018.
R. Xi, M. Hou, M. Fu, H. Qu, and D. Liu. Deep dilated convolution multimodality time series for human activity recognition. In International Joint Conference on Neural Networks, pp. 1-8, 2018.
K. Xia, J. Huang, and H. Wang. Lstm-enn architecture for human activity recognition. IEEE Access, 8:56855-56866, 2020.
C. Xu, D. Chai, He, X. Zhang, and S. Duan. Innohar: A deep neural network for complex human activity recognition. IEEE Access, 7:9893-9902 2019.
Y. Yuki, K. Nozaki, J.and Hiroi, and N. Kaji, K.and Kawaguchi. Activity recognition using dual- couvlstm extracting local and global features for shl recognition challenge. Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium Pervasive and Ubiquitous Computing and Wearable Computers, 2018
D. Zhang, K. Chen, D. Jian Zhang, K D. Chen, and L. Yao. Motor imagery classification via temporal attention cues of graph embedded eeg signals. IEEE Journal of Biomedical and Health Informatics, 2020.
12


Reviews:
[{"review_id": "PcyN6z9nqp", "reviewer": "Reviewer_jwqd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is quite an interesting approach for the real world time series signal analyses, and seems promising in terms of accuracy. \n\nMajor point.\nOne thing missing is an experiment for the authors' argument about an advantage on a limited amount of training data. This point, the robustness and performance of the T-WaveNet, should be proofed by experiments where training data size is gradually decreasing (75%, 50%, 25%, etc.).\n\nMinor point.\nWhat is the unit of \"Timing windos size\" in the table 1?", "summary_of_the_paper": "The authors proposed T-WaveNet, a novel tree-structured wavelet neural network for time series signal analysis. It utilizes the dominant frequency range to extract informative representation from raw signals. The tree-structure network consists of invertible neural networks as a frequency bisection operator, and of a feature fusion module. T-WaveNet was tested against various other methods using 4 different datasets and showed better results consistently.", "summary_of_the_review": "T-WaveNet, a novel tree-structured wavelet neural network is proposed for real world time series signal analysis. If the authors could show robustness of the method against a limited amount of training data by experiment, this paper would be valuable for many participants of ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "P5A62I5MWx", "reviewer": "Reviewer_VTwR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a truly innovative technique, which seems to be effective and versatile enough to be applied on several tasks and different biomedical signals (EEG, EMG,…)\nThe technique has been thoroughly evaluated and the results are convincing. The sue of the ablation study demonstrates the right decisions and design choices have been taken throughout the architecture design.\nI have some minor points to raise:\nIn section 3.2, the frequency bisection operator aims at building a predictor for estimation odds samples from even ones. If I am not mistaking, this predictor is applied also for downsampled signals, but I fail to understand how hard it is to find a suitable predictor. How can the authors explain, why trying to achieve such a trivial task allows to build such an effective network.\nCould the authors discuss how the use of scattering network with the suggested fusion module could be working on the different teste tasks?\nCould the authors discuss how much computing time is required for training and deploying the suggested technique on real examples?\n", "summary_of_the_paper": "This paper introduces a novel technique for the analysis of biomedical signals. The technique is inspired from Wavelet transform and consists in analyzing the original signal in different frequency subband. The technique differs from wavelet transform by the fact that instead of using a fixe wavelet function, a dedicated invertible neural network is learnt for each frequency subband. Finally, features from each subbands are fused using a Transformer like model.\nThe proposed techniques has been thoroughly tested on separate datasets and separate tasks, and compared to various state-of-the-art techniques, and the proposed technique has consistently outperformed other techniques on these various tasks. Finally, an ablation study has been performed in order to demonstrate the benefits of the different choices of the model architecture.\n", "summary_of_the_review": "The authors suggested an innovative solution for the analysis of biomedical time-series. They have performed thorough testing of their approach on several tasks, and obtained convincing results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GhHxYLoHOL2", "reviewer": "Reviewer_jjB7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros.\n\n-The well-founded idea combines frequency-based features (hand-imposed representation) and deep learning enhancement for time-series discrimination.\n-As a result, a data-driven wavelet approach is obtained using INN schemes.\n-Overall, the paper is clear and easy to follow. Moreover, the experiments dñemonstrate convincing evidence regarding the time-series discrimination results.\n-The codes are available, and the main training details are exposed.\n\nCons. and comments\n\n-The formant-based feature extraction seems to be confusing. How do you test the formant (maximum) estimation from noisy signals?.\n-Authors claim that a fixed window is used to compute de FFT; however, how to set such a parameter for different non-stationary patterns?\n-The loss function in eq 11 depicted an L2 regularization; how to deal with outliers?\n-The bisection operation includes the estimation of even and odd signal parts. Nonetheless, It would be helpful to discuss better the relationship between the sampling frequency, the window size, and the bisection operation.\n-Concerning the BCI2a database results, the acc seems to be poor. Can you explain better why?. Maybe, confusion matrices and class-dependent time-series analysis could be discussed.", "summary_of_the_paper": "The authors introduce a deep learning approach to classify time series. In particular, a wavelet-based algorithm is carried out using invertible neural networks and a formant-based strategy. Besides, self-attention mechanisms are utilized to code short and long time-series correlations. Finally, different databases are tested, showing an interesting performance for non-stationary pattern coding. ", "summary_of_the_review": "A good paper is presented founded on a deep learning-based representation strategy that can be used as a data-driven wavelet representation holding attention mechanism. The experiments show the potential impact in different databases devoted to time-series classification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1VZeE3EOtNu", "reviewer": "Reviewer_4oWV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper gives a reasonable way to combine classic frequent-temporal analysis (i.e., Fourier transformation and wavelet) and deep learning operations (e.g., attention for feature fusion). I list the main advantages and disadvantages below:\n\nAdvantages:\n+  The combination of deep learning and classic frequent-temporal analysis is effective, e.g., to use attention based mechanism for feature fusion across different domain is a solid design.\n+ The organization of empirical study is good, the author not only reports the end-to-end performance boost over the state-of-the-art approaches, but also provides some ablation study to support the design of the wavenet modules.\n\nDisadvantages:\n+ The biggest concern I have is that, the author claims \"Such a disentangled representation learning method facilitates a more effective extraction of the discriminative features, as demonstrated by our extensive experiments on various real-life time series classification datasets.\" Along with the tune of the introduction, it seems that the author tends to argue this approach should be considered as a general approach for various time series analysis tasks. However, in the empirical study only four classification tasks over human activity or health monitoring data are considered. I think such empirical finding may not be sufficient for a claim of an effective general-purpose time series analysis framework.\n\n+ One technique section is a little confusing to me, i.e., why would the FSEA balance the energy of each subband? It seems that the motivation of such frequent domain analysis is to identify the dominating frequency and remove the noisy in other frequencies. Additionally, shouldn't the wavelet transform include the functionality of co-locating salient frequent and temporal signal simultaneously? Why would you still need a FSEA module?\n\n+ Lastly, a tiny issue would be a lack of introduction about how the training and test datasets are split in the text, perhaps one additional col in Table 1 would fix this. \n \n", "summary_of_the_paper": "This paper introduces T-Wavenet, a fused approach including both classic feature engineering and recent advance of deep learning, in an attempt to improve the state-of-the-art analysis of time series. Empirical study is mainly conducted on human activity or health monitoring data to support the design decisions. ", "summary_of_the_review": "This paper presents a solid and reasonable way to combine classic frequent-temporal analysis and the recent advance of deep learning based approaches. \nHowever, it seems a little incautious to claim the effectiveness of the proposed model as a general purpose approach for time series analysis based on the narrow scope of the benchmark datasets. I personally suggest two options for fixing this:\n\n+ To conduct a more extensive evaluation over diversified datasets, e.g. there are 128 datasets available in the latest UCR time series website (https://www.cs.ucr.edu/~eamonn/time_series_data_2018/), all of them or a representative subset of them would be a good collection of benchmarks. \n\n+ To narrow-down the scope of the approach as an effective method for health monitoring related time series analysis. In this way, some concrete analysis should be added about some unique domain-specific properties of such signals, and the author would also justify why the proposed approach would be able to learn the representation of these properties more effectively. \n\nPost rebuttal comments:\nMy concerns have been mostly resolved given the update and additional experimental results. \n \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a tree-structured wavelet deep neural network to effectively extract more discriminative and expressive feature representations in time series signals.  Based on a frequency spectrum energy analysis,  the approach decomposes input signals into multiple subbands and builds a tree structure with data-driven wavelet transforms the bases of which are learned using invertible neural networks. In the end, the scattering subband features are fused using a self-attention-like mechanism.  The effectiveness of the proposed approach is verified extensively on a variety of datasets from different domains including follow-up experiments in the rebuttal.  Overall,  the work is technically novel and provides an interesting way of extracting adaptive finer-grained features to deal with time series signals.  The authors' rebuttal is solid which has cleared most of the concerns raised by the reviewers with additional supportive experimental evidence. 

I would recommend accept.
Decision: Accept (Poster)