Title: DYNAMICAL CLUSTERING OF TIME SERIES DATA USING MULTI-DECODER RNN AUTOENCODER

Abstract: Clustering algorithms have wide applications and play an important role in data analysis fields including time series data analysis. The performance of a clustering algorithm depends on the features extracted from the data. However, in time series analysis, there has been a problem that the conventional methods based on the signal shape are unstable for phase shift, amplitude and signal length variations. In this paper, we propose a new clustering algorithm focused on the dynamical system aspect of the signal using recurrent neural network and variational Bayes method. Our experiments show that our proposed algorithm has a robustness against above variations and boost the classification performance.

1. INTRODUCTION:

The rapid progress of IoT technology has brought huge data in wide fields such as traffic, industries, medical research and so on. Most of these data are gathered continuously and accumulated as time series data, and the extraction of features from a time series have been studied intensively in recent years. The difficulty of time series analysis is the variation of the signal in time which gives rise to phase shift, compress/stretch and length variation. Many methods have been proposed to solve these problems. Dynamic Time Warping (DTW) was designed to measure the distance between warping signals (Rabiner & Juang, 1993) . This method solved the compress/stretch problem by applying a dynamic planning method. Fourier transfer or wavelet transfer can extract the features based on the frequency components of signals. The phase shift independent features are obtained by calculating the power spectrum of the transform result. In recent years, the recurrent neural network (RNN), which has recursive neural network structure, has been widely used in time series analysis (Elman, 1990; 1991) . This recursive network structure makes it possible to retain the past information of time series. Furthermore, this architecture enables us to apply this algorithm to signals with different lengths. Although the methods mentioned above are effective solutions for the compress/stretch, phase shift and signal length variation issues respectively, little has been studied about these problems comprehensively.

Let us turn our attention to feature extraction again. Unsupervised learning using a neural network architecture autoencoder (AE) has been studied as a feature extraction method (Hinton & Salakhutdinov, 2006; Vincent et al., 2008; Rifai et al., 2011) . AE using RNN structure (RNN-AE) has also been proposed (Srivastava et al., 2015) and it has been applied to real data such as driving data (Dong et al., 2017) and others. RNN-AE can be also interpreted as the discrete dynamical system: chaotic behavior and the deterrent method have been studied from this point of view (Zerroug et al., 2013; Laurent & von Brecht, 2016) .

In this paper, we propose a new clustering algorithm for feature extraction focused on the dynamical system aspect of RNN-AE. In order to achieve this, we employed a multi-decoder autoencoder with multiple decoders to describe different dynamical systems. We also applied the variational Bayes method (Attias, 1999; Ghahramani & Beal, 2001; Kaji & Watanabe, 2011) as the clustering algorithm.

This paper is composed as follows: in Section 4, we explain AE from a dynamical system view, then we define our model and from this, derive its learning algorithm. In Section 5, we describe the application of our algorithm to an actual time series to show its robustness, including running two experiments using periodic data and driving data. Finally we summarize our study and describe our future work in Section 7.

2. RELATED WORK:

A lot of excellent clustering/representation algorithms of data using AE have been studied so far (Tschannen et al., 2018) . Song et al. (2013) 

X = (X 1 , • • • , X n , • • • , X N ), X n = (x 1 n , • • • , x t n , • • • , x T n ), x t n ∈ R D , n = 1, • • • , N, t = 1, • • • , T

, where D denotes data dimension. RNN, unlike the usual feed-forward neural network, operates the same transform matrix to the hidden valuable recursively,

EQUATION

where σ(•) is an activation function and z t , h t , b ∈ R L . This recursive architecture makes it possible to handle signals with different lengths, although it is vulnerable to the vanishing gradient problem as with the deep neural network (DNN) (Elman, 1990; 1991) . Long short-term memory (LSTM) and gated recurrent unit (GRU) are widely known solutions to this problem (Gers et al., 2000; Hochreiter & Schmidhuber, 1997; Cho et al., 2014) . These methods have the extra mechanism called a gate structure to control output scaling and retaining/forgetting of the signal information. Though this mechanism works effectively in many application fields (Malhotra et al., 2015; Rana, 2016) , the architecture of network is relatively complicated. As an alternative simpler method to solve this problem, the algorithm using a unitary matrix as the transfer matrix V was proposed in recent years. Since the unitary matrix does not change the norm of the variable vector, we avoid the vanishing gradient problem. In addition, the network architecture remains unchanged from the original RNN.

In this paper, we focus on the dynamical system aspect of the original RNN. We employ the unitary matrix type RNN to take advantage of this dynamical system structure. However, to implement the above method, we need to find the transform matrix V in the space of unitary matrices U = {U (L) ∈ GL(L)|U (L) * U (L) = I}, where GL(L) is the set of complex-valued general linear matrices with size L × L and * means the adjoint matrix. Several methods to find the transform matrix from the U has been reported so far (Pascanu et al., 2013; Jing et al., 2017; Wisdom et al., 2016; Arjovsky et al., 2016; Jing et al., 2019) . Here, we adopt the method proposed by Jing et al. (2017) .

3.2. RNN AUTOENCODER AND DYNAMICAL SYSTEM:

The architecture of AE using RNN is shown in Fig. 2 . AE is composed of an encoder unit and a decoder unit. The parameters

(V enc , U enc , V dec , U dec ) are trained by minimizing ∥X -X dec ∥ 2 F = ∑ T t=1 ∥x t -x t dec ∥ 2

, where X is the input data and X dec is the decoded data. The input data is recovered from only the encoded signal h using the matrix (V dec , U dec ), therefore h is considered as the essential information of the input signal. When focusing on the transformation Figure 2 : Architecture of RNN Autoencoder of the hidden variable, this recursive operation has the same structure of a discrete dynamical system expression as described in the following equation:

h t = f (h t-1 ),

where f is given by Eq. ( 1). From this point of view, we can understand that RNN describes the universal dynamical system structure which is common to the all input signals.

4. DERIVATION OF MULTI-DECODER RNN AE ALGORITHM:

In this section, we will give the architecture of the Multi-Decoder RNN AE (MDRA) and its learning algorithm. As we discussed in the previous section, RNN can extract the dynamical system characteristics of the time series. In the case of the original RNN, the model expresses just one dynamical system, hence all input data are recovered from the encoded result h by the same recovery rule. Therefore h is usually used as the feature value of the input data. In contrast, in this paper, we focus on the transformation rule itself. For this purpose, we propose MDRA which has multiple decoders to extract various dynamical system features. The architecture of MDRA is shown in Fig. 3 

. Let us put W k dec = (V k dec , U k dec ) for k = 1, • • • , K, W enc = (V enc , U enc ), and W = (W enc , W 1 dec , • • • , W K dec ).

We will derive the learning algorithm to optimize the whole set of parameters W in the following section.

4.1. DECOMPOSITION OF FREE ENERGY:

We applied a clustering method to derive the learning algorithm of MDRA. Many clustering algorithms have been proposed: here we employ the variational Bayes (VB) method, because the VB method enabled us to adjust the number of clusters by tuning the hyperparameters of a prior distribution. We first define free energy, which is negative marginal log-likelihood, by the following equation,

EQUATION

)

where X is data tensor defined in Section 3 and W is parameter tensor of MDRA defined above.

Y = (y 1 , y 2 , • • • , y N )

is the set of latent variables each of which means an allocation for a decoder. That is, These specific distributions are given as follows:

y n = (y n1 , • • • , y nK ) T ∈ R K , where y nk = 1 if X n is allocated to the k-th decoder

p W (X, Y , α, β|H) = p W (X|Y , H, β)p(Y |α)p(α)p(β), p(α) = Γ (θ 0 K) Γ(θ 0 ) K K ∏ k=1 α θ0-1 k , p(β) = λ ν0 0 Γ(ν 0 ) β ν0-1 exp(-λ 0 β), p W (X|Y , H, β) = N ∏ n=1 p W (X n |y n , h n , β), p(Y |α) = N ∏ n=1 p(y n |α), p W (X n |y n , h n , β) = K ∏ k=1 { ( β π ) TnD 2 exp(-β∥X n -g(h n |W k dec )∥ 2 F ) }y nk , p(y n |α) = K ∏ k=1 α y nk k .

Here, θ 0 > 0, ν 0 > 0 and λ 0 > 0 are hyperparameters and g(h

n |W k dec ) = X n dec,k denotes decoder mapping of RNN from the encoded n-th data h n , H = (h 1 , • • • , h N )

and T n D is the total signal dimension of input signal X n including dimension of input data. To apply the variational Bayes algorithm, we the derive the upper bound of the free energy by applying Jensen's inequality,

F (X|W) = -log ∫ ∫ { N ∏ n=1 ∑ yn p W (X n |y n , h n , β)p(y n |α) } p(α)p(β)dαdβ = -log E q(Y )q(α)q(β) [ p W (X|Y , H, β)p(Y |α)p(α)p(β) q(Y )q(α)q(β) ] ≤ D KL (q(Y )q(α)q(β)∥p(Y , α, β|X)) + F (X|W) = D KL (q(Y )q(α)q(β)∥p(Y |α)p(α)p(β)) - N ∑ n=1 E q(yn)q(β) [log p W (X n |y n , h n , β)] ≡ F (X|W),

where D KL (•∥•) is the Kullback-Leibler divergence. The upper bound F (X|W) is called the variational free energy or (negated) evidence lower bound (ELBO). The variational free energy is minimized using the variational Bayes method under the fixed parameters W. Furthermore, it is also minimized with respect to the parameters W by applying the RNN learning algorithm to the second term of F (X|W),

- N ∑ n=1 E q(yn)q(β) [log p W (X n |y n , h n , β)] ∝ N ∑ n=1 E q(yn)q(β) [ K ∑ k=1 y nk β∥X n -g(h n |W k dec )∥ 2 F ] .

4.2. MINIMIZATION OF THE VARIATIONAL FREE ENERGY:

In this section, we derive the variational Bayes algorithm for MDRA to minimize the variational free energy. We show the outline of the derivation below (for a detailed derivation, see Appendix A). The general formula of the variational Bayes algorithm is given by

log q(Y ) = E q(α,β) [log p W (X, Y , H, α, β)] + const., log q(α, β) = E q(Y ) [log p W (X, Y , H, α, β)] + const.

. By applying the above equations to the above probabilistic models, we obtained the specific algorithm shown in Appendix A.1. Then we minimize the following term using RNN algorithm:

N ∑ n=1 K ∑ k=1 { r nk ∥X n -g(h n |W k dec )∥ 2 F } ,

where

r nk = E q(yn) [y nk ] as detailed in Appendix A.2. We denote R = (r 1 , • • • , r N ), where r n = (r n1 , • • • , r nK ) T ∈ R K .

From the above discussion, we finally obtained the following MDRA algorithm. 

N ∑ n=1 K ∑ k=1 { r nk ∥X n -g(h n |W k dec )∥ 2 F } .

Calculate R = (r nk ) by the algorithm VB part of MDRA (Algorithm 2). until the difference of variational free energy F (X|W) < Threshold , and 8 periods respectively in 64 steps. Each signal is added a phase shift (maximum one period), amplitude variation (from 50% to 100% of the maximum amplitude), additional noise (maximum 2% of maximum amplitude) and signal length variation (maximum 80% of the maximum signal length). Examples of input data are illustrated in Fig. 5 .

We compared RNN-AE to MDRA on its feature extraction performance using the above periodic signals. Fig. 6 and Fig. 7 show the results of RNN-AE and MDRA respectively. The parameter 3 in Appendix B. We used multi-dimensional scaling (MDS) as the dimension reduction method to visualize the distributions of features in Fig. 6 and Fig. 7 . Fig. 6 shows the distribution of the encoded data h n which is the initial value of the decoder unit in Fig. 2 . We found that RNN-AE can separate the input data into three regions corresponding to each frequency. However each frequency region is distributed widely, therefore some part of the region overlap each other. 7 , is completely separated into each frequency component without overlap. This result shows that the distribution of r n as the feature extraction has robustness for a phase shift, amplitude and signal length variation. We also show that MDRA can boost the classification accuracy using an actual driving data in the next section.

5.2. EXPERIMENT OF REAL DRIVING DATA:

We applied our algorithm to a driver identification problem. We use the data consisting of 3 drivers signals when driving, including speed, acceleration, braking and steering angle signals. 1 The input signal was about 10 seconds differential data (128 steps), which was cut out from the original data by a sliding window. 2 The detailed information of the input data is shown in Table 1 . We also show samples of data (difference of acceleration) in Fig. 10 and MDRA respectively, the right figure is the distribution of r n of MDRA. We can find different trends in the distributions of the latent variable h n and r n of MDRA. The distribution of r n spreads wider than that of h n . Table 2 shows the accuracy of the driver identification results using the above 

6. DISCUSSION:

We verified the feature extraction peformance of the MDRA using actual time series data. In Section 5.1, we saw that the periodic signals are completely classified by the frequency using clustering weight r n . In this experiment, the average clustering weights, the elements of are (3.31e-01, 8.31e-47, 8.31e-47, 3.46e-01, 8.31e-47, 3.19e-01, 8.31e-47) , with only three components having effective weights. This weight narrowing-down is one of the advantages of VB learning. The left of Fig. 9 shows an enlarged view of around "freq 4" in Fig. 7 (right). We found that the distribution of "freq 4" is in fact spread linearly. The right of Fig. 9 is the result Hinton, 2008) . We found that each frequency data formed several spreading clusters without overlapping.

1 N ∑ N n=1 r n ∈ R K ,

As we saw earlier, the distribution of r n has a spreading distribution compared to that of h n . We inferred that the spreading of the distribution r n was caused by extracting the diversity on the driving scenes. In addition, the identification result shows that the combination of the features given by r n and h n can improve the performance. Dong et al. (2017) , which studied a driver identify algorithm using the AE, proposed the minimization of the error integrating the reconstruction error of AE and the classification error of deep neural network. This algorithm can avoid the over-fitting by using unlabeled data whose data collection cost is smaller than labeled data. From these results, we can expect that the MDRA can contribute to not only boost identification performance but also restrain the over-fitting. In this paper, we proposed a new algorithm MDRA that can extract dynamical system features of a time series data. We conducted experiments using periodic signals and actual driving data to verify the advantages of MDRA. The results show that our algorithm not only has robustness for the phase shift, amplitude and signal length variation, but also can boost classification performance.

8. FUTURE WORK:

The phase transition phenomenon of the variational Bayes learning method, depending on the hyperparameters, has been reported in (Watanabe & Watanabe, 2006) ．The hyperparameter setting of the prior distribution has a great effect on the clustering result and classification performance. We intend to undertake a detailed study of the relation between the feature extraction performance and hyperparameter setting of the prior distributions in the future.

A DERIVATION OF ALGORITHM:

A.1 MINIMIZATION ALGORITHM OF THE VARIATIONAL FREE ENERGY Initially, we suppose that the posterior is expressed by q(Y , α, β) = q(Y )q(α, β). Then [log p(α) ]

log q(Y ) = E q(α,β) [log p W (X, Y , H, α, β)] + const. = E q(α,β) [log p W (X|Y , H, β)] + E q(α,β) [log p(Y |α)] + E q(α,β) [log p(α)] + E q(α,β) [log p(β)] + const. = E q(α) [log p(Y |α)] + E q(α)

+ E q(β) [log p W (X|Y , H, β)] + E q(β) [log p(β)] + const..

In addition,

E q(α) [log p(Y |α)] = E q(α) [ log N ∏ n=1 K ∏ k=1 α y nk k ] = N ∑ n=1 K ∑ k=1 y nk E q(α) [log α k ] , E q(β) [log p W (X|Y , H, β)] = N ∑ n=1 K ∑ k=1 y nk E q(β) [ -β∥X n -g(h n |W k dec )∥ 2 F + T n D 2 (log β -log π) ] ,

where T n D means total signal dimension. Therefore, we obtain

log q(Y ) = N ∑ n=1 K ∑ k=1 y nk log ρ nk + const..

We here put

EQUATION

) Hence q(Y ) ∝ ∏ N n=1 ∏ K k=1 ρ y nk nk , by putting r nk = ρ nk ∑ K k=1 ρ nk , we obtain q(Y ) = N ∏ n=1 K ∏ k=1 r y nk nk .

Next we calculate log q(α, β),

log q(α, β) = E q(Y ) [log p W (X, Y , H, α, β)] + const. = E q(Y ) [log p(Y |α)] + log p(α) + E q(Y ) [log p W (X|Y , H, β)] + log p(β) + const..

Above equation can be divided into the two terms including α and β respectively,

log q(α) ∝ E q(Y ) [log p(Y |α)] + log p(α) + const. = N ∑ n=1 K ∑ k=1 log α k E q(yn) [y nk ] + (θ 0 -1) K ∑ k=1 log α k + const..

Substituting E q(yn) [y nk ] = 1 • q(y nk = 1) + 0 • q(y nk = 0) = q(y nk = 1) = r nk to the above equation, we obtain

log q(α) = N ∑ n=1 K ∑ k=1 log α k r nk + (θ 0 -1) K ∑ k=1 log α k + const..

On the other hand,

log q(β) = E q(Y ) [log p W (X|Y , H, β)] + log p(β) + const. = N ∑ n=1 K ∑ k=1 [ E q(yn) [y nk ] { -β∥X n -g(h n |W k dec )∥ 2 F + T n D 2 (log β -log π) }] + (ν 0 -1) log β + λβ + const..

Similarly, by applying E q(yn) [y nk ] = r nk , we obtain log q(β)

= N ∑ n=1 K ∑ k=1 [ r nk { -β∥X n -g(h n |W k dec )∥ 2 F + T n D 2 (log β -log π) }] + (ν 0 -1) log β + λβ + const..

We finally calculate log ρ nk in Eq. ( 3).We first calculate log q(α) and log q(β),

log q(β) = N ∑ n=1 K ∑ k=1 [ -r nk { β∥X n -g(h n |W k dec )∥ 2 F + T n D 2 (log β -log π) }] + (ν 0 -1) log β -λ 0 β + const. = βf + ( ν 0 + 1 2 N ∑ n=1 T n D -1 ) log β -λβ + const., where we put f = ∑ K k=1 ∑ N n=1 -r nk ∥X n -g(h n |W k dec )∥ 2 F . In addition, putting λ = λ 0 -f , ν = ν 0 + 1 2 ∑ N n=1 T n D, q(β) = e βf β ν0+ 1 2 ∑ N n=1 TnD-1 e -λ0β • const. = e -λβ β ν-1 • const. = λν Γ(ν) β ν-1 e -λβ = Gamma(β|ν, λ).

By using the expectations of β and log β by gamma distribution

E q(β) [β] = νλ -1 , E q(β) [log β] = ψ(ν) -log λ (Appendix A.3), we obtain E q(β) [ -β∥X n -g(h n |W k dec )∥ 2 F + T n D 2 (log β -log π) ] = -∥X n -g(h n |W k dec )∥ 2 F ν λ-1 + T n D 2 (ψ(ν) -log λ) - T n D 2 log π.

Similarly, q(α) turns out to be the Dirichlet distribution with parameters ( θ1 , • • • , θK ), and

E q(α) [log α k ] = ψ( θk )-ψ ( ∑ K k=1 θk

) is calculated by the same way in the general mixture model. Therefore we finally obtain

log ρ nk = ψ( θk ) -ψ ( K ∑ k=1 θk ) -∥X n -g(h n |W k dec )∥ 2 F ν λ-1 + T n D 2 (ψ(ν) -log λ) - T n D 2 log π.

From the above results, the following variational Bayes algorithm is derived.

A.2 MINIMIZATION ALGORITHM OF THE SECOND TERM OF THE RNN TERM (THE SECOND TERM OF VARIATIONAL FREE ENERGY):

In this section, we minimize

-E q(Y )q(β) [ ∑ N n=1 log p W (X n |y n , h n , β)

] to minimize the free energy Eq. ( 2) with respect to W. More specifically, we minimize Algorithm 2 VB part of MDRA Input: X: set of input signals Output:R: allocation weight Set hyperparameters θ 0 , ν 0 , λ 0 and the number of iterations I for i ← 0 to I do VB E-step:

log ρ nk = ψ( θk ) -ψ ( K ∑ k=1 θk ) -∥X n -g(h n |W k dec )∥ 2 F ν λ-1 + T n D 2 (ψ(ν) -log λ) - T n D 2 log π r nk = ρ nk ∑ K k=1 ρ nk VB M-step: N k = N ∑ n=1 r nk , θk = θ 0 + N k λ = λ 0 + K ∑ k=1 N ∑ n=1 r nk ∥X n -g(h n |W k dec )∥ 2 F , ν = ν 0 + 1 2 N ∑ n=1 T n D end for - N ∑ n=1 E q(yn)q(β)   log    K ∏ k=1 { ( β π ) Tn D 2 e -β∥X n -g(hn|W k dec )∥ 2 F }y nk      = - N ∑ n=1 E q(yn)q(β) [ K ∑ k=1 y nk { T n D 2 (log β -log π) -β∥X n -g(h n |W k dec )∥ 2 F } ] = - N ∑ n=1 K ∑ k=1 E q(yn) [y nk ] { T n D 2 (E q(β) [log β] -log π) -E q(β) [β]∥X n -g(h n |W k dec )∥ 2 F } = E q(β) [β] N ∑ n=1 K ∑ k=1 r nk ∥X n -g(h n |W k dec )∥ 2 F - N ∑ n=1 K ∑ k=1 r nk T n D 2 (E q(β) [log β] -log π) ∝ N ∑ n=1 K ∑ k=1 r nk ∥X n -g(h n |W k dec )∥ 2 F + const.

where we used r nk = E q(yn) [y nk ]. We achieve this by applying RNN algorithm. From the above discussion including Appendix A.1, we obtain the MDRA algorithm. In addition, ∫ ∞ 0 x ν-1 e -λe x dx is the normalization constant of gamma distribution, therefore it equals to Γ(ν)/λ ν . Hence we finally obtain 

B PARAMETER SETTING:

In this section, we show the parameter setting of the experiments in Section 5. Here L is the dimension of hidden variable h, capacity, fft and cpx are parameters of EUNN (Jing et al., 2017) , K is the number of the decoders, θ 0 , ν 0 , λ 0 are hyperparameters of prior distributions.

C EXAMPLE OF DRIVING DATA:

We applied our algorithm to the driving data in the Section 5.2. We used the differential signals of speed, acceleration, braking and steering angle signals in the experiment. Fig. 10 shows examples of acceleration signals. 

:

This data was created by HQL (Research Institute of Human Engineering for Quality Life: https:// www.hql.jp/ howhql/ spirit.html).

We use only the data of which the maximum acceleration difference is more than a certain threshold.

Figures and Tables:

integrated the distance between data and centroids into an objective function to obtain a cluster structure in the encoded data space. Pineau & Lelarge (2018) proposed a generative model based on the variational autoencoder (VAE) (Kingma & Welling, 2014) with a clustering structure as a prior distribution. Wang et al. (2019) achieved a high separability clustering result by adding a regularization term for the orthogonality and balanced clusters of the encoded data. These, however, are regularization methods of the objective function, and focused on only the distribution of the encoded data. They did not give the clustering policy based on the decoder structure, namely, the reconstruction process of the data. From dynamical system point of view, one decoder of RNN-AE corresponds to a single dynamics in the space of latent representation. Hence, it is natural to equip RNN-AE with multiple decoders to implement multiple dynamics. Such an extension of RNN-AE, however, has yet to be proposed in related works to the best of our knowledge. 3 RECURRENT NEURAL NETWORK AND DYNAMICAL SYSTEM 3.1 RECURRENT NEURAL NETWORK USING UNITARY MATRIX RNN is a neural network designed for time series data. The architecture of the main unit is called cell, and mathematical expressions are shown in Fig. 1 and Eq. (1).

Figure 1: RNN Cell

Figure 3: Architecture of MDRA

Fig. 4 describes the relation of the VB and RNN steps of MDRA algorithm. Algorithm 1 MDRA Input: X: set of input signals Output: W: weight tensors, R: allocation weights, H: encoded signals Set hyperparameters θ 0 , ν 0 , λ 0 and the initial value of W randomly. repeat Calculate W that minimizes the following value by RNN algorithm:

Figure 4: MDRA algorithm

Figure 5: Examples of periodic signals

Figure 6: Visualization of features extracted by RNN-AE (periodic signals)

Figure 7: Visualization of features extracted by MDRA (periodic signals) left: h n , right: r n

Figure 8: Extraction result of driving features

Figure 9: Left: Enlarged view of "freq 4" in Fig. 7 (right) Right: t-SNE result of r n distribution

DERIVATION OF E Gamma(β|ν,λ) [log β]By putting β = e x , we obtain x = log β, dβ = e x dx,E Gamma(β|ν,λ) [log β] e x(ν-1) e -λe x e x dx = ∫ x λ ν Γ(ν) e xν-λe x dx.We here used dν e xν-λe x = xe xν-λe x ,then the above equation isE Gamma(β|ν,λ) [log β]

Gamma(β|ν,λ)  [log β] ν) -log λ = ψ(ν) -log λ.

Figure 10: Examples of driving data

Driving data specification

Table content: <table><tr><td>Training</td><td>Test</td><td colspan="4">Driver Signal length Sampling pitch Slide</td></tr><tr><td>44851</td><td>11213</td><td>3</td><td>128</td><td>0.1 sec.</td><td>4</td></tr></table>

Driving identification results

Table content: <table><tr><td>RNN-AE</td><td>MDRA</td></tr><tr><td colspan="2">49.22%(+/-0.74%) 53.80%(+/-0.46%)</td></tr><tr><td colspan="2">49.30%(+/-0.52%) 53.77%(+/-0.55%)</td></tr><tr><td colspan="2">49.38%(+/-0.71%) 54.01%(+/-0.50%)</td></tr><tr><td colspan="2">49.10%(+/-0.47%) 52.78%(+/-0.54%)</td></tr><tr><td colspan="2">49.46%(+/-0.55%) 53.51%(+/-0.63%)</td></tr><tr><td>Ave. 49.29</td><td>Ave. 53.59</td></tr><tr><td colspan="2">of the dimension reduction of features by t-Distributed Stochastic Neighbor Embedding (t-SNE)</td></tr><tr><td>(van der Maaten &amp;</td><td/></tr></table>

Parameter setting (periodic signals)

Table content: <table><tr><td>L</td><td colspan="4">EUNN cap. fft cpx K θ 0</td><td>VB ν 0</td><td>λ 0</td></tr><tr><td>RNN-AE 16 MDRA 16</td><td>4</td><td>T</td><td>F</td><td colspan="2">-7 5.0 1.0 0.01 ---</td></tr></table>

Parameter setting of Driving identification

Table content: <table><tr><td>L</td><td colspan="4">EUNN cap. fft cpx K</td><td>θ 0</td><td>VB ν 0</td><td>λ 0</td></tr><tr><td>RNN-AE 32 MDRA 32</td><td>8</td><td>T</td><td>F</td><td colspan="2">-7 0.83 100.0 0.01 ---</td></tr></table>

Peer review (iclr2020):
This paper introduces a variation of clustering algorithm for feature extraction, by employing the variational Bayes method, on time series data that’s prone to modulations in frequency, amplitude and phase. This approach includes the proposal of a new architecture, multiple decoder RNN-AE (MDRA), in order to capture the dynamical system characteristics. The proposed contribution has been validated by its performance in a classification task on a synthetically modulated time series data as well as driving behavior data. Main Argument The paper does not do a good job of demonstrating if the problem being addressed is a significant one. The authors incorrectly attribute RNN-AE to Srivastava et al 2015 to establish their motivation and that RNN-AE is indeed a problem which they are addressing by including multiple decoders. But, the architecture presented in Srivastava et al. 2015 is not an RNN-AE but an LSTM-AE. This leaves us wondering what could be the overall impact of the current proposed architecture to the literature. For example, the authors could have instead tried multiple decoders on the LSTM-AE. Also, the use of EUNN (Jing et al. 2016) to avoid vanishing gradient is rather naive, given that sophisticated training techniques for RNNs are available in the literature. In fact, GORU has been proposed by the same authors (Jing et al 2017) of EUNN (Jing et al. 2016) and has shown to be superior to EUNN. This choice of EUNN over GORU or even the use of RNN over LSTM leaves us with little about the impact of this work. The authors could have indeed focused on comparing their proposed method with the current state-of-the-art architectures which include LSTM-VAE/GMVAE and many others. The experiments, although look fine, lack the robustness/generalizability often desired of such complex architectures by trying it on more complex sequence data. Specifically, a section on experimental settings (initialization scheme, hyperparameter optimization, etc.) haven’t been touched upon. Further supporting comments: 1. The MDRA architecture was proposed based on the premise of capturing the dynamical system structure. The interpretation of RNN as describing the dynamical system structure aka h, although good, is particularly serving no specific purpose in this work as the authors themselves point out the emphasis on the transformation rule, in Section 4 Paragraph 1 Sentence 5. Equating the U’s, V’s and W’s to the dynamical system features is a very shallow reasoning and is too limited in explaining anything about the actual dynamics of the system under study. 2. In the last sentence of Section 3 the authors indicate that they adopted the method, EUNN, proposed by Jing et al (2017). However, the paper does not justify the choice. Another method, GORU was proposed by Jing et al (2019) and it seems to perform better than EUNN. Why not this one, for example, or anything else? 3. In fact, given the incredible success of LSTMs in many application areas and the gamut of innovations that have been proposed for superior training of LSTMs, the paper may need to justify the use of a more simpler and problematic RNN-AE in the face of a more successful LSTM-AE (Srivastava et al. 2015). Maybe, a comparison to a similar but LSTM-based multi-decoder architecture would demonstrate the reasoning. 4. Do the authors have a reason for not trying RNN-VAE/LSTM-VAE with multiple decoders instead of MDRA? What are the distinct benefits of MDRA, in terms of latent representations, over a VAE/GM-VAE/Normalizing flows? 5. As the discussion of the work relates to RNN, I believe this work should have a wide range of applications beyond just time series data i.e. it should generalize well to any sequence data (speech recognition, sentence classification), The current experiments although decent could have been extended to include more complex datasets and thereby demonstrate the generalizability of this architecture. Other comments: 1. Section 2 last paragraph, Pascanu et al is out of place. Arjovsky et al should have been cited in the last sentence of the previous paragraph. 2. In Section 1 Paragraph 2 Sentence 2, the authors incorrectly attribute RNN-AE to Srivastava et al when in fact they proposed LSTM-AE. 3. Section 1 Paragraph 2 may benefit from a rewrite as it is really vague in building-up the motivation behind the proposed work. 4. Some use of terminology is unclear 1. Example: In Section 1, Paragraph 1, Sentence 3 I doubt if phase shift, compress/stretch, and length variation are outcomes of time series variation or if they are sources of time series variability. 2. Example: Section 3, Paragraph 1, Sentence 2: The architecture of the main unit is called cell. Instead, the primary component of an RNN architecture is called a cell (hidden node in other words). 5. Typos 1. Section 4 Heading: DERIVATION not DEVIATION 2. Section 4.1 Paragraph 2. Variables not valuables 3. Section 6 Heading: DISCUSSION instead of DISUCUSSION 4. Appendix C Heading: EXAMPLE not EXAPMPLE 5. Appendix D: quiet not quite 6. Figure 11 box 2: output not outout 7. Figure 11 caption: Fully not Full 8. Section 5.2 Line 1: drivers’ not driver’s 9. Appendix A3. In the expectation of log_beta with respect to a Gamma distribution, ‘x’ should have been ‘beta’ instead.