Translation Between Waves,  wave2wave

1 INTRODUCTION . The problem of shortage of training data but can be supplied by other sensor data is called an opportunistic sensor problem ( Roggen et al. , 2013 ) . For example in human activity logs , the video data can be missing in bathrooms by ethical reasons but can be supplied by environmental sensors which have less ethical problems . For this purpose we propose to extend the sequence-to-sequence ( seq2seq ) model ( Cho et al. , 2014 ; Sutskever et al. , 2014 ; Dzmitry Bahdanau , 2014 ; Luong et al. , 2015 ) to translate signal wave x ( continuous time-series signals ) into other signal wave y . The straight-forward extension does not apply by two reasons : ( 1 ) the lengths of x and y are radically different , and ( 2 ) both x and y are high dimensionals . First , while most of the conventional seq2seq models handle the input and output signals whose lengths are in the same order , we need to handle the output signals whose length are sometimes considerably different than the input signals . For example , the sampling rate of ground motion sensor is 100Hz and the duration of an earthquake is about 10sec . That is , the length of the output signal wave is 10000 times longer in this case . Therefore , the segmentation along temporal axis and discarding uninformative signal waves are required . Second , signal waves could be high dimensionals ; motion capture data has 129 dimensions and acceleormeter data has 18 dimensions . While most of the conventional seq2seq does not require the high-dimensional settings , meaning that it is not usual to translate multiple languages simultaneously , we need to translate signal waves in high dimensions into other signal waves in high dimensions simultaneously . To overcome these two problems we propose 1 ) the window-based representation function and 2 ) the wave2wave iterative back-translation model in this paper . Our contributions are the following : • We propose a sliding window-based seq2seq model wave2wave ( Section 4.1 ) , • We propose the wave2wave iterative back-translation model ( Section 4.2 ) which is the key to outperform for high-dimensional data . 2 RELATED WORKS . Related works include various encoder-decoder architectures and generative adversarial networks ( GANs ) . First , the encoder-decoder architecture has several variations : ( 1 ) CNNs in both sides ( Badrinarayanan et al. , 2015 ) , ( 2 ) RNNs in both sides ( Cho et al. , 2014 ; Sutskever et al. , 2014 ; Dzmitry Bahdanau , 2014 ; Luong et al. , 2015 ) , or ( 3 ) one side is CNN and the other is RNN ( Xu et al. , 2015 ) . When one side is related to autoregressive model ( van den Oord et al. , 2016 ) , further variations are appeared . These architectures are considered to be distinctive . The pros of CNN is an efficient extraction of features and overall execution while the pros of RNN is its excellent handling of time-series or sequential data . CNN is relatively weak in handling time-series data . In this reason , the time domain is often handled by RNN . The encoder-decoder architecture using CNNs in both sides is used for semantic segmentation ( Badrinarayanan et al. , 2015 ) , image denoising ( Mao et al. , 2016 ) , and super-resolution ( Chen et al. , 2018 ) , which are often not related to time-series . In the context of time-series , GluonTS ( Alexandrov et al. , 2019 ) uses the encoder-decoder approach which aims at time-series prediction task where parameters in encoder and decoder are shared . Apart from the difference of tasks , our approach does not share the parameters in encoder and those in decoder . All the more our model assumes that the time-series multi-modal data are related to the multi-view of the same targeted object which results in multiple modalities . Second , among various GAN architectures , several GANs aims at handling time-series aspect . Vid2vid ( Wang et al. , 2018 ) is an extension of pix2pix ( Isola et al. , 2016 ) which aims at handling video signals . ForGAN ( Koochali et al. , 2019 ) aims at time-series prediction task . 3 SEQ2SEQ . Architecture with context vector Let x1 : S denotes a source sentence consisting of time-series S words , i.e. , x1 : S = ( x1 , x2 , . . . , xS ) . Meanwhile , y1 : T = ( y1 , . . . , yT ) denotes a target sentence corresponding to x1 : S . With the assumption of a Markov property , the conditional probability p ( y1 : T |x1 : S ) , translation from a source sentence to a target sentence , is decomposed into a time-step translation p ( y|x ) as in ( 1 ) : log p ( y1 : T |x1 : S ) = T∑ t=1 log p ( yt|y < t , ct ) ( 1 ) where y < s = ( y1 , y2 , . . . , ys−1 ) and cs is a context vector representing the information of source sentence x1 : S to generate an output word yt . To realize such time-step translation , the seq2seq architecture consists of ( a ) a RNN ( Reccurent Neural Network ) encoder and ( b ) a RNN decoder . The RNN encoder computes the current hidden state hencs given the previous hidden state h enc s−1 and the current input xs , as in ( 2 ) : hencs = RNNenc ( xs , h enc s−1 ) ( 2 ) where RNNenc denotes a multi-layered RNN unit . The RNN decoder computes a current hidden state hdect given the previous hidden state and then compute an output yt . hdect = RNNdec ( h dec t−1 ) ( 3 ) pθ ( yt|y < t , ct ) = softmax ( gθ ( h dec t , ct ) ) ( 4 ) where RNNdec denotes a conditional RNN unit , gθ ( · ) is the output function to convert h dec t and ct to the logit of yt , and θ denotes parameters in RNN units . With training data D = { yn1 : T , xn1 : S } Nn=1 , the parameters θ are optimized so as to minimize the loss function L ( θ ) of log-likelihood : L ( θ ) = − 1 N N∑ n=1 T∑ t=1 log pθ ( y n t |yn < t , ct ) ( 5 ) or squared error : L ( θ ) = 1 N N∑ n=1 T∑ t=1 ( ynt − gθ ( hdec n t , c n t ) ) 2 ( 6 ) Global Attention To obtain the context vector cs , we use global attention mechanism ( Luong et al. , 2015 ) . The global attention considers an alignment mapping in a global manner , between encoder hidden states hencs and a decoder hidden step h dec t . at ( s ) = align ( hdect , h enc s ) ( 7 ) = exp ( score ( hdect , h enc s ) ∑T s exp ( score ( h dec t , h enc s ) ) ( 8 ) where the score is computed by weighted inner product as follows score ( hdect , h enc s ) = h dec t > Wah enc s ( 9 ) where the weight parameter Wa is obtained so as to minimize the loss function L ( θ ) . Then , the context vector ct is obtained as a weighted average of encoder hidden states as ct = S∑ s=1 at ( s ) h enc s ( 10 ) 4 PROPOSED METHOD : WAVE2WAVE . The problems of global attention model are that ( 1 ) the lengths of input and output are radically different , and that ( 2 ) both input and output sequences are high dimensionals . For example in activity translation , there are 48 motion sensors and 3 accelerometer sensors . Their frequency rates are as high as 50Hz and 30Hz respectively . Therefore , the number of steps S , T in both encoder and decoders are prohibitively large so that the capturing information of source sentence x1 : S is precluded in the context vector c . 4.1 WINDOW-BASED REPRESENTATION . Let us consider the case that source and target sentences are multi-dimensional continuous timeseries , signal waves , as shown in Figure 1 1 . That is , each signal at time-step x1 : S is expressed as dx-dimensional vector xs—there are dx sensors in the source side . Then a source signal wave x1 : S consists of S-step dx-dimensional signal vectors , i.e. , x1 : S = ( x1 , x2 , . . . , xS ) . 1We note that signal waves in Figure 1 are depicted as one-dimensional waves for clear visualization . To capture an important shape informaion from complex signal waves ( see Figure 1 ) , we introduce trainable window-based representation function R ( · ) as rencs′ = R ( W enc s′ ) ( 11 ) where W encs′ is a s ′-th window with fixed window-width wenc , expressed as dx × wenc-matrix as W encs′ = [ xwenc ( s′−1 ) +1 , xwenc ( s′−1 ) +2 , . . . , xwenc ( s′−1 ) +wenc ] , ( 12 ) and rencs′ is extracted representation vector inputted to the seq2seq encoder as shown in Figure 1 —the dimension of renc is the same as the one of the hidden vector h enc . Similarly , to approximate the complex target waves well , we introduce inverse representation function , R−1 ( · ) which is separately trained from R−1 ( · ) as W dect′ = R −1 ( rdect′ ) ( 13 ) where rdect′ is the t ′-th output vector from seq2seq decoder as shown in Figure 1 , and W dect′ is a window matrix which is corresponding to a partial wave of target waves y1 : T = ( y1 , . . . , yT ) . The advantage of window-based architecture are three-fold : firstly , the number of steps in both encoder and decoder could be largely reduced and make the seq2seq with context vector work stably . Secondly , the complexity and variation in the shape inside windows are also largely reduced in comparison with the entire waves . Thus , important information could be extracted from source waves and the output sequence could be accurately approximated by relatively simple representation R ( · ) and inverse-representation R−1 ( · ) functions respectively . Thirdly , both representation R ( · ) and inverse-representation R−1 ( · ) functions are trained end-to-end manner by minimizing the loss L ( θ ) where both functions are modeled by fully-connected ( FC ) networks . Figure 1 depicts the overall architecture of our wave2wave with an example of toy-data . The wave2wave consists of encoder and decoder with long-short term memory ( LSTM ) nodes in their inside , representation function R ( W encs′ ) and inverse-representation function R −1 ( W dect′ ) . In this figure , one-dimensional 10000-time-step continuous time-series are considered as an input and an output and the width of window is set to 2000— there are 5 window steps for both encoder and decoder , i.e. , wenc = wdec = 2000 and S′ = T ′ = 5 . Then , 1 × 2000 encoder-window-matrix W encs′ is converted to dr dimensional encoder-representation vector rencs′ by the representation function R ( W encs′ ) . Meanwhile , the output decoder , dr dimensional decoder-representation r dec t′ , is converted to 1× 2000 decoder-window-matrix W dect′ by the inverse representation function R−1 ( rdect′ ) .

Summary: This work explains how to use a seq2seq encoder-decoder neural network on the case of multivariate time series. The authors name this particular application of seq2seq the wave2wave network. Given a multivariate time series covering a time interval, it is split into subintervals of equal length, such that each block is a matrix. This matrix becomes an input into a recurrent encoder. On the decoder side, the similar matrix is produced at the output. The proposed neural network is tested on two data sets: an earthquake and activity translation.