Translation Between Waves,  wave2wave
Authors: Tsuyoshi Okita
Hirotaka Hachiya
Sozo Inoue
Naonori Ueda
Venue:  (2020)
Keywords: sequence to sequence model
signal to signal
deep learning
RNN
encoder-decoder model

Abstract: The understanding of sensor data has been greatly improved by advanced deep learning methods with big data. However, available sensor data in the real world are still limited, which is called the opportunistic sensor problem.  This paper proposes a new variant of neural machine translation seq2seq to deal with continuous signal waves by introducing the window-based (inverse-) representation to adaptively represent partial shapes of waves and the iterative back-translation model for high-dimensional data.  Experimental results are shown for two real-life data: earthquake and activity translation.  The performance improvements of one-dimensional data was about 46 % in test loss and that of high-dimensional data was about 1625 % in perplexity with regard to the original seq2seq.


Under review as a conference paper at ICLR 2019
TRANSLATION BETWEEN WAVES, wave2wave
Anonymous authors Paper under double-bling review
ABSTRACT
The understanding of sensor data has been greatly improved by advanced deep learning methods with big data. However, available sensor data the real world are still limited, which is called the opportunistic sensor problem. This paper proposes new variant of neural machine translation seq2xeq to deal with con- tinuous signal waves by introducing the window-base based (inverse-) representation to adaptively represent partial shapes waves and the iterative back-translation model for high-dimenniooal data. Experimental results are shown for two real-life data: earthquake and activity translation. The performance improvements of one- dimensional data was about 46 % in test loss and that of sigh-dimensional data was about 1625 % in perplexity with regard to the original seq2seq.
INTRODUCTION
The problem of shortage of training data but can be supplied by other sensor data is called an opportumistic sensor problem (Roggen al., 2013). For example in human activity logs, the video data can be missing in bathrooms by ethical reasons but can be supplied by environmental sensors which have less ethical problems. For this purpose we propose to extend the set-eennee----queeee (seq2seq) model (Cho et al. 2014, [Sutskever et al. 2014, Dzmitry Bahdanau 2014 [uong et al.) 2015) to translate signal wave I (continuous time-series signals) into other signal wave y. The straight-forward extension does not apply by two reasons: (1) the lengths of x and are radically different, and (2) both x and are high dimensionals.
First, while most of the conventional seq2seq models handle the input and output signals whose lengths are in the same order, we need to handle the output signals whose length are sometimes con- siderably different than the input signals. For example, the sampling rate of ground motion sensor 100Hz and the duration of an earthquake is about 10sec. That is, the length of the output signal wave is 10000 times longer in this case. Therefore, the segmentation along temporal axis and dis- carding uninformative signal waves are required. Second, signal waves could be high dimensionals; motion capture data has 129 dimensions and acceleormeter data has 18 dimensions. While most of the conventional seq2seq does not require the nigh-dimensional settings, meaning that it is not usual translate multiple languages simultaneously, we need translate signal waves high dimensions into other signal waves in high dimensions simultaneously.
To overcome these two problems we propose 1) the window-based representation function and 2) the wave2wave iterative back-translation model in this paper. Our contributions are the following:
We propose a sliding window-based seq2seq model wave2wave (Section
We propose the wave2wave iterative back-translation model (Section 4.2 which is the key to outperform for high-dimensional data.
RELATED WORKS
Related works include various encoder-decoder architectures and generative adversarial networks (GANs). First, the encoder-decoder architecture has several variations:
(1) CNNs in both sides
(Badrinarayanan et al.] 2015). (2) RNNs in both sides (Cho et al., 2014 Sutskever et al. 2014] Dzmitry Bahdanau, 2014 Luong et al. 2015), or (3) one side is CNN and the other RNN (Xu ct al.] 2013). When one side is related to autoregressive model (van den Oord et al. 2016). further variations are appeared. These architectures considered be distinctive. The pros of CNN is an

---

Under review as a conference paper at ICLR 2019
efficient extraction of features and overall execution while the pros of RNN is its excellent handling of time-scries or sequential data. CNN is relatively weak in handling time-series data. In this reason, the time domain is often handled by RNN. The encoder-decoder architecture using CNNs in both sides is used for semantic segmentation (Badrinarayanan et al. 2015), image denoising (Mao et al.) 2016), and super-resolution (Chen al. 20018), which are often not related to time-series. In the context of time-series, GluonTS Alexandrov et al. 2019) uses the encoder-decode aims time-series prediction task where parameters in encoder and decoder are shared. Apart from the difference of tasks, our approach does not share the parameters in encoder and those in decoder. All the more our model assumes that the time-series multi-modal data are related the multi-view of the same targeted object which results in multiple modalities. Second, among various GAN architectures, several GANs aims at handling time-series aspect. Vid2vid (Wang et al. 2018) an extension of pix2pix (Isola et al. (016)) which aims at handling video signals. ForGAN (Koochali ctt al. 2019) aims at time-series prediction task.
approach which
SEQ2SEQ
Architecture with context vector Let 21.5 denotes a source sentence consisting of time-series words, i.e., Fi.S (21, (g)) Meanwhile, /1.7 (y1,.... Mr) denotes a target sen- tence corresponding 21.s. With the assumption of Markov property, the conditional probability p(y1r(z11s) translation from a source sentence to a target sentence, is decomposed into a time-step translation p(ylar) in
log pep(1r//11ss) =
log pggllueee...
where V<. (y1, ................................ .--) and c, a context vector representing the information of source sentence 21.9 to generate an output word yr-
To realize such time-step translation, the seq2seq architecture consists of (a) RNN (Reccurent Neural Network) encoder and (b) RNN decoder. The RNN encoder computes the current hidden state h hene given the previous hidden state hete and the current input V's, as in 2
hm RRNNee enc
(2)
where RNNenc denotes multi-layered RNN unit.
The RNN decoder computes current hidden state h given the previous hidden state and then compute an output Vt-
h dec RNN (h
dec
(3)
Po (v Meee C2) = softmax (oohtte,
(4)
where RNN dee denotes a conditional RNN unit, ge(-) is the output function to convert hyee and c to the logit of y/. and 0 denotes parameters in RNN units.
W the parameters 0 are optimized so as minimize the loss function C(0) of log- ligeliiklihood:
With training data D (Mi.r sT.s)
N
T
((0)
log pa (v 1u
(5)
N
n
squared error:
N
T
(())
1 N
ve
(o decn
1, (,,,))
(6)
Global Attention To obtain the context vector C., we use global attention mechanism Luong et al., 2015). The global attention considers an alignment mapping in a global manner, between encoder hidden states hene and a decoder hidden step hdee.
(s) align hdec h
exp(score(h) (he h
(8)
exp(score chee hgnc))
2

---

Under review as a conference paper at ICLR 2019
decoder-winnlow wee
output yı
output
representation (R
representationn
dec
aligment (2)
context
softmac
encoder-hideen hidden An
0.6
0.1
concal
uncoder-hidden A
Encoder
Decoder
encoder-represetttion enc
sin
R(W
decoder-hicdden af
decader-hidden
Impur
Micoo2
encoder-winddw Wi
lengthhw
Figure 1: Overall architecture of our method, wave2wave, consisting RNN encoder and decoder with context vector and sliding window representation. Input and output time-series data are toy examples where the input is generated by combining sine waves with random magnitudes and periods. The output the version of the input flipped horizontally.
where the score is computed by weighted inner product as follows
score(h hgme))
(heee
dee N.eWWh W.h.
where the weight parameter Wa is obtained context vector C₁ is obtained as a weighted average of encoder hidden states as
minimize the loss function L(O). Then, the
az(s)h.
(10)
PROPOSED METHOD: WAVE2WAVE
The problems of global attention model are that (1) the lengths of input and output are radically different, and that (2) both input and output sequences high dimensionals. For example in activity translation, there are 48 motion sensors and 3 accelerometer sensors. Their frequency rates are high 50Hz and 30Hz respectively. Therefore, the number of steps S, T in both encoder and decoders are prohibitively large so that the capturing information of source sentence #1.s is precluded in the context vector c.
4.1 WINDOW-BASED REPRESENTATION
multi-diimensonall continuous time- series, signal waves, as shown in Figure 1 That is, each signal at time-step &1.s is expressed as dy di-dimensol vector x, there are d sensors in the source side. Then source signal wave X1:5 consists of S-step dy-dimensional signal vectors, i.c., X1.5 = (x1, xg..,,))....)..
Let consider the that source and target sentences
We note that signal waves Figure depicted as one-dimensional for clear visualization.
3

---

Under review as a conference paper at ICLR 2019
To capture important shape informaion from complex signal waves (see Figure Db.
introduce
trainable window-based representation function R(-) as
" = R(
(11)
where Were is 8' -'tth window with fixed window-width w expressed as dx wene- matrix
W = [x X
x Xu 1)+
(12)
and & is extracted representation vector inputted the seq2seq encoder as shown in Figure the dimension of ronc is the same as the one of the hidden vector h
Similarly, to approximate the complex target waves well, we introduce inverse representation func- tion, R (.) which is separately trained from (-)
Widee R (r
(13)
where rdre is the f'-th output vector from seq2seq decoder as shown in Figure and Wabe is window matrix which is corresponding to partial wave of target waves yI.7 ( .T))
The advantage of window-based architecture are three-fold: firstly, the number of steps in both encoder and decoder could be largely reduced and make the seq2seq with context vector work stably. Secondly, the complexity and variation in the shape inside windows are also largely reduced in comparison with the entire waves. Thus, important information could be extracted from source waves and the output sequence could be accurately approximated by relatively simple representation R(-) and inverse-repreeentee twerse-represenntttine R (.) functions are trained end-to-end manner by minimizing the loss [(0) where both functions modeled by fully-connected (FC) networks.
R R-'(-) functions respectively. Thirdly, both representation R(-) and
Figure I depicts the overall architecture of our wave2wave with an example of toy-data. The wave2wave consists of encoder and decoder with long-short term memory (LSTM) nodes in their inside, representation function R(Wers) and inverse-represennation function R R-1 (Wddee. In this figure, one-dimensional 10000-time-step continuous time-series considered as input and an output and the width of window is set to 2000 there are 5 window steps for both encoder and de- weadder-winowwmaaee W
coder, i.e., nm dec 2000 and S" T Then, 1 x 2000 is converted to d, dimensional encoder-represennaaii vector rer by the representation function R(WerC) Meanwhile, the output decoder, de dimensional decoder--epressentaion decc is con- verted to 1 x 2000 decoder-window-maale Wdee by the inverse representation function R (rde()))).
4.2 WAVE2WAVE ITERATIVE MODEL
We consider two different ways tackle with high-dimensional sensor data. Since NMT for ma- chine translation handles embeddings of words, the straightforward extention to high-dimensional settings uses the dx-dimensional source signal at the same time step as source embeddings. and the dy di-deeniiionll target signal at the same time step target embeddings. We call this a wave2wave model. i.c. our standard model. Alternatively, we can build dy independent embeddings separately for corresponding individual 1-dimensional target signal at each time step while we use the same di-dimensoonal source signal embeddings. We call this a Waveeaaveeeerative model. We suppose that the former model would be effective when sensor data are correlated while the latter model would be effective when sensor data are independent. Algorithm shows the latter algorithm.
Algorithm 1: Wave2wavelterattiee model
Data: STCd igtd, xT', Care t- x²*,, Cigt, f- y def train Wave waaaeeeeeveeeeratieeee.... for j = (1,dy) do
dy
eigt x T):
f(j) train TrainWaeeWavv(essr x S,ergt xT); end
Note that the embedding esre is equivalent to dx de-dimensionally decomposed representation of rone, and Cigs is equivalent to dy dy-dimensionally decomposed representation of rdee. The back-translation is technique to improve the performance by bi-directional translation removing the noise under a coutral-biased translation (Hoang et al. 2018). We deploy this technique which we call the wave2wave iterative back-translation model.
4

---

Under review as a conference paper at ICLR 2019
method
train loss
test loss
simple encoder-decoder d, 100
1.13
0.53
simple encoder-decoder dN 500
0.90
0.47
simple encoder-decoder dN = 1000
0.41
0.63
simple seq2seq w Wider 500
9.27
2.87
simple seq2seq wTne = Wder 1000
9.87
2.79
simple seq2sequane Wdee
2000
6.82
2.60
wave2wave wene Wdde
500
0.67
0.44
wave2wave wone Wddee
1000
0.17
0.34
wave2wave woe Wdde 2000
0.25
0.43
Table 1: Mean squared loss of simple encoder-decoder methods, simple seq2seq methods and our
wave2wave in earthquake ground motion data
5 EVALUATION ON REAL-LIFE DATA: GROUND MOTION TRANSLATION
In this section, we apply our proposed method, wave2wave, to predict broadband-ground mo- tion from only its long-period motion, caused by the same earthquake. In this section, wave2wave translates one dimensional signal wave into one dimensional signal wave.
Ground motions of carthquakes cause fatal damages on buildings and infrastructures. Physics-basee numerical simulators are used generate ground motions specific place, given the property of earthquake, e.g., location and scale to estimate the damages on buildings and infrastructures (Iwaki Fujiwara 2013). However, the motion generated by simulators are limited only long periods, longer than second due to heavy computational costs, and the lack of detailed knowledge of the subsurface structure.
A large amount of ground motion data have been collected by K(kyosin)-NEF over the past 20 years in Japan. Machine learning approaches would effective to predict broadhand-ge ground motions including periods less than 1 second, from simulated long period motions. From this perspective, we apply our method wave2wave to this problem by setting long-ground motion as an input and broadband-gron ground motion an output, with the squared loss function L(0).
As for training data, we use 365 ground motion data collected at the observation station, IBR011, located at Ibaraki prefecture, Japan from Jan. 1, 2000 to Dec. 31, 2017-originall originally there are 374 data but 10 data related Tohoku earthquakes and the source deeper than 300m are removed. As for testing,
9 ground-motion data of earthquakes occurred at the beginning of 2018.
In addition, both long and broadband ground motion data are cropped to the fixed length, i.e., $ = t 10000ms and its amplitude is smoothed using RMS (Root Mean Square) envelope with 200ms windows to capture essential property of earthquake motion. Moreover, as for data augmentation, in- phase and quadrature components, and those absolute values are extracted from each ground motion. That is, there are totally 365 x 3 training data. Figure 2a shows an example of 3 components of ground motion of earthquake occurred on May 17, 2018. Chiba in Japan, and corresponding RMS envelopes.
Table depiets the mean equaree loss training of three methods, simple encoder-decoder, simple seq2seq, and our proposed method with the same setting as the toy data except h hec 50. This table shows that our wave2wave methods basically outperform other methods although wave2wave with the small Hindow-width eue dec 500 is lost by simple encoder-decoder with large hidden layer dz = 1000 in train loss. This indicates that window-based representation and unverse-representatio functions are helpful similarly in toy data.
Figure depicts examples of predicted broadband ground motions of earthquakes occurred on Jan. 24 and May 17, 2018. These show that method wave2wave predict enveloped broadband ground motion well given long- long-period ground motion although there is little overfitting due to small training data.
is expected that predicted broadband-motfe combined with simulated long-period motion could be used for more accurate estimation of the damages buildings and infrastructures.
5

---

Under review as a conference paper at ICLR 2019
Enveloped
30
20
20
10
15.
-20
110
5
Absolure [gal) Quadrapture [ga)
-30
30
2000
4000
600D
8000
10000
20
2000
4000
6000
8000
10000
20
10
15
10-
5
30
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
20
20000
4000
6000
8000
10000
2000
4000
6000
8000
10000
time [10ms)
time [10ms]
(a) Example of enveloped ground motion
Long-period
Broadband
Absolute [gel] Qusdrrrprerrreee (awl) In-phasee (gel)
2
ground truis
teit 500
teed 1000
2000
4000
6000
8000
10000
20000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
2,000
4000
6000
8000
10000
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
time
[10ms]
time [10eg]
(h) Predicted broadband ground motion Jan. 24, 2018
Long-period
Broadband
Absoolute [gal] Quaddrapee (nelllip (gal)
ground truth
#ay3Ewave tost x- 500
M
10
wavazeeve Leat w- 1000
5
ueve2 test w 2000
0-
2
6000
4000
10000
15
2.
10-
W
20000
4000
6000
8000
10000
20
2.000
4000
6000
10000
15
2.
10:
2000
4000
6000
8000
10000
20000
4000
6000
8000
10000
[10ms]
(c) Predicted broadband ground motion on May. 17, 2018
Figure 2: top: Example of original and enveloped ground motion data with in-phase, quadrature components and these absolute values. middle and bottom: predicted broadband ground motion by methods wave2wave for earthquakes occurred Jan. 24, 2018 and May. 17, 2018.
EVALUATION ON REAL-LIFE DATA: ACTIVITY TRANSLATION
This section deploys wave2wave for activity translation (Refer Figure B). Until the previous section, the signals one dimensions. The signals in this section high-dimensional in their inputs

---

Under review as a conference paper at ICLR 2019
well as outputs. The dimensions of motion capture, video, and accelerometer are 129, 48, and 18 dimensions, respectively, in the case of MHAD datase P
Under the mild assumption that the
Aclivity translation task
Motion capture
wavadwave
Figure 3: Figure shows activity translation task and activity recognition task which we conduct experiments.
targeted person which are recorded in three different modalities, including motion capture, video, and accelerometer, are synchronized and the noise such as the effect of other surrounding persons is eliminated. Hence, assume that each signal shows one of the multi-view projections of single person. That is, we can intuitively think that they are equivalent. Under this condition, we do a translation from motion capture to video (Similarly, accelerometer to motion capture, and video to accelerometer, and these inverse directions).
6.0.1 OVERALL ARCHITECTURE
signal. When video is input, We rec takes the form of pose vectors which are converted by OpenPose li- brary (Cao al. 2017p. Then, this representation is convereted into the window representation by R R(We)r)) When motion capture is input, Wene takes the form of motion capture vectors. In this way we used these signals for imput well output for wave2wave. The raw output are reconstructed by R (Wddrr) for the output of representation Wides.
Wave Signal Figure 3 shows that motion capture and video be considered
s'
Wave Signal Dimensionality Reduction As an alternative to use FC layer before the input, we use the clustering algorithm, specifically an affinity propagation (Frey Dueck, (200)), in order to reduce the size of representation as whole. While most clustering algorithms need supply the number of clusters beforehand, this affinity propagation algorithm solves the appropriate cluster number result.
RestiiReeooltion Spatial Pyramid Additinaly structures in wave2wave is the multi-scalabilitt since the frame rate of multimodal data are considerably different. We adopted the approach of multi-resolution spatial pyramid by dynamic pose (Neverova al., 2014). We assume that the sequence of frames across modalities is synchronized and sampled at a given temporal step v and concatenated form a spatio-tempon temporal 3-d volume.
6.0.2 EXPERIMENTAL EVALUATION
Experimental Setups We used the MHAD dataset from Berkeley, We used video, accelerometer, and mocap modalities. We used video with Cluster-0l/Cam01-02 subsets, and the whole mocap (optical) and accelerometer data with 12 persons/5 trials. Video input was preprocessed by Open- Pose which identifies 48 dimensions of vectors. Optical mocap had the position of the keypoints whose dimension was 129. Accelerometer were placed in 6 places in the body whose dimension was 18. We used the parameters in wave2wave with entropy loss function L(Ø) - \/2 Σ log po (vf.r .riis) with LSTM modules 500, embedding size 500, dropout 3, maximum sentence length 400, and batch size 100. We used Adam optimizer. We used 2, 3, for multi-resolution spatial pyramid. We used the same parameter set for wave2wave interactive model. We use Titan Xp.
hhttp://teee inte-iimmeiioo htwpr/s--eocrrg/eeeeeeeeeeee
http://tele-immmssionnccoo

---

Under review as a conference paper at ICLR 2019
(wene, Widee)
ppl dz
ppl
dz
ppl dz
ppl
dz
ppl dz
ppl
dz
129
129
129
seq2seq baseline
seq2seq clustering
58000.42
52000.33
5.20
30.22
wave2wave
wave2wavellt
wave2wavvlleBaackrrans
(1,16)
2.13
19.74
2.13
4.72
2.13
4.73
(5,80)
0.33
10.73
0.33
3.44
0.32
3.40
(10,160)
0.42
11.28
0.42
3.49
0.42
3.48
(20,320)
0.72
13.67
0.72
3.78
0.72
3.75
(30,480)
1.21
15.03
1.21
4.11
1.21
4.11
(60,960)
4.30
35.98
4.30
6.81
4.30
6.82
Table 2: Figure shows major experimental results for acc2moc.
Human Understandability One characteristic of activity translation can be observed in the direc- tion of wave2wave translation with accelerometer to video, e.g. acc2cam. That is, the accelerometer data takes the form that is understandable by human beings by nature but translation video makes this visible. By selecting 50 test cases, the human could understand 48 cases. 96 % is fairly good. The second characteristic of activity translation is opportunistic sensor problem, e-g. when cannot use video camera in bathrooms, then translate it video which can use this opportunity. This corresponds to the case of ac- celeromter video, c.g. acc2cam. We conduct this experiments. Upon watching the video signals on screen we could observe the basic human movements. By selecting 50 test cases, the human could understand 43 cases.
use other sensor modality, e.g. accelerometer, and
Experimental Results Major experimental results are shown in Table 2 (1,5, 10, 20, 30, 60). For each window size we measured one target with perplexity (ppl) and the whole target with perplexity (ppl). We compared several wave2wave results with (1) the seq2seq model without dimensionality reduction (via clustering). (2) the seq2seq model with dimensionality reduction. All the experiments are done with the direction from accelerometer to motion capture (ace2moc).
We used wene
Firstly, the original seg2seq model did not work well without dimensionality reduction of input space. The perplexity was 58000.45 This figure suggests that the optimization of deep learning did not go progress due to the complexity of the training data or the bad initialization. However, the results were improved fairly well we do dimensionality reduction using clustering. This figure is close to the results by wave2wave (iterative) with uue = 60.
Secondly, weee 5 performed better than other window size for perplexity when d, 1. When this became high dimensional, the wave2wave iterative model performed better than the wave2wave mode: 3.44 vs 10.73 in perplexity. Since motion capture has d2 = 129 dimensions, the representa- tion space becomes Rd when we let R denote the parameter space of point in motion capture. Compared with this the wave2wave iterative model equipped with the representation space linear with R. The wave2wave iterative model has advantage in this point. Moreover, the wave2wave iterative brak-trannlation model made the best score in perplexity when dN = as well as dz 129.
7 CONCLUSION
We proposed a method to translate between waves wave2wave with sliding window-based mech- anism and iterative Back-translation model for high-dimensional data. Experimental results for two real-life data show that this is positive. Performance improvements about 46 % in test loss for one dimensional case and about 1625 % in perplexity for high-dimensional case using the iterative mack-translation model.
8

---

Under review as a conference paper at ICLR 2019
REFERENCES
Alexander Alexandrov, Konstantinos Benidis, Michael Bchhke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Sundar Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. Gluonts: Probabilistic time series models in python. CoRR, abs/1906.05264, 2019.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. CoRR, abs/1511.00561 2015. URL http://
faxxiv.org/ abs /1111. 00561
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi multi-person 2d pose estimation using part affinity fields. CVPR, 2017.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Com- bining recent advances in neural machine translation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 76-86, 2018.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. EMNLP2014, 2014.
Yoshua Bengio Dzmitry Bahdanau, Kyunghyun Cho. Neural machine translation by jointly learning to align and translate. arxiv, 2014.
Brendan J. Frey and Delbert Dueck. Clustering by passing messages between data points, Science, 315(5814), pp. 972-976, 2007.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back- translation for neural machine translation. Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 18-24, 2018.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.
A. Iwaki and H. Fujiwara. Synthesis of high-frequency ground motion using information extracted from low-frequency ground motion: case study kanto area. journal of Japan sssociation for earthquake engineering, 13:1-18, 2013.
Alireza Koochali, Peter Schichtel, Sheraz Ahmed, and Andreas Dengel. Probabilistic forecasting of
sensory data with generative adversarial networks forgan. CoRR, abs/1903. 12549, 2019.
Minh-Thang Luong, Hicu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 1412-1421, September 2015.
Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. Image denoising using very deep fully convolu- tional encoder-decoder
URL networr networks with symmetric skip connections. CoRR, abs/ 009566 2016.
org/ abs/l 1003 09056
Natalia Neverova, Christian Wolf, Graham W.Taylor, and Florian Nebout. Multi-scale deep learn- ing for gesture detection and localization. Workshop on Looking at People (ECCV), 2014.
D. Roggen, G. Trster, P. Lukowicz, A. Ferscha, and R. Chavarriaga. Opportunistic human activity and context recognition. Computer, 46(2):36-45, 2013. ISSN 0018-9162. doi: 10.1 109/MC. 2012.393.
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. NIPS 2014, 2014.
Aäron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. CoRR, abs/16009033ee 2016. URI.http::/ wwwp:/wwwwwww./ooottoo.oooo.ss 609. 03449

---

Under review as a conference paper at ICLR 2019
Ting- Chun Wang, Ming- Yu Liu, Jun- Jun-Yan Zhu, Guilin Liu, Andrew Tao. Jan Kautz, and Bryan Catan- zaro. Video-to-vided synthesis. CoRR, has/1808806601. 2018. URL http:://axxi..o/////
abs/ /eos/188...0 06601
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arxiv, 2015.
10


Reviews:
[{"review_id": "S1x8MOPqYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a variant of sequence to sequence models that operates on the fixed segments (waves). The paper contains significant flaws both in its writing and its experimental setup.\n\n1) Authors write: \"For example in human activity logs, the video data can be missing in bathrooms by ethical reasons but can be supplied by environmental sensors which have less ethical problems.\". I believe that this statement is completely irrelevant to the paper.\n\n2) Authors write: \"The straight-forward extension does not apply by two reasons: (1) the lengths of x and y are radically different, and (2) both x and y are high dimensionals.\" This statement is not justified and to the contrary of authors statement there has been recent work on wave2wave modeling including end-to-end speech-to-speech neural translation by Jia et al 2019, Liu et al 2019.\n\n3) Authors write: \"CNN is relatively weak in handling time-series data.\" Again this statement is not justified and to the contrary there has been work by Gehring et al 2017 showing success of CNNs in neural machine translation and Bai et al 2018 showing success of Temporal Convolutional Nets on various sequence modeling tasks. \n\n4) Several typos in the paper: Reccurent -> Recurrent on page 2, informaion -> information on page 5\nI believe that this paper is far from worthy of being considered to be accepted by ICLR.\n\n5) Authors claim that they learn several inverse function R^{-1} to capture target waves. On the first glance it may seem that they invert the same weight matrix used for learning source waves R, however upon closer inspection it seems that they learn separate weight matrices for source and target weights which makes inverse notation very confusing.\n\n6) I believe experimental setup is flawed. Authors consider the tasks that are not used by the community which makes it impossible to compare their method to related literature. Furthermore it is not clear to me how capturing the sequence semi-autoregressively (by modeling chunks of waves) produces such as big boost over vanilla sequence to sequence baseline.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1ekz4gaKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose modifications to baseline seq-to-seq systems for wave-to-wave translation. To handle possibly long inputs and outputs, as well as significant length differences, they propose to use sliding windows. For high-dimensional outputs, they use an iterative approach predicting each dimension independently. They evaluate their models on earthquake data on on activity translation (video to motion capture).\n\nI would reject this paper because it largely ignores the litterature on speech recognition, text-to-speech synthesis and speech-to-speech translation (e.g. Jia et al. Direct speech-to-speech translation with a sequence-to-sequence model). I would suggest to the authors to familiarize themselves with work presented at conferences such as Interspeech and ICASSP.\n\nThe contributions of the paper are unfortunately minimal. Window-based representations are already used, for example to obtain spectrograms. Predicting each feature independently appears very inefficient and mostly suitable for small datasets.\n\nWhile it wouldn't change my rating of the paper, there are many typos (e.g. acceleormeter, informaion) that could easily be corrected.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ltkbSRKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work explains how to use a seq2seq encoder-decoder neural network on the case of multivariate time series. The authors name this particular application of seq2seq the wave2wave network. Given a multivariate time series covering a time interval, it is split into subintervals of equal length, such that each block is a matrix. This matrix becomes an input into a recurrent encoder. On the decoder side, the similar matrix is produced at the output. The proposed neural network is tested on two data sets: an earthquake and activity translation.\nStrenghts:\n+ application on activity translation and evaluation by human raters is interesting\n+ the proposed work is a nice application of an encoder-decoder architecture in case of multivariate time series\nWeaknesses:\n- there is no methodological novelty. The proposed network uses a standard architecture and the authors use straightforward partitioning of the signal in equal-sized windows. Due to this, inventing a new name wave2wave is not fully justified.\n- the paper has many grammar mistakes and typos (for example, first sentence in the introduction seems to miss a few words). The writing is vague and imprecise and occasionally interferes with understanding. It would be important for the authors to carefully proofread the paper for language and clarity of presentation.\n- the experimental results are not explained in a sufficient detail. I did not understand what is the objective of the earthquake application and how does the data set looks like (e.g., what do you mean by “365 x 3 training data”). The details of experimental design, architecture and hyperparameter choice are not provided in the earthquake application. It is not clear what the authors mean by “simple encoder-decoder” and “simple seq2seq”, I guess those are some baselines. For the activity application, it would be useful to see some illustration of the neural network output when video is predicted from motion capture. It would help to clarify the accuracy measures – perplexity and human understability. How many humans were involved in rating and what were they asked to do? How did the humans rate the non-video outputs?\n\nOverall, given the quality of writing and lack of methodological novelty this paper is not ready for publication.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the task of sequence to sequence modelling with multivariate, real-valued time series.
The authors propose an encoder-decoder based architecture that operates on fixed windows of the original signals.

The reviewers unanimously criticise the lack of novelty in this paper and the lack of comparison to existing baselines.
While Rev #1 positively highlights human evaluation contained in the experiments, they nevertheless do not think this paper is good enough for publication as is.
The authors did not submit a rebuttal.

I therefore recommend to reject the paper.
Decision: Reject