Title: TRENET: HYBRID NEURAL NETWORKS FOR LEARN-ING THE LOCAL TREND IN TIME SERIES

Abstract: Local trends of time series characterize the intermediate upward and downward patterns of time series. Learning and forecasting the local trend in time series data play an important role in many real applications, ranging from investing in the stock market, resource allocation in data centers and load schedule in smart grid. Inspired by the recent successes of neural networks, in this paper we propose TreNet, a novel end-to-end hybrid neural network that predicts the local trend of time series based on local and global contextual features. TreNet leverages convolutional neural networks (CNNs) to extract salient features from local raw data of time series. Meanwhile, considering long-range dependencies existing in the sequence of historical local trends, TreNet uses a long-short term memory recurrent neural network (LSTM) to capture such dependency. Furthermore, for predicting the local trend, a feature fusion layer is designed in TreNet to learn joint representation from the features captured by CNN and LSTM. Our proposed TreNet demonstrates its effectiveness by outperforming conventional CNN, LSTM, HMM method and various kernel based baselines on real datasets.

1. INTRODUCTION:

Time series, which is a sequence of data points in time order, is being generated in a wide spectrum of domains, such as daily fluctuation of the stock market, power consumption records of households, performance monitoring data of clusters in data centres, and so on. In many applications, users are interested in understanding the evolving trend in time series and forecasting the trend, since the conventional prediction on specific data points could deliver very little information about the semantics and dynamics of the underlying process generating the time series. For instance, time series in Figure 1 are from the household power consumption dataset 1 . Figure 1 (a) shows some raw data points of time series. Though point A and B have approximately the same value, the underlying system is likely to be in two different states when it outputs A and B, because A is in an upward trend while B is in a downward trend (Wang et al., 2011; Matsubara et al., 2014) . On the other hand, even when two points with the similar value are both in the upward trend, e.g, point A and C, the different slopes and durations of the trends where point A and C locate, could also indicate different states of the underlying process.

Particularly, in this paper we are interested in the local trend of time series which measures the intermediate local behaviour, i.e., upward or downward pattern of time series that characterized by the slope and duration (Wang et al., 2011) . For instance, in Figure 1 (b) the linear segments over raw data points of time series represent the local trends extracted from a real household power consumption time series. For the ease of presentation, we will use the term trend and local trend interchangeably in the rest of the paper. Learning and forecasting local trends are quite useful in a wide range of applications. For instance, in the stock market, due to its high volatility and noisy environment, in reality predicting stock price trends is preferred over the prediction of the stock market absolute values (Atsalakis & Valavanis, 2009) . Predicting the local trend of stock price time series empowers traders to design profitable trading strategies (Chang et al., 2012b; Atsalakis & Valavanis, 2009) . In the smart energy domain, knowing the predictive local trend of power consumption time series enables energy providers to schedule power supply and maximize energy utilization (Zhao & Magoulès, 2012) .

Meanwhile, in recent years neural networks have shown the dramatical power in a wide spectrum of domains, e.g., natural language processing, computer vision, speech recognition, time series analysis, etc. (Wang et al., 2016b; Sutskever et al., 2014; Yang et al., 2015; Lipton et al., 2015) . For time series data, two mainstream architectures, convolutional neural network (CNN) and recurrent neural network (RNN) have been exploited in different time series related tasks, e.g., RNN in time series classification (Lipton et al., 2015) and CNN in activity recognition and snippet learning (Liu et al., 2015; Yang et al., 2015) . RNN is powerful in discovering the dependency in sequence data (Jain et al., 2014; Graves, 2012) and particularly the Long Short-Term Memory (LSTM) RNN works well on sequence data with long-term dependencies (Chung et al., 2014; Hochreiter & Schmidhuber, 1997) due to the internal memory mechanism. CNN excels in exacting effective representation of local salience from raw data of time series by enforcing a local connectivity between neurons. (Yang et al., 2015; Hammerla et al., 2016) . In this paper, we focus on learning and forecasting the local trends in time series via neural networks. This involves learning different aspects of the data. On one hand, the sequence of historical local trends describes the long-term contextual information of time series and thus naturally affects the evolution of the following local trend. On the other hand, the recent raw data points of time series (Wang et al., 2011; Batal et al., 2012) , which represent the local variation and behaviour of time series, affect the evolving of the following trend as well and have particular predictive power for abruptly changing local trends (Wang et al., 2011) . For instance, in Figure 1 (c), trend 1, 2 and 3 present a continuous upward pattern. Then when we aim at predicting the subsequent trend of time series at the end of the third local trend, the previous three successive upward trends outline a probable increasing trend afterwards. However, the local data around the end of the third trend, e.g., data points in the red circle, indicate that time series could stabilize and even decrease. The data points after the third trend indeed present a decreasing trend indicated by the red dotted segment. In this case, the subsequent trend has more dependency on the local data points. Therefore, it is highly desired to develop a systematic way to model such various hidden and complementary dependencies in time series for the local trend forecasting problem.

To this end, we propose a end-to-end hybrid neural network, referred to as TreNet. In particular, it consists of a LSTM recurrent neural network to capture the long dependency in historical local trends, a convolutional neural network to extract local features from local raw data of time series, and a feature fusion layer to learn joint representation to take advantage of both features drawn from CNN and LSTM. Such joint representation is used for the local trend forecasting. The experimental analysis on real datasets demonstrates that TreNet outperforms individual recurrent neural network, convolutional neural network and a variety of baselines in term of local trend prediction accuracy.

The rest of the paper is organized as follows. Section 2 presents related work, while Section 3 defines the problem to be solved and introduces the notations. In Section 4, we present the proposed TreNet. Section 5 demonstrates the performance of our method and baselines on real datasets. Finally, the paper is concluded in Section 6. Refer to Section 7 and Section 8 for more experiment results and discussion.

2. RELATED WORK:

Traditional learning approaches over local trends of time series mainly make use of Hidden Markov Models (HMMs) (Wang et al., 2011; Matsubara et al., 2014) . HMMs maintain short-term state dependences, i.e., the memoryless Markov property and predefined number of states, which requires significant task specific knowledge. RNNs instead use high dimensional, distributed hidden states that could take into account long-term dependencies in sequence data. Previous time series segmentation approaches (Keogh et al., 2001; Matsubara et al., 2014; Yuan, 2015) focus on achieving a meaningful segmentation and finding patterns, rather than modeling the relation in segments and therefore are not suitable for forecasting local trends. Multi-step ahead prediction is another way to realize local trend prediction by fitting the predicted values to estimate the local trend. However, multi-step ahead prediction is a non-trivial problem itself (Chang et al., 2012a) . In this paper, we concentrate on directly learning local trends through neural networks.

RNNs have recently shown promising results in a variety of applications, especially when there exist sequential dependencies in data (Lyu & Zhu, 2014; Chung et al., 2014; Sutskever et al., 2014) . Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997; Lyu & Zhu, 2014; Chung et al., 2014) , a class of recurrent neural networks with sophisticated recurrent hidden and gated units, are particularly successful and popular due to its ability to learn hidden long-term sequential dependencies. (Lipton et al., 2015) uses LSTMs to recognize patterns in multivariate time series, especially for multi-label classification of diagnoses. (Chauhan & Vig, 2015; Malhotra et al., 2015) evaluate the ability of LSTMs to detect anomalies in ECG time series. Bidirectional LSTM (Graves & Schmidhuber, 2005) is usually intended for speech processing rather than time series forecasting problems. Our paper focuses on using LSTM to capture the dependency in the sequence of historical local trends and meanwhile the hidden states in LSTM are further used to learn joint feature representations for the local trend forecasting.

CNN is often used to learn effective representation of local salience from raw data (Vinyals et al., 2015; Donahue et al., 2015; Karpathy et al., 2014) . (Hammerla et al., 2016; Yang et al., 2015; Lea et al., 2016 ) make use of CNNs to extract features from raw time series data for activity/action recognition. (Liu et al., 2015) focuses on the prediction of periodical time series values by using CNN and embedding time series with the potential neighbors in the temporal domain. Our proposed TreNet will combine the strengths of both LSTM and CNN and form a novel and unified neural network architecture for local trend forecasting.

Hybrid neural networks, which combines the strengths of various neural networks, are receiving increasing interest in the computer vision domain, such as image captioning (Mao et al., 2014; Vinyals et al., 2015; Donahue et al., 2015) , image classification (Wang et al., 2016a) , protein structure prediction (Li & Yu, 2016) , action recognition (Ballas et al., 2015; Donahue et al., 2015) and so on. But efficient exploitation of such hybrid architectures has not been well studied for time series data, especially the trend forecasting problem. (Li & Yu, 2016; Ballas et al., 2015) utilize CNNs over images in cascade of RNNs in order to capture the temporal features for classification. (Bashivan et al., 2015) transforms EEG data into a sequence of topology-preserving multi-spectral images and then trains a cascaded convolutional-recurrent network over such images for EEG classification. (Wang et al., 2016a; Mao et al., 2014) propose the CNN-RNN framework to learn a shared representation for image captioning and classification problems. In our proposed TreNet, LSTM and CNN first respectively learn the trend evolution and local raw data of time series and then TreNet fuses the features captured by LSTM and CNN to predict the trend.

3. PROBLEM FORMULATION:

In this section, we provide the formal definition of the trend learning and forecasting problem in this paper.

We define time series as a sequence of data points X = {x 1 , . . . , x T }, where each data point x t is real-valued and subscript t represents the time instant. The corresponding local trend sequence of X is a series of piecewise linear representations of X , denoted by T = { k , s k }. Each element of T , e.g., k , s k describes a linear function over a certain subsequence (or segment) of X and corresponds to a local trend in X . Such local trends in T are extracted from X by time series segmentation and fitting a linear function w.r.t. time t over each segment (Keogh et al., 2001; Wang et al., 2011) . 

(t) = { k , s k | k k ≤ t}.

Meanwhile, as we discussed in Section 1, local raw data of time series affects the varying of trend as well and thus we define the local data w.r.t. a certain time instant t as a sequence of data points in a window of size w, denoted by L(t) = {x t-w , . . . , x t }.

At certain time t, trend forecasting is meant to predict the duration and slope of the following trend based on a given sequence of historical trends T (t) and local data set L(t). The predicted duration and slope at time t are denoted by ˆ t and ŝt . Our proposed TreNet can be trained for predicting either ˆ t or ŝt . For simplicity, we use ŷt to represent the predicted value of TreNet throughout the paper.

Therefore, given the training dataset D = X ∪T , we aim to propose a neural network based approach to learn a function ŷt = f (T (t), L(t)) for the trend forecasting. In this paper, we focus on univariate time series. The proposed method can be naturally generalized to multivariate time series as well by augmenting the input to the neural network. Refer to Section 8 for more discussion.

4. HYBRID NEURAL NETWORKS FOR TREND LEARNING AND FORECASTING:

In this section, we first present an overview about the proposed TreNet for the trend forecasting. Then we will detail the components of TreNet. k or the activation of this neuron is then expressed as (Hochreiter & Schmidhuber, 1997; Chung et al., 2014) :

EQUATION

where o j k is an output gate and calculated as:

o j k = σ(W o [ k s k ] + U o h k-1 + V o c k ) j (2)

where [ k s k ] is the concatenation of the duration and slope of the trend k, h k-1 and c k are the vectorization of the activations of {h j k-1 } and {c j k }, and σ is a logistic sigmoid function. Then, the memory cell c j k is updated through partially forgetting the existing memory and adding a new memory content cj k :

EQUATION

The extent to which the existing memory is forgotten is modulated by a forget gate f j k , and the degree to which the new memory content is added to the memory cell is modulated by an input gate i j k . Then, such gates are computed by

EQUATION

At each step k, the hidden activation h k is the output to the feature fusion layer. Specifically, given a T (t) containing n local trends (i.e., |T (t

)| = n), the output of R(T (t)) is R(T (t)) = h n .

Learning features from the local raw data of time series.

When the k-th trend in T is fed to LSTM, the corresponding local raw time series data input to the CNN part of TreNet is L(t), where t = k i=1 i . CNN consists of H stacked layers of 1-d convolutional, activation and pooling operations. Denote by a i the input signal of layer i and thus at the first layer a 1 = L(t). Each layer has a specified number of filters n i of a specified filter size d i . Each filter on a layer sweeps through the entire input signal to exact local features as follows:

v i,j m = φ(b i,j + m+d i /2 z=m-d i /2 W i,j z a i z ) , ∀m = 1, . . . , |a i | (6)

where v i,j m is the activation of j-th filter of layer i on m position of the input signal. Here φ is the Leaky Rectified Linear Unit, which is shown to perform better (Xu et al., 2015) . Then the maxpooling is performed over the v i,j m of each filter. Finally, the output of CNN in TreNet is the concatenation of max-pooling of each filter on the last layer H, namely:

EQUATION

where q is the pooling size.

Feature fusion and output layers.:

The feature fusion layer combines the representations R(T (t)) and C(L(t)), to form a joint feature. Then, such joint feature is fed to the output layer to provide the trend prediction. Particularly, we first map R(T (t)) and C(L(t)) to the same feature space and add them together to obtain the activation of the feature fusion layer (Mao et al., 2014) . The output layer is a fully-connect layer following the feature fusion layer. Mathematically, the prediction of TreNet is expressed as:

ŷt = f (R(T (t)), C(L(t))) = W o • φ(W r • R(T (t)) + W c • C(L(t))) f eature f usion +b o (8)

where φ(•) is element-wise leaky ReLU activation function and + denotes the element-wise addition. W o and b o are the weights and bias of the output layer.

To train TreNet, we adopt the squared error function plus a regularization term as:

J(W , b ; T , X ) = 1 |T | |T | k=1 (ŷ k -y k ) 2 + λ W 2 (9)

where W and b represent the weight and bias parameters in TreNet, λ is a hyperparameter for the regularization term and y k is the true value of trend slope or duration.

The cost function is differentiable and the architecture of TreNet allows the gradients from the loss function ( 9) to be backpropagated to both LSTM and CNN parts. TreNet can be trained respectively for the slope and duration of local trends using T and X . When performing forecasting, T (t) and L(t) are fed to TreNet and the prediction value ŷk could be either the slope or duration depending on the training target.

5. EXPERIMENTAL ANALYSIS:

In this section, we conduct extensive experiments to demonstrate the prediction performance of TreNet by comparing to a variety of baselines. Due to the page limit, refer to Section 7 for more experiment results.

5.1. EXPERIMENT SETUP:

Dataset: We test our method and baselines on three real time series datasets.

• Daily Household Power Consumption (HousePC). This dataset2 contains measurements of electric power consumption in one household with a one-minute sampling rate over a period of almost 4 years. Different electrical quantities and some sub-metering values are available. We use the voltage time series throughout the experiments. • Gas Sensor (GasSensor). This dataset3 contains the recordings of chemical sensors exposed to dynamic gas mixtures at varying concentrations. The measurement was constructed by the continuous acquisition of the sensor array signals for a duration of about 12 hours without interruption. We mainly use the gas mixture time series regarding Ethylene and Methane in air. All datasets are preprocessed by (Keogh et al., 2001) to extract local trends. Alternative time series segmentation and local trend extraction approaches can be used as well. We choose (Keogh et al., 2001) here due to its high efficiency. Totally, we obtain 42591, 4720 and 1316 local trends respectively from above datasets. For the ease of experimental result interpretation, the slope of extracted local trends is represented by the angle of the corresponding linear function and thus in a bounded value range [-90, 90] . The duration of local trends is measured by the number of data points within the local trend. Then, the obtained trend sequences and the set of local data are split into training (80%), validation (10%) and test (10%) datasets.

Baselines: We compare TreNet with the following six baselines:

• CNN. This baseline method predicts the trend by only using CNN over the set of local raw data of time series to learn features for the forecasting. The size of local data is set at w as is defined in Section 3. • LSTM. This method uses LSTM to learn dependencies in the trend sequence T and predicts the trend only using the trained LSTM. commonly used kernels (Liu et al., 2015) , i.e., Radial Basis kernel (SVRBF), Polynomial kernel (SVPOLY), Sigmoid kernel (SVSIG). The trend sequence and the corresponding set of local time series data are concatenated as the input features to such SVR approaches. • Pattern-based Hidden Markov Model (pHMM). (Wang et al., 2011) proposed a patternbased hidden Markov model (HMM), which segments the time series and models the dependency in segments via HMM. The derived HMM model is used to the state of time series and then to estimate the trend based on the state. • Naive. This is the naive approach which takes the duration and slope of the last trend as the prediction for the next one. • ConvNet+LSTM(CLSTM). It is based on the cascade structure of ConvNet and LSTM in (Bashivan et al., 2015) which feeds the features learnt by ConvNet over time series to a LSTM and obtains the prediction from the LSTM.

Evaluation metric: We evaluate the predictive performance of TreNet and baselines in terms of Root Mean Square Error (RMSE). The lower the RMSE, the more accurate the predictions.

Training::

The training procedure of TreNet and baselines in our paper follows the schema below.

The CNN and LSTM components in TreNet share the same network structure (e.g., number of layers, neurons in each layer) as CNN and LSTM baselines. CNN has two stacked convolutional layers, which have 32 filters of size 2 and 4. The number of memory cells in LSTM is 600. For baseline CNN and LSTM, we tune the learning rate for each approach from {10 -1 , 10 -2 , 10 -3 , 10 -4 , 10 -5 } (Sutskever et al., 2013) , in order to achieve the least prediction errors and then fix the learning rate.

For TreNet, in addition to the learning rate, the number of neurons in the feature fusion layer is chosen from the range {300, 600, 900, 1200} to achieve the best performance. We use dropout and L2 regularization to control the capacity of neural networks to prevent overfitting, and set the values to 0.5 and 5 × 10 -4 respectively for all datasets (Mao et al., 2014) . The Adam optimizer (Kingma & Ba, 2014 ) is chosen to learn the weights in neural networks.

Regarding the SVR based approaches, we carefully tune the parameters c (error penalty), d (degree of kernel function), and γ (kernel coefficient) for kernels. Each parameter is selected from the sets c ∈ {10 -5 , 10 -4 , . . . , 1, . . . , 10 4 , 10 5 }, d ∈ {1, 2, 3}, γ ∈ {10 -5 , 10 -4 , . . . , 1, . . . , 10 5 } respectively. We iterate through candidate values of each combination of c, d and γ to train our model, and keep the parameters that generate the lowest RMSE on the validation set, and then use them to predict on the test set.

The training datasets of SVR and pHMM baselines are consistent as that of TreNet. Likewise, CNN and LSTM baselines are respectively fed by the set of local data and the trend sequence of the same size as TreNet. In addition, since the window size of local data is tunable, we vary the window size of local data, i.e. w, from the range {100, 300, 500, 700, 900}, so as to investigate how the size of local data influences the predication performance. The results will be presented in Section 5.2. The model's performance on the validation set will be evaluated after each epoch of training. Each model is trained for at least 50 epochs. Meanwhile, the training process adopts early stopping if no further improvement in the performance of validation shows up after 50 epochs.

5.2. EXPERIMENT RESULTS:

Table 1 studies the prediction performances of TreNet and baselines. For each dataset, the window size of local data is constant for approaches (i.e., CNN, SVRBF, SVPOLY, SVSIG, pHMM and TreNet) that take local data as input. Then, the results of each approach are obtained by tuning the corresponding parameter as described in Section 5.1.

In Table 1 , we observe that TreNet consistently outperforms baselines on the duration and slope prediction by achieving around 30% less errors at the maximum. It verifies that the hybrid architecture of TreNet can improve the performance by utilizing the information captured by both CNN and LSTM. Specifically, pHMM method performs worse due to the limited representation capability of HMM. On the slope prediction, SVR based approaches can get comparable results as TreNet.

In the following group of experiments, we investigate the effect of local data size (i.e., w) on the prediction. In particular, we tune the value of local data size for the approaches whose input fea-tures contains local data and observe the prediction errors. Such approaches include CNN, SVRBF, SVPOLY, SVSIG, pHMM and TreNet. LSTM only consumes the trend sequence and thus is not included. Due to the page limit, we report the results on the HousePC dataset in Table 2 and Table 3 The merging process repeats until every possible merge gives rise to a segment with errors above a specified threshold. We use the relative mean squared error as the error metric and specify the threshold as 0.05. In this group of experiments, we visualize the trend prediction using the sample testing data instance from each dataset in Figure 4 . We can observe that in HousePC TreNet successfully predicts the changed trend, though there are successive upward trends before. In Stock and GasSensor datasets, the succeeding upward and downward trends are correctly predicted as well. 4 , Table 5 , Table 6 and Table 7 .

7.2. ADDITIONAL EXPERIMENT RESULTS:

From the results, we observe that TreNet outperforms baselines almost on all window sizes. Meanwhile, the prediction errors often present the decreasing and stable pattern as the window size varies.

Window size of local data: The observation in above experiments w.r.t. the varying window size provides inspiration for choosing the window size of local data. Given the training dataset, we can find out the maximum duration of local trends and takes it as the local data size. This is because doing so can ensure that the range of local data in each training instance can cover the most recent local trend, whose raw data is believed to have strong predictive power for the subsequent trend. Additionally, we observe that setting the window size of local data of CNN and TreNet in this way can achieve comparable prediction errors compared to the cases with larger window sizes . 

8. DISCUSSION:

For multivariate time series, we can augment the input of TreNet by including the trend sequences and local data of exogenous time series and then train TreNet for a certain target time series to predict its trend. Another line of research is to explore equipping TreNet with multi-task learning. This is motivated by the observation that if we decompose the trend forecasting problem into classification and regression respectively for the slope and duration, we can utilize the correlation between slope

:

https://archive.ics.uci.edu/ml/datasets/Individual+household+electric+power+consumption

https://archive.ics.uci.edu/ml/datasets/Gas+sensor+array+under+dynamic+gas+mixtures

Figures and Tables:

Figure 1: (a) Time series of household power consumption. (b) Local trends in time series. (c) Effect of local raw data on the trend forecasting.

k and s k respectively represent the duration and slope of trend k. k is measured in terms of the time range covered by trend k. Local trends in T are time ordered and non-overlapping. The durations of all the local trends in T address k k = T . In addition, a local trend sequence ending by time t is denoted by T

Overview.The idea of our TreNet is to combine CNN with LSTM to utilize their representation abilities on different aspects of training data D (D = X ∪ T ) and then to learn a joint feature for the trend prediction. Technically, TreNet is designed to learn a predictive function ŷt = f (R(T (t)), C(L(t))). R(T (t)) is derived by training the LSTM over sequence T to capture the dependency in the trend evolving, while C(L(t)) corresponds to local features extracted by CNN from L(t). The long-term and local features captured by LSTM and CNN, i.e., R(T (t)) and C(L(t)) convey complementary information pertaining to the trend varying. Therefore, the feature fusion layer is supposed to take advantages of both features to produce a fused representation for improved performance. Finally, the trend prediction is realized by the function f (•, •), which corresponds to the feature fusion and output layers in Figure2.

Figure 2: Illustration of the hybrid architecture of TreNet. (best viewed in colour)

Figure 3: Illustration of local trend extraction via time series segmentation. (Best viewed in colour)

Figure 4: Visualization of the trend prediction by TreNet in HousePC, Stock and GasSensor datasets. The blue line in each figure represents the historical trend sequence. The yellow line represents the predicted local trend.

Stock Transaction (Stock): This dataset is extracted from Yahoo Finance and contains the daily stock transaction information in New York Stock Exchange from 1950-10 to 2016-4.

Table content: <table/>

• Support Vector Regression (SVR). A family of support vector regression based approaches with different kernel methods is used for the trend forecasting. We consider three RMSE of the prediction of local trend duration and slope on each dataset.

Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="2">RMSE @ Duration RMSE @ Slope</td></tr><tr><td/><td>CNN</td><td>27.51</td><td>13.56</td></tr><tr><td/><td>LSTM</td><td>27.27</td><td>13.27</td></tr><tr><td/><td>SVRBF</td><td>31.81</td><td>12.94</td></tr><tr><td>HousePC</td><td>SVPOLY</td><td>31.81</td><td>12.93</td></tr><tr><td/><td>SVSIG</td><td>31.80</td><td>12.93</td></tr><tr><td/><td>pHMM</td><td>34.06</td><td>26.00</td></tr><tr><td/><td>Naive</td><td>39.68</td><td>21.17</td></tr><tr><td/><td>CLSTM</td><td>25.97</td><td>13.77</td></tr><tr><td/><td>TreNet</td><td>25.89</td><td>12.89</td></tr><tr><td/><td>CNN</td><td>18.87</td><td>12.78</td></tr><tr><td/><td>LSTM</td><td>11.07</td><td>8.40</td></tr><tr><td/><td>SVRBF</td><td>11.38</td><td>7.40</td></tr><tr><td>Stock</td><td>SVPOLY</td><td>11.40</td><td>7.42</td></tr><tr><td/><td>SVSIG</td><td>11.49</td><td>7.41</td></tr><tr><td/><td>pHMM</td><td>36.37</td><td>8.70</td></tr><tr><td/><td>Naive</td><td>11.36</td><td>8.58</td></tr><tr><td/><td>CLSTM</td><td>9.26</td><td>7.31</td></tr><tr><td/><td>TreNet</td><td>8.86</td><td>6.84</td></tr><tr><td/><td>CNN</td><td>53.99</td><td>11.51</td></tr><tr><td/><td>LSTM</td><td>55.77</td><td>11.22</td></tr><tr><td/><td>SVRBF</td><td>62.81</td><td>10.21</td></tr><tr><td>GasSensor</td><td>SVPOLY</td><td>70.91</td><td>10.95</td></tr><tr><td/><td>SVSIG</td><td>85.69</td><td>11.92</td></tr><tr><td/><td>pHMM</td><td>111.62</td><td>13.07</td></tr><tr><td/><td>Naive</td><td>53.76</td><td>10.57</td></tr><tr><td/><td>CLSTM</td><td>54.20</td><td>14.86</td></tr><tr><td/><td>TreNet</td><td>52.28</td><td>9.57</td></tr></table>

. The results on Stock and GasSensor datasets can be referred to Section 7.Baseline Naive has no original time series data as input CLSTM works on the whole time and has no local data. Thus they are excluded from this set of experiments.In Table2, we observe that compared to baselines TreNet has the lowest errors on the duration prediction across different window sizes. pHMM requires sufficient data points to model the relations of segments and fails to work on 100 size. As the window size increases and more local data points are fed to the training process, the prediction errors of CNN and TreNet decrease or nearly stabilize. This could be because only the certain amount of local data has predictive power. The filtering and pooling mechanism enables CNN to focus on the certain local data having strong predictive power and thus giving more local data only gives rise to marginal improvements. Such similar phenomenon is observed on the slope prediction as is shown in Table3. For more results and discussion, please refer to Section 7. RMSE of the duration predictions w.r.t. different sizes of local data in HousePC dataset

Table content: <table><tr><td colspan="7">Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet</td></tr><tr><td>100</td><td>29.37</td><td>31.48</td><td>31.96</td><td>31.88</td><td>-</td><td>25.93</td></tr><tr><td>300</td><td>27.33</td><td>31.17</td><td>31.61</td><td>31.66</td><td>30.03</td><td>25.94</td></tr><tr><td>500</td><td>27.51</td><td>31.81</td><td>31.81</td><td>31.80</td><td>34.06</td><td>25.89</td></tr><tr><td>700</td><td>27.41</td><td>31.10</td><td>31.09</td><td>31.11</td><td>27.37</td><td>25.72</td></tr><tr><td>900</td><td>27.42</td><td>31.28</td><td>31.27</td><td>31.27</td><td>28.45</td><td>25.62</td></tr><tr><td colspan="7">Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet</td></tr><tr><td>100</td><td>13.68</td><td>12.93</td><td>12.9352</td><td>12.9346</td><td>-</td><td>13.14</td></tr><tr><td>300</td><td>13.60</td><td>12.93</td><td>12.9346</td><td>12.9345</td><td>27.75</td><td>13.15</td></tr><tr><td>500</td><td>13.56</td><td>12.94</td><td>12.9342</td><td>12.9346</td><td>26.00</td><td>12.89</td></tr><tr><td>700</td><td>13.52</td><td>12.93</td><td>12.9345</td><td>12.9345</td><td>35.32</td><td>12.86</td></tr><tr><td>900</td><td>13.60</td><td>12.94</td><td>12.9350</td><td>12.9346</td><td>37.60</td><td>12.96</td></tr></table>

RMSE of the slope predictions w.r.t. different sizes of local data in HousePC dataset6 CONCLUSIONIn this paper we propose TreNet, a novel hybrid neural network to learn and predict the local trend behaviour of time series. The experimental results demonstrate that such a hybrid framework can indeed utilize complementary information extracted by CNN and LSTM to enhance the prediction performance. Moreover, such architecture is generic and extendible in that additional exogenous time series can be fed to TreNet, so as to boost the performance and investigate the effect of different data sources on the trend evolving.

Table content: <table/>

RMSE of the duration predictions on different sizes of local data in Stock dataset

Table content: <table><tr><td colspan="7">Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet</td></tr><tr><td>100</td><td>18.87</td><td>11.38</td><td>11.40</td><td>11.49</td><td>-</td><td>8.86</td></tr><tr><td>300</td><td>18.17</td><td>11.41</td><td>11.44</td><td>11.42</td><td>39.84</td><td>8.85</td></tr><tr><td>500</td><td>18.06</td><td>11.39</td><td>11.44</td><td>11.36</td><td>32.10</td><td>8.51</td></tr><tr><td>700</td><td>18.10</td><td>11.45</td><td>11.59</td><td>11.58</td><td>36.37</td><td>8.58</td></tr><tr><td>900</td><td>18.07</td><td>11.32</td><td>11.47</td><td>11.59</td><td>38.36</td><td>8.78</td></tr><tr><td colspan="7">Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet</td></tr><tr><td>100</td><td>12.78</td><td>7.40</td><td>7.42</td><td>7.41</td><td>-</td><td>6.84</td></tr><tr><td>300</td><td>12.24</td><td>7.42</td><td>7.51</td><td>7.38</td><td>6.67</td><td>6.53</td></tr><tr><td>500</td><td>12.13</td><td>7.47</td><td>7.41</td><td>7.42</td><td>7.59</td><td>6.58</td></tr><tr><td>700</td><td>12.24</td><td>7.53</td><td>7.58</td><td>7.51</td><td>9.74</td><td>6.75</td></tr><tr><td>900</td><td>12.25</td><td>7.61</td><td>7.45</td><td>7.59</td><td>14.00</td><td>6.73</td></tr></table>

RMSE of the slope predictions on different sizes of local data in Stock datasetThen, we provide the RMSE w.r.t. the varying window size on Stock and GasSensor datasets in Table

Table content: <table/>

RMSE of the duration predictions on different sizes of local data in GasSensor dataset

Table content: <table><tr><td colspan="7">Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet</td></tr><tr><td>100</td><td>54.23</td><td>57.77</td><td>65.99</td><td>99.78</td><td>-</td><td>53.91</td></tr><tr><td>300</td><td>53.99</td><td>62.81</td><td>70.91</td><td>85.69</td><td>-</td><td>52.28</td></tr><tr><td>500</td><td>53.82</td><td>61.86</td><td>64.33</td><td>91.51</td><td>111.62</td><td>51.77</td></tr><tr><td>700</td><td>53.14</td><td>61.20</td><td>63.89</td><td>78.20</td><td>175.36</td><td>51.15</td></tr><tr><td>900</td><td>53.19</td><td>61.45</td><td>63.83</td><td>68.09</td><td>255.73</td><td>51.25</td></tr><tr><td colspan="7">Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet</td></tr><tr><td>100</td><td>11.98</td><td>11.16</td><td>11.19</td><td>12.48</td><td>-</td><td>10.30</td></tr><tr><td>300</td><td>11.51</td><td>10.21</td><td>10.95</td><td>11.92</td><td>-</td><td>9.57</td></tr><tr><td>500</td><td>11.75</td><td>10.08</td><td>10.65</td><td>11.64</td><td>13.07</td><td>9.60</td></tr><tr><td>700</td><td>11.59</td><td>9.54</td><td>10.44</td><td>11.72</td><td>12.29</td><td>9.55</td></tr><tr><td>900</td><td>12.10</td><td>9.61</td><td>10.37</td><td>11.54</td><td>12.37</td><td>9.46</td></tr></table>

RMSE of the slope predictions on different sizes of local data in GasSensor dataset

Table content: <table/>

Peer review (iclr2017):
Updated review: the authors did an admirable job of responding to and incorporating reviewer feedback. In particular, they put a lot of effort into additional experiments, even incorporating a new and much stronger baseline (the ConvNet -> LSTM baseline requested by multiple reviewers). I still have two lingering concerns previously stated -- that each model*s architecture (# hidden units, etc.) should be tuned independently and that a pure time series forecasting baselines (without the trend preprocessing) should be tried. I*m going to bump up my score from a clear rejection to a borderline. ----- This paper is concerned with time series prediction problems for which the prediction targets include the slope and duration of upcoming local trends. This setting is of great interest in several real world problem settings (e.g., financial markets) where decisions (e.g., buy or sell) are often driven by local changes and trends. The primary challenge in these problems is distinguishing true changes and trends (i.e., a downturn in share price) from noise. The authors tackle this with an interesting hybrid architecture (TreNet) with four parts: (1) preprocessing to extract trends, (2) an LSTM that accepts those trends as inputs to ostensibly capture long term dependencies, (3) a ConvNet that accepts a local window of raw data as its input at each time step, and (4) a higher *feature fusion* (i.e., dense) layer to combine the LSTM*s and ConvNet*s outputs. On three univariate time series data sets, the TreNet outperforms the competing baselines including those based on its constituent parts (LSTM + trend inputs, CNN). Strengths: - A very interesting problem setting that can plausibly be argued to differ from other sequential modeling problems in deep learning (e.g., video classification). This is a nice example of fairly thoughtful task-driven machine learning. - Accepting the author*s assumptions as true for the moment, the proposed architecture seems intuitive and well-designed. Weaknesses: - Although this is an interesting problem setting (decisions driven by trends and changes), the authors did not make a strong argument for why they formulated the machine learning task as they did. Trend targets are not provided from *on high* (by data oracle) but extracted from raw data using a deterministic algorithm. Thus, one could just easily formulate this as plain time series forecasting problem in which we forecast the next 100 steps and then apply the trend extractor to convert those predictions into a trend. If the forecasts are accurate, so will be the extracted trends. - The proposed architecture, while interesting, is not justified, in particular the choice to feed the extracted trends and raw data into separate LSTM and ConvNet layers that are only combined at the end by a shallow MLP. An equally straightforward but more intuitive choice would have been to feed the output of the ConvNet into the LSTM, perhaps augmented by the trend input. Without a solid rationale, this unconventional choice comes across as arbitrary. - Following up on that point, the raw->ConvNet->LSTM and {raw->ConvNet,trends}->LSTM architectures are natural baselines for experiments. - The paper presupposes, rather than argues, the value of the extracted trends and durations as inputs. It is not unreasonable to think that, with enough training data, a sufficiently powerful ConvNet->LSTM architecture should be able to learn to detect these trends in raw data, if they are predictive. - Following up on that point, two other obvious baselines that were omitted: raw->LSTM and {raw->ConvNet,trends}->MLP. Basically, the authors propose a complex architecture without demonstrating the value of each part (trend extraction, LSTM, ConvNet, MLP). The baselines are unnecessarily weak. One thing I am uncertain about in general: the validity of the practice of using the same LSTM and ConvNet architectures in both the baselines and the TreNet. This *sounds* like an apples-to-apples comparison, but in the world of hyperparameter tuning, it could in fact disadvantage either. It seems like a more thorough approach would be to optimize each architecture independently. Regarding related work and baselines: I think it is fair to limit the scope of in-depth analysis and experiments to a set of reasonable, representative baselines, at least in a conference paper submitted to a deep learning conference. That said, the authors ignored a large body of work on financial time series modeling using probabilistic models and related techniques. This is another way to frame the above *separate trends from noise* problem: treat the observations as noisy. One semi-recent example: J. Hernandez-Lobato, J. Lloyds, and D. Hernandez-Lobato. Gaussian process conditional copulas with applications to financial time series. NIPS 2013. I appreciate this research direction in general, but at the moment, I believe that the work described in this manuscript is not suitable for inclusion at ICLR. My policy for interactive review is to keep an open mind and willingness to change my score, but a large revision is unlikely. I would encourage the authors to instead use their time and energy -- and reviewer feedback -- in order to prepare for a future conference deadline (e.g., ICML).