Title: AUTOREGRESSIVE CONVOLUTIONAL NEURAL NETWORKS FOR ASYNCHRONOUS TIME SERIES

Abstract: We propose Significance-Offset Convolutional Neural Network, a deep convolutional network architecture for regression of multivariate asynchronous time series. The model is inspired by standard autoregressive (AR) models and gating mechanisms used in recurrent neural networks. It involves an AR-like weighting system, where the final predictor is obtained as a weighted sum of adjusted regressors, while the weights are data-dependent functions learnt through a convolutional network. The architecture was designed for applications on asynchronous time series and is evaluated on such datasets: a hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, an artificially generated noisy autoregressive series and household electricity consumption dataset. The proposed architecture achieves promising results as compared to convolutional and recurrent neural networks. The code for the numerical experiments and the architecture implementation will be shared online to make the research reproducible.

1. INTRODUCTION:

Time series forecasting is focused on modeling the predictors of future values of time series given their past. As in many cases the relationship between past and future observations is not deterministic, this amounts to expressing the conditional probability distribution as a function of the past observations:

p(X t+d |X t , X t-1 , . . .) = f (X t , X t-1 , . . .).

(1)

This forecasting problem has been approached almost independently by econometrics and machine learning communities.

In this paper we examine the capabilities of convolutional neural networks (CNNs), (Lecun et al., 1998) in modeling the conditional mean of the distribution of future observations; in other words, the problem of autoregression. We focus on time series with multivariate and noisy signal. In particular, we work with financial data which has received limited public attention from the deep learning community and for which nonparametric methods are not commonly applied. Financial time series are particularly challenging to predict due to their low signal-to-noise ratio (cf. applications of Random Matrix Theory in econophysics (Laloux et al., 2000; Bun et al., 2017) ) and heavy-tailed distributions (Cont, 2001) . Moreover, the predictability of financial market returns remains an open problem and is discussed in many publications (cf. efficient market hypothesis of Fama (1970) ).

A common situation with financial data is that the same signal (e.g. value of an asset) is observed from different sources (e.g. financial news, analysts, portfolio managers in hedge funds, marketmakers in investment banks) in asynchronous moments of time. Each of these sources may have a different bias and noise with respect to the original signal that needs to be recovered (cf. time series in Figure 1 ). Moreover, these sources are usually strongly correlated and lead-lag relationships are possible (e.g. a market-maker with more clients can update its view more frequently and precisely than one with fewer clients). Therefore, the significance of each of the available past observations might be dependent on some other factors that can change in time. Hence, the traditional econometric models such as AR, VAR, VARMA (Hamilton, 1994) might not be sufficient. Yet their relatively good performance motivates coupling such linear models with deep neural networks that are capable of learning highly nonlinear relationships.

Figure 1 : Quotes from four different market participants (sources) for the same CDS 1 throughout one day. Each trader displays from time to time the prices for which he offers to buy (bid) and sell (ask) the underlying CDS. The filled area marks the difference between the best sell and buy offers (spread) at each time.

For these reasons, we propose Significance-Offset Convolutional Neural Network, a Convolutional Network extension of standard autoregressive models (Sims, 1972; 1980) equipped with a nonlinear weighting mechanism and provide empirical evidence on its competitiveness with standard multilayer CNN and recurrent Long-Short Term Memory network (Hochreiter & Schmidhuber, 1997) . The mechanism is inspired by the gating systems that proved successful in recurrent neural networks (Hochreiter & Schmidhuber, 1997; Chung et al., 2014) and highway networks (Srivastava et al., 2015) .

2. RELATED WORK:



2.1. TIME SERIES FORECASTING:

Literature in time series forecasting is rich and has a long history in the field of econometrics which makes extensive use of linear stochastic models such as AR, ARIMA and GARCH processes to mention a few. Unlike in machine learning, research in econometrics is more focused on explaining variables rather than improving out-of-sample prediction power. In practice, one can notice that these models 'over-fit' on financial time series: their parameters are unstable and out-of-sample performance is poor.

Reading through recent proceedings of the main machine learning venues (e.g. ICML, NIPS, AIS-TATS, UAI), one can notice that time series are often forecast using Gaussian processes (Petelin et al., 2011; Tobar et al., 2015; Hwang et al., 2016) , especially when time series are irregularly sampled (Cunningham et al., 2012; Li & Marlin, 2016) . Though still largely independent, researchers have started to "bring together the machine learning and econometrics communities" by building on top of their respective fundamental models yielding to, for example, the Gaussian Copula Process Volatility model (Wilson & Ghahramani, 2010) . Our paper is in line with this emerging trend by coupling AR models and neural networks.

Over the past 5 years, deep neural networks have surpassed results from most of the existing literature in many fields (Schmidhuber, 2015) : computer vision (Krizhevsky et al., 2012) , audio signal processing and speech recognition (Sak et al., 2014) , natural language processing (NLP) (Bengio et al., 2003; Collobert & Weston, 2008; Grave et al., 2016; Jozefowicz et al., 2016) . Although sequence modeling in NLP, i.e. prediction of the next character or word, is related to our forecasting problem (1), the nature of the sequences is too dissimilar to allow using the same cost functions and architectures. Same applies to the adversarial training proposed by Mathieu et al. (2016) for video frame prediciton, as such approach favors most plausible scenarios rather than outputs close to all possible outputs, while the latter is usually required in financial time series due to stochasticity of the considered processes.

Literature on deep learning for time series forecasting is still scarce (cf. Gamboa (2017) for a recent review). Literature on deep learning for financial time series forecasting is even scarcer though interest in using neural networks for financial predictions is not new (Mozer, 1993; McNelis, 2005) . More recent papers include Sirignano (2016) 

2.2. GATING AND WEIGHTING MECHANISMS:

Gating mechanisms for neural networks were first proposed by Hochreiter & Schmidhuber (1997) and proved essential in training recurrent architectures (Jozefowicz et al., 2016) due to their ability to overcome the problem of vanishing gradient. In general, they can be expressed as

EQUATION

where f is the output function, c is a 'candidate output' (usually a nonlinear function of x), ⊗ is an element-wise matrix product and σ : R → [0, 1] is a sigmoid nonlinearity that controls the amount of the output passed to the next layer (or to further operations within a layer). Appropriate compositions of functions of type 2 lead to the popular recurrent architectures such as LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Chung et al., 2014) .

A 

EQUATION

where (f l ) L l=1 are candidate outputs (composition operators in MuFuRu) and ( p l ) L l=1 are linear functions of the inputs.

The idea of weighting outputs of the intermediate layers within a neural networks is also used in attention networks (See e.g. Cho et al. (2015) ) that proved successful in such tasks as image captioning and machine translation. Our approach is similar as the separate inputs (time series steps) are weighted in accordance with learned functions of these inputs, yet different since we model these functions as multi-layer CNNs (instead of projections on learned directions) and since we do not use recurrent layers. The latter is important in the above mentioned tasks as it enables the network to remember the parts of the sentence/image already translated/described.

3. MOTIVATION:

Time series observed in irregular moments of time make up significant challenges for learning algorithms. Gaussian processes provide useful theoretical framework capable of handling asynchronous data; however, due to assumed Gaussianity they are inappropriate for financial datasets, which often follow fat-tailed distributions (Cont, 2001) . On the other hand, prediction of even a simple autoregressive time series such us AR(2) given by X(t) = αX(t -1) + βX(t -2) + ε(t)2 may involve highly nonlinear functions when sampled irregularly. Precisely, it can be shown that the conditional expectation

EQUATION

) where a k and b k are rational functions of α and β (See Appendix A for the proof). This would not be a problem if k was fixed, as then one would be interested in estimating of a k and b k directly; this, however, is not the case with asynchronous sampling. When X is an autoregressive series of higher order and more past observations are available, the analogous expectation E[X(t n )|{X(t n-m ), m = 1, . . . , M }] would involve more complicated functions that in general may not possess closed forms.

In real-world applications we often deal with multivariate time series whose dimensions are observed separately and asynchronously. This adds even more difficulty to assigning appropriate weights to the past values, even if the underlying data structure is linear. Furthermore, appropriate representation of such series might be not obvious as aligning such series at fixed frequency may lead to loss of information (if too low frequency is chosen) or prohibitive enlargement of the dataset (especially when durations have varying magnitudes), see Figure 2A . As an alternative, we might consider representing separate dimensions as a single one with dimension and duration indicators as additional features. Figure 2B presents this approach, which is going to be at the core of the proposed architecture. A natural model for prediction of such series could be an LSTM, which, given consecutive input values and respective durations (X(t n ), t n -t n-1 ) =: x n in each step would memorize the series values and weight them at the output according to the durations. However, the drawback of such approach lies in imbalance between the needs for memory and for nonlinearity: the weights that such network needs to assign to the memorized observations potentially require several layers of nonlinearity to be computed properly, while past observations might just need to be memorized as they are.

For these reasons we shall consider a model that combines simple autoregressive approach with neural network in order to allow learning meaningful data-dependent weights

E[x n |{x n-m , m = 1, . . . , M }] = = M m=1 α m (x n-m ) • x n-m (5)

where (α m ) M m=1 satisfying α 1 + • • • + α M ≤ 1 are modeled using neural network. To allow more flexibility and cover situations when e.g. observed values of x are biased, we should consider the summation over terms α m (x n-m ) • f (x n-m ), where f is also a neural network. We formalize this idea in Section 4.

4. MODEL ARCHITECTURE:

Suppose that we are given a multivariate time series (x n ) n ⊂ R d and we aim to predict the conditional future values of a subset of elements of x n

EQUATION

where

EQUATION

where

• F, S : R d×M → R d I ×M are neural networks described below,

• σ is a normalized activation function independent on each row, i.e.

σ((a T 1 , . . . , a T d I ) T ) = (σ(a 1 ) T , . . . , σ(a d I ) T ) T (8)

for any a 1 , . . . , a d I ∈ R M and σ such that σ(a) T 1 M = 1 for any a ∈ R M .

• ⊗ is Hadamard (element-wise) matrix multiplication.

The summation in 7 goes over the columns of the matrix in bracket; hence the i-th element of the output vector ŷn is a linear combination of the i-th row of the matrix F (x -M n ). We are going to consider S to be a fully convolutional network (composed solely of convolutional layers) and F of the form

F (x -M n ) = W ⊗ off(x n-m ) + x I n-m ) M m=1 (9)

where W ∈ R d I ×M and off : R d → R d I is a multilayer perceptron. In that case F can be seen as a sum of projection (x → x I ) and a convolutional network with all kernels of length 1. Equation ( 7) can be rewritten as

EQUATION

where W m , S m (•) are m-th columns of matrices W and S(•). We will call the proposed network a Significance-Offset Convolutional Neural Network (SOCNN), while off and S respectively the offset and significance (sub)networks.

The network scheme is shown in Figure 3 . Note that when off ≡ 0 and σ ≡ 1 the model simplifies to the collection of d I separate AR(M ) models for each dimension.

Interpretation of the components:

Note that the form of Equation ( 10) enforces the separation of temporal dependence (obtained in weights W m ), the local significance of observations S m (S as a convolutional network is determined by its filters which capture local dependencies and are independent on the relative position in time) and the predictors off(x n-m ) that are completely independent on position in time. This provides some amount of interpretability of the fitted functions and weights. For instance, each of the past observations provides an adjusted single regressor for the target variable through the offset network. Note that due to asynchronous sampling procedure, consecutive values of x might be heterogenous, hence On the other hand, significance network provides data-dependent weights for all regressors and sums them up in autoregressive manner. Figure 7 in Appendix E.2 shows sample significance and offset activations for the trained network.

Relation to asynchronous data:

As mentioned before, one of the common problems with time series are the varying durations between consecutive observations. A simple approach at data-preprocessing level is aligning the observations at some chosen frequency by e.g. duplicating or interpolating observations. This, however, might extend the size of an input and, therefore, model complexity.

The other idea is to treat the duration and/or time of the observation as another feature, as presented in Figure 2B . This approach is at the core of the SOCNN architecture: the significance network is aimed at learning the high-level features that indicate the relative importance of past observations, which, as shown in Section 3, could be predominantly dependent on time and duration between observations.

Loss function:

L 2 error is a natural loss function for the estimators of expected value

EQUATION

)

As mentioned above, the output of the offset network can be seen as a collection of separate predictors of the changes between corresponding observations x I n-m and the target variable

EQUATION

)

For that reason, we consider the auxiliary loss function equal to mean squared error of such intermediate predictions

EQUATION

)

The total loss for the sample (x -M n , y n ) is therefore given by

EQUATION

where y n is given by Eq. 10 and α ≥ 0 is a constant. In Section 5.2 we discuss the empirical findings on the impact of positive values of α on the model training and performance, as compared to α = 0 (lack of auxiliary loss).

5. EXPERIMENTS:

We evaluate the proposed model on a financial dataset of bid/ask quotes sent by several market participants active in the credit derivatives market, artificially generated datasets and household electric power consumption dataset available from UCI repository (Lichman, 2013) , comparing its performance with simple CNN, single-and multi-layer LSTM (Hochreiter & Schmidhuber, 1997) and 25-layer ResNet (He et al., 2015) .

Apart from performance evaluation of SOCNNs, we discuss the impact of the network components, such as auxiliary loss and the depth of the offset sub-network.

The details of the training process and hyperparameters used in the proposed architecture as well as in benchmark models are presented in C.

5.1. DATASETS:



Artificial data:

We test our network architecture on the artificially generated datasets of multivariate time series. We consider two types of series:

1. Synchronous series. The series of K noisy copies ('sources') of the same univariate autoregressive series ('base series'), observed together at random times. The noise of each copy is of different type.

2. Asynchronous series. The series of observations of one of the sources in the above dataset. At each time, the source is selected randomly and its value at this time is added to form a new univariate series. The final series is composed of this series, the durations between random times and the indicators of the 'available source' at each time.

The details of the simulation process are presented in Appendix D. We consider synchronous and asynchronous series X K×N where K ∈ {16, 64} is the number of sources and N = 10, 000, which gives 4 artificial series in total3 .

Electricity data:

The household electricity dataset4 contains measurements of 7 different quantities related to electricity consumption in a single household, recorded every minute for 47 months, yielding over 2 million observations. Since we aim to focus on asynchronous time-series, we alter it so that a single observation contains only value of one of the seven features, while durations between consecutive observations range from 1 to 7 minutes5 .The regression aim is to predict all of the features at the next time step.

Non-anonymous quotes:

The proposed model was designed primarily for forecasting incoming non-anonymous quotes received from the credit default swap market. The dataset contains 2.1 million quotes from 28 different sources, i.e. market participants. Each quote is characterized by 31 features: the offered price, 28 indicators of the quoting source, the direction indicator (the quote refers to either a buy or a sell offer) and duration from the previous quote. For each source and direction we aim at predicting the next quoted price from this given source and direction considering the last 60 quotes. We formed 15 separate prediction tasks; in each task the model was trained to predict the next quote by one of the fifteen most active market participants6 .

This dataset, which is proprietary, motivated the aforementioned construction of artificial asynchronous time series datasets based on its statistical features for reproducible research purpose.

5.2. RESULTS:

Table 1 presents the detailed results from the artificial and electricity datasets. The proposed networks outperform significantly the benchmark networks on the asynchronous, electricity and quotes datasets. For the synchronous datasets, on the other hand, SOCNN almost matches the results of the benchmarks. This similar performance could have been anticipated -the correct weights of the past values in synchronous artificial datasets are far less nonlinear than in case when separate dimensions are observed asynchronously. For this reason, the significance network's potential is not fully utilized. We can also observe that the depth of the offset network has negligible or negative For Quotes dataset, the proposed model was the best one for all 15 tasks and the only one to always beat VAR model. Surprisingly, for each of the other networks it was difficult to excel the benchmark set by simple linear model. We also found benchmark networks to have very unstable test loss during training in some cases, despite convergence of the training error. Especially, for one of the tasks LSTM and ResNet obtained very high test errors7 . The auxiliary loss was usually found to have minor importance, though in some cases it led to best results.

The small positive auxiliary weight helped achieve more stable test error throughout training in many cases. The higher weights of auxiliary loss considerably improved the test error on asynchronous datasets (See Table 2 ); however for other datasets its impact was negligible. In general, the proposed SOCNN had significantly lower variance of the test and validation errors, especially in the early stage of the training process and for quotes dataset. Figure 4 presents the learning curves for two different artificial datasets.

Model robustness:

To understand better why SOCNN obtained better results than the other networks, we check how these networks react to the presence of additional noise in the input terms8 . Figure 5 presents changes in the mean squared error and significance and offset network outputs with respect to the level of noise. SOCNN is the most robust out of the compared networks and, together with singlelayer LSTM, least prone to overfitting. Despite the use of dropout and cross-validation, the other models tend to overfit the training set and have non-symmetric error curves on test dataset. dataset. The plots show how the error would change if an additional noise term was added to the input series. The dotted curves show the total significance and average absolute offset (not to scale) outputs for the noisy observations. Interestingly, significance of the noisy observations increases with the magnitude of noise; i.e. noisy observations are far from being discarded by SOCNN.

6. CONCLUSION AND DISCUSSION:

In this article, we proposed a weighting mechanism that, coupled with convolutional networks, forms a new neural network architecture for time series prediction. The proposed architecture is designed for regression tasks on asynchronous signals in the presence of high amount of noise. This approach has proved to be successful in forecasting financial and artificially generated asynchronous time series outperforming popular convolutional and recurrent networks.

The proposed model can be further extended by adding intermediate weighting layers of the same type in the network structure. Another possible generalization that requires further empirical studies can be obtained by leaving the assumption of independent offset values for each past observation, i.e. considering not only 1x1 convolutional kernels in the offset sub-network.

Finally, we aim at testing the performance of the proposed architecture on other real-life datasets with relevant characteristics. We observe that there exists a strong need for common 'econometric' datasets benchmark and, more generally, for time series (stochastic processes) regression.

Proof. The proof follows a simple induction. It is sufficient to show that

EQUATION

where

EQUATION

and E k (t) is a linear combination of {ε(t -i), i = 0, 1, . . . , k -2}. Basis of the induction is trivially satisfied via 15. In the induction step, we assume that 17 holds for k. For t > k + 1 we have

w k X(t -1) = v k X(t -2) + b k-1 X(t -k -1) + E k (t -1).

Multiplying sides of this equation by b and adding av k X(t -1) we obtain

EQUATION

)

Since aX(t -1) + bX(t -2) = X(t) -ε(t) we get

EQUATION

As E k+1 (t) = bE k (t -1) -v k ε(t) is a linear combination of {ε(t -i), i = 0, 1, . . . , k -1}, the above equation proves 17 for k = k + 1.

C.2 BENCHMARK NETWORKS:

We compare the performance of the proposed model with CNN, ResNet, multi-layer LSTM networks and linear (VAR) model. The benchmark networks were designed so that they have a comparable number of parameters as the proposed model. Consequently, LeakyReLU activation function ( 23) with leak rate .1 was used in all layers except the top ones where linear activation was applied.

For CNN we provided the same number of layers, same stride (1) and similar kernel size structure. In each trained CNN, we applied max pooling with the pool size of 2 every two convolutional layers10 . Table 3 presents the configurations of the network hyperparameters used in comparison. The training and validation sets were sampled randomly from the first 80% of timesteps in each series, with ratio 3 to 1. The remaining 20% of data was used as a test set. All models were trained using Adam optimizer (Kingma & Ba, 2015) which we found much faster than standard Stochastic Gradient Descent in early tests. We used batch size of 128 for artificial data and 256 for quotes dataset. We also applied batch normalization (Ioffe & Szegedy, 2015) in between each convolution and the following activation. At the beginning of each epoch, the training samples were shuffled.

To prevent overfitting we applied dropout and early stopping12 . Weights were initialized following the normalized uniform procedure proposed by Glorot & Bengio (2010) . Experiments were carried out using implementation relying on Tensorflow (Abadi et al., 2016) and Keras front end (Chollet, 2015) . For artificial data we optimized the models using one K20s NVIDIA GPU while for quotes dataset we used 8-core Intel Core i7-6700 CPU machine only.

APPENDIX D ARTIFICIAL DATA GENERATION:

We simulate a multivariate time series composed of K noisy observations of the same autoregressive signal. The simulated series are constructed as follows:

1. We simulate univariate stationary AR(10) time series x with randomly chosen weights.

2. The series is copied K times and each copy x (k) is associated with a separate noise process ε (k) . We consider Gaussian or Binomial noise of different scales; for each copy it is either added to or multiplied by the initial series (x

(k) = x + ε (k) or x (k) = x × ε (k) ).

3. We simulate a random time process T where differences between consecutive events are iid exponential random variables.

4a. The final series is composed of K noisy copies of the original process observed at times indicated by the random time process, and a duration between observations. 4b. At each time T (t) indicated by the random time process T , one of the noisy copies k is drawn and its value at this time x T (t) is selected to form a new noisy series x * . The final multivariate series is composed of x * , the series of durations between observations and K indicators of which observation was drawn at each time.

Assume that (x t ) t=1,2,... is a stationary AR(ν) series and consider the following (random) noise functions ε 0 (x, c, p) = x + c(2 -1), ∼ Bernoulli(p), ε 1 (x, c, p) = x(1 + c(2 -1)), ∼ Bernoulli(p), ε 2 (x, c, p) = x + c , ∼ N (0, 1), ε 3 (x, c, p) = x(1 + c ), ∼ N (0, 1).

(24)

Note that argument p of ε 2 and ε 3 is redundant and was added just for notational convenience.

Let N t ∼ Exp(λ) be a series of i.i.d. exponential random variables with some constant rate λ and let T (t) = t s=1 N s + 1 . Then T (t) is a strictly increasing series of times, at which we will observe the noisy observations. Let p 1 , p 2 , . . . , p K ∈ (0, 1) and define 

:

iTraxx Europe Main Index, a tradable Credit Default Swap index of 125 investment grade rated European entities.

Where ε(t) is an error term independent of {X(s) : s < t}.

Note that a series with K sources is K + 1-dimensional in synchronous case and K + 2-dimensional in asynchronous case. The base series in all processes was a stationary AR(10) series. Although that series has the true order of 10, in the experimental setting the input data included past 60 observations. The rationale behind that is twofold: not only is the data observed in irregular random times but also in real-life problems the order of the model is unknown. Figure6(available in Appendix D) presents samples from the two of the simulated series.

Available at UCI Machine Learning Repository website https://archive.ics.uci.edu/ml/datasets/individual+household+electric+power+consumption

The exact details of preprocessing can be found in Appendix E.1.

This separation is related to data normalization purposes and different magnitudes of the levels of predictability for different market participants. The quotes from the remaining 13 participants were not selected for prediction as their market presence was too short or too irregular to form reliable training, validation and test samples.

The exact results for all tasks for Quotes dataset can be found in Appendix F.

The details of the added noise terms are presented in the Appendix B.

The asynchronous series has one dimension representing the value of the quote, one representing duration and others representing indicators of the source. See ?? for details.

Hence layers 3, 6 and 9 were pooling layers, while layers 1, 2, 4, 5, . . . were convolutional layers.

We found LSTMs very unstable on quotes dataset without gradient clipping or with higher clipping boundary.

12 Whenever 10 consecutive epochs did not bring improvement in the validation error, the learning rate was reduced by a factor of 10 and the best weights obtained till then were restored. After the second reduction and another 10 consecutive epochs without improvement, the training was stopped. The initial learning rate was set to .001.

Figures and Tables:

Figure 2: (A) Fixed sampling frequency and it's drawbacks; keeping all available information leads to much more datapoints. (B) Proposed data representation for the asynchronous series. Consecutive observations are stored together as a single value series, regardless of which series they belong to; this information, however, is stored in indicator features, alongside durations between observations.

Figure 3: A scheme of the proposed SOCNN architecture. The network preserves the timedimension up to the top layer, while the number of features per timestep (filters) in the hidden layers is custom. The last convolutional layer, however, has the number of filters equal to dimension of the output. The Weighting frame shows how outputs from offset and significance networks are combined in accordance with Eq. 10.

Figure 4: Learning curves for CNNs and SOCNNs with different auxiliary weights for two datasets. The solid lines indicate the test error while the dashed lines indicate the training error. Note the different scales on the horizontal axes.

Figure5: Experiment comparing robustness of the considered networks for Asynchronous 16 dataset. The plots show how the error would change if an additional noise term was added to the input series. The dotted curves show the total significance and average absolute offset (not to scale) outputs for the noisy observations. Interestingly, significance of the noisy observations increases with the magnitude of noise; i.e. noisy observations are far from being discarded by SOCNN.

Figure6: Simulated synchronous (left) and asynchronous (right) artificial series. Note the different durations between the observations from different sources in the latter plot. For clarity, we present only 6 out of 16 total dimensions. K) ∝ q K for some q > 0. Define

Detailed results for all datasets. For each model, we present the mean squared error obtained on the out-of-sample test set. The best results for each dataset are marked by bold font. SOCNN1 (SOCNN1+) denote proposed models with one (10 or 7) offset sub-network layers. For quotes dataset the presented values are averaged mean-squared errors from 6 separate prediction tasks, normalized according to the error obtained by VAR model.

Table content: <table><tr><td>model</td><td colspan="5">VAR CNN ResNet LSTM SOCNN1 SOCNN1+</td></tr><tr><td>Synchronous 16</td><td>0.841 0.152</td><td>0.150</td><td>0.152</td><td>0.154</td><td>0.173</td></tr><tr><td>Synchronous 64</td><td>0.364 0.028</td><td>0.028</td><td>0.028</td><td>0.029</td><td>0.031</td></tr><tr><td colspan="2">Asynchronous 16 0.577 0.040</td><td>0.032</td><td>0.027</td><td>0.017</td><td>0.020</td></tr><tr><td colspan="2">Asynchronous 64 0.318 0.041</td><td>0.046</td><td>0.050</td><td>0.032</td><td>0.034</td></tr><tr><td>Electricity</td><td>0.729 0.366</td><td>0.359</td><td>0.463</td><td>0.158</td><td>0.158</td></tr><tr><td>Quotes</td><td>1.000 0.891</td><td>2.245</td><td>0.841</td><td>0.374</td><td>-</td></tr><tr><td colspan="6">impact on the results achieved by the SOCNN network. This means that the significance network</td></tr><tr><td colspan="6">has crucial impact on the performance, which is in-line with the potential drawbacks of the LSTM</td></tr><tr><td colspan="6">network discussed in Section 3: obtaining proper weights for the past observations is much more</td></tr><tr><td colspan="4">challenging than getting good predictors from the single past values.</td><td/><td/></tr></table>

MSE for different values of α for two artificial datasets.

Table content: <table><tr><td/><td/><td/><td>HOHFWULFLW\GDWD</td><td colspan="2">DV\QFKURQRXVDULWILFLDOGLP</td></tr><tr><td/><td/><td/><td>&amp;11</td><td/><td>&amp;11</td></tr><tr><td/><td/><td/><td>/670</td><td/><td>/670</td></tr><tr><td/><td/><td>PHDQVTXDUHGHUURU</td><td>5HV1HW 62&amp;11 = 0 62&amp;11 = 0.01</td><td>PHDQVTXDUHGHUURU</td><td>5HV1HW 62&amp;11 = 0 62&amp;11 = 0.01</td></tr><tr><td>α</td><td colspan="2">Async 16 async 64</td><td/><td/></tr><tr><td>0.0</td><td>0.0284</td><td>0.0624</td><td>WLPHVHFRQGV</td><td>HSRFK</td></tr><tr><td>0.01</td><td>0.0253</td><td>0.0434</td><td/><td/></tr><tr><td>0.1</td><td>0.0172</td><td>0.0323</td><td/><td/></tr></table>

Configurations of the trained models. f -number of convolutional filters/memory cell size in LSTM, ks -kernel size, p -dropout rate, clip -gradient clipping threshold, conv -(k × 1) convolution with kernel size k indicated in the ks column, conv1 -(1 × 1) convolution. Apart from the listed layers, each network has a single fully connected layer on the top.

Table content: <table><tr><td>Kernel sizes (3, 1) ((1,</td></tr></table>

Detailed results for each prediction task for the quotes dataset. Each task involves prediction of the next quote by one of the banks. Numbers represent the mean squared errors on out-of-sample test set.

Table content: <table><tr><td>task</td><td colspan="4">CNN VAR LSTM ResNet SOCNN</td></tr><tr><td colspan="2">bank A 0.993 1.123</td><td>0.999</td><td>1.086</td><td>0.530</td></tr><tr><td colspan="2">bank B 1.225 2.116</td><td colspan="2">1.673 31.598</td><td>0.613</td></tr><tr><td colspan="2">bank C 3.208 3.952</td><td>2.957</td><td>3.805</td><td>0.617</td></tr><tr><td colspan="2">bank D 3.634 4.134</td><td>3.436</td><td>4.635</td><td>0.649</td></tr><tr><td colspan="2">bank E 3.558 4.367</td><td>3.344</td><td>3.717</td><td>1.154</td></tr><tr><td>bank F</td><td>8.541 8.150</td><td>7.880</td><td>8.274</td><td>1.553</td></tr><tr><td colspan="2">bank G 0.248 0.278</td><td>0.132</td><td>1.462</td><td>0.063</td></tr><tr><td>bank I</td><td>4.777 4.853</td><td>3.933</td><td>4.936</td><td>0.400</td></tr><tr><td>bank J</td><td>1.094 1.172</td><td>1.097</td><td>1.216</td><td>0.773</td></tr><tr><td colspan="2">bank K 2.521 4.307</td><td>2.573</td><td>4.731</td><td>0.926</td></tr><tr><td colspan="2">bank L 1.108 1.448</td><td>1.186</td><td>1.312</td><td>0.743</td></tr><tr><td colspan="2">bank M 1.743 1.816</td><td>1.741</td><td>1.808</td><td>1.271</td></tr><tr><td colspan="2">bank N 3.058 3.232</td><td>2.943</td><td>3.229</td><td>1.509</td></tr><tr><td colspan="2">bank O 0.539 0.532</td><td>0.420</td><td>0.566</td><td>0.218</td></tr><tr><td>bank P</td><td>0.447 0.354</td><td>0.470</td><td>0.510</td><td>0.283</td></tr></table>

Peer review (iclr2018):
 The author proposed: 1. A data augmentation technique for asynchronous time series data. 2. A convolutional *Significance* weighting neural network that assigns normalised weights to the outputs of a fully-connected autoregressive *Offset* neural network, such that the output is a weighted average of the *Offset* neural net. 3. An *auxiliary* loss function. The experiments showed that: 1. The proposed method beat VAR/CNN/ResNet/LSTM 2 synthetic asynchronous data sets, 1 real electricity meter data set and 1 real financial bid/ask data set. It*s not immediately clear how hyper-parameters for the benchmark models were chosen. 2. The author observed from the experiments that the depth of the offset network has negligible effect, and concluded that the *Significance* network has crucial impact. (I don*t see how this conclusion can be made.) 3. The proposed auxiliary loss is not useful. 4. The proposed architecture is more robust to noise in the synthetic data set compared to the benchmarks, and together with LSTM, are least prone to overfitting. Pros - Proposed a useful way of augmenting asynchronous multivariate time series for fitting autoregressive models - The convolutional Significance/weighting networks appears to reduce test errors (not entirely clear) Cons - The novelties aren*t very well-justified. The *Significance* network was described as critical to the performance, but there is no experimental result to show the sensitivity of the model*s performance with respect to the architecture of the *Significance* network. At the very least, I*d like to see what happens if the weighting was forced to be uniform while keeping the *Offset* network and loss unchanged. - It*s entirely unclear how the train and test data was split. This may be quite important in the case of the financial data set. - It*s also unclear if model training was done on a rolling basis, which is common for time series forecasting. - The auxiliary loss function does not appear to be very helpful, but was described as a key component in the paper. Quality: The quality of the paper was okay. More details of the experiments should be included in the main text to help interpret the significance of the experimental results. The experiment also did not really probe the significance of the *Significance* network even though it*s claimed to be important. Clarity: Above average. Originality: Mediocre. Nothing really shines. Weighted average-type architecture has been proposed many times in neural networks (e.g., attention mechanisms). Significance: Low. It*s unclear how useful the architecture really is.