# Improved ACD-Based Financial Trade Durations Prediction Leveraging LSTM Networks and Attention Mechanism

Authors: Yong Shi; Wei Dai; Wen Long; Bo Li
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

The liquidity risk factor of security market plays an important role in the formulation of trading strategies. A more liquid stock market means that the securities can be bought or sold more easily. As a sound indicator of market liquidity, the transaction duration is the focus of this study. We concentrate on estimating the probability density functionpΔti+1|Gi, where Δti+1 represents the duration of the (i + 1)-th transaction and Gi represents the historical information at the time when the (i + 1)-th transaction occurs. In this paper, we propose a new ultrahigh-frequency (UHF) duration modelling framework by utilizing long short-term memory (LSTM) networks to extend the conditional mean equation of classic autoregressive conditional duration (ACD) model while retaining the probabilistic inference ability. And then, the attention mechanism is leveraged to unveil the internal mechanism of the constructed model. In order to minimize the impact of manual parameter tuning, we adopt fixed hyperparameters during the training process. The experiments applied to a large-scale dataset prove the superiority of the proposed hybrid models. In the input sequence, the temporal positions which are more important for predicting the next duration can be efficiently highlighted via the added attention mechanism layer.


---

## Body

## 1. Introduction

Market liquidity refers to the degree to which an asset can be bought and sold easily for a fair price [1]. In other words, market liquidity can be regarded as the speed at which transactions can be concluded while maintaining a basically stable price [1]. Therefore, market liquidity risk is one of the most common factors considered by security investors especially by high-frequency traders in building a trading strategy.With the rapid development of computer storage technology, transaction by transaction financial trading data is accessible to researchers. Letti stand for the time at which the i−thtrade occurs so that the duration between the (i + 1)-th and i-th trade is Δti+1=ti+1−ti, which can directly measure the transaction speed of financial trading. The autoregressive conditional duration (ACD) model proposed by Engle and Russell has been the primary framework used for analyzing trading durations of ultrahigh-frequency (UHF) data, which are irregularly time-spaced and convey meaningful information [2]. In ACD models, the transaction duration is decomposed into the multiplicative product of two components: the conditional (expected) duration and the unexpected duration. The expected component is the portion of transaction duration that is linearly conditional on past durations, whereas the unexpected duration is the fraction of duration beyond that which could be predicted from past durations and is usually characterized by an exponential distribution.Based on the work of Engle and Russell [2], many works tried to improve the ability of capturing the relation between the conditional duration and the lagged durations. For example, the logarithmic version of ACD model was provided in [3], the threshold autoregressive conditional duration model was proposed in [4], the asymmetric autoregressive conditional duration model was put forward in [5], and the smooth transition ACD model and the time-varying ACD model were introduced in [6]. There are also many other works focusing on choosing a suitable distribution to characterize the unexpected duration. The distributions which have been applied to the ACD models include the generalized Gamma distribution in [7], generalized F distribution in [8], the mixture of two exponential distributions in [9], the regime-switching Pareto distribution in [10], and the mixture of an exponential and a generalized beta of type 2 (GB2) distribution in [11]. Like many other statistical models, the ACD family models require strong assumptions which are difficult to satisfy in realistic situations [12].In recent years, machine learning methods have been widely applied to image identification and natural language processing problems. Compared with traditional statistical models, machine learning methods have looser model assumptions and better generalization ability. The artificial neural network (ANN), inspired by the biological neural network, is one of the most widely used machine learning methods. According to Universal Approximation theorem [13], feedforward neural networks can approximate a Borel measurable function to any desired degree of accuracy if sufficiently many hidden units with arbitrary squashing functions are provided. Recurrent neural networks (RNNs) are a family of specially designed artificial neural network networks capable of extracting temporal information via the cycle architecture [14]. For the development in optimization techniques and computation hardware, RNNs have been widely used in many different domains recently [15]. To solve the vanishing/exploding gradient problem of simple RNNs, Hochreiter S. proposed the long short-term memory (LSTM) neural networks which can help us to utilize a longer sequence of historical information [16]. Although having the merit of strong fitting ability, LSTMs cannot provide probabilistic output compared with ACD family models.Inspired by the work from Kristjanpoller and Minutolo [17], we propose a new architecture called LSTM-ACD to predict the UHF transaction durations by combing the ANN networks and ACD framework. We take a fully data-driven approach to extend the mean equation of classic ACD models while retaining the probabilistic inference ability. In addition, attention layer is added into our model to make a visualization of the proposed network and to improve the interpretability. The proposed architecture is applied to real-world stock duration datasets. The result shows that the proposed model produces more accurate estimation and prediction, outperforming the traditional ACD family models.The rest of this paper is organized as follows: Section2 introduces the methodology in detail, while Section 3 contains the experiment design and the corresponding results in this study. Section 4 concludes this paper and points out the possible direction of future research.


## 2. Methodology

In Section2, the ACD framework is integrated with LSTM networks to propose a new LSTM-ACD model for predicting the trading durations of UHF data. This section is organized as follows. Section 2.1 introduces the classic ACD model. Section 2.2 describes the proposed LSTM-ACD architecture in detail. In addition the attention mechanism layer is utilized to unveil the internal mechanism of the proposed model.


### 2.1. Traditional ACD Family Models

A classic ACD model assumes that the durations are conditionally exponentially distributed with a mean that follows an ARMA process [2]. As shown in (1), the duration Δti between the i-th and (i − 1)-th trade is the multiplicative product of μi and εi, which represents expected and unexpected portion of the transaction duration, respectively. In the conditional mean equation, μi linearly depends on the lagged durations and the lagged terms of itself. pandq in formula (2) represent the lagged order:(1)Δti=μiεi,(2)μi=ω+∑j=1pαjΔti−j+∑j=1qβjμi−j.By adding the lagged terms of error termεi as the independent variables, Hautsch proposed the Additive and Multiplicative ACD (AMACD) [18] model, as follows:(3)μi=ω+∑j=1pαjxi−j+∑j=1rνjεi−j+∑j=1qβjμi−j.A major limitation of basic ACD model and AMACD model is the assumption that the variables in the conditional mean equation behave in strict stationarity and linearity, but the duration sequences are usually in a nonlinear or nonstationary state. Based on power transformation, a nonlinear Box–Cox ACD model was proposed in [19], as follows:(4)μiδ1=ω+∑j=1pαjεi−jδ2+∑j=1rβjμi−jδ1.This paper also extends the linear conditional mean equation by LSTM networks to propose a new LSTM-ACD framework due to the strong fitting ability of deep learning techniques.


### 2.2. The Proposed Attention-LSTM-ACD Model

#### 2.2.1. LSTM-ACD Model

It has been generally known that the LSTM cell is able to store information over longer time range compared with simple RNNs. As depicted in Figure1, the information flow propagating across timesteps is controlled by three LSTM gates: the forget gate, the input gate, and the output gate.Figure 1
Structure of LSTM cell.Assuming thatWf, Wi, Wo, and Wc represent the LSTM weight matrices, and bf, bi, bc, and bo represent the bias vectors. The input vector, output vector, and cell state vector at time t are denoted as xt,ht,andCt, respectively [20]. The operating process of a LSTM cell can be mathematically described as follows:(5)forget gate:ft=σWf⋅ht−1,xt+bf,input gate:it=σWi⋅ht−1,xt+bi,outputgate:ot=σWo⋅ht−1,xt+bo,candidatestate values calculation:C˜t=tanhWC⋅ht−1,xt+bC,Ct=ft×Ct−1+it×C˜t,ht=ot⋅tanhCt.As a type of RNNs specially designed to avoid the exponentially fast decaying factor, the LSTM networks can effectively prevent the gradient vanishing/explosion problem. Due to their ability to learn long-term dependencies, LSTMs are particularly suitable for financial prediction problems. Hence, we have the conjecture that extending the linear mean equation to LSTM network will improve the ability of extracting long-term dependencies for duration sequence. To verify this hypothesis, we take theΔti−1 and lnμ^i−1 as the input for the LSTM cell at the time point of i-th transaction where Δti−1 is the duration of last transaction and lnμ^i−1 is the logarithmic value of the output of the proposed LSTM-ACD model at time i−1. To retain the ability of probabilistic inference, the objective function is still the log likelihood function of Δti=μiεi which follows an exponential distribution. The log-likelihood function can be mathematically described as follows:(6)l=∑​ln1μ^iexp−1μ^iΔti,(7)lnμ^i=φΔti−1,lnμ^i−1,hi−1,where φ represents a mapping from Δti−1,lnμ^i−1,hi−1 to lnμ^i by a LSTM cell.


#### 2.2.2. Visualization and Promotion by Attention Mechanism

Attention mechanism was firstly proposed to improve the image processing accuracy by mimicking the perceptual system of human beings [21]. In the work of [22], attention mechanism was introduced to extend the basic encoder-decoder architecture and enhance the interpretability on the task of machine translation. Unlike the sequence-to-sequence modelling in sentence translation, the problem we focus on in this paper is to predict the financial duration one-step ahead. The attention weights which help automatically search for import hidden states of the sequence-to-one LSTM architecture can be calculated by the following formulas:(8)ei−k=vαTtanhwαhi−k,αi−k=expei−k∑k=1Texpei−k,where hi−k represents the hidden state lagged k timesteps and αi−k represents the attention weight of hi−k. The wα and vα are parameter matrices in the attention mechanism. By allocating different attention weights for different hidden states, a new vector ci is produced as the input of a feedforward network f for predicting the target variable yi:(9)ci=∑k=1Tαkhi−k,yi=fci.In this study, the attention layer is integrated with LSTM to characterize the dynamics oflnμi in the abovementioned mean equation of ACD model. The proposed Attention-LSTM-ACD model can be described by the following equations:(10)hi−k′=LSTMhi−k−1′,si−k−1′,Δti−k−1,lnμ^i−k−1,ci′=∑k=1T′αk′hi−k′,lnμ^i=f′ci′,where si−k−1′ represents the cell state of LSTM lagged k+1 timesteps. Figure 2 shows the Attention-LSTM-ACD model in more detail.Figure 2
Architecture of Attention-LSTM-ACD model.


## 2.1. Traditional ACD Family Models

A classic ACD model assumes that the durations are conditionally exponentially distributed with a mean that follows an ARMA process [2]. As shown in (1), the duration Δti between the i-th and (i − 1)-th trade is the multiplicative product of μi and εi, which represents expected and unexpected portion of the transaction duration, respectively. In the conditional mean equation, μi linearly depends on the lagged durations and the lagged terms of itself. pandq in formula (2) represent the lagged order:(1)Δti=μiεi,(2)μi=ω+∑j=1pαjΔti−j+∑j=1qβjμi−j.By adding the lagged terms of error termεi as the independent variables, Hautsch proposed the Additive and Multiplicative ACD (AMACD) [18] model, as follows:(3)μi=ω+∑j=1pαjxi−j+∑j=1rνjεi−j+∑j=1qβjμi−j.A major limitation of basic ACD model and AMACD model is the assumption that the variables in the conditional mean equation behave in strict stationarity and linearity, but the duration sequences are usually in a nonlinear or nonstationary state. Based on power transformation, a nonlinear Box–Cox ACD model was proposed in [19], as follows:(4)μiδ1=ω+∑j=1pαjεi−jδ2+∑j=1rβjμi−jδ1.This paper also extends the linear conditional mean equation by LSTM networks to propose a new LSTM-ACD framework due to the strong fitting ability of deep learning techniques.


## 2.2. The Proposed Attention-LSTM-ACD Model

### 2.2.1. LSTM-ACD Model

It has been generally known that the LSTM cell is able to store information over longer time range compared with simple RNNs. As depicted in Figure1, the information flow propagating across timesteps is controlled by three LSTM gates: the forget gate, the input gate, and the output gate.Figure 1
Structure of LSTM cell.Assuming thatWf, Wi, Wo, and Wc represent the LSTM weight matrices, and bf, bi, bc, and bo represent the bias vectors. The input vector, output vector, and cell state vector at time t are denoted as xt,ht,andCt, respectively [20]. The operating process of a LSTM cell can be mathematically described as follows:(5)forget gate:ft=σWf⋅ht−1,xt+bf,input gate:it=σWi⋅ht−1,xt+bi,outputgate:ot=σWo⋅ht−1,xt+bo,candidatestate values calculation:C˜t=tanhWC⋅ht−1,xt+bC,Ct=ft×Ct−1+it×C˜t,ht=ot⋅tanhCt.As a type of RNNs specially designed to avoid the exponentially fast decaying factor, the LSTM networks can effectively prevent the gradient vanishing/explosion problem. Due to their ability to learn long-term dependencies, LSTMs are particularly suitable for financial prediction problems. Hence, we have the conjecture that extending the linear mean equation to LSTM network will improve the ability of extracting long-term dependencies for duration sequence. To verify this hypothesis, we take theΔti−1 and lnμ^i−1 as the input for the LSTM cell at the time point of i-th transaction where Δti−1 is the duration of last transaction and lnμ^i−1 is the logarithmic value of the output of the proposed LSTM-ACD model at time i−1. To retain the ability of probabilistic inference, the objective function is still the log likelihood function of Δti=μiεi which follows an exponential distribution. The log-likelihood function can be mathematically described as follows:(6)l=∑​ln1μ^iexp−1μ^iΔti,(7)lnμ^i=φΔti−1,lnμ^i−1,hi−1,where φ represents a mapping from Δti−1,lnμ^i−1,hi−1 to lnμ^i by a LSTM cell.


### 2.2.2. Visualization and Promotion by Attention Mechanism

Attention mechanism was firstly proposed to improve the image processing accuracy by mimicking the perceptual system of human beings [21]. In the work of [22], attention mechanism was introduced to extend the basic encoder-decoder architecture and enhance the interpretability on the task of machine translation. Unlike the sequence-to-sequence modelling in sentence translation, the problem we focus on in this paper is to predict the financial duration one-step ahead. The attention weights which help automatically search for import hidden states of the sequence-to-one LSTM architecture can be calculated by the following formulas:(8)ei−k=vαTtanhwαhi−k,αi−k=expei−k∑k=1Texpei−k,where hi−k represents the hidden state lagged k timesteps and αi−k represents the attention weight of hi−k. The wα and vα are parameter matrices in the attention mechanism. By allocating different attention weights for different hidden states, a new vector ci is produced as the input of a feedforward network f for predicting the target variable yi:(9)ci=∑k=1Tαkhi−k,yi=fci.In this study, the attention layer is integrated with LSTM to characterize the dynamics oflnμi in the abovementioned mean equation of ACD model. The proposed Attention-LSTM-ACD model can be described by the following equations:(10)hi−k′=LSTMhi−k−1′,si−k−1′,Δti−k−1,lnμ^i−k−1,ci′=∑k=1T′αk′hi−k′,lnμ^i=f′ci′,where si−k−1′ represents the cell state of LSTM lagged k+1 timesteps. Figure 2 shows the Attention-LSTM-ACD model in more detail.Figure 2
Architecture of Attention-LSTM-ACD model.


## 2.2.1. LSTM-ACD Model

It has been generally known that the LSTM cell is able to store information over longer time range compared with simple RNNs. As depicted in Figure1, the information flow propagating across timesteps is controlled by three LSTM gates: the forget gate, the input gate, and the output gate.Figure 1
Structure of LSTM cell.Assuming thatWf, Wi, Wo, and Wc represent the LSTM weight matrices, and bf, bi, bc, and bo represent the bias vectors. The input vector, output vector, and cell state vector at time t are denoted as xt,ht,andCt, respectively [20]. The operating process of a LSTM cell can be mathematically described as follows:(5)forget gate:ft=σWf⋅ht−1,xt+bf,input gate:it=σWi⋅ht−1,xt+bi,outputgate:ot=σWo⋅ht−1,xt+bo,candidatestate values calculation:C˜t=tanhWC⋅ht−1,xt+bC,Ct=ft×Ct−1+it×C˜t,ht=ot⋅tanhCt.As a type of RNNs specially designed to avoid the exponentially fast decaying factor, the LSTM networks can effectively prevent the gradient vanishing/explosion problem. Due to their ability to learn long-term dependencies, LSTMs are particularly suitable for financial prediction problems. Hence, we have the conjecture that extending the linear mean equation to LSTM network will improve the ability of extracting long-term dependencies for duration sequence. To verify this hypothesis, we take theΔti−1 and lnμ^i−1 as the input for the LSTM cell at the time point of i-th transaction where Δti−1 is the duration of last transaction and lnμ^i−1 is the logarithmic value of the output of the proposed LSTM-ACD model at time i−1. To retain the ability of probabilistic inference, the objective function is still the log likelihood function of Δti=μiεi which follows an exponential distribution. The log-likelihood function can be mathematically described as follows:(6)l=∑​ln1μ^iexp−1μ^iΔti,(7)lnμ^i=φΔti−1,lnμ^i−1,hi−1,where φ represents a mapping from Δti−1,lnμ^i−1,hi−1 to lnμ^i by a LSTM cell.


## 2.2.2. Visualization and Promotion by Attention Mechanism

Attention mechanism was firstly proposed to improve the image processing accuracy by mimicking the perceptual system of human beings [21]. In the work of [22], attention mechanism was introduced to extend the basic encoder-decoder architecture and enhance the interpretability on the task of machine translation. Unlike the sequence-to-sequence modelling in sentence translation, the problem we focus on in this paper is to predict the financial duration one-step ahead. The attention weights which help automatically search for import hidden states of the sequence-to-one LSTM architecture can be calculated by the following formulas:(8)ei−k=vαTtanhwαhi−k,αi−k=expei−k∑k=1Texpei−k,where hi−k represents the hidden state lagged k timesteps and αi−k represents the attention weight of hi−k. The wα and vα are parameter matrices in the attention mechanism. By allocating different attention weights for different hidden states, a new vector ci is produced as the input of a feedforward network f for predicting the target variable yi:(9)ci=∑k=1Tαkhi−k,yi=fci.In this study, the attention layer is integrated with LSTM to characterize the dynamics oflnμi in the abovementioned mean equation of ACD model. The proposed Attention-LSTM-ACD model can be described by the following equations:(10)hi−k′=LSTMhi−k−1′,si−k−1′,Δti−k−1,lnμ^i−k−1,ci′=∑k=1T′αk′hi−k′,lnμ^i=f′ci′,where si−k−1′ represents the cell state of LSTM lagged k+1 timesteps. Figure 2 shows the Attention-LSTM-ACD model in more detail.Figure 2
Architecture of Attention-LSTM-ACD model.


## 3. Experiment

### 3.1. Data Description

#### 3.1.1. Data Source

The Shenzhen Stock Exchange 100 Index (SZSE 100) is the first index designed for reflecting the multiple level market conditions of Chinese stock market. The constituent stocks of SZSE 100 represent the core high-quality assets in the Shenzhen A-share market, with strong growth, low valuation, and high investment value. In this paper, we collect duration data of the first 100,000 transactions which has excluded the transactions during premarket opening session, for each constituent stock from SZSE 100. The readers can acquire the data from the Transend DataBase System of Wind Information Co., Ltd (https://www.wind.com.cn/). Since the stock Tianjin Zhonghuan Semiconductor Co., Ltd., which is coding in 002129.SZ has no transactions during 2017, we totally have 99 stocks listed in SZSE COMP on December 31st, 2016, as our research dataset, which sums to 9900,000 transactions.


#### 3.1.2. Data Characteristics

As the box plots in Figure3 demonstrate, transaction durations of each constituent stock from SZSE 100 Index reveal a very long tail compared with the interquartile range. The large amount of data located in the tail means the existence of liquidity risk.Figure 3
Box plots of durations of 99 stocks from SZSE 100 (minimum time unit: millisecond). The 99 stocks are listed in thex-axis, and the y-axis represents the duration dimension.
(a)(b)(c)(d)(e)(f)(g)(h)(i)To further dig the dynamic characteristics of the duration sequence, the averaged coefficients of autocorrelation function (acf) and partial correlation function (pacf) coefficients are plotted. As shown in the following Figure 4, we can see that time series duration data show a longer memory in that both acfcoefficients and pacf coefficients decay very slowly as the lagged term increases. Hence, the higher complexity of the UHF duration data requires a forecasting algorithm with strong fitting ability.Figure 4
Averaged (a) acf and (b) pacf.
(a)(b)


### 3.2. Evaluation Criteria

#### 3.2.1. Mean Absolute Error (MAE) and Mean Squared Logarithmic Error (MSLE)

As two frequently used metrics,MAE and MSLE are both used to directly evaluate the performance of duration prediction and can be calculated by the following formulas:(11)MAE=1N∑i=1Ndurationiforecast−durationireal,MSLE=1N∑i=1Nloge1+durationiforecast−loge1+durationireal2.A smallerMAE or MSLE means that we have a more precise forecast of the transaction duration.


#### 3.2.2. Performance Measure for Quantile Prediction

To evaluate the forecasting performance of quantile points, we utilize the loss function in quantile regression minimization problems [23]. Let TaRi:i=1,…,h be the prediction quantile points of the probability level α, xi be the realistic duration of the i−th transaction, and the I represent an indicative function, and the performance measure QLα,t [11] can be calculated as follows:(12)QLα,t=∑i=T+1T+hxi−TaRi,αα−Ixi<TaRi,α.


#### 3.2.3. Experiment Models

In Section2, we have created a new framework for the one-step ahead prediction. The sequence of 50 lagged durations (1 feature, 50 timesteps) is firstly chosen as the input data, and we hence construct the LSTM-ACD model and Attention-LSTM-ACD model, which are presented in Sections 2.2.1 and 2.2.2, respectively. The only difference between the two models is the attention layer. To further utilize the information of transaction by transaction data, one-dimensional duration feature is extended to multidimensional feature vector by adding the transaction volume and transaction type information. And then, two other models are constructed, named as the Attention-LSTM-ACD (M) model and the LSTM-ACD (M) model. The experiments will be performed with the following seven models: the ACD model, the AMACD model, the BACD model, the LSTM-ACD model, the Attention-LSTM-ACD model, the Attention-LSTM-ACD (M) model, and the LSTM-ACD (M) model.


### 3.3. Training

During the training process, configurations are determined with as few exogenous inputs as possible because of the various drawbacks of manual tuning. We adopt fixed hyperparameters including learning rate, number of neurons of each layer, batch size, and timesteps for each constituent stock of SZSE 100.


#### 3.3.1. Generation of Training Sets, Validation Sets, and Test Sets

As mentioned above, the sample used in this study is the 100,000 durations in 2017 for each stock collected from SZSE 100. We select the last 30% of data as the test set, while the remaining data are divided into training set and validation set according to the ratio of 8 : 2.


#### 3.3.2. Training Process

During the experiment, fixed hyperparameter combination is selected for each model based on LSTM-ACD framework. Table1 lists the hyperparameters used in our experiment. The attention size represents the height of the tensor wα in formula (6). The initial learning rate is 0.5, and it is reduced by 50% after 1000 training steps. Besides the selection of hyperparameter combination, the remaining parameters of the proposed hybrid models are learned by taking advantage the early-stopping technique to avoid the overfitting problem. We evaluate model performance on the validation set every 100 training steps, and the early-stopping patience represents the number of times that there is continuously no improvement in the log likelihood function calculated on the validation set.Table 1
The hyperparameters of each model.
Attention-LSTM-ACD (M) and LSTM-ACD (M)Attention-LSTM-ACD and LSTM-ACDInput layer3 features, 50 timesteps1 feature, 50 timestepsLSTM layer5 hidden neurons5 hidden neuronsAttention size2 (for model Attention-LSTM-ACD (M))2 (for model Attention-LSTM-ACD)Fully connected layer2 hidden neurons2 hidden neuronsBatch size300300Start learning rate0.50.5Decay steps10001000Decay rate50%50%Early stopping patience1010In this paper, the four models based on LSTM-ACD framework are coded in Tensorflow1.0, and the traditional ACD family methods are modelling by the ACDm package based onR language.


### 3.4. Experiment Results

#### 3.4.1. Comparison of Different Models inMAE and MSLE

The out-of-sample forecasting errors of the seven types of experiment models are calculated. Table2 lists the average MAE and average MSLE in the test sets when the seven models are applied to SZSE 100 Index constituent stocks, respectively. The LSTM-ACD (M) model has the minimum average MAE and average MSLE, while the remaining three models based on LSTM-ACD framework all perform a bit worse than the traditional ACD family models in the two metrics. As mentioned above, uniform hyperparameter combination is chosen when applying the hybrid models. If we select different hyperparameters when focusing on different stocks, the performance of these hybrid models will be much better. In BACD model (see equation (4)), εi=xi/μi and we calculate ε^i by Durationireal/Durationiforecast during the process of iterated prediction. The estimation value of δ1 could be much smaller than δ2, so we will get an extremely large forecasted value if Durationireal is much larger than Durationiforecast. This situation appears when we are applying BACD model to some stocks and results in extremely large average MAE and average quantile loss. Hence, the average MAE and average quantile loss of BACD model are not listed in this paper.Table 2
The averageMAE and MSLE on SZSE 100 Index constituent stocks of each model.
Average MAEAverage MAElaggedDifferenceAverage MSLEAverage MSLElaggedDifferenceAttention-LSTM-ACD(M)2.02641.97620.05020.70880.68920.0195LSTM-ACD(M)1.79901.76020.03880.59470.56770.0270Attention-LSTM-ACD1.97581.93670.03900.69350.66290.0306LSTM-ACD1.89641.83680.05960.65200.60430.0477ACD1.86411.66120.20300.62850.51840.1100BACD———0.69800.59280.1052AMACD1.81491.64460.17020.60070.49880.1019As the AMACD model places second in both averageMAE and average MSLE, we compare LSTM-ACD (M) model with AMACD model in detail. We can see from Figure 5 that the LSTM-ACD (M) is also superior to the AMCD model on more stocks in both metrics of MAE and MSLE.Figure 5
The contrasts between LSTM-ACD (M) and AMACD in (a) MAE and (b) MSLE (the proportion of each slice in a pie chart represents the quantity of stocks on which the corresponding model performs better than the other).
(a)(b)In addition, we calculate theMAE and MSLE one-step lagged of each model for the durations by the following formulas:(13)MAElagged=1N∑i=1NDurationiforecast−Durationi−1real,MSLElagged=1N∑i=1Nloge1+Durationiforecast−loge1+Durationi−1real2.Suppose there are two models A and B for the same duration prediction task. We getMAElaggedA,MAEA for model A and get MAElaggedB,MAEB for model B. If MAEA=MAEB and MAElaggedA<MAElaggedB, we can deduce that model B is more likely to extract the long-term dependency in the time series data because model A utilizes a higher proportion of Durationi−1real value to forecast Durationireal. In an extreme situation that MAEA=MAEB and MAElaggedA=0<MAElaggedB, obviously, model A just uses Durationi−1real as the value of Durationiforecast. By an extension of this logic, if MAEA−MAElaggedA>MAEB−MAElaggedB, we can also deduce that model B has a stronger ability in capturing the long-term dependency relationship. Similarly, a smaller MSLE−MSLElagged also indicates a higher ability in modelling long-term dependency. The results in columns 4 and 7 of Table 2 show that the average MAElagged and average MSLElagged of each traditional ACD model are significantly smaller than the average MAE and average MSLE, respectively. It means the four models based on LSTM-ACD framework are superior to the traditional ACD family models in reflecting the long-term relationship of the sequential data.Further detail forMAE and MSLE is provided in Figure 6. It can be seen that the LSTM-ACD (M) and LSTM-ACD are both superior to the previous ACD family models on more stocks in the metric of MAE. Moreover, LSTM-ACD (M) also has the minimum MSLE on more stocks than all other models. BACD model and LSTM-ACD model performs best on 21.21% and 22.22% of the assets, respectively, in terms of MSLE. Take the unstable performance of BACD model into consideration, the LSTM-ACD model still places second overall.Figure 6
Detailed comparison of the 7 models inMAE and MSLE (the size of each pie slice represents the quantity of stocks on which the corresponding model achieves the best performance). (a) Minimum MAE. (b) Minimum MSLE.
(a)(b)


#### 3.4.2. Comparison of Different Models in Quantile Forecasts

Table3 lists the quantile forecast measure QL of different upper quantile levels for the seven models. It can be found that the Attention-LSTM-ACD (M) model is the supreme one at all three quantile levels among the models based on LSTM-ACD framework. In terms of the Attention-LSTM-ACD model, it also provides a better quantile forecasting than LSTM-ACD model at all quantile levels. Although our proposed models are not superior to the traditional ACD family models in extreme quantile forecasting, these results still indicate that the attention layer can improve the accuracy in conditional distribution forecasting. The QL of BACD model is not presented in Table 3 due to the extreme large values of QL. Besides, the performance of our proposed models can be further improved by selecting different hyperparameters for different stocks.Table 3
Quantile loss for the models at different upper quantile levels.
ModelUpper quantile level0.10.050.01Attention-LSTM-ACD (M)23323.4217772.658964.29LSTM-ACD (M)24252.6718974.4810296.03Attention-LSTM-ACD24423.3919732.8811523.59LSTM-ACD24691.7519866.1311676.77ACD20040.6614566.036484.51BACD———AMACD20084.5914713.166663.78


#### 3.4.3. Attention Weights of Different Lag Orders

This section makes visualization for the Attention-LSTM-ACD model and Attention-LSTM-ACD (M) model. As can be seen in Table4 and Figure 7, the weights learned by the attention layer in both the two models decrease exponentially with the increase in lag order. This means that the closer transaction has a more important effect on the current duration, which is consistent to our intuition.Table 4
Average weights of the Attention-LSTM-ACD (M) model and the Attention-LSTM-ACD model on SZSE 100 Index constituent stocks.
Attention-LSTM-ACD (M)Attention-LSTM-ACDLag orderWeightLag orderWeightLag 10.034797791Lag 10.078697926Lag 20.028296111Lag 20.034143126Lag 30.024825037Lag 30.027711418Lag 40.023690568Lag 40.025372255Lag 50.022575405Lag 50.023460835Lag 60.022012244Lag 60.022699354Lag 70.02148028Lag 70.021109585Lag 80.020997523Lag 80.020365109Lag 90.020678233Lag 90.02006378Lag 100.020491562Lag 100.01944402……………………Lag 410.018665664Lag 410.017536173Lag 420.018792729Lag 420.017302261Lag 430.018596823Lag 430.017404814Lag 440.018732648Lag 440.017530387Lag 450.018695344Lag 450.017503974Lag 460.018753901Lag 460.017434779Lag 470.018770904Lag 470.017377114Lag 480.01898069Lag 480.017801802Lag 490.018924108Lag 490.017549126Lag 500.018977175Lag 500.01779628Figure 7
Attention weights of the (a) Attention-LSTM-ACD (M) model and (b) Attention-LSTM-ACD model of different lags on SZSE 100 Index constituent stocks (each blue line represents the attention weight sequence of a stock for the corresponding model, and each red line represents the average attention weights on SZSE 100 Index constituent stocks for the corresponding model.
(a)(b)


## 3.1. Data Description

### 3.1.1. Data Source

The Shenzhen Stock Exchange 100 Index (SZSE 100) is the first index designed for reflecting the multiple level market conditions of Chinese stock market. The constituent stocks of SZSE 100 represent the core high-quality assets in the Shenzhen A-share market, with strong growth, low valuation, and high investment value. In this paper, we collect duration data of the first 100,000 transactions which has excluded the transactions during premarket opening session, for each constituent stock from SZSE 100. The readers can acquire the data from the Transend DataBase System of Wind Information Co., Ltd (https://www.wind.com.cn/). Since the stock Tianjin Zhonghuan Semiconductor Co., Ltd., which is coding in 002129.SZ has no transactions during 2017, we totally have 99 stocks listed in SZSE COMP on December 31st, 2016, as our research dataset, which sums to 9900,000 transactions.


### 3.1.2. Data Characteristics

As the box plots in Figure3 demonstrate, transaction durations of each constituent stock from SZSE 100 Index reveal a very long tail compared with the interquartile range. The large amount of data located in the tail means the existence of liquidity risk.Figure 3
Box plots of durations of 99 stocks from SZSE 100 (minimum time unit: millisecond). The 99 stocks are listed in thex-axis, and the y-axis represents the duration dimension.
(a)(b)(c)(d)(e)(f)(g)(h)(i)To further dig the dynamic characteristics of the duration sequence, the averaged coefficients of autocorrelation function (acf) and partial correlation function (pacf) coefficients are plotted. As shown in the following Figure 4, we can see that time series duration data show a longer memory in that both acfcoefficients and pacf coefficients decay very slowly as the lagged term increases. Hence, the higher complexity of the UHF duration data requires a forecasting algorithm with strong fitting ability.Figure 4
Averaged (a) acf and (b) pacf.
(a)(b)


## 3.1.1. Data Source

The Shenzhen Stock Exchange 100 Index (SZSE 100) is the first index designed for reflecting the multiple level market conditions of Chinese stock market. The constituent stocks of SZSE 100 represent the core high-quality assets in the Shenzhen A-share market, with strong growth, low valuation, and high investment value. In this paper, we collect duration data of the first 100,000 transactions which has excluded the transactions during premarket opening session, for each constituent stock from SZSE 100. The readers can acquire the data from the Transend DataBase System of Wind Information Co., Ltd (https://www.wind.com.cn/). Since the stock Tianjin Zhonghuan Semiconductor Co., Ltd., which is coding in 002129.SZ has no transactions during 2017, we totally have 99 stocks listed in SZSE COMP on December 31st, 2016, as our research dataset, which sums to 9900,000 transactions.


## 3.1.2. Data Characteristics

As the box plots in Figure3 demonstrate, transaction durations of each constituent stock from SZSE 100 Index reveal a very long tail compared with the interquartile range. The large amount of data located in the tail means the existence of liquidity risk.Figure 3
Box plots of durations of 99 stocks from SZSE 100 (minimum time unit: millisecond). The 99 stocks are listed in thex-axis, and the y-axis represents the duration dimension.
(a)(b)(c)(d)(e)(f)(g)(h)(i)To further dig the dynamic characteristics of the duration sequence, the averaged coefficients of autocorrelation function (acf) and partial correlation function (pacf) coefficients are plotted. As shown in the following Figure 4, we can see that time series duration data show a longer memory in that both acfcoefficients and pacf coefficients decay very slowly as the lagged term increases. Hence, the higher complexity of the UHF duration data requires a forecasting algorithm with strong fitting ability.Figure 4
Averaged (a) acf and (b) pacf.
(a)(b)


## 3.2. Evaluation Criteria

### 3.2.1. Mean Absolute Error (MAE) and Mean Squared Logarithmic Error (MSLE)

As two frequently used metrics,MAE and MSLE are both used to directly evaluate the performance of duration prediction and can be calculated by the following formulas:(11)MAE=1N∑i=1Ndurationiforecast−durationireal,MSLE=1N∑i=1Nloge1+durationiforecast−loge1+durationireal2.A smallerMAE or MSLE means that we have a more precise forecast of the transaction duration.


### 3.2.2. Performance Measure for Quantile Prediction

To evaluate the forecasting performance of quantile points, we utilize the loss function in quantile regression minimization problems [23]. Let TaRi:i=1,…,h be the prediction quantile points of the probability level α, xi be the realistic duration of the i−th transaction, and the I represent an indicative function, and the performance measure QLα,t [11] can be calculated as follows:(12)QLα,t=∑i=T+1T+hxi−TaRi,αα−Ixi<TaRi,α.


### 3.2.3. Experiment Models

In Section2, we have created a new framework for the one-step ahead prediction. The sequence of 50 lagged durations (1 feature, 50 timesteps) is firstly chosen as the input data, and we hence construct the LSTM-ACD model and Attention-LSTM-ACD model, which are presented in Sections 2.2.1 and 2.2.2, respectively. The only difference between the two models is the attention layer. To further utilize the information of transaction by transaction data, one-dimensional duration feature is extended to multidimensional feature vector by adding the transaction volume and transaction type information. And then, two other models are constructed, named as the Attention-LSTM-ACD (M) model and the LSTM-ACD (M) model. The experiments will be performed with the following seven models: the ACD model, the AMACD model, the BACD model, the LSTM-ACD model, the Attention-LSTM-ACD model, the Attention-LSTM-ACD (M) model, and the LSTM-ACD (M) model.


## 3.2.1. Mean Absolute Error (MAE) and Mean Squared Logarithmic Error (MSLE)

As two frequently used metrics,MAE and MSLE are both used to directly evaluate the performance of duration prediction and can be calculated by the following formulas:(11)MAE=1N∑i=1Ndurationiforecast−durationireal,MSLE=1N∑i=1Nloge1+durationiforecast−loge1+durationireal2.A smallerMAE or MSLE means that we have a more precise forecast of the transaction duration.


## 3.2.2. Performance Measure for Quantile Prediction

To evaluate the forecasting performance of quantile points, we utilize the loss function in quantile regression minimization problems [23]. Let TaRi:i=1,…,h be the prediction quantile points of the probability level α, xi be the realistic duration of the i−th transaction, and the I represent an indicative function, and the performance measure QLα,t [11] can be calculated as follows:(12)QLα,t=∑i=T+1T+hxi−TaRi,αα−Ixi<TaRi,α.


## 3.2.3. Experiment Models

In Section2, we have created a new framework for the one-step ahead prediction. The sequence of 50 lagged durations (1 feature, 50 timesteps) is firstly chosen as the input data, and we hence construct the LSTM-ACD model and Attention-LSTM-ACD model, which are presented in Sections 2.2.1 and 2.2.2, respectively. The only difference between the two models is the attention layer. To further utilize the information of transaction by transaction data, one-dimensional duration feature is extended to multidimensional feature vector by adding the transaction volume and transaction type information. And then, two other models are constructed, named as the Attention-LSTM-ACD (M) model and the LSTM-ACD (M) model. The experiments will be performed with the following seven models: the ACD model, the AMACD model, the BACD model, the LSTM-ACD model, the Attention-LSTM-ACD model, the Attention-LSTM-ACD (M) model, and the LSTM-ACD (M) model.


## 3.3. Training

During the training process, configurations are determined with as few exogenous inputs as possible because of the various drawbacks of manual tuning. We adopt fixed hyperparameters including learning rate, number of neurons of each layer, batch size, and timesteps for each constituent stock of SZSE 100.


### 3.3.1. Generation of Training Sets, Validation Sets, and Test Sets

As mentioned above, the sample used in this study is the 100,000 durations in 2017 for each stock collected from SZSE 100. We select the last 30% of data as the test set, while the remaining data are divided into training set and validation set according to the ratio of 8 : 2.


### 3.3.2. Training Process

During the experiment, fixed hyperparameter combination is selected for each model based on LSTM-ACD framework. Table1 lists the hyperparameters used in our experiment. The attention size represents the height of the tensor wα in formula (6). The initial learning rate is 0.5, and it is reduced by 50% after 1000 training steps. Besides the selection of hyperparameter combination, the remaining parameters of the proposed hybrid models are learned by taking advantage the early-stopping technique to avoid the overfitting problem. We evaluate model performance on the validation set every 100 training steps, and the early-stopping patience represents the number of times that there is continuously no improvement in the log likelihood function calculated on the validation set.Table 1
The hyperparameters of each model.
Attention-LSTM-ACD (M) and LSTM-ACD (M)Attention-LSTM-ACD and LSTM-ACDInput layer3 features, 50 timesteps1 feature, 50 timestepsLSTM layer5 hidden neurons5 hidden neuronsAttention size2 (for model Attention-LSTM-ACD (M))2 (for model Attention-LSTM-ACD)Fully connected layer2 hidden neurons2 hidden neuronsBatch size300300Start learning rate0.50.5Decay steps10001000Decay rate50%50%Early stopping patience1010In this paper, the four models based on LSTM-ACD framework are coded in Tensorflow1.0, and the traditional ACD family methods are modelling by the ACDm package based onR language.


## 3.3.1. Generation of Training Sets, Validation Sets, and Test Sets

As mentioned above, the sample used in this study is the 100,000 durations in 2017 for each stock collected from SZSE 100. We select the last 30% of data as the test set, while the remaining data are divided into training set and validation set according to the ratio of 8 : 2.


## 3.3.2. Training Process

During the experiment, fixed hyperparameter combination is selected for each model based on LSTM-ACD framework. Table1 lists the hyperparameters used in our experiment. The attention size represents the height of the tensor wα in formula (6). The initial learning rate is 0.5, and it is reduced by 50% after 1000 training steps. Besides the selection of hyperparameter combination, the remaining parameters of the proposed hybrid models are learned by taking advantage the early-stopping technique to avoid the overfitting problem. We evaluate model performance on the validation set every 100 training steps, and the early-stopping patience represents the number of times that there is continuously no improvement in the log likelihood function calculated on the validation set.Table 1
The hyperparameters of each model.
Attention-LSTM-ACD (M) and LSTM-ACD (M)Attention-LSTM-ACD and LSTM-ACDInput layer3 features, 50 timesteps1 feature, 50 timestepsLSTM layer5 hidden neurons5 hidden neuronsAttention size2 (for model Attention-LSTM-ACD (M))2 (for model Attention-LSTM-ACD)Fully connected layer2 hidden neurons2 hidden neuronsBatch size300300Start learning rate0.50.5Decay steps10001000Decay rate50%50%Early stopping patience1010In this paper, the four models based on LSTM-ACD framework are coded in Tensorflow1.0, and the traditional ACD family methods are modelling by the ACDm package based onR language.


## 3.4. Experiment Results

### 3.4.1. Comparison of Different Models inMAE and MSLE

The out-of-sample forecasting errors of the seven types of experiment models are calculated. Table2 lists the average MAE and average MSLE in the test sets when the seven models are applied to SZSE 100 Index constituent stocks, respectively. The LSTM-ACD (M) model has the minimum average MAE and average MSLE, while the remaining three models based on LSTM-ACD framework all perform a bit worse than the traditional ACD family models in the two metrics. As mentioned above, uniform hyperparameter combination is chosen when applying the hybrid models. If we select different hyperparameters when focusing on different stocks, the performance of these hybrid models will be much better. In BACD model (see equation (4)), εi=xi/μi and we calculate ε^i by Durationireal/Durationiforecast during the process of iterated prediction. The estimation value of δ1 could be much smaller than δ2, so we will get an extremely large forecasted value if Durationireal is much larger than Durationiforecast. This situation appears when we are applying BACD model to some stocks and results in extremely large average MAE and average quantile loss. Hence, the average MAE and average quantile loss of BACD model are not listed in this paper.Table 2
The averageMAE and MSLE on SZSE 100 Index constituent stocks of each model.
Average MAEAverage MAElaggedDifferenceAverage MSLEAverage MSLElaggedDifferenceAttention-LSTM-ACD(M)2.02641.97620.05020.70880.68920.0195LSTM-ACD(M)1.79901.76020.03880.59470.56770.0270Attention-LSTM-ACD1.97581.93670.03900.69350.66290.0306LSTM-ACD1.89641.83680.05960.65200.60430.0477ACD1.86411.66120.20300.62850.51840.1100BACD———0.69800.59280.1052AMACD1.81491.64460.17020.60070.49880.1019As the AMACD model places second in both averageMAE and average MSLE, we compare LSTM-ACD (M) model with AMACD model in detail. We can see from Figure 5 that the LSTM-ACD (M) is also superior to the AMCD model on more stocks in both metrics of MAE and MSLE.Figure 5
The contrasts between LSTM-ACD (M) and AMACD in (a) MAE and (b) MSLE (the proportion of each slice in a pie chart represents the quantity of stocks on which the corresponding model performs better than the other).
(a)(b)In addition, we calculate theMAE and MSLE one-step lagged of each model for the durations by the following formulas:(13)MAElagged=1N∑i=1NDurationiforecast−Durationi−1real,MSLElagged=1N∑i=1Nloge1+Durationiforecast−loge1+Durationi−1real2.Suppose there are two models A and B for the same duration prediction task. We getMAElaggedA,MAEA for model A and get MAElaggedB,MAEB for model B. If MAEA=MAEB and MAElaggedA<MAElaggedB, we can deduce that model B is more likely to extract the long-term dependency in the time series data because model A utilizes a higher proportion of Durationi−1real value to forecast Durationireal. In an extreme situation that MAEA=MAEB and MAElaggedA=0<MAElaggedB, obviously, model A just uses Durationi−1real as the value of Durationiforecast. By an extension of this logic, if MAEA−MAElaggedA>MAEB−MAElaggedB, we can also deduce that model B has a stronger ability in capturing the long-term dependency relationship. Similarly, a smaller MSLE−MSLElagged also indicates a higher ability in modelling long-term dependency. The results in columns 4 and 7 of Table 2 show that the average MAElagged and average MSLElagged of each traditional ACD model are significantly smaller than the average MAE and average MSLE, respectively. It means the four models based on LSTM-ACD framework are superior to the traditional ACD family models in reflecting the long-term relationship of the sequential data.Further detail forMAE and MSLE is provided in Figure 6. It can be seen that the LSTM-ACD (M) and LSTM-ACD are both superior to the previous ACD family models on more stocks in the metric of MAE. Moreover, LSTM-ACD (M) also has the minimum MSLE on more stocks than all other models. BACD model and LSTM-ACD model performs best on 21.21% and 22.22% of the assets, respectively, in terms of MSLE. Take the unstable performance of BACD model into consideration, the LSTM-ACD model still places second overall.Figure 6
Detailed comparison of the 7 models inMAE and MSLE (the size of each pie slice represents the quantity of stocks on which the corresponding model achieves the best performance). (a) Minimum MAE. (b) Minimum MSLE.
(a)(b)


### 3.4.2. Comparison of Different Models in Quantile Forecasts

Table3 lists the quantile forecast measure QL of different upper quantile levels for the seven models. It can be found that the Attention-LSTM-ACD (M) model is the supreme one at all three quantile levels among the models based on LSTM-ACD framework. In terms of the Attention-LSTM-ACD model, it also provides a better quantile forecasting than LSTM-ACD model at all quantile levels. Although our proposed models are not superior to the traditional ACD family models in extreme quantile forecasting, these results still indicate that the attention layer can improve the accuracy in conditional distribution forecasting. The QL of BACD model is not presented in Table 3 due to the extreme large values of QL. Besides, the performance of our proposed models can be further improved by selecting different hyperparameters for different stocks.Table 3
Quantile loss for the models at different upper quantile levels.
ModelUpper quantile level0.10.050.01Attention-LSTM-ACD (M)23323.4217772.658964.29LSTM-ACD (M)24252.6718974.4810296.03Attention-LSTM-ACD24423.3919732.8811523.59LSTM-ACD24691.7519866.1311676.77ACD20040.6614566.036484.51BACD———AMACD20084.5914713.166663.78


### 3.4.3. Attention Weights of Different Lag Orders

This section makes visualization for the Attention-LSTM-ACD model and Attention-LSTM-ACD (M) model. As can be seen in Table4 and Figure 7, the weights learned by the attention layer in both the two models decrease exponentially with the increase in lag order. This means that the closer transaction has a more important effect on the current duration, which is consistent to our intuition.Table 4
Average weights of the Attention-LSTM-ACD (M) model and the Attention-LSTM-ACD model on SZSE 100 Index constituent stocks.
Attention-LSTM-ACD (M)Attention-LSTM-ACDLag orderWeightLag orderWeightLag 10.034797791Lag 10.078697926Lag 20.028296111Lag 20.034143126Lag 30.024825037Lag 30.027711418Lag 40.023690568Lag 40.025372255Lag 50.022575405Lag 50.023460835Lag 60.022012244Lag 60.022699354Lag 70.02148028Lag 70.021109585Lag 80.020997523Lag 80.020365109Lag 90.020678233Lag 90.02006378Lag 100.020491562Lag 100.01944402……………………Lag 410.018665664Lag 410.017536173Lag 420.018792729Lag 420.017302261Lag 430.018596823Lag 430.017404814Lag 440.018732648Lag 440.017530387Lag 450.018695344Lag 450.017503974Lag 460.018753901Lag 460.017434779Lag 470.018770904Lag 470.017377114Lag 480.01898069Lag 480.017801802Lag 490.018924108Lag 490.017549126Lag 500.018977175Lag 500.01779628Figure 7
Attention weights of the (a) Attention-LSTM-ACD (M) model and (b) Attention-LSTM-ACD model of different lags on SZSE 100 Index constituent stocks (each blue line represents the attention weight sequence of a stock for the corresponding model, and each red line represents the average attention weights on SZSE 100 Index constituent stocks for the corresponding model.
(a)(b)


## 3.4.1. Comparison of Different Models inMAE and MSLE

The out-of-sample forecasting errors of the seven types of experiment models are calculated. Table2 lists the average MAE and average MSLE in the test sets when the seven models are applied to SZSE 100 Index constituent stocks, respectively. The LSTM-ACD (M) model has the minimum average MAE and average MSLE, while the remaining three models based on LSTM-ACD framework all perform a bit worse than the traditional ACD family models in the two metrics. As mentioned above, uniform hyperparameter combination is chosen when applying the hybrid models. If we select different hyperparameters when focusing on different stocks, the performance of these hybrid models will be much better. In BACD model (see equation (4)), εi=xi/μi and we calculate ε^i by Durationireal/Durationiforecast during the process of iterated prediction. The estimation value of δ1 could be much smaller than δ2, so we will get an extremely large forecasted value if Durationireal is much larger than Durationiforecast. This situation appears when we are applying BACD model to some stocks and results in extremely large average MAE and average quantile loss. Hence, the average MAE and average quantile loss of BACD model are not listed in this paper.Table 2
The averageMAE and MSLE on SZSE 100 Index constituent stocks of each model.
Average MAEAverage MAElaggedDifferenceAverage MSLEAverage MSLElaggedDifferenceAttention-LSTM-ACD(M)2.02641.97620.05020.70880.68920.0195LSTM-ACD(M)1.79901.76020.03880.59470.56770.0270Attention-LSTM-ACD1.97581.93670.03900.69350.66290.0306LSTM-ACD1.89641.83680.05960.65200.60430.0477ACD1.86411.66120.20300.62850.51840.1100BACD———0.69800.59280.1052AMACD1.81491.64460.17020.60070.49880.1019As the AMACD model places second in both averageMAE and average MSLE, we compare LSTM-ACD (M) model with AMACD model in detail. We can see from Figure 5 that the LSTM-ACD (M) is also superior to the AMCD model on more stocks in both metrics of MAE and MSLE.Figure 5
The contrasts between LSTM-ACD (M) and AMACD in (a) MAE and (b) MSLE (the proportion of each slice in a pie chart represents the quantity of stocks on which the corresponding model performs better than the other).
(a)(b)In addition, we calculate theMAE and MSLE one-step lagged of each model for the durations by the following formulas:(13)MAElagged=1N∑i=1NDurationiforecast−Durationi−1real,MSLElagged=1N∑i=1Nloge1+Durationiforecast−loge1+Durationi−1real2.Suppose there are two models A and B for the same duration prediction task. We getMAElaggedA,MAEA for model A and get MAElaggedB,MAEB for model B. If MAEA=MAEB and MAElaggedA<MAElaggedB, we can deduce that model B is more likely to extract the long-term dependency in the time series data because model A utilizes a higher proportion of Durationi−1real value to forecast Durationireal. In an extreme situation that MAEA=MAEB and MAElaggedA=0<MAElaggedB, obviously, model A just uses Durationi−1real as the value of Durationiforecast. By an extension of this logic, if MAEA−MAElaggedA>MAEB−MAElaggedB, we can also deduce that model B has a stronger ability in capturing the long-term dependency relationship. Similarly, a smaller MSLE−MSLElagged also indicates a higher ability in modelling long-term dependency. The results in columns 4 and 7 of Table 2 show that the average MAElagged and average MSLElagged of each traditional ACD model are significantly smaller than the average MAE and average MSLE, respectively. It means the four models based on LSTM-ACD framework are superior to the traditional ACD family models in reflecting the long-term relationship of the sequential data.Further detail forMAE and MSLE is provided in Figure 6. It can be seen that the LSTM-ACD (M) and LSTM-ACD are both superior to the previous ACD family models on more stocks in the metric of MAE. Moreover, LSTM-ACD (M) also has the minimum MSLE on more stocks than all other models. BACD model and LSTM-ACD model performs best on 21.21% and 22.22% of the assets, respectively, in terms of MSLE. Take the unstable performance of BACD model into consideration, the LSTM-ACD model still places second overall.Figure 6
Detailed comparison of the 7 models inMAE and MSLE (the size of each pie slice represents the quantity of stocks on which the corresponding model achieves the best performance). (a) Minimum MAE. (b) Minimum MSLE.
(a)(b)


## 3.4.2. Comparison of Different Models in Quantile Forecasts

Table3 lists the quantile forecast measure QL of different upper quantile levels for the seven models. It can be found that the Attention-LSTM-ACD (M) model is the supreme one at all three quantile levels among the models based on LSTM-ACD framework. In terms of the Attention-LSTM-ACD model, it also provides a better quantile forecasting than LSTM-ACD model at all quantile levels. Although our proposed models are not superior to the traditional ACD family models in extreme quantile forecasting, these results still indicate that the attention layer can improve the accuracy in conditional distribution forecasting. The QL of BACD model is not presented in Table 3 due to the extreme large values of QL. Besides, the performance of our proposed models can be further improved by selecting different hyperparameters for different stocks.Table 3
Quantile loss for the models at different upper quantile levels.
ModelUpper quantile level0.10.050.01Attention-LSTM-ACD (M)23323.4217772.658964.29LSTM-ACD (M)24252.6718974.4810296.03Attention-LSTM-ACD24423.3919732.8811523.59LSTM-ACD24691.7519866.1311676.77ACD20040.6614566.036484.51BACD———AMACD20084.5914713.166663.78


## 3.4.3. Attention Weights of Different Lag Orders

This section makes visualization for the Attention-LSTM-ACD model and Attention-LSTM-ACD (M) model. As can be seen in Table4 and Figure 7, the weights learned by the attention layer in both the two models decrease exponentially with the increase in lag order. This means that the closer transaction has a more important effect on the current duration, which is consistent to our intuition.Table 4
Average weights of the Attention-LSTM-ACD (M) model and the Attention-LSTM-ACD model on SZSE 100 Index constituent stocks.
Attention-LSTM-ACD (M)Attention-LSTM-ACDLag orderWeightLag orderWeightLag 10.034797791Lag 10.078697926Lag 20.028296111Lag 20.034143126Lag 30.024825037Lag 30.027711418Lag 40.023690568Lag 40.025372255Lag 50.022575405Lag 50.023460835Lag 60.022012244Lag 60.022699354Lag 70.02148028Lag 70.021109585Lag 80.020997523Lag 80.020365109Lag 90.020678233Lag 90.02006378Lag 100.020491562Lag 100.01944402……………………Lag 410.018665664Lag 410.017536173Lag 420.018792729Lag 420.017302261Lag 430.018596823Lag 430.017404814Lag 440.018732648Lag 440.017530387Lag 450.018695344Lag 450.017503974Lag 460.018753901Lag 460.017434779Lag 470.018770904Lag 470.017377114Lag 480.01898069Lag 480.017801802Lag 490.018924108Lag 490.017549126Lag 500.018977175Lag 500.01779628Figure 7
Attention weights of the (a) Attention-LSTM-ACD (M) model and (b) Attention-LSTM-ACD model of different lags on SZSE 100 Index constituent stocks (each blue line represents the attention weight sequence of a stock for the corresponding model, and each red line represents the average attention weights on SZSE 100 Index constituent stocks for the corresponding model.
(a)(b)


## 4. Conclusion and Future Research

In this paper, we review the studies of transaction duration modelling based on ACD framework and find that these studies can be classified into two categories: (a) propose a new nonlinear equation form to describe the dynamics of conditional (expected) duration; (b) choose a more flexible distribution for the unexpected portion of the duration.This study constructs a new framework for transaction duration modelling from the perspective of extending the mean equation of ACD model by machine learning methods. Firstly, we build a LSTM-ACD model by combining the LSTM networks with classic ACD model to characterize the complexity of the conditional mean process while retaining the advantage of providing probabilistic output. And then, attention layer is added to construct the Attention-LSTM-ACD model with the ability of unveiling importance of each hidden state in the LSTM networks.Our proposed new framework is applied to a large-scale dataset. The fixed hyperparameters are chosen for all constituent stocks of SZSE 100 Index to reduce the impact of manual tuning, and the parameters (and consequently the underlying distributions) are learned via maximize the log-likelihood function. The results show that LSTM-ACD (M) model can present highest accuracy on the task of forecasting on real-world financial datasets among all the presented models. Although Attention-LSTM-ACD model and Attention-LSTM-ACD (M) model could not provide a more accurate performance inMAEmetric, the attention layer vividly depicts the importance of different temporal points of the input sequence and outperforms the corresponding LSTM-ACD model and LSTM-ACD (M) model in QL loss metric, respectively. In addition, the average MAElagged of ACD model is significantly smaller than the average MAE, which means the predictions of the LSTM-ACD framework models to some extent convey more meaningful information. As a suitable chosen residual distribution does matters, the exponential distribution used in our framework can be extended to more flexible distributions in future research.



---

*Source: 7854512-2021-01-30.xml*