Transformers versus LSTMs for electronic trading

## Abstract

Abstract With the rapid development of artificial intelligence, long short term memory (LSTM), one kind of recurrent neural network (RNN), has been widely applied in time series prediction. Like RNN, Transformer is designed to handle the sequential data. As Transformer achieved great success in Natural Language Processing (NLP), researchers got interested in Transformer’s performance on time series prediction, and plenty of Transformer-based solutions on long time series forecasting have come out recently. However, when it comes to financial time series prediction, LSTM is still a dominant architecture. Therefore, the question this study wants to answer is: whether the Transformer-based model can be applied in financial time series prediction and beat LSTM. To answer this question, various LSTM-based and Transformer-based models are compared on multiple financial prediction tasks based on high-frequency limit order book data. A new LSTM-based model called DLSTM is built and new architecture for the Transformer-based model is designed to adapt for financial prediction. The experiment result reflects that the Transformer-based model only has the limited advantage in absolute price sequence prediction. The LSTM-based models show better and more robust performance on difference sequence prediction, such as price difference and price movement.

## 1 Introduction

Financial time series prediction is a significant task in investing and market-making activities. The Efficient Market Hypothesis proposed by Eugene [] states that all the information of the asset’s inner value has already precisely and completely reflected on the asset price, and it is impossible to beat the market by financial prediction. However, whether the market is efficient is questionable because technical analysis [] believes the financial market is the physical movement of price (or features derived from prices). The price information can be interpreted by waves and patterns that can repeat themselves, where it is possible to make profitable buy or sell decisions in advance [, ]. During the prediction, challenging factors are noise and volatile features because price information is generally non-linear and non-stationary []. Lots of models are proposed to solve the financial time series problem. A typical linear model for regression is Auto-Regressive Integrated Moving average (ARIMA) [] and its variations, which requires domain expertise to handcraft features. With the development of machine learning, Artificial Neural Networks (ANN) raises great interest because of their capability to extract more abstract features from data and find a hidden non-linear relationship without assumptions or human expertise by adding more parameters. Long short-term memory (LSTM), which is a special recurrent neural network (RNN) architecture that has been proven successful in the application of sequential data, is widely applied to handwriting recognition [] and speech recognition []. Like RNN, the Transformer [] is also used to handle the sequential data. Compared to LSTM, the Transformer does not need to handle the sequence data in order, which instead confers the meaning of the sequence by the Self-attention mechanism.

Applying LSTM and Transformer for financial time series prediction is a popular trend nowadays. Depending on the historical financial data, researchers usually make predictions for the future numerical prices, price difference, return or future price movement (rise, stationary, fall). Although LSTM and Transformer are applied in different aspects for this problem. There are mainly two research directions:

- 1)
Make predictions based on high-frequency Limit Order Book (LOB) data and its derived features, such as Volume Order Imbalance (VOI) and Trade Flow Imbalance (TFI) [, , , , ].
- 2)
Make predictions based on OHLC (Open, High, Low, Close) data and its derived financial indices, such as Relative Strength Index (RSI) and Moving average convergence divergence (MACD) [, , , , , , , , , , ].

Since 2017, the Transformer has been increasingly used for Natural Language Processing (NLP) problems. It produces more impressive results than RNN, such as machine translation [] and speech applications [], replacing RNN models such as LSTM in NLP tasks. Recently, a surge of Transformer-based solutions for less explored long time series forecasting problem has appeared []. However, as for the financial time series prediction, LSTM remains the dominant architecture.

Whether Transformer-based methods can be the right solution for financial time series forecasting is a problem worth investigating. Therefore, this paper is going to compare the performance of different Transformer-based and LSTM-based methods on financial time series prediction problems based on LOB data and attempt to adapt new models based on Transformer and LSTM. The contributions of this study are summed up as follows:

- 1.
Systematically compare Transformer-based and LSTM-based methods in different financial prediction tasks based on high frequency LOB data collected from Binance Exchange. Tasks include (1) mid-price prediction, (2) mid-price difference prediction and (3) mid-price movement prediction.
- 2.
For the first and second tasks, comparisons are all conducted on previous LSTM-based and Transformer-based methods. In the first task, the Transformer-based method has around $10\%-25\%$ prediction error less than the LSTM-based method, but the prediction result quality is insufficient for trading. In the second task, the LSTM-based method performs better than the Transformer-based method, where its highest out-of-sample $R^{2}$ reaches around $11.5\%$.
- 3.
The most significant contribution of this study is in the last task, mid-price movement prediction. A new LSTM-based model named DLSTM is developed for this task, which combines LSTM and the time series decomposition method. This model achieves $63.73\%$ to $73.31\%$ accuracy and shows strong profitability and robustness in simulated trading, outperforming previous LSTM and Transformer-based methods. In addition, the architecture of previous Transformer-based methods is also changed in order to adapt movement prediction task.

The later parts of this study are structured as follows: The background and related work are introduced in Section [2](#S2). Section [3](#S3) describes the formulation of three financial prediction tasks. Section [4](#S4) explains the details of previous Transformer-based and LSTM-based methods used for comparison. Section [5](#S5) is the analysis of experiment results. Please note that the details of the newly developed DLSTM model and the architecture changes of Transformer models are explained in Section [5.3](#S5.SS3) in Section [5](#S5). The study is organized in this way to provide readers with a better understanding of the relationship among three financial prediction tasks.

The Source Code of this study is available at: [https://github.com/772435284/transformers_versus_lstms_for_electronic_trading](https://github.com/772435284/transformers_versus_lstms_for_electronic_trading)

##### Acknowledgements

We would like to express our gratitude to Zhipeng Wang for constructive comments and suggestions.

## 2 Background and Related work

Time series prediction has been applied in different financial activities. For example, the trader or market maker predicts the future price or the price movement of the assets so that he/she can design trading/market making strategies based on the prediction results to make profit from it. This part examines papers related to time series prediction using LSTM and Transformer. Most of them are related to financial time series prediction.

### 2.1 LSTM-based Time Series Prediction Solutions

LSTM has been widely utilized in financial time series prediction depending on OHLC data and its derived financial indices. Many works [, , , , ] present predicting stock prices as successful by LSTM. Bidirectional LSTM (BiLSTM) is applied to increase the prediction performance []. With the rise of NLP, Sequence-to-Sequence Model (S2S) [] is proposed and applied in machine translation and question answering. It is now also applied in financial time series prediction by combining LSTM structure and attention mechanisms contributing to higher performance [, ].

LSTM has successfully forecasted high-frequency data depending on the large datasets extracted from the limit order book (LOB). In terms of making predictions upon order book data, Convolution Neural Network (CNN) and LSTM are both in favour of research and sometimes they are combined. Sirignano et al. [] trained a universal model using LSTM to predict the LOB price movement by data from all stocks, which outperforms linear and non-linear models trained on a specific asset. Zhang et al. [] combine CNN and LSTM to form a new deep neural network architecture called DeepLOB to predict future stock price movements in LOB data outperforming architecture only containing LSTM. Zhang et al. [] also combine the DeepLOB architecture with Seq2Seq and Attention model to forecast multi-horizon future price movements in one forward procedure, which reduces the training effort and achieves better performance in long horizon prediction than DeepLOB. According to Tsantekidis et al. [], the structure of CNN-LSTM architecture is able to outperform other models on LOB price movement prediction because of its more stable behaviour. Kolm et al. [] use the OFI feature derived from the LOB to predict the future min-price return, where the CNN-LSTM structure still achieves the best performance.

### 2.2 Transformer-based Time Series Prediction Solutions

As Transformer makes a great contribution to NLP [], many advanced models are proposed, such as BERT and GPT-3, which now have more influence in time series prediction. As the canonical Self-attention mechanism has $O(L^{2})$ time and memory complexity, many modifications have been made to the Transformer in order to adapt to the time series prediction problem to process the long sequence efficiently. There are many alternative Transformer models for long time series forecasting problem have been developed recently []: LogTrans[], Reformer [], Informer [], Autoformer [], Pyraformer [] and the recent FEDformer [].

Theses Transformer models mentioned above are mainly evaluated on non-financial datasets, such as electricity consumption, traffic usage, and solar energy dataset. They achieve a considerable performance increase in accuracy over the LSTM model. Some works start applying Transformers for financial time series prediction. Hu [] uses a Temporal Fusion Transformer with support vector regression (SVR) and LSTM to predict stock price. Sridhar et al. [] predict the Dogecoin price through Transformer, which has superior performance compared to LSTM. Sonkiya et al. [] do sentiment analysis by BERT to generate the sentiment score, which is then combined with other financial indices and passed into the Generative Adversarial Network (GAN) for stock price prediction. These works [, , ] use Transformer-based methods to make predictions based on OHLC data and the research on applying Transformer for LOB prediction is limited.

Overall, LSTM is applied broadly in financial time series prediction and has been tested on various datasets, while the Transformer is limited. Therefore, this study wants to apply these Transformer-based models to a wider region in financial time series prediction to compare their performance to LSTM.

## 3 Financial Time Series Prediction Tasks Formulation

This study compares LSTM-based and Transformer-based methods among three financial prediction tasks based on LOB data. In this section, the basic concept of LOB will be first introduced, and then the formulation of three financial prediction tasks will be explained in detail. Three tasks are listed below:

- •
Task 1: LOB Mid-Price Prediction
- •
Task 2: LOB Mid-Price Difference Prediction
- •
Task 3: LOB Mid-Price Movement Prediction

### 3.1 Limit Order Book

With computers and the Internet, most financial products such as stocks, forex and cryptocurrency are traded on the electronic market nowadays. Two types of orders exist in the electronic market: limit order and market order. According to Gould et al. [], a limit order is the order to execute buy or sell direction at a specific price, where the orders can be succeeded, overdue, or cancelled and are recorded by the LOB. There are bid limit orders and ask limit orders used to buy and sell products by the trader or sell and buy products by the market marker. The highest bid price the buyers are ready to buy is referred to as the best bid price, and the lowest ask price the sellers are ready to sell is called the best ask price. The average of these two prices is called the mid-price, which reflects the current value of the financial product. The difference between them is the spread, which the market marker can usually make a profit. The illustration of the LOB is shown in Fig [1](#S3.F1).

Figure: Figure 1: An illustration of LOB based on Zhang et al. [] and Kolm et al. [].
Refer to caption: /html/2309.11400/assets/figures/Limit_order_book.png

Another type of order is the market order, where the trader can immediately buy/sell the product at the best price.
There exists a matching mechanism in the LOB. Most exchanges adopt the price and time priority matching mechanism. The limit orders will be first executed in order with a better price. If two orders have the same execution price, then the order that comes first in time will be executed first, following the first in first out (FIFO) principle.

### 3.2 Task 1: LOB Mid-Price Prediction

The first task is to predict the LOB Mid-Price Prediction, which is to compare the ability to predict absolute price values similar to non-financial datasets in previous works [, , , , ]. The definition of time series prediction is given below and shown in Figure [2](#S3.F2):

Figure: Figure 2: The illustration of time series prediction.
Refer to caption: /html/2309.11400/assets/figures/TSF.png

First, define a sliding window size $L_{x}$ for the past data. The input data at each time step $t$ is defined as:

$$ $X_{t}=\left\{x_{1},x_{2},\ldots,x_{L_{x}}\right\}_{t}$ (1) $$

Then define a prediction window size $k$, where the goal is to predict the information in future $L_{x}+k$ steps. It will be the single-step prediction when $k=1$ and be multi-horizon prediction when $k>1$. Then the output at time step t is defined as:

$$ $Y_{t}=\left\{y_{1},y_{2},\ldots,y_{k}\right\}_{t}$ (2) $$

The next step is to define the $x_{t}$ and $y_{t}$ in the input and output for mid-price prediction. Assume the market depth is 10. For a limit bid order at time t, the bid price is denoted as $p_{i,t}^{bid}$ and the volume is $v_{i,t}^{bid}$, where $i$ is the market depth. Same for the limit ask order, ask price is $p_{i,t}^{ask}$ and volume is $v_{i,t}^{ask}$. Then the LOB data at time t is defined as:

$$ $x_{t}=\left[p_{i,t}^{ask},v_{i,t}^{ask},p_{i,t}^{bid},v_{i,t}^{bid}\right]_{i=1}^{n=10}\in R^{40}$ (3) $$

The past mid-price will be added to LOB data as input, and the mid-price is represented as:

$$ $p_{t}^{\text{mid }}=\frac{p_{1,t}^{ask}+p_{1,t}^{bid}}{2}$ (4) $$

Finally, the $x_{t}$ will be:

$$ $x_{t}=\left[p_{i,t}^{ask},v_{i,t}^{ask},p_{i,t}^{bid},v_{i,t}^{bid},p_{t}^{\text{mid }}\right]_{i=1}^{n=10}\in R^{41}$ (5) $$

The target is to predict the future mid-price, so $y_{t}=p_{t}^{\text{mid }}$.

### 3.3 Task 2: LOB Mid-Price Difference Prediction

The second task is to predict the mid-price change, which is the the difference of two mid-prices in different time step. Trading strategies can be designed if the price change becomes negative or positive. The input of this task is the same as the mid-price prediction, as described in Equation [3](#S3.E3). The target is to regress the future difference between current mid-price $p_{t}^{\text{mid }}$ and the future mid-price $p_{t+\tau}^{\text{mid}}$:

$$ $d_{t+\tau}=p_{t+\tau}^{\text{mid}}-p_{t}^{\text{mid }}$ (6) $$

Like the mid-price prediction, a prediction window size is defined as $k$, then the output of this task in each timestamp $t$ is represented as:

$$ $Y_{t}=\left\{d_{t+1},d_{t+2},\ldots,d_{t+k}\right\}_{t}$ (7) $$

### 3.4 Task 3: LOB Mid-Price Movement Prediction

According to Ruppert [], the absolute price information is generally non-stationary, while the price change information, such as price difference and return and approximately stationary. The the mid-price difference is a difficult target for deep learning methods to predict because it is hard to extract meaningful pattern from it, although it helps design trading strategies. An example of non-stationary and stationary sequence is shown in Figure [3](#S3.F3).

Figure: Figure 3: An example of non-stationary sequence vs stationary sequence.
Refer to caption: /html/2309.11400/assets/figures/s_ns.png

For this reason, an easier classification task for predicting mid-price movement is introduced here. To train a model to predict mid-price movement, the first step is to create price movement labels for each timestamp. This study follows the smoothing labelling method from Tsantekidis et al. [] and Zhang et al. []: Use $m^{-}$ to represent the average of the last $k$ mid-price and $m^{+}$ to represent the average of the next $k$ mid-price:

$$ $m^{-}(t)=\frac{1}{k}\sum_{i=0}^{k}p_{t-k}^{mid}$ (8) $$

$$ $m^{+}(t)=\frac{1}{k}\sum_{i=1}^{k}p_{t+k}^{mid}$ (9) $$

$k$ is set to $20,30,50,100$ in this study following previous work of Zhang et al. [].
And then, define a percentage change $l_{t}$ to decide the price change direction.

$$ $l_{t}=\frac{m^{+}(t)-m^{-}(t)}{m^{-}(t)}$ (10) $$

The label is dependent on the value of $l_{t}$. A threshold $\delta$ is set to decide the corresponding label. There are three labels for the price movement:

$$ $\text{ label }=\left\{\begin{array}[]{c}0(\text{ fall }),\text{ when }l_{t}>\delta\\ 1(\text{ stationary }),\text{ when }-\delta\leq l_{t}\leq\delta\\ 2(\text{ rise }),\text{ when }l_{t}<-\delta\end{array}\right.$ (11) $$

Figure: Figure 4: An example of labelling on horizon 100 on ETH-USDT dataset with fixed threshold $\delta$. The green colour represents the rise signal. Purple represents the price is stationary and red colour means the price fall.
Refer to caption: /html/2309.11400/assets/figures/labeling.png

An example of labelling for horizon 100 is shown in Figure [4](#S3.F4).
Assume there is an input in Equation [3](#S3.E3) at timestamp $t$, predicting mid-price movement is a one-step ahead prediction, which is to predict the mid-price movement in timestamp $t+1$.

## 4 Methodology

### 4.1 LSTM

LSTM was introduced by Hochreiter et al. [] is one of the RNNs with structural adaptability in time series data input. Although the traditional RNN has the capacity to store data, but it suffers from the exploding gradient problem and vanishing gradient problem. Exploding/vanishing gradient means the gradient that is used to update the neural networks increases/decreases exponentially, which makes the neural network untrainable []. Therefore, RNN is not successful in studying long-time series relations []. LSTM neural network utilizes the coordination of three gates to keep long-term dependency and short-term memory. According to Gers et al. [], the three gates that LSTM utilizes are 1) forget gate, 2) input gate 3) output gate. The structure of an LSTM cell is shown in Figure [5](#S4.F5).

Figure: Figure 5: LSTM Structure based on Graves [] and Fisher [].
Refer to caption: /html/2309.11400/assets/figures/LSTM.png

The calculations of the LSTM are as follows []:
First, the LSTM need to decide to forget some information from the cell state, which is done by the forget gate. The forget gate has its sigmoid function $\sigma$. Then the function of forget gate is:

$$ $\Gamma_{f}^{\langle t\rangle}=\sigma\left(W_{f}\left[h^{\langle t-1\rangle},x^{\langle t\rangle}\right]+b_{f}\right)$ (12) $$

Where the $W_{f}$ is the weight of the last hidden state and input, $b_{f}$ is the bias of the hidden state.
The next step is to design what information the neural cell should remember. To update the information, the input gate should coordinate with a $tanh$ layer containing a vector of new candidate values $\tilde{c}^{\langle t\rangle}$. The calculation is as follows:

$$ $\Gamma_{i}^{\langle t\rangle}=\sigma\left(W_{i}\left[h^{\langle t-1\rangle},x^{\langle t\rangle}\right]+b_{i}\right)$ (13) $$

$$ $\tilde{c}^{\langle t\rangle}=\tanh\left(W_{c}\left[h^{\langle t-1\rangle},x^{\langle t\rangle}\right]+b_{c}\right)$ (14) $$

And then, with the previous steps of calculations, the cell state can be updated:

$$ $c^{\langle t\rangle}=\Gamma_{f}^{\langle t\rangle}\circ c^{\langle t-1\rangle}+\Gamma_{i}^{\langle t\rangle}\circ\tilde{c}^{\langle t\rangle}$ (15) $$

Lastly, calculate the result from the output gate to get the new hidden state:

$$ $\Gamma_{o}^{\langle t\rangle}=\sigma\left(W_{o}\left[h^{\langle t-1\rangle},x^{\langle t\rangle}\right]+b_{o}\right)$ (16) $$

$$ $h^{\langle t-1\rangle}=\Gamma_{o}^{\langle t\rangle}\circ\tanh\left(\tilde{c}^{\langle t\rangle}\right)$ (17) $$

The output of LSTM can be every hidden state or the final hidden state, which depends on the application. In the implementation, this hidden state will be fed into a multi-layer perceptron (MLP), also known as feedforward-backwards propagation neural network (FFBPN). The output of this layer will pass through an activation function to generate the final output.
Usually, the final hidden state will be utilized in financial time series prediction, which produces absolute price prediction or price movement prediction.

#### 4.1.1 Alternative LSTM-based Models

Besides the canonical LSTM, three more LSTM based-models are chosen for comparison to Transformer-based models. They are DeepLOB [], DeepLOB-Seq2Seq [] and DeepLOB-Attention [] created by Zhang et al. The architecture of these three models are shown in Figure [6](#S4.F6) and [7](#S4.F7). Here the structures of these three models are briefly explained:

Figure: Figure 6: DeepLOB architecture sourced from Zhang et al. []. ”Conv $1\times 2$ @16” means there is 16 filters of size $1\times 2$ in this convolutional layer.
Refer to caption: /html/2309.11400/assets/figures/DeepLOB.png

DeepLOB [] DeepLOB’s architecture consists of three main components: Convolutional Blocks, an Inception Module and an LSTM layer.
A. Convolutional Blocks The LOB inputs mentioned in Equation [3](#S3.E3) are fed into the convolutional blocks that contain multiple convolutional layers, where the first and second convolutional blocks are more important than the third one. The first convolutional block has a layer with filter size of ($1\times 2$) and stride of ($1\times 2$). At each order book level, this layer summarise the price and volume information $\left\{p^{(i)},v^{(i)}\right\}$. For the second convolutional block, it has a layer with the same filter size and stride as the first one, but it is a feature mapping for the micro-price defined by []:

$$ $p^{\text{micro }}=Ip_{i}^{ask}+(1-I)p_{i}^{\text{bid }}$ (18) $$

$$ $I=\frac{v_{i}^{bid}}{v_{i}^{ask}+v_{i}^{bid}}$ (19) $$

$I$ is called the imbalance. Then the last convolutional block integrates the feature information from the previous two layers. The whole convolutional blocks work as a feature extractor.

B. Inception Module the Inception Module employs the time series decomposition method. The input is decomposed by two $1\times 1$ convolutions and one max-pooling layer into three lower-dimensional representations. Then these representations pass through convolution layers with 32 channels to be merged together. This decomposition method improves the prediction accuracy.

C. LSTM Layer Finally, the extracted features are inputted into one LSTM layer to capture the underlying pattern and dependencies. The last output layer can be a SoftMax layer or a linear layer, which depends on the specific tasks.

Figure: Figure 7: DeepLOB-Seq2Seq and DeepLOB-Attention architecture sourced from Zhang et al. [].
Refer to caption: /html/2309.11400/assets/figures/deepLOB_Seq2Seq.png

DeepLOB-Seq2Seq [] To generate multi-horizon predictions, Zhang et al. developed DeepLOB-Seq2Seq. The main idea is to feed the output of the Inception Module into a Seq2Seq architecture to do iterated multi-step (IMS) prediction. Seq2Seq [] architecture contains encoder and decoder constructed by recurrent neural network (RNN). Assume the sequence input is $X_{T}=\left(x_{1},x_{2},\cdots,x_{T}\right)$, the encoder will output a hidden state at each timestamp t:

$$ $h_{t}=f\left(h_{t-1},x_{t}\right)$ (20) $$

After obtaining the hidden states from the encoder, a context vector $c$ has to be constructed from these hidden states. The last hidden state or the mean of all hidden states can be taken as a context vector. Context vector work as a ”bridge” between the encoder and decoder, where the context vector is utilized to initialize the decoder, and the hidden state output of the decoder at each timestamp is:

$$ $d_{t}=f\left(d_{t-1},y_{t-1},c\right)$ (21) $$

Then the distribution of the output $y_{t}$ is:

$$ $P\left(y_{t}\mid y_{<t},c\right)=g\left(d_{t},c\right)$ (22) $$

where the output of the decoder not only depends on the previous true value as input but is also conditioned on the context vector.

DeepLOB-Attention [] With the same idea as the DeepLOB-Seq2Seq model, the difference of the DeepLOB-Attention model is changing the Seq2Seq architecture into Attention. The attention model [] constructs the context vector differently instead of using the last hidden state or the mean of all hidden states. Same as Seq2Seq model, the encoder outputs hidden state $h_{t}$ and decoder outputs hidden state $d_{t}$. The first step is to compute a similarity score between the hidden state $d_{t}$ of the decoder and each encoder state $h_{i}$, where the similarity score is usually calculated by dot product:

$$ $s_{i}=h_{i}^{T}d_{t}$ (23) $$

And then, normalize the similarity scores to obtain a weight distribution by softmax:

$$ $\left\{\alpha_{1},\alpha_{2},\ldots,\alpha_{S}\right\}=\operatorname{softmax}\left(\left\{s_{1},s_{2},\ldots,s_{S}\right\}\right)$ (24) $$

Finally, generate the context vector from the attention weights:

$$ $c_{t}=\sum_{i=1}^{S}\alpha_{i}h_{i}$ (25) $$

After that, the process of producing the output $y_{t}$ is the same as the Seq2Seq model.

### 4.2 Transformer

The Transformer [] has an encoder-decoder structure that relies on the Self-attention mechanism without relying on CNN and RNN. This architecture allows the Transformer to process the long sequence and has no problem with the vanishing gradient, which means it can model the dependency regardless of the length of the input/output. A series of components forms the Transformer: Multi-head Self-attention, positional-wise feed-forward neural network, layer-normalization, and residual connection. According to Vaswani et al. [] and Farsani et al. [], the architecture of the Transformer for financial time series prediction is shown in Fig.[8](#S4.F8).

Figure: Figure 8: Transformer Structure based on Vaswani et al. [] and Farsani et al. [].
Refer to caption: /html/2309.11400/assets/figures/Transformer.png

There is a slight difference between this transformer and the vanilla one used for NLP tasks. The word embedding process is omitted, and the financial time series is fed into the transformer using time-stamp encoding. The details of time-stamp encoding will be explained in Section [4.2.2](#S4.SS2.SSS2).

#### 4.2.1 Multi-head Self-attention Mechanism

Self-attention is a mechanism for finding the relevant vector in a sequence. The target of Self-attention is to compute the attention score and then extract information based on the score. According to Vaswani et al. [], the calculations of scaled dot-production Self-attention are as follows:
First, to compute the attention score, multiply the input vector $I$ with different learnable weight matrices $W^{q}$ and $W^{k}$ to obtain the query and key:

$$ $\displaystyle Q=W^{q}I$ (26) $\displaystyle K=W^{k}I$ (27) $$

The next step is to calculate the attention matrix, where each element in it is an attention score:

$$ $A=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)$ (28) $$

Where $d_{k}$ is the dimension of query and key.
Lastly, multiply the attention matrix with the values, and the final output can be obtained:

$$ $\text{ Attention }(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$ (29) $$

The output is the weighted sum of the relevance of different vectors in the sequence.
In the implementation of the transformer, Multi-head Self-attention is used, which is beneficial for finding different types of relevance.

#### 4.2.2 Learnable Time-stamp Encoding

Different from the fixed sinusoid encoding used in the vanilla transformer [], timestamp encoding from the Informer [] is more informative for time series data and a similar method is applied in Autoformer [] and FEDformer []. The timestamp encoding method uses learnable embedding layers to produce positional information to add to the sequence, where timestamp information like a minute, hour, week, year and extra time stamps like event or holiday can be incorporated. To obtain the time-step encoding, the first step is to calculate fixed sinusoid encoding. Assume the input is $X_{t}=\left\{x_{1},x_{2},\ldots,x_{L_{x}}\mid x_{i}\in R^{d_{x}}\right\}_{t}$ at timestamp $t$ , where $L_{x}$ is the sliding window size and $d_{x}$ is the model dimensionality, then the encoding is calculated as follows:

$$ $\displaystyle PE_{pos,2i}=\sin\left(\frac{pos}{\left(2L_{x}\right)^{\frac{2i}{d}}}\right)$ (30) $\displaystyle PE_{pos,2i+1}=\cos\left(\frac{pos}{\left(2L_{x}\right)^{\frac{2i}{d}}}\right)$ (31) $$

And then, project the original input $x_{i}^{t}$ into the model dimensionality vector $u_{i}^{t}$ using convolutional filters.
The next step is to use a learnable embedding layer $\text{SE}_{pos}$ to incorporate the timestamp information. The structure of the timestamp embedding is shown in Fig.[9](#S4.F9).

Figure: Figure 9: Time-stamp embeddings based on Zhou et al. [].
Refer to caption: /html/2309.11400/assets/figures/Timestamp_encode.png

Finally, add up all the calculation results above to get the final encoding:

$$ $X_{feed[i]}^{t}=\alpha u_{i}^{t}+PE_{\left(L_{x}\times(t-1)+i\right)}+\sum_{p}\left[SE_{L_{x}(t-1)+i}\right]_{p}$ (32) $$

Where $i\in\left\{1,\ldots,L_{x}\right\}$ and $\alpha$ is the parameter balancing the ratio of scalar projection and embeddings.

#### 4.2.3 Alternative Transformer-based Models

As mentioned in Section [2](#S2), several new Transformer-based models [, , , , ] are dedicated for long time series forecasting. However, they have not been tested on financial time series data. In this study, they are chosen as the alternative models to compare with the vanilla Transformer and LSTM. These models and their relationships are briefly summarized as follows (see Figure [10](#S4.F10)):

Figure: Figure 10: Architecture designs for alternative Transformer models sourced from Zeng et al.[]. In step (1) and (2), the operations inside the solid box are required, while those in the dashed box can be applied optionally.
Refer to caption: /html/2309.11400/assets/figures/Transformer_relationship.png

LogTrans Li et al. [] put forward LogTrans with convolutional Self-attention generating queries and keys in the Self-attention layer. This work makes the convolutional layer widely used in the attention module in the later studies [, , ]. It uses a Logsparse mask to reduce the time complexity from $O(L^{2})$ to $O(L\log(L))$.

Reformer Reformer [] changes the time complexity of the Transformer to $O(L\log(L))$ as well by using sensitive hashing instead of dot-product in the calculate of attention. It also replaces the residual connection in the vanilla Transformer with a reversible residual connection, which makes the Transformer more efficient.

Informer Although Reformer and LogTrans reduced the time complexity to $O(L\log(L))$, the memory consumption is still the same, so the efficiency gain is not high. Also, LogTrans and Reformer use iterated multi-step prediction (IMS), generating a single prediction in each timestamp and using that prediction iteratively to obtain multi-step prediction [], which suffers from error accumulation. Informer [] proposed by Zhou et al. employs a ProbSparse Self-attention mechanism to achieve $O(L\log(L))$ time and memory complexity. They propose an innovative generative style decoder to make direct multi-step(DMS) prediction, which is to generate the multi-step prediction at once in one forward procedure []. This method speed up the long-term forecast compared to the LogTrans and Reformer. The DMS forecast method and learnable timestamp encoding are applied in the Autoformer [] and FEDformer [].

Autoformer The optimization of Transformer on time series prediction is a trade-off between efficiency and information utilization. Depending on the structure of Informer, Autoformer [] reduced the time complexity with an Auto-Correlation mechanism rather than making Self-attention sparse in LogTrans and Informer, which can preserve the information well and measure the sub-series dependency. Time series decomposition is a method commonly used in time series analysis to deconstruct the time series into several components [, ]. The underlying temporal pattern can be revealed from these components to make the time series more predictable []. Autoformer first embeds the time series decomposition as an inner neural block to derive the trend-cyclical component from the input sequence and the seasonal component from the difference between the trend-cyclical component and the input sequence. This new decomposition architecture can deconstruct time series to use the series periodicity to update attention.

FEDformer Based on the decomposition architecture of Autoformer, Zhou et al. [] builds the FEDformer to use Fourier transform and Wavelet transform, which are in the frequency domain as a new decomposition method to reach linear complexity $O(L)$. .

## 5 Experimentation Result and Evaluation

### 5.1 Comparison of LOB Mid-Price Prediction

The first task to compare transformer versus LSTM is the mid-price prediction. In this task, predicting the absolute value of the mid-price is similar to the previous work’s [, , , , ] experiment on non-financial datasets.

#### 5.1.1 Experiment Setting for LOB Mid-Price Prediction

Dataset All the experiments are based on cryptocurrency LOB data, which are collected in real-time from Binance Exchange using cryptofeed [] WebSocket API and saved to the database using kdb+tick triplet []. In this experiment, one-day LOB data of product BTC-USDT (Bitcoin-U.S. dollar tether) on $2022.07.15$. containing $863397$ ticks is utilized. The time interval between each ticks is not evenly spaced. The time interval is $0.1$ second on average. The first $70\%$ data is used to construct the training set, and the rest $10\%$ and $20\%$ of data are used for validation and testing. The reason why only using one day of LOB data is that it is enough to train for a task prediction for absolute value without overfitting according to previous works’ experiments on non-financial datasets.
Models For the comparison purpose, I choose canonical LSTM and vanilla Transformers along with four Transformer-based models: FEDformer [], Autoformer [], Informer [] and Reformer []. For the implementation of Transformer-based models, they are taken from open-source code repositories [, ]. The implementation of vanilla Transformer [], Reformer [], Informer [] and Autoformer [] are from the Autoformer repository []. The implementation of FEDformer [] is from its own repository [].
Training setting The dataset is normalized by the z-score normalization method. The validation set and the test set are normalized by the mean and standard deviation of the training set. All the models are trained for 10 epochs using the Adaptive Momentum Estimation optimizer and L2 loss with early stopping. The batch size is 32, and the initial learning rate is 1e-4. All models are implemented by Pytorch [] and trained on a single NVIDIA RTX A5000 GPU with 24 GB memory.

#### 5.1.2 Result and Analysis for LOB Mid-Price Prediction

**Table 1: Mid price prediction result with different prediction lengths $k\in\{96,192,336,720\}$ in test set. The input window size is set to $96$ (MSE’s unit is in $10^{-2}$ and MAE’s unit is in $10^{-1}$).**
| Models | FEDformer | Autoformer | Informer | Reformer | Transformer | LSTM |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| 96 | 0.0793 | 0.179 | $0.0926$ | $0.201$ | $1.411$ | $0.543$ | $2.186$ | $0.619$ | $2.836$ | $0.696$ | $0.104$ | $0.204$ |
| 192 | 0.155 | 0.257 | $0.176$ | $0.279$ | $1.782$ | $0.749$ | $1.842$ | $0.824$ | $2.799$ | $0.832$ | $0.195$ | $0.287$ |
| 336 | 0.274 | 0.348 | $0.319$ | $0.376$ | $2.080$ | $0.830$ | $9.218$ | $1.947$ | $1.456$ | $0.665$ | $0.315$ | $0.369$ |
| 720 | 0.608 | 0.514 | $0.643$ | $0.539$ | $2.808$ | $1.093$ | $72.57$ | $6.824$ | $4.306$ | $1.297$ | $0.771$ | $0.587$ |

Quantitative result To evaluate the performance of different models, following the previous works [, , , , ], the performance metrics consist of Mean Square Error (MSE) and Mean Absolute Error (MAE), representing the prediction error. Lower MSE or MAE indicates the model has less prediction error. MSE and MAE are calculated by:

$$ $MSE=\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-\widehat{Y}_{i}\right)^{2}$ (33) $$

$$ $MAE=\frac{1}{n}\sum_{i=1}^{n}\left|Y_{i}-\hat{Y}_{i}\right|$ (34) $$

where $Y_{i}$ is the true value and $\widehat{Y}_{i}$ is the predicted value. $n$ is the number of ticks.
The results of all models are shown in Table [1](#S5.T1). From the table, these outcomes can be summarized:
1. Both FEDformer and Autoformer outperform LSTM and FEDformer has the best performance in all the prediction lengths. FEDformer and Autoformer give a large increase in performance in terms of MSE and MAE compared to LSTM. For FEDformer, it gives $24\%(0.104\rightarrow 0.0793)$ MSE reduction on 96 prediction length and $21\%(0.771\rightarrow 0.608)$ on 336 prediction length. For Autoformer, it gives $11\%(0.104\rightarrow 0.0926)$ MSE reduction on 96 prediction length and $16\%(0.771\rightarrow 0.643)$ MSE reduction on 336 prediction length. These results show that both Autoformer and FEDformer perform well in terms of MSE and MAE because of their low error and long-term robustness.
2. Although FEDformer and Autoformer’s MSE and MAE are low on this task, LSTM is relatively not bad on mid-price prediction. LSTM outperforms the other three models: Informer, Reformer, and vanilla Transformer, which indicates that LSTM is robust in handling LOB data, while transformer-based models require lots of modification to perform well.
3. Vanilla Transformer model has worse performance on prediction lengths $96$ and $192$ and Reformer has worse performance on prediction lengths $336$ and $720$ because they suffered from error accumulation during the IMS prediction process. Informer’s worse performance than LSTM is mainly due to its sparse version of attention, leading to information loss on the time series.

Figure: Figure 11: Illustration of normalized forecasting outputs with 96 input window size and $\{96,192,336,720\}$ prediction lengths. Each timestamp is one tick. Reformer’s results are not plotted in the lower two panels for better visualisation.
Refer to caption: /html/2309.11400/assets/figures/reg_all.png

Qualitative results The prediction results of compared models on all the prediction horizons are shown in Figure [11](#S5.F11). When the prediction horizon is $96$, Autoformer and Reformer are able to generate a proper trend for the future mid-price, while other models generate almost a flat line as predictions. On the prediction horizon of $192$, almost all the models’ predictions plateau except the Reformer, but Reformer’s result becomes more stochastic than the prediction horizon of $96$. For larger prediction horizons $336$ and $720$, all the models can hardly predict a proper trend and Reformer’s result in not plotted because it becomes too stochastic.

Based on the qualitative results above, although Autoformer and FEDformer outperform LSTM in terms of MSE and MAE, their actual prediction performance is far more inadequate for high-frequency trading. The analysis from Figure [11](#S5.F11) is just “eyeballing” whether the model is good. In this case, another formal metric out of sample $R^{2}$ is added here to judge the prediction quality. According to Lewis-Beck [], $R^{2}$ determine how well the prediction result $Y$ can be explained by the input $X$, and higher $R^{2}$ indicates the model has a better fit for the predicted value. Out of sample $R^{2}$ is defined by:

$$ $R^{2}=1-\frac{\sum_{i=1}^{n}\left(Y_{i}-\widehat{Y}_{i}\right)^{2}}{\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)}$ (35) $$

where $\bar{Y}=\frac{1}{n}\sum_{i=1}^{n}Y_{i}$; $\widehat{Y}_{i}$ is the predicted value and $Y_{i}$ is the ground truth.
Here the out-of-sample $R^{2}$ is calculated based on the price difference. Please note that the price difference here differs from the one mentioned in Section [3.3](#S3.SS3). The price difference here is calculated from the absolute price prediction result, while the one in Section [3.3](#S3.SS3) is the direct prediction target. The result of $R^{2}$ is shown in Table [2](#S5.T2).

**Table 2: Average of out of sample $R^{2}$ result with different prediction lengths $k\in\{96,192,336,720\}$.**
| Models | Autoformer | FEDformer | Informer | Reformer | LSTM | Transformer |
| --- | --- | --- | --- | --- | --- | --- |
| 96 | -0.753 | -0.237 | -43.811 | -69.080 | -0.946 | -87.899 |
| 192 | -0.596 | -0.205 | -25.281 | -26.792 | -0.644 | -43.368 |
| 336 | -1.032 | -0.364 | -20.123 | -63.252 | -0.414 | -13.035 |
| 720 | -0.521 | -0.189 | -7.760 | -137.322 | -0.589 | -16.314 |

From the table, all the out-of-sample $R^{2}$ values are negative for all models. However, according to Lewis-Beck [], $R^{2}$ at least needs to be larger than zero to indicate that input $X$ can explain output $Y$. This indicates that predictions for absolute mid-price are useless for trading purposes. The metrics of MSE and MAE obscure the real quality of prediction results, highlighting the importance of $R^{2}$ calculated based on the price difference.

To sum up, although Autoformer and FEDformer have lower MSE and MAE than LSTM, their prediction result is not helpful and practical for trading. The more sensible way is to directly use the price difference as the prediction target.

### 5.2 Comparison of LOB Mid-Price Diff Prediction

Both transformer-based models and LSTM are unable to generate the satisfactory result on the mid-price prediction, so this task turns to the mid-price difference prediction. The mid-price difference is a useful alpha-term structure in trading for traders and market makers [].

#### 5.2.1 Experiment Setting for LOB Mid-Price Diff Prediction

Dataset The dataset for this experiment is collected the same way as the last experiment, but the dataset size becomes larger to avoid overfitting. In this experiment, four days of LOB data for the product BTC-USDT from 2022.07.03 (inclusive) to 2022.07.06 (inclusive) is used, containing 3432211 ticks. The first $80\%$ of data is used as a training set, and the rest $20\%$ is split in half for validation and testing.
Models and Limitations Five models are being compared in this experiment: canonical LSTM [], vanilla transformer [], CNN-LSTM (DeepLOB [] model used for regression), Informer [] and Reformer []. State-of-the-art FEDformer and Autoformer are not compared in this task because their time decomposition structure is limited in handling price difference series. As mentioned in Section [3.4](#S3.SS4), the price difference is approximately stationary. The time decomposition method is useful for the non-stationary time series, such as the mid-price series, where it can extract meaningful patterns. In contrast, the price difference series is approximately stationary, and little meaningful information can be extracted from it. Therefore, FEDformer and Autoformer can only produce a poor result in this task and will not be compared below.
Training settings the training setting is the same as the last experiment.

#### 5.2.2 Result and analysis for LOB Mid-Price Diff Prediction

Figure: Figure 12: Performance of price difference prediction with input window size 100 and prediction length 100. Negative data points are not plotted for ease of visualization.
Refer to caption: /html/2309.11400/assets/figures/price_diff.png

Following the previous works [], out of sample $R^{2}$ is the evaluation metric for this task. The performance of all the models is shown in Figure [12](#S5.F12). The canonical LSTM achieves the best performance among all models, which reaches the highest $R^{2}$ around $11.5\%$ in forecast length $5$ to $15$. For CNN-LSTM, it has comparable performance to LSTM. On the other hand, Informer, Reformer and Transformer have worse $R^{2}$ than LSTM, but their $R^{2}$ trend is similar. In short, for the price difference prediction task, LSTM-based models is more stable and more robust than Transformer-based models. This result is in expectation because Reformer, Informer and Transformer already have worse performance than LSTM in mid-price prediction task because of their shortcomings. At the same time, the state-of-art FEDformer and Autoformer cannot be applied because of their limitation. In order to let these state-of-the-art transformer-based models make a meaningful prediction, a new structure is designed in the next part, and it is applied to the price movement prediction task.

### 5.3 Comparison of LOB Mid-Price Movement Prediction

#### 5.3.1 Innovative Architecture on Transformer-based Methods

The alternative Transformer-based models mentioned in Section [4.2.3](#S4.SS2.SSS3) mainly focus on the long time series forecasting problem, which is a regression task. For the LOB data, it is easy to adapt these models for the mid-price prediction and mid-price difference prediction because both are regression problems. For the mid-price movement prediction task, the model needs to produce a classification result for the future, and there are few existing Transformer models specialized available for this task. In contrast, most of the Transformer-based models are designed for classification tasks without forecasting, such as sentiment analysis, spam detection and pos-tagging in NLP. In this case, adapting the existing Transformer-based models to do price movement forecasting is necessary. I first adapt Transformer-based models in price movement forecasting and want to facilitate transformer development in this specific task. The new architecture of the transformer-based model is shown in Figure [13](#S5.F13). The details are explained as follows:
Predicting the next mid-price movement based on the past price and volume information is an one step ahead of prediction.

Figure: Figure 13: New architecture of transformer-based model for LOB mid-price movement prediction.
Refer to caption: /html/2309.11400/assets/figures/trans_innov.png

A straightforward method to adapt the transformer-based model is to pass the next predicted mid-price into a softmax activation. However, this method performs poorly because it only considers the past mid-price information and ignores future ones. It is worth noting that in the labelling process in Section [3.4](#S3.SS4), previous and next $k$ mid-price information are utilized. In this case, I adapt the existing transformer-based models to feed the whole predicted mid-price sequence into a linear layer and finally pass through a softmax activation function to generate price movement output. This adaptation will benefit those transformer-based models using the DMS forecasting method because they have fewer errors in the long-time series prediction process.

#### 5.3.2 DLSTM: Innovation on LSTM-based Methods

Inspired by the Dlinear model [] and Autoformer, combining the merits of time decomposition with the LSTM, a new model named DLSTM is designed.
DLSTM is designed based on these three observations: Firstly, the time series decomposition method is capable of increasing the performance, especially embedding this process by neural blocks in previous works [, , ]. Secondly, LSTM is a robust and simple model for multiple forecasting tasks. Thirdly, Dlinear beats other Transformer-based models in some long time series forecasting tasks thanks to the time series decomposition method and DMS prediction. However, predicting price movement is one step ahead prediction, where the model will not suffer from the error accumulation effect. In this case, it is sensible to replace linear with LSTM, because LSTM is a model well-known better than linear in handing time series.

Figure: Figure 14: Architecture of DLSTM
Refer to caption: /html/2309.11400/assets/figures/DLSTM.png

The architecture of DLSTM is shown in Figure [14](#S5.F14). The main difference between Dlinear and DLSTM is that the LSTM layers replace the Linear layer. According to the time decomposition method introduced in Autoformer [], in the prediction process, assume there is a time series $X_{T}=\left(x_{1},x_{2},\ldots,x_{T}\right)$, first decompose it into Trend series by the moving average:

$$ $X_{t}=AvgPool(Padding(X_{T}))$ (36) $$

where $AvgPool(\cdot)$ is the average pooling operation and the $Padding(\cdot)$ is to fix the input length.
And then the Remainder series is calculated by $X_{r}=X_{T}-X_{t}$. After that, these two series are inputted into two LSTM layers. Finally, the hidden states $H_{t}$ and $H_{r}$ produced by two LSTM layers will be added together and then pass through a linear and softmax activation to generate the final price movement result.

#### 5.3.3 Setting for LOB Mid-Price Movement Prediction

Dataset In this experiment, the largest dataset among three tasks is utilized to avoid over-fitting and test the model’s robustness. The whole dataset contains 12 days of LOB data of product ETH-USDT (Ethereum-U.S. dollar tether) from 2022.07.03 (inclusive) to 2022.07.14 (inclusive), containing 10255144 ticks. The training and testing data are taken from the first six days and the last three days, and the left data are used for validation. The test set is also used for the simple trading simulation.
Models Thanks to the innovative structure mentioned in Section [5.3.1](#S5.SS3.SSS1), most of the transformer-based models can be adapted and applied in this task for comparison, which are: Vanilla Transformer [], Reformer [], Informer [], Autoformer [], FEDformer []. On the other hand, all the LSTM-based models are compared in this task as well, which are: canonical LSTM [], DLSTM, DeepLOB [], DeepLOB-Seq2Seq [], DeepLOB-Attention []. Besides these models, a simple MLP model is built as a baseline. The implementation of the Transformer-based models are based on the code repository mentioned above. The implementation of DeepLOB [], DeepLOB-Seq2Seq [], DeepLOB-Attention [] are based on two repositories [, ]. For the DLSTM, it is inspired by code of Dlinear[] model from its repository [].
Training settings The batch size for training is set to 64 and the loss function is changed to Crossentropy loss. Other training settings are the same as the last experiment.

#### 5.3.4 Result and analysis for LOB Mid-Price Movement Prediction

The performance of models is evaluated by classification metrics: accuracy and the mean of precision, recall and F1-score. Result are shown in Table [3](#S5.T3) and Table [4](#S5.T4).

**Table 3: Experiment results of Mid Price Movement for prediction horizons 20 and 30. Red Bold represents the best result and \ulblue underline represents the second best result.**
| Model | Accuracy | Precision | Recall | F1 |
| --- | --- | --- | --- | --- |
| Prediction Horizon k = 20 |  |  |  |  |
| MLP | 61.58 | 61.70 | 61.58 | 61.47 |
| LSTM | 62.77 | 62.91 | 62.77 | 62.78 |
| DeepLOB | 70.29 | 70.58 | 70.30 | 70.24 |
| DeepLOB-Seq2Seq | \ul70.40 | \ul70.79 | \ul70.42 | \ul70.37 |
| DeepLOB-Attention | 70.04 | 70.26 | 70.03 | 70.01 |
| Autoformer | 68.89 | 68.99 | 68.89 | 68.91 |
| FEDformer | 65.37 | 65.70 | 65.37 | 65.20 |
| Informer | 68.71 | 68.82 | 68.72 | 68.71 |
| Reformer | 68.01 | 68.26 | 68.00 | 67.95 |
| Transformer | 67.80 | 67.99 | 67.81 | 67.77 |
| DLSTM | 73.10 | 74.01 | 73.11 | 73.11 |
| Prediction Horizon k = 30 |  |  |  |  |
| MLP | 59.19 | 59.30 | 58.70 | 58.48 |
| LSTM | 60.64 | 60.47 | 60.45 | 60.45 |
| DeepLOB | 67.23 | 67.26 | 67.17 | 67.15 |
| DeepLOB-Seq2Seq | 67.56 | 67.73 | 67.53 | 67.49 |
| DeepLOB-Attention | 67.21 | 67.39 | 66.98 | 66.96 |
| Autoformer | \ul67.93 | \ul67.86 | \ul67.77 | \ul67.77 |
| FEDformer | 66.57 | 66.44 | 66.05 | 65.83 |
| Informer | 65.41 | 65.33 | 65.14 | 65.13 |
| Reformer | 64.28 | 64.31 | 64.08 | 64.06 |
| Transformer | 64.25 | 64.16 | 64.13 | 64.13 |
| DLSTM | 70.61 | 70.83 | 70.63 | 70.59 |

**Table 4: Experiment results of Mid Price Movement for prediction horizons 50 and 100. Red Bold represents the best result and \ulblue underline represents the second best result.**
| Model | Accuracy | Precision | Recall | F1 |
| --- | --- | --- | --- | --- |
| Prediction Horizon k = 50 |  |  |  |  |
| MLP | 55.65 | 55.71 | 55.62 | 54.98 |
| LSTM | 62.77 | 62.91 | 62.77 | 62.78 |
| DeepLOB | 63.32 | 63.69 | 63.32 | 63.37 |
| DeepLOB-Seq2Seq | 63.62 | 64.04 | 63.61 | 63.59 |
| DeepLOB-Attention | \ul64.05 | \ul64.19 | \ul64.04 | \ul63.94 |
| Autoformer | 60.17 | 60.64 | 60.12 | 58.40 |
| FEDformer | 63.46 | 63.44 | 63.42 | 62.52 |
| Informer | 61.76 | 61.64 | 61.74 | 61.55 |
| Reformer | 60.43 | 60.79 | 60.42 | 60.37 |
| Transformer | 59.51 | 59.78 | 59.51 | 59.46 |
| DLSTM | 67.45 | 67.96 | 67.45 | 67.59 |
| Prediction Horizon k = 100 |  |  |  |  |
| MLP | 57.03 | 56.03 | 56.36 | 56.01 |
| LSTM | 53.49 | 52.83 | 52.82 | 52.36 |
| DeepLOB | 58.12 | 58.50 | 57.92 | 57.86 |
| DeepLOB-Seq2Seq | 58.30 | 58.43 | 57.93 | 57.77 |
| DeepLOB-Attention | 59.16 | \ul58.59 | \ul58.65 | \ul58.50 |
| Autoformer | \ul59.18 | 58.34 | 58.40 | 57.83 |
| FEDformer | 57.97 | 56.97 | 56.62 | 54.14 |
| Informer | 56.11 | 56.15 | 55.85 | 55.81 |
| Reformer | 54.92 | 54.47 | 54.53 | 54.47 |
| Transformer | 55.42 | 55.04 | 54.92 | 54.72 |
| DLSTM | 63.73 | 63.02 | 63.18 | 63.05 |

A few outcomes can be observed from the result:
1. DLSTM outperforms all the previous LSTM-based and Transformer-based models. It achieves the highest accuracy, precision, recall and F1 score in all the prediction horizons. This result shows that the time series decomposition structure originating from Autoformer can effectively handle time series, especially when combined with a simple LSTM model. DLSTM is making one step ahead prediction for the mid-price movement, so it will not suffer from error accumulation from the DMS prediction process.
2. DeepLOB-Attention model has the second best result in horizon 50 and 100 (excluding accuracy). DeepLOB-Seq2Seq has the second best result for prediction horizon 20. This result indicates that the encode-decoder structure and attention mechanism can contribute to the prediction performance because the autoregressive process can correlate the mid-price movement from different prediction horizons.
3. The DeepLOB-Attention and DeepLOB-Seq2Seq performance is comparable to DeepLOB but better than DeepLOB, especially in the long prediction horizon. This result accords with the result in the previous paper [], which proves the correctness of the result.
4. The Autoformer gets the second-best result in prediction horizon 30. Although it is not the best model, it still means that Autoformer is usable for the time series prediction, and its time decomposition structure is adequate. The shortcoming of Autoformer and the latest FEDformer is that they are huge models compared to LSTM, and they need to be fine-tuned to work well in a specific task. In contrast, LSTM-based models’ sizes are much smaller and do not need much hyper-parameters tuning. More analysis of the efficiency will be discussed in the Section [5.3.7](#S5.SS3.SSS7).

To summarize, combing the results of this task and previous tasks, LSTM-based models generally have their advantage in financial time series for their robustness and good compatibility. Although the Transformer-based model is large and complicated to tune and requires a long training time, they are still usable for the forecasting task. Furthermore, the research of Transformer-based method in time series prediction is meaningful because the time decomposition method from Autoformer contributes back to the original LSTM model.

Figure: Figure 15: Confusion matrix of DLSTM model with prediction horizon 20, 30, 50, 100.
Refer to caption: /html/2309.11400/assets/figures/confusion.png

#### 5.3.5 Simple Trading Simulation without transaction cost

In order to show the models and their predictions are practical and useful in trading, a simple trading simulation (backtesting) is designed. Three models with good classification metrics performance are chosen for comparison: DLSTM, DeepLOB [], Autoformer []. Canonical LSTM [] and Vanilla Transformer [] are used as baselines. The three-day test set is used for this trading simulation. To make a fair comparison among models, the trading simulation follows the simple setup in the previous work []:
The number of shares pre-trade $\mu$ (volume) is set to one. At each timestamp, the model will predict the price movement (0 0: fall, $1$: stationary, $2$: rise) as a trading signal. When the prediction is $2$, enter the long position, and the position is held until it encounters 0 0. The same rule is applied to the short position when the prediction is 0 0, and only one direction of position can exist in this simulation trading. A delay is set between the prediction and the order execution to simulate the high-frequency trading latency. For example, assume the model generates a prediction $2$ at time $t$, $\mu$ shares will be bought at time $t+5$.

**Table 5: Cumulative price returns and annualized sharpe ratio of different models.**
| Forecast Horizon | Prediction Horizon = 20 | Prediction Horizon = 30 | Prediction Horizon =50 | Prediction Horizon=100 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | CPR | SR | CPR | SR | CPR | SR | CPR | SR |
| LSTM | 15.396 | 51.489 | 12.458 | \ul41.411 | 8.484 | 28.817 | 4.914 | 20.941 |
| DLSTM | \ul14.966 | 46.949 | 12.634 | 37.432 | 6.194 | 22.027 | 3.215 | \ul16.346 |
| DeepLOB | 13.859 | \ul56.094 | 12.789 | 42.567 | 5.726 | 21.014 | 2.646 | 14.992 |
| Transformer | 14.553 | 59.995 | \ul12.737 | 41.044 | 6.896 | \ul28.147 | 2.859 | 16.981 |
| Autoformer | 9.942 | 32.688 | 8.617 | 30.576 | \ul8.214 | 25.882 | \ul3.620 | 17.765 |

Several assumptions are made for the simulation trading:
1) Since the trading product is in cryptocurrency exchange, the trading volume is considerable sufficient in the market, which means the simulated trades will not have a market impact.
2) The focus of this part of the experiment is to show the practicality of the prediction result and make relative comparisons among models instead of inventing a fully developed high-frequency trading strategy. Industrial HFT trading strategies usually require the combination of different prediction signals and precise entry exit rules []. For simplicity, the order is assumed to be executed at the mid-price without transaction cost.

Figure: Figure 16: Cumulative return curve for different models in trading simulation.
Refer to caption: /html/2309.11400/assets/figures/backtest.png

As displayed in Table [5](#S5.T5) and Figure [16](#S5.F16), each model’s profitability is presented. The performance of simulated trading is evaluated by cumulative price return (CPR) and the Annualized Sharpe Ratio (SR). The CPR is formulated by:

$$ $CPR=\sum_{1}^{t}s*\mu*\left(p_{\text{mid }}^{\text{holding },t}-p_{\text{mid }}^{\text{settlement },t}\right)$ (37) $$

where $s$ is the trading position, which is $1$ for long position and $-1$ for short position. $\mu$ is the number of shares.
And the Sharpe Ratio is calculated by:

$$ $SR=\sqrt{365}\times\frac{\text{ Average }(\text{ daily }CPR)}{\text{ StandardDeviation }(\text{ daily }CPR)}$ (38) $$

The value of annualized SR is enormous because the assumptions mentioned above are not realistic for practical trading.
Based on the results, LSTM based-model’s performance in simulated trading is generally better than Transformer-based model. The canonical LSTM model achieves highest CPR and SR in prediction horizon 20 and 30 and DeepLOB has the best performance in prediction horizon 50. For DLSTM, it has comparable performance to canonical LSTM and DeepLOB model. This result shows that the prediction result from LSTM-based models are robust and practical for trading. Autoformer’s CPR is the lowest in prediction horizon 20 and 30. The state-of-the-art Autoformer sometimes has even worse performance than the vanilla Transformer in simulated trading, although it obtains a better classification metrics. To summarize, LSTM-based models are relatively the better models for electronic trading.

#### 5.3.6 Simple Trading Simulation with transaction cost

In the real-world market, all operations, including buying or selling, need a commission fee, and sometimes the transaction cost might outweight the return. This section will introduce a hypothetical transaction cost of $0.002\%$ to further compare the robustness among different models. The results are shown in Table [6](#S5.T6) and Figure [17](#S5.F17).

**Table 6: Cumulative price returns and annualized sharpe ratio of different models under $0.002\%$ transaction cost.**
| Forecast Horizon | Prediction Horizon = 20 | Prediction Horizon = 30 | Prediction Horizon =50 | Prediction Horizon=100 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | CPR | SR | CPR | SR | CPR | SR | CPR | SR |
| LSTM | \ul2.102 | \ul15.160 | 1.767 | 12.429 | \ul1.596 | \ul11.536 | 0.778 | 6.014 |
| DLSTM | 3.039 | 19.962 | 2.716 | 16.523 | 1.957 | 12.359 | 1.180 | 9.811 |
| DeepLOB | 1.964 | 15.082 | \ul1.924 | \ul13.128 | 1.450 | 10.273 | \ul0.823 | \ul7.993 |
| Transformer | 1.860 | 13.894 | 1.561 | 10.917 | 1.047 | 6.612 | 0.118 | -23.496 |
| Autoformer | 0.189 | -8.704 | 0.873 | 5.118 | -0.225 | -9.193 | -0.061 | -14.835 |

From the table, DLSTM has the highest CPRs and SRs for all the prediction horizons, outperforming all other models. This shows DLSTM’s strong profitability and robustness against the risk brought by the transaction cost. LSTM-based methods’ performance is generally better than Transformer-based methods. Canonical LSTM and DeepLOB achieve the second-best CPRs and SRs in different prediction horizons. This indicates that the LSTM-based model’s prediction results are more practical and effective in electronic trading. Interestingly, Transformer-based models’ performance drops significantly under the transaction cost. The state-of-the-art Autoformer produces even less profit than vanilla Transformer, yielding negative CPRs and SRs in prediction horizon 50 and 100, although its prediction classification metrics is better than Transformer.

Figure: Figure 17: Cumulative return curve under $0.002\%$ transaction cost.
Refer to caption: /html/2309.11400/assets/figures/backtest_fee.png

To further investigate the impact of transaction cost on simulated trading. An experiment is extended to see how the CPR and SR change as the transaction cost increase. The experiment on CPR is done on the three-day test set from $2022.07.12$ (inclusive) to $2022.07.14$ (inclusive) as mentioned in Section [5.3.3](#S5.SS3.SSS3). The experiment on SR has a longer backtesting period ranging from $2022.07.13$ (inclusive) to $2022.07.24$ (inclusive) because annualized Sharpe Ratio is calculated based on daily CPR, so using a longer backtesting period can produce a more accurate SR. The experiment results are shown in Figure [18](#S5.F18) and Figure [19](#S5.F19).

Figure: Figure 18: Cumulative price return change with increasing transaction cost (back testing period: $2022.07.12$ (inclusive) to $2022.07.14$ (inclusive)).
Refer to caption: /html/2309.11400/assets/figures/cpr_tx.png

Figure: Figure 19: Sharpe ratio change with increasing transaction cost (back testings period: $2022.07.13$ (inclusive) to $2022.07.24$ (inclusive)).
Refer to caption: /html/2309.11400/assets/figures/sr_tx.png

In terms of the CPR, all models’ CPR decreases as the transaction cost increases. For the prediction horizon $20$ and $30$, DLSTM still outperforms other models when the transaction cost increases. All the models generate comparable CPR except the Autoformer in the prediction horizon $50$ and $100$. Autoformer’s CPR is the lowest in all prediction horizons for most transaction cost settings. Regarding the SR, all models’ SR decrease as the transaction cost increases. However, DLSTM maintains a higher SR than other models as the transaction cost increases. At the same time, for the Autoformer, its SR drops significantly and even becomes the lowest as the transaction cost increases. Overall, DLSTM keeps its profitability and robustness for different transaction costs, while the Transformer-based method’s performance can be largely affected by the transaction cost. This result further indicates that the LSTM-based method is superior for electronic trading.

#### 5.3.7 Efficiency Comparison on Transformers versus LSTM

**Table 7: Efficiency comparison of Transformer-based and LSTM-based models on price movement prediction. MACs represent the number of Multiply-accumulate operations.**
| Method | MACs | Parameter | Time | Memory | Time | Memory | Test Step |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DLSTM | 6.72 M | 193.9 k | 4.4ms | 1404MiB | $O(L)$ | $O(L)$ | 1 |
| DeepLOB | 36.42 M | 143.91 k | 6.3ms | 2250MiB | $O(L)$ | $O(L)$ | 1 |
| Transformer | 1.25 G | 10.64 M | 17.7ms | 3534MiB | $O(L^{2})$ | $O(L^{2})$ | 1 |
| Reformer | 1.17 G | 5.84 M | 23.6ms | 4966MiB | $O(LlogL)$ | $O(L^{2})$ | 1 |
| Informer | 1.15 G | 11.43 M | 22.1ms | 4361MiB | $O(LlogL)$ | $O(LlogL)$ | 1 |
| Autoformer | 1.25 G | 10.64M | 75.2ms | 5394MiB | $O(L)$ | $O(L)$ | 1 |
| FEDformer | 1.25 G | 16.47 M | 38.3ms | 3556MiB | $O(L)$ | $O(L)$ | 1 |

The efficiency comparison of Transformer-based and LSTM-based models on price movement prediction is shown in Table [7](#S5.T7). The comparison is separated into two parts, the left panel is the practical efficiency, and the right is the theoretical efficiency.
The latest transformer-based models have a focus on lowering the time and memory complexity. Autoformer and FEDformer claim to achieve $O(L)$ time and memory complexity in theory. However, their actual inference time and memory consumption are higher than the vanilla transformer models because of their complex design. For the training process, it usually takes more than $12$ hours to train an Autoformer and FEDformer model, even with the cutting-edge GPU device (e.g., 24GB NVIDIA RTX 3090 GPU is used here), which is not efficient to retrain the model on new data. The researchers should reconsider the focus of the Transformer-based model on time series application. The time and memory complexity is not a big threshold for the vanilla Transformer, where its inference speed and memory consumption is acceptable depending on today’s computing power.

The LSTM-based model has higher efficiency than the Transformer-based model for its low inference time and small model size, where its theoretical efficiency corresponds to its practical efficiency. This gives the LSTM-based model advantage in high-frequency trading, which requires fast execution speed. This also again emphasizes that LSTM-based models are the better model in electronic trading.

## 6 Conclusion and Future work

This study systematically compares LSTM-based and Transformer-based models among three financial time series prediction tasks based on cryptocurrency LOB data. The first task is to predict the LOB mid-price. FEDformer and Autoformer have less error than other models, and LSTM is still a strong model that surpasses Informer, Reformer and vanilla Transformer. Although the mid-price prediction error is low, the quality of the mid-price prediction result is far from sufficient for practical use in high-frequency trading. The second task is to predict LOB mid-price difference. LSTM-based methods show their robustness in time series prediction and perform better than Transformer-based models, which reach the highest $11.5\%$ $R^{2}$ in around $10$ prediction steps. State-of-the-art Autoformer and FEDformer are limited in this task because their time decomposition architecture can not handle the difference sequence. However, in a separate study [], it was shown that custom transformer configurations can outperform the standard transformers.

The last task is to predict the LOB mid-price movement. New architecture for the Transformer-based model is designed for adapting the classification task. A new DLSTM model is proposed combining the merits of LSTM and time decomposition architecture from Autoformer. DLSTM outperforms all other models in classification metrics, and Autoformer shows comparable performance to LSTM-based models. A simple trading simulation is done to verify the practicality of the prediction. LSTM-based models have overall better performance than Transformer-based models and DLSTM model beats all other models under the transaction cost.

In conclusion, based on all the experiments on three different tasks, the Transformer-based model can only outperform LSTM-based models by a large margin in terms of the limited metrics for mid-price prediction. In comparison, the LSTM-based model is still dominant in the later two tasks, so LSTM-based models are generally the better model in financial time series prediction for electronic trading.

For future research, applying LSTM-based and Transformer-based models in Deep Reinforcement Learning (DRL) can be a proper direction. A complete high-frequency trading strategy usually requires the combination of different prediction signals and needs an experienced trader to control the take-profit and stop-loss. In this case, using DRL to generate the optimal trading strategy directly can get us one step closer to the actual trading.

## Appendix A Labelling Details

As mentioned in Section [3.4](#S3.SS4), a threshold $\delta$ needs to be set to decide the corresponding labels. The choice of $\delta$ follows a simple rule, which is to make the labelling roughly balanced. The choice of $\delta$ for different prediction horizon on ETH-USDT dataset is shown in Table [8](#A1.T8) and the distribution of labelling is shown in Figure [20](#A1.F20).

**Table 8: $\delta$ for different prediction horizons for ETH-USDT dataset.(units in $10^{-4}$)**
| Horizon | 20 | 30 | 50 | 100 |
| --- | --- | --- | --- | --- |
| $\delta$ | 0.17 | 0.3 | 0.6 | 0.92 |

Figure: Figure 20: Labelling distribution for different prediction horizon in ETH-USDT dataset
Refer to caption: /html/2309.11400/assets/figures/all_label.png

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. What are the fundamental architectural characteristics that make LSTM models more effective for differential sequences compared to Transformers?
2. Can you provide deeper analysis to support the generalizability of your findings of LSTM vs Transformer?
3. How does financial time series forecasting different from other time series forecasting (like weather, traffic, etc.?)
4. To address the remaining limitations identified in *Weaknesses*: a) Could you provide detailed model implementations and hyperparameter configurations? b) How would decomposition techniques benefit Transformer architectures? c) Please include comparisons with state-of-the-art models
soundness: 2
strengths: 1. The paper addresses a relevant and significant question by comparing LSTM and Transformer models in financial time series forecasting.
2. The experimental setup is extensive and provides substantial data.
summary: This research compares the effectiveness of Transformer and LSTM architectures in financial forecasting. The study examines both model types using high-frequency trading data and introduces DLSTM and a finance-specific Transformer. Results show that Transformers only slightly outperform in absolute price predictions, while LSTMs showing more reliable performance overall.
weaknesses: 1. The paper lacks code and detailed implementation information for both the Transformer and LSTM models, which limits reproducibility.
2. The novelty of the proposed approach is limited. While the authors introduce a DLSTM model to improve performance, the idea of decomposition was previously explored in models like DLinear [1], diminishing the originality of the contribution. Beyond the comparative analysis, additional innovation is also limited.
3. The decomposition strategy appears to be applied only to the LSTM model. For a fair comparison, a decomposition approach for the Transformer model should also be included. In Table 3, DLSTM significantly outperforms LSTM, which suggests that a decomposed Transformer might also show improved results.
4. The paper does not include several state-of-the-art (SOTA) Transformer-based models, such as PatchTST [2], Crossformer [3], and iTransformer [4], in the comparison, which limits the comprehensiveness of the analysis.
5. The statement "Transformer-based models exhibit only a marginal advantage in predicting absolute price sequences, whereas LSTM-based models demonstrate superior and more consistent performance in predicting differential sequences such as price differences and movements" requires further investigation. A deeper analysis into the underlying causes of this observed difference is missing, which weakens the interpretability of the results.

[1] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023.

[2] Nie, Yuqi, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." The Eleventh International Conference on Learning Representations.

[3] Zhang, Yunhao, and Junchi Yan. "Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting." The eleventh international conference on learning representations. 2023.

[4] Liu, Yong, et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." The Twelfth International Conference on Learning Representations.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Dataset diversity and generalizability: Can you provide more insights into the choice of using only Binance LOB data for a single cryptocurrency pair in your experiments? How do you expect the proposed DLSTM model and the comparative analysis between LSTM-based and Transformer-based models to perform on a wider range of financial instruments, such as stocks, forex, or other cryptocurrencies, as well as data from multiple exchanges? Providing results on more diverse datasets could strengthen the claims of generalizability and robustness of the findings.
Ablation studies and component contributions: Can you conduct ablation studies to investigate the individual contributions of the time series decomposition approach in the proposed DLSTM model? It would be helpful to compare the performance of DLSTM with and without this specific modification to assess its impact on the model's effectiveness. Additionally, can you provide a more detailed analysis of the adapted Transformer-based models' architecture for the movement prediction task, highlighting the importance of each proposed change?
Model interpretability: Can you elaborate on the interpretability of the proposed DLSTM model and the adapted Transformer-based models? How do these models compare with other LSTM-based and Transformer-based models in terms of interpretability? Providing insights into the factors driving the models' predictions and their relative importance could be valuable for understanding the models' decision-making process and enhancing trust in their applications for electronic trading.
Hyperparameter tuning and model selection: Can you provide more details on the hyperparameter tuning process and model selection criteria used for the various models in your experiments? Specifically, what approach was used for hyperparameter optimization (e.g., grid search, random search, Bayesian optimization), and which hyperparameters were tuned for each model? Additionally, how were the validation sets or cross-validation techniques employed in the model selection process?
Robustness to market conditions: Have you considered evaluating the performance of the proposed DLSTM model and the comparative analysis between LSTM-based and Transformer-based models under different market conditions, such as periods of high volatility, market crashes, or significant news events? Demonstrating the models' ability to generalize and adapt to various market scenarios could provide a more comprehensive assessment of their robustness and practical applicability in electronic trading.
soundness: 3
strengths: Originality: The study offers a novel perspective on the application of LSTM-based and Transformer-based models in financial time series forecasting, specifically in the context of electronic trading using high-frequency LOB data. The authors introduce a new LSTM-based model, DLSTM, which creatively combines LSTM with a time series decomposition approach inspired by the Autoformer architecture. This innovative integration of existing ideas allows DLSTM to outperform other models in the mid-price movement prediction task.
Quality: The paper demonstrates a high level of quality in its experimental design and analysis. The authors conduct a comprehensive comparative study across three prediction tasks (mid-price prediction, mid-price difference prediction, and mid-price movement prediction), using a diverse range of LSTM-based and Transformer-based models. The experiments are well-structured, and the results are thoroughly analyzed, providing valuable insights into the performance of different models in each task.
Clarity: The paper is well-written and easy to follow. The authors provide clear explanations of the problem formulation, the proposed DLSTM model, and the experimental setup. The use of tables and figures enhances the clarity of the results, making it easy for readers to compare the performance of different models across various metrics and prediction horizons.
Significance: The findings of this study have significant implications for the application of deep learning models in financial time series forecasting, particularly in the context of electronic trading. The authors demonstrate that while Transformer-based models may excel in certain aspects of mid-price prediction, LSTM-based models, especially the proposed DLSTM, exhibit superior and more consistent performance in tasks such as mid-price difference prediction and mid-price movement prediction. The incorporation of trading simulations with and without transaction costs further highlights the practical significance of the proposed DLSTM model for real-world trading scenarios.

Moreover, the paper's adaptation of existing Transformer-based models' architecture to better suit the demands of the movement prediction task showcases the potential for further improvements in this domain. By incorporating both past and projected mid-price data, followed by a linear layer and softmax activation, the authors demonstrate a creative approach to enhancing the performance of Transformer-based models in financial time series forecasting.
In summary, the paper's originality, quality, clarity, and significance make it a valuable contribution to the field of financial time series forecasting using deep learning models, offering new insights and directions for future research in this domain.
summary: 5348_Transformers_versus_LSTMs
pdf
LT
Here is a new paper needs to be reviewed. Summary*
Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.
*

Summary

This paper conducts a comparative study between LSTM-based and Transformer-based models for financial time series forecasting, specifically in the context of electronic trading using high-frequency limit order book (LOB) data. The authors investigate the performance of these models across three prediction tasks: mid-price prediction, mid-price difference prediction, and mid-price movement prediction.

For the mid-price prediction task, the study finds that Transformer-based models like FEDformer and Autoformer achieve lower prediction errors than LSTM-based models. However, the authors note that the practical utility of these results for high-frequency trading is limited due to insufficient prediction quality.

In the mid-price difference prediction task, LSTM-based models demonstrate superior performance and robustness compared to Transformer-based models. The canonical LSTM achieves the highest R^2 of around 11.5% within about 10 prediction steps, while state-of-the-art Transformer models struggle to effectively process difference sequences.

The paper's main contribution lies in the mid-price movement prediction task, where the authors introduce a novel LSTM-based model called DLSTM. This model integrates LSTM with a time series decomposition approach inspired by the Autoformer architecture. DLSTM significantly outperforms all other models in classification metrics and proves its effectiveness in trading simulations, particularly when transaction costs are considered.

Additionally, the authors adapt the architecture of existing Transformer-based models to better suit the demands of the movement prediction task. They incorporate both past and projected mid-price data, followed by a linear layer and softmax activation, to determine price movements.

Overall, the study highlights that while Transformer-based models may excel in certain aspects of mid-price prediction, LSTM-based models, particularly the proposed DLSTM, demonstrate consistent superiority and practicality in financial time series prediction for electronic trading.
weaknesses: While the paper presents valuable insights and contributions, there are a few areas that could be improved or require further clarification:

Limited dataset diversity: The experiments in this study are conducted using LOB data from a single cryptocurrency pair (BTC-USDT or ETH-USDT) on one exchange (Binance). To demonstrate the generalizability of the proposed DLSTM model and the comparative analysis between LSTM-based and Transformer-based models, it would be beneficial to include a wider range of financial instruments, such as stocks, forex, or other cryptocurrencies, as well as data from multiple exchanges. This would strengthen the paper's conclusions and provide a more comprehensive assessment of the models' performance across diverse financial time series.
Lack of ablation studies: While the paper introduces the novel DLSTM model, which integrates LSTM with a time series decomposition approach, there is a lack of ablation studies to investigate the individual contributions of each component. For example, the authors could compare the performance of DLSTM with and without the time series decomposition to assess the impact of this specific modification. Additionally, a more detailed analysis of the adapted Transformer-based models' architecture for the movement prediction task would provide valuable insights into the effectiveness of the proposed changes.
Limited discussion on model interpretability: Interpretability is a crucial aspect of financial time series forecasting models, especially in the context of electronic trading, where understanding the factors driving the model's predictions is essential for risk management and decision-making. The paper could benefit from a more in-depth discussion on the interpretability of the proposed DLSTM model and the adapted Transformer-based models, as well as a comparison with the interpretability of other LSTM-based and Transformer-based models.
Hyperparameter tuning and model selection: The paper does not provide a detailed description of the hyperparameter tuning process and model selection criteria for the various models used in the experiments. It is essential to discuss the approach used for hyperparameter optimization, such as grid search, random search, or Bayesian optimization, and the specific hyperparameters tuned for each model. Additionally, the authors could provide more information on the model selection process, such as the use of validation sets or cross-validation techniques.
Robustness to market conditions: The experiments in this study are conducted using LOB data from a specific time period (e.g., July 2022). To demonstrate the robustness of the proposed DLSTM model and the comparative analysis between LSTM-based and Transformer-based models, it would be valuable to evaluate the models' performance under different market conditions, such as periods of high volatility, market crashes, or significant news events. This would provide a more comprehensive assessment of the models' ability to generalize and adapt to various market scenarios.

Addressing these weaknesses would further strengthen the paper's contributions and provide a more comprehensive and robust analysis of the proposed DLSTM model and the comparative study between LSTM-based and Transformer-based models in financial time series forecasting for electronic trading.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: What specific modifications were made to the Transformer architecture to adapt it to financial prediction tasks?

Can the authors elaborate on the metrics used to evaluate the models' performance? What criteria were significant in determining the practical utility of the models?
soundness: 3
strengths: Relevant Application: The use of LSTM and Transformer models for financial predictions on LOB data is timely and relevant given the growing interest in high-frequency trading and predictive models in finance.

Comparative Scope: The study covers multiple models and tasks, providing a broad comparison between LSTM- and Transformer-based architectures on real-world financial data.
summary: The paper explores the use of Transformer and LSTM-based models for financial time series forecasting tasks using high-frequency limit order book (LOB) data. A new LSTM-based model, DLSTM, is proposed alongside a modified Transformer architecture tailored for financial predictions. The study compares these models across three tasks: mid-price prediction, mid-price difference prediction, and mid-price movement prediction. Results suggest that Transformer-based models offer only marginal improvements in specific tasks, while LSTM models, particularly DLSTM, are more reliable in predicting mid-price differences and movements.
weaknesses: Unconvincing Novelty: The paper lacks substantial novelty. The DLSTM model is essentially a combination of existing methods, such as time series decomposition and LSTM layers, without a clear innovation. Similarly, the Transformer modifications are incremental and do not provide a compelling improvement. As a result, the contributions seem incremental and insufficiently distinct from existing work in financial time series forecasting.

Interpretability Issues: The added complexity of Transformer-based models raises interpretability concerns, especially given the unclear benefit over simpler LSTM-based models. Without a more interpretable mechanism or explanation for its performance gains, the model’s added complexity appears unnecessary.

Insufficient Performance Gain for Complexity: The study demonstrates only marginal improvements from the proposed Transformer modifications over traditional LSTMs, particularly in mid-price prediction. Despite the significant computational complexity introduced by Transformer-based models, the improvements are minimal and do not convincingly justify their adoption for practical trading applications.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: If possible, include LOB data from the LOBSTER dataset, to increase the generalizability of the experiment. If possible, include latest transformer based model (e.g. iTransformer, PatchTST). Recommend to use benchmarking frameworks such as LOBFrame or LOBCAST  in the experimental design to ensure that the results can be more comparable to existing studies. A more detailed discussion of the specific differences and advantages of DLSTM over other temporal decomposition methods (e.g., DLinear) could be added. could also include some ablation studies. include code for reproducibility.
soundness: 2
strengths: Even relatively simple LSTM models perform well in financial time series forecasting tasks, compared with transformer-based model.
summary: This paper compares the performance of LSTM and Transformer models in financial time series forecasting (limit order book data). They compared with FEDformer, Autoformer, Informer, Reformer, Transformer and LSTM. The main results show that Transformer has a slight advantage in predicting absolute price series, but the LSTM model performs more consistently and accurately in the prediction of price changes and price movements. In addition, the paper introduces DLSTM inspired by DLinear and Autoformer.
weaknesses: 1. The writing quality of the paper is low, especially the format of literature citation is not uniform and some of the citations are not standardized in formatting and arrangement. 
2. The experimental setup lacks comparison with the frameworks and standards widely used in the current research field and fails to demonstrate the advantages of the selected model. For example, the authors failed to cite and use the latest limit order book (LOB) benchmark frameworks, such as LOBFrame (https://github.com/FinancialComputingUCL/LOBFrame) and LOBCAST (https://arxiv.org/abs/2308.01915), both of which are open source frameworks currently widely used for Limit Order Book Forecasting. In addition, the authors did not include some of the latest Transformer based models (e.g., iTransformer and PatchTST), which have demonstrated advantages in terms of performance and efficiency in time series forecasting. Comparing these latest models would make the experimental results more convincing and practical.
3. The experimental data used in this paper is limit order book data from three cryptocurrencies, which, although suitable for high-frequency forecasting tests, is not representative of the financial market, and the volatility and noise characteristics of the cryptocurrency market are quite different from those of the traditional financial market. Data from LOBSTER (https://lobsterdata.com/) are more common and widely used in the literature currently.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: Please refer to questions to be addressed, the weaknesses section.
soundness: 1
strengths: The structure and logic of the paper is well organized. 

The experimental setup, description, and analysis are clearly stated with sufficient detail.
summary: In this paper, the authors conduct a comparative analysis of various LSTM-based and Transformer-based models for multiple financial prediction tasks using high-frequency limit order book data. They introduce a novel LSTM-based model called DLSTM and a newly designed Transformer-based model specifically tailored for financial predictions. Their results reveal that Transformer-based models offer a slight advantage in predicting absolute price sequences. However, LSTM-based models show superior and more consistent performance in predicting differential sequences, such as price differences and movements.
weaknesses: 1. The authors compare Transformers and LSTMs, concluding that LSTMs have advantages in multiple electronic trading tasks. However, the selection of Transformer-based models is limited to earlier studies (prior to 2023) and does not include recent state-of-the-art (SOTA) works, such as those mentioned in references [1], [2], and [3]. Notably, Liu et al. [2] claim significant improvements on similar tasks. Excluding these recent studies makes it premature to conclude that Transformer-based models underperform compared to LSTMs. Additionally, there is insufficient evidence to assert that the authors' proposed DLSTM model is the optimal choice for this application. Could you please include comparisons with some of these SOTA results to more robustly justify the conclusion?

[1] Garza, A., Challu, C., & Mergenthaler-Canseco, M. (2023). TimeGPT-1. arXiv preprint arXiv:2310.03589.

[2] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2023). itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625.

[3] Das, A., Kong, W., Sen, R., & Zhou, Y. (2023). A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688.

2. The authors' conclusion lacks novelty and largely aligns with the findings and conclusions of Zeng et al. [4] It appears to apply established approaches and conclusions to domain-specific practices. While retaining empirical relevance, the study does not offer methodological breakthroughs.

[4] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023, June). Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 9, pp. 11121-11128).

3. The experimental setup could be made more representative by incorporating additional metrics such as Mean Absolute Scaled Error and Relative Mean Absolute Error.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1.Given the limited dataset used and the lack of detailed experimental information (settings of baselines), I am very concerned about the reliability of this paper's conclusions. How would you address or demonstrate the robustness of your findings under these limitations?

2.How do you explain the significant differences in experimental results with and without transaction costs? What factors contribute to this discrepancy?

3.What are the specific advantages of your time series decomposition method compared to other decomposition approaches, and why do these advantages arise?

4.Other questions can refer to the weaknesses.
soundness: 1
strengths: (1) This research presents findings that compare the performance of two types of models.
(2) It successfully highlights the weaknesses in current measurement metrics.
(3) Interesting task definition.
summary: This research examines the performance differences between Transformer-based models and LSTMs across three cryptocurrency limit order book data prediction tasks. It also introduces DLSTM, a LSTM-based model, and a Transformer-based model redesigned for financial forecasting.
weaknesses: 1.Baseline Selection Rationale: The paper does not clearly explain why specific Transformer and LSTM variants, such as Autoformer and FEDformer, were chosen in the comparison. It remains unclear if these variants have unique advantages for financial time series forecasting. Providing additional theoretical support or rationale for model selection would enhance the scientific basis of this choice.

2.Data Risk: The study only tests on a single asset (BTC-USDT), lacking a broader dataset. This limited scope may mean the model’s performance does not generalize well to other financial data. Testing on a single asset is insufficient to comprehensively assess the model’s generalizability.

3.Lack of Experimental Details: The paper lacks adequate details on the experimental setup, especially regarding hyperparameter settings and baseline model architectures. This omission makes replication challenging and affects the reliability of the results. Sufficient information is not provided to ensure a fair comparison among baseline models.

4.Unclear Result Interpretation: The paper does not adequately explain the significant differences in performance between experiments with and without transaction costs. Lacking theoretical support or data analysis, it's hard for me to understand the causes behind these variations under different settings.

5.Limited Community Contribution: Time series decomposition, used in this study, appears to be a common approach, closely resembling classical time series decomposition methods. It is unclear how this study provides any specific advantage over the standard decomposition methods.

6.Although the paper points out shortcomings in MSE and MAE metrics, it fails to propose a robust method to address these deficiencies.

7.Some capitalization inconsistencies, eg. in line 034  Self-attention mechanism.

Meta-review: A. Scientific Claims and Findings:

The paper compares the effectiveness of Transformer and LSTM architectures for financial forecasting using high-frequency trading data. The authors introduce a new LSTM-based model called DLSTM and a finance-specific Transformer. Their results indicate that Transformers have a slight advantage in absolute price predictions, but LSTMs are more reliable overall.

B. Strengths:

The paper addresses a relevant and significant question in financial time series forecasting.
The experimental setup is extensive and provides substantial data.
The paper is well-written and easy to follow.

C. Weaknesses:

Lack of code and detailed implementation information.
Limited novelty of the proposed approach.
The decomposition strategy is only applied to the LSTM model.
Several state-of-the-art Transformer-based models are not included in the comparison.
The statement about Transformers and LSTMs requires further investigation.
Limited dataset diversity.
Lack of ablation studies.
Limited discussion on model interpretability.

D. Reasons for Rejection:

The paper has several weaknesses, including limited novelty, lack of detailed information, and a limited comparison with state-of-the-art models. These issues raise concerns about the reproducibility and comprehensiveness of the research. Additionally, the paper lacks a deeper analysis of the observed differences between Transformer and LSTM models, which limits the interpretability of the results.

Decision: Reject