# Reinforcement Learning with Self-Attention Networks for Cryptocurrency Trading
## 5 DRL for Asset Trading
### The Deep NN for Asset Trading  
An NN processes numeric data of the market's assets, such as prices and volumes, and outputs the distribution of assets to be held in the subsequent periods of the process. The initial layers of the network extract independent high-level features for each asset, and a set of self-attention layers [24] blend that information to produce the output. The components of our NN are shown in Figure 5. The data fed to the NN is an arrangement of financial indicators with dimensions \(F\times N\times K\), where \(F\) is the number of financial indicators, \(N\) is the number of assets, and \(K\) is the number of past steps the agent is allowed to observe. The number of financial indicators equals six, as mentioned in Section 5.1. \(F\) is considered to be the number of channels of the input data. The number of assets in the dataset increases over time from 3 to 227 (see Table 2); therefore, \(N\) is not a constant. The number of past steps (\(K\)) depends on two factors: the number of days and sampling time. The data used in our experiments has a sampling time of 30 min, and the number of days our agent is allowed to observe is 20; therefore, \(K\) equals 960. \begin{table}
\begin{tabular}{c c} \hline \hline
**Feature** & **Value** \\ \hline Initial date & 17 August 2017 \\ Final date & 3 November 2020 \\ \# of days (training) & 809 \\ \# of days (test) & 365 \\ \# of assets & 3 to 227 \\ \# of indicators & 9 \\ Sampling time & 30 min \\ \# of entries (training) & 38,832 \\ \# of entries (test) & 17,520 \\ Fees (except BNB) & 0.1\% \\ Fees BNB & 0.05\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Dataset properties. Since the channels of the data have a very different range of values, normalization was applied before feeding to the NN. The normalization consists of subtracting the mean from each channel and dividing the results by the standard deviation of the channel, resulting in a normal distribution with zero mean and unit standard deviation. Then, for each asset, the normalized inputs are independently passed through a sequence of two convolutional layers, which are the primal feature extraction mechanism of the NN. The extracted features are flattened and passed through a fully connected (FC) layer to produce a set of feature vectors. The feature vectors are directed one by one to the blending mechanism of the network, consisting of a set of three self-attention encoders [24]. The design of the self-attention encoders used in our NN is shown in Figure 6a. The self-attention encoders consist of two different sub-layers. The first sub-layer is a multi-head self-attention block, meaning it contains parallel feature extractors (heads), which generate independent feature vectors from the inputs. The number of heads in our self-attention blocks is eight. The distinct feature vectors generated by the heads are combined into a single feature vector using a linear layer, as shown in Figure 6b. Then, these results are fed to a normalization layer [25], which has a residual connection [25] to generate the output of the first sub-layer. Figure 5: Self-attention network diagram for cryptocurrency trading. The activation function of all FC layers is the ReLU function. The output size of the final FC layers is one. Each head in a self-attention encoder has three matrices: query (\(W_{\text{Q}}\)), key (\(W_{\text{K}}\)), and value (\(W_{V}\)), which are the parameters to be trained. When a self-attention encoder receives the feature vectors corresponding to the assets, the feature vectors are concatenated and multiplied by the matrices \(W_{\text{Q}}\), \(W_{\text{K}}\), and \(W_{V}\) to obtain the matrices \(Q\), \(K\) and \(V\), which are used to compute the output of the self-attention head using the Equation (1) of [24], which computes the amounts of attention having to be given to all the assets when generating the output of each asset. The second sub-layer of the self-attention encoder also has two blocks. The first block is a feed-forward NN with two layers.