# Reinforcement Learning with Self-Attention Networks for Cryptocurrency Trading
## 3 Problem Formulation  
A trading session is the total amount of time in which an agent is allowed to conduct transactions to generate profits. The session is divided into equal-length steps, where at each of them, the following steps take place: selection, exchange, and waiting. The selection step is carried out by a neural network, which computes a non-negative scalar for each asset, representing the weight or percentage of that asset with respect to the entire set of investments to be held in the coming period. The design of the NN used in the selection step and its training procedure is described in detail in Section 5. After the selection process, some of the assets held by the agent have to be exchanged to obtain the assets chosen by the selection procedure. The transaction fees pre-established by the trading service provider are considered in computing the quantities to be exchanged. The algorithms developed for computing the transactions that lead to the lowest possible transaction fees are described in Section 4. After computing the transactions, these are executed in the market and a waiting period begins. This period allows the investments to mature and generate profits or losses to the agent. Once the waiting period finishes, the process moves to a new step, beginning with the selection of new assets for the new period. The process is repeated until the trading session is complete. Let an asset in the market and a step in the trading session be \(i\) and \(k\), respectively. The shares corresponding to each asset \(i\) held by the agent at the beginning of step \(k\) are denoted by \(s_{i}^{k}\). If there are \(N\) assets in the market, then \(i\in\{0,1,...,N-1\}\). The index \(0\) is reserved for the cash asset, whose value is assumed to be constant throughout the trading \begin{table}
\begin{tabular}{c c l} \hline \hline
**Work** & **Year** & **Main Contributions** \\ \hline [PERSON] et al. [3] & 1998 & Proposed the RRL method. \\ [PERSON] and [PERSON] [4] & 2001 & Expanded RRL and compared to other approaches. \\ Gold [5] & 2003 & Applied RRL to the FX market. \\ [PERSON] and [PERSON] [6] & 2006 & Proposed an adaptive reinforcement method for trading in the FX market. \\ [PERSON] and [PERSON] [7] & 2010 & Proposed a threshold RRL method. \\ [PERSON] and [PERSON] [8] & 2013 & Combined RRL with a technical and fundamental analysis for stock trading. \\ [PERSON] et al. [10] & 2016 & Combined fuzzy logic and RNNs for stock trading. \\ [PERSON] and [PERSON] [11] & 2017 & Trained CNNs using Policy Gradient for cryptocurrency trading. \\ [PERSON] and [PERSON] [12] & 2018 & Used RNNs and Boltzmann machines for cryptocurrency trading. \\ [PERSON] and [PERSON] [13] & 2018 & Compared the performance of multiple DRL techniques for stock market trading. \\ [PERSON] et al. [14] & 2018 & Proposed an adversarial training method to enhance DRL methods. \\ [PERSON] and [PERSON] [15] & 2019 & Used Q-learning and transfer learning for stock trading. \\ [PERSON] et al. [2] & 2020 & Used GRUs for stock market trading. \\ [PERSON] and [PERSON] [18] & 2020 & Proposed the method SDDRL for stock market trading. \\ [PERSON] et al. [19] & 2020 & Trained an LSTM network using Q-learning for stock market trading. \\ [PERSON] et al. [21] & 2020 & Trained a GRU network using Policy Gradient for stock market trading. \\ [PERSON] and [PERSON] [22] & 2021 & Proposed a NN architecture for markets with a dynamic number of assets. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Literature review summary. process. Therefore, the asset is used as a reference, i.e., the shares of all the assets (\(s_{i}^{k+1}\)) are measured in units of this asset. In stock markets, the USD is used for this purpose, but in cryptocurrency markets, the most popular choice is the USDT. The sum of \(s_{i}^{k+1}\) for all \(i\), denoted \(s^{k+1}\), is the total capital of the investor at the beginning of step \(k\).