# Reinforcement Learning with Self-Attention Networks for Cryptocurrency Trading
## 3 Problem Formulation
The variables corresponding to the shares held by the agent at the end of period \(k\) are denoted similarly; \(s_{i}^{k}\) represents the individual asset shares and \(s^{k}\) is their sum. The trading process can be summarized as follows. The investor begins the trading session with a capital \(s^{0}\) distributed among some assets \(s_{i}^{0}\). Then, the neural network suggests some suitable assets for period \(k=1\), and some assets are exchanged. The result of those transactions is the \(s_{i}^{1}\). Then, a waiting period follows, allowing the values of the assets to evolve into \(\tilde{s}_{i}^{1}\), reaching the end of the step, where a new set of assets is chosen by the NN for period \(k=2\). This process is repeated until reaching the final step \(K\), where the performance of the agent is computed using the formula: \(s^{K}-s^{0}\), which is the difference between the final and initial capital. The transition between period \(k\) and \(k+1\) consists of a set of transactions summarized in Equation (1), where both \(i\) and \(j\) represent each integer from \(0\) to \(N-1\). Thus, Equation (1) is in fact a system of \(N\) equations, one for each asset. Note, for simplicity, the upper limit in the summation symbols has been dropped; the upper limit in all expressions is \(n-1\) unless otherwise specified. The amount transferred from asset \(i\) to asset \(j\) and the fee for a transaction is written as \(t_{ij}^{k}\) and \(f_{ij}\), respectively. Equation (1) can be interpreted as follows. The shares of asset \(i\) after executing the transactions (\(s_{i}^{k+1}\)) are equal to the shares before the transactions (\(\tilde{s}_{i}^{k}\)), minus the shares transferred from that specific asset to each of the other asset (\(t_{ij}^{k}\)), minus the fees spent for transactions (\(f_{ij}t_{ij}^{k}\)), plus the shares received from the rest of assets (\(t_{ij}^{k}\)). Note, all the variables in Equation (1) are non-negative numbers. \[s_{i}^{k+1}=\tilde{s}_{i}^{k}-\sum_{j}t_{ij}^{k}-\sum_{j}f_{ij}t_{ij}^{k}+\sum _{j}t_{ji}^{k},\quad i,j\in\{0,1,...,N-1\} \tag{1}\]  
Then, since the output of the NN is the relative weights of the assets with respect to the total capital, Equation (1) has to be modified to contain the asset weights instead of the shares. This is accomplished by dividing Equation (1) by \(\tilde{s}^{k}\), as in [22], resulting in Equation (2), where the old and new weights are denoted by \(\tilde{w}_{i}^{k}\) and \(w_{i}^{k+1}\), respectively, \(\tau_{ij}^{k}\) are the transfers with respect to the weights, and \(\rho^{k+1}\) is a rate representing the capital lost due to the transactions. The definition of these variables is shown in Equation (3). \[\rho^{k+1}w_{i}^{k+1}=\tilde{w}_{i}^{k}-\sum_{j}(1+f_{ij})\tau_{ij}^{k}+\sum _{j}\tau_{ji}^{k} \tag{2}\]  
\[w_{i}^{k+1}\stackrel{{\text{\tiny{def}}}}{{=}}\frac{s_{i}^{k+1}} {\tilde{s}^{k+1}},\tilde{w}_{i}^{k}\stackrel{{\text{\tiny{def}}} }{{=}}\frac{\tilde{s}_{i}^{k}}{\tilde{s}^{k}},\tau_{ij}^{k}\stackrel{{ \text{\tiny{def}}}}{{=}}\frac{t_{ij}^{k}}{\tilde{s}^{k}}\text{ and }\rho^{k+1}\stackrel{{\text{\tiny{def}}}}{{=}} \frac{s^{k+1}}{\tilde{s}^{k}} \tag{3}\]  
Note, the sum of all the weights \(w_{i}^{k+1}\) equals one, and the same is true for the weights \(\tilde{w}_{i}^{k}\). Therefore, by adding the \(N\) expressions represented by Equation (2), i.e., from \(i\) equals \(0\) to \(N-1\), and simplifying the result, we arrive at Equation (4), providing an easy way to evaluate the discount rate \(\rho^{k+1}\) that becomes one minus the fees due to all the transactions. Therefore, it can be used as the objective function of an optimization process to compute the optimal exchange amounts. Note, since it is not possible to transfer any amount from an asset to itself, both \(\tau_{ii}^{k}\) and \(f_{ii}\) are assumed to be zero, leading to Equation (4).