# Smart Robotic Strategies and Advice for Stock Trading Using Deep Transformer Reinforcement Learning
## 3 Methodology and Proposed Model
### Preliminaries
#### 3.1.6 Reward Function  
It is necessary to provide feedback to the agent through reward signals, in order to teach it which behaviours are appropriate and inappropriate, depending on the context in which they are performed. The agent's trading approach is significantly impacted, both directly and immediately, by this feedback [32]. In our case, the reward function \(r=(s,a,s^{\prime})\) the reward that the agent will earn after it has acted in each state. Our goal is to encourage profits that are sustained over a long period of time, while controlling risk as much as possible. The metrics we use to reward our agents are varied, thus promoting our efforts to improve their performance. The measures covered here include the profit and loss function (PnL), the Sharpe ratio, the Sortino ratio, and the Calmar ratio. PnL is a regularly used metric in trading system research to examine the efficiency of reward functions in RL [7; 27; 33]. Nonetheless, PnL does not consider the risks associated with making a profit. As a result, the risk-adjusted return measure is presumed to be capable of accounting for this factor. * The Profit and Loss (PnL) The profit function is the one that of the most-used reward functions. Its mathematical formula is as follows: \[r_{t}=\left(1+a_{t}\times\frac{p_{t}-p_{t-1}}{p_{t-1}}\right)\frac{p_{t-1}}{p_ {t-n}},\] (2)where \(p_{t}\) is the closing price on the market at time \(t\), while \(a_{t}\) is the agent's action at the same time \(t\). Equation (2) is comprises of both one-day and long-term gross returns over \(n\) periods. * Volatility-Based Metrics: Sharpe and Sortino ratio According to [34], the Sharpe ratio is a frequently used measure of the risk-adjusted return, which can indicate both profit and volatility. The Sharpe ratio is calculated by dividing the average risk-free return of the investor by the standard deviation of that return: \[S_{t}=\frac{\text{Average}\left(\sum_{i=1}^{W}R_{i}\right)}{\sigma\Big{(}\sum _{i=1}^{W}R_{i}\Big{)}}.\] (3) At time \(t\), the Sharpe ratio reward function is \(S_{t}\), the daily return on multiple shares of a stock is \(R_{i}\), and return averages and standard deviations are estimated over returns for periods of \(W\), where \(W\) denotes the window size, which is used to calculate the average and standard deviation of the returns. Despite this fact, the Sharpe ratio considers volatility in portfolio values. However, the ratio equally regards both upward and downward movements; that is, it also penalizes upward volatility [35; 36; 37]. As a matter of fact, upward volatility (upwards price movement) contributes to positive returns, while downward volatility causes losses. In contrast to the Sharpe ratio, the Sortino ratio only considers downward volatility to be a risk, rather than overall volatility. The upward volatility, therefore, is not penalized by this ratio. Mathematically, it is formulated as follows: \[SR_{t}=\frac{\text{Average}\left(\sum_{i=1}^{W}R_{i}\right)}{\sigma_{down} \Big{(}\sum_{i=1}^{W}R_{i}\Big{)}},\] (4) where \(SR_{t}\) and \(\sigma_{down}\) represent the reward function of the Sortino ratio at time \(t\) and the standard deviation of the daily return below zero for the period \(W\), respectively. According to the Sortino ratio, the volatility of the loss under the profit conditions is the only factor considered, as the volatility of the loss under loss conditions is irrelevant. Modern portfolio theory indicates that the Sharpe and Sortino ratios represent high profits without significant volatility. Moreover, according to [38], the Sharpe ratio and the Sortino ratio allow RL agents to perform significantly better over other benchmarks. However, neither the PnL, Sharpe ratio, or Sortino ratio metrics take into account the maximum drawdown, defined as the maximum loss noted between a peak and a bottom of a portfolio before a new peak is reached. It provides a measure of the rate of change in price over a specified time period, and is an indicator of downside risk [39; 40].