# Optimizing Automated Trading Systems with Deep Reinforcement Learning
## 4 Experiment
### Results and Discussion  
#### 4.3.1 DDQN and D-DDQN Comparision  
The results with different settings are presented in the tables and figures below. First, the trading system with the cumulative return reward function is considered. In Figure (a)a, the average returns in percentage over the training data sets are reported, and the average returns over the 100 testing sets with different start dates and end dates are plotted in Figure (b)b. The DDQN setting can beat the D-DDQN setting in terms of return for all the periods; for example, in the training period, DDQN provides the maximum return of 3.95% and maximum loss of \(-3.48\%\) while the D-DDQN return is 2.87% and loss is \(-3.61\%\). More specifically, Table 3 shows statistical results where the reward function is the cumulative return. Compared with the D-DDQN setting, the system based on DDQN achieves higher average returns in both the training and testing period. However, the standard deviation is also larger, which indicates the instability of the results when trading with short-term periods. In the real market, trading performance is evaluated by the profit achieved after a larger period of time, e.g., weekly or monthly, while this experiment focuses on the daily profit and thus, high volatility is acceptable. In future work, the system could consider different time intervals to compare the stability of the profit achieved. \begin{table}
\begin{tabular}{c l c} \hline \hline
**Parameters** & **Description** & **Value** \\ \hline \(N_{e}\) & Number of episodes & 100 \\ \(N_{B}\) & Max capacity of replay memory & 10,000 \\ \(batch\_size\) & Batch size & 40 \\ \(N_{u}\) & Period of Q target network updates & 10 \\ \(\gamma\) & Discount factor for future rewards & 0.98 \\ \(\epsilon_{start}\) & Initial value for epsilon of the \(\epsilon\)-greedy & 1 \\ \(\epsilon_{end}\) & Final value for epsilon of the \(\epsilon\)-greedy & 0.12 \\ \(learning\_rate\) & Learning rate of ADAM optimizer & 0.001 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Parameters for training the AI agent. Figure 4: Average returns from DRL approach with return reward function. (**a**) Training period; (**b**) Testing period. An example of backtesting results and trading details of the trading system in a trading day is shown in Figure 5. To see more clearly the performance of the D-DDQN setting compared with the DDQN setting, we consider the cumulative average returns in all periods. Figure 5(a) shows the cumulative average return achieved when trading for the entire training period and the performance for the entire testing period is plotted in Figure 5(b). The returns of DDQN in the training and testing periods are 6.26% (8.94% per month) and 2.23% (4.46% per month) while the returns of D-DDQN are 1.14% (1.63% per month) and \(-\)1.08% (\(-\)2.16% per month), respectively. The results show that DDQN setting has better performance than D-DDQN setting with the return reward function. Next, the trading system with the Sharpe reward function is considered. In Figure 7, the average Sharpe value over all the periods is plotted and statistical indicators are summarized in Table 4. Although the D-DDQN setting provides better average returns than DDQN over the training period, other statistical indicators all show that DDQN provides better performance. Furthermore, the DDQN setting provides positive returns and less volatility in all periods. Figure 8a,b report the cumulative average returns over the entire training and testing periods, respectively. The returns of DDQN in the training and testing periods are 1.08% (1.54% per month) and 7.98% (15.96% per month) while the returns of D-DDQN are 2.60% (3.71% per month) and \(-9.32\%\) (\(-18.64\%\) per month), respectively. We can see strong fluctuations in the returns of the D-DDQN setting during the training and testing period, whereas DDQN setting provides positive returns in both periods.