# Optimizing Automated Trading Systems with Deep Reinforcement Learning
## 3 Research Methodology
### The Trading System  
From the concepts for parameter optimization described, in this section, we proceed to build a trading system that can both give trading signals and optimize strategies automatically. To provide the trading environment, we study the state representation and present it in a form that the agent can understand. We propose a definition of an agent's parameter selection and a mapping from the act of choosing parameters for trading strategies to investment decisions. Each decision can be scored using the proposed reward function. The main components of the proposed system are described as follows. * Learning environment: The trading scenario or state of environment, \(s_{t}\), is represented as a one-hot encoded vector and decomposed into two parts: the data price \(d_{t}\in D\), and the sequence of selected parameter configurations and their corresponding rewards,\((\lambda_{t},r_{t})\in(\Lambda\times\mathcal{R})\). Given a certain state of the environment, the agent navigates the parameter response surface to select a set of parameters to optimize the reward. He then applies the chosen set of parameters to his trading strategy and executes a sequence of orders (buy, hold or sell) based on the trading rules. These orders are sent to the trading environment to compute the reward and generate the next scenario, \(s^{\prime}_{t}\). * Artificial intelligent agent: The aim of the agent is to learn an optimal policy, which defines the probability of selecting a parameter set that maximizes the discounted cumulative return or Sharpe ratio generated from trading strategies. * Learning mechanism: Figure 1 illustrates the interaction between the trader and the trading environment where the arrows show the steps in Algorithm 1. The blue arrows in Figure 1 are the main steps to illustrate a general DRL problem with experience replay. In the proposed environment, the agent can take a random action with probability, \(\epsilon\), or follow the policy that is believed to be optimal with probability, \(1-\epsilon\). An initial value for epsilon of the \(\epsilon\)-greedy action, \(\epsilon_{start}\), is selected for the first observations and then is set to a new value, \(\epsilon_{end}\), after a number of observations. The learning process of agents can be built on a Deep Q-Learning Network. During the trading process, the trader executes orders and calculates the performance through a backtesting step. In our experiment, the trading strategy is built with the common and simple indicator RSI (see [30] for detailed definition). However, the algorithm can be applied to any other technical indicator. The trading rules are presented as follows. A buy signal is produced when RSI falls below oversold zone (\(RSI<30\)) and rises above 30 again. When the RSI rises above the overbought zone (\(RSI>70\)) and falls below 70 again, a sell signal is obtained. In our novel DRL-based parameter optimization method, the DQN algorithm is modified and improved to adapt to the specific trading decision-making problem. The modifications and improvements are summarized as below. * Neural network architecture: to approximate the action-value function, architecture of the Deep Neural Network (DNN) can be built using Convolutional Neural Network (CNN) or classical feedforward DNN. In [27], the classical feedforward DNN with leaky rectified linear unit (Leaky ReLU) activation function is used due to the different nature of the input which is time series in our case. CNN is usually used with image input; however, CNN can still be used with an univariate time series, as input as in [31]. CNNs are appropriate for multivariate time series with use of features extracted via the convolutional and the pooling layers. Because of the potential for applying data  
Figure 1: Illustration of the learning mechanism in trading environment. other than price such as volume, multiple moving average time series, CNN is applied in our network. * Double DQN and Dueling DQN: These two networks are improved versions of regular DQN.