# Optimizing Automated Trading Systems with Deep Reinforcement Learning
## 3 Research Methodology
### Learning Mechanism  
The interaction between AI agent and learning environment is the core mechanism for training and testing in the RL framework. Starting at a random location in the parameter space of a random data set, the agent needs to navigate the parameter response surface, including the parameter configuration and corresponding reward of a given model. At each step, the agent explores the environment, selects the next best parameter set with the \(\epsilon\)-greedy technique, and sends it to the environment. The policy \(\epsilon\)-greedy(Q) to select action \(a_{t}\) is defined, as follows:  
\[a_{t}= \begin{cases}\sim Unif(\Lambda),&\text{if }x\sim Unif([0,1])<\epsilon,\\ argmax_{a}Q^{*}(s_{t},a),&\text{otherwise}.\end{cases}  \tag{7}\]  
Learning environment will update the current state, evaluate it and send feedback back to agent as a reward. When the agent runs out of a number of episodes, he will be relocated to the response surface of another data set. The transitions are stored in the replay memory where a small batch of experiences is sampled to update the Q-network. By using experience replay, it breaks down the successive correlation among samples and also allows the network to make use of experiences better. Algorithm 1 can be used to describe the learning process of the agent, including training phase and testing phase. The main purposes of the training phase include generating learning samples, training the Deep Q-network using DRL algorithm. While in the testing phase, given an unseen scenario, the target network is used to predict an optimal parameter set. The step by step algorithm for training is as follows. 1. Learning environment randomly sampling a data set \(D=\{d_{0},...,d_{T}\}\) where each data \(d_{t}\sim Unif(\mathcal{D})\) for \(t=1,...,T\). 2. The input for DRL algorithm is represented as a one-hot encoded state vector, \(s_{0}=(d_{0},(\{\lambda_{init}\}^{dim(\Lambda)},0))\). 3. Given the state vector \(s_{t}\), a candidate action \(a_{t}\) is selected with the \(\epsilon\)-greedy technique. 4. The parameter \(\lambda_{t}\) is computed. Then, it is sent to the learning environment to compute the reward \(r_{t}\) and generate the next scenario \(s_{t}^{\prime}\) (or \(s_{t+1}\)). 5. The sample tuples \((s_{t},a_{t},r_{t},s_{t+1})\) is stored in replay buffer for later use in training model. 6. When the replay buffer has stored enough samples (\(\geq\) the minimum replay buffer size, \(N_{B}\)), the oldest tuple will be replaced. A batch of samples is sampled randomly from a replay buffer for training. 7. The Q-network is updated by minimizing the defined loss function which is similar to training supervised learning model. 8. Finally, the target networks are updated after a preset number of steps \(N_{u}\). 9. If the end of the episode is reached, the searching step will be stopped and go back to step 1. Else, increase \(t=t+1\) and go back to step 3. ```
1: Initialize network \(Q\) and target network \(\hat{Q}\)
2: Initialize experience replay memory \(B\)
3: Initialize the Agent to interact with the Environment
4:for\(N_{e}\) iterations do\(\triangleright\)\(N_{e}\) - number of episodes
5: Randomly sampling a data set \(D=\{d_{0},...,d_{T}\}\), \(d\sim Unif(\mathcal{D})\)
6: Get state \(s_{0}=(d_{0},(\{\lambda_{init}\}^{dim(\Lambda)},0))\)
7:for\(t\in\{0,...,T\}\) and while \(s_{t}\) is not terminal do
8: Determine next action \(a_{t}\) from state \(s_{t}\) using policy \(\epsilon\)-greedy(Q) \(\triangleright\) Equation (7)
9: Receive reward \(r_{t}=R(d_{t},\lambda=g(a_{t}))\)\(\triangleright\) Equation (3)
10: Generate new state \(s^{\prime}_{t}=\tau(s_{t},a_{t},r_{t})\)\(\triangleright\) Equation (4)
11: Store transition \((s_{t},a_{t},r_{t},s^{\prime}_{t})\) in the experience replay memory \(B\)
12: Replace oldest tuple if \(|B|>N_{B}\)\(\triangleright\)\(N_{B}\) - replay buffer size
13:if enough experience in \(B\)then
14: Sample a random minibatch of \(N\) transitions from \(B\)
15:for every transition \((s_{i},a_{i},r_{i},s^{\prime}_{i})\) in minibatch do \(y_{i}=