# Optimizing Automated Trading Systems with Deep Reinforcement Learning
## 3 Research Methodology
### Learning Environment  
The parameter optimization problem is formulated as a Markov Decision Process represented by a tuple \((\mathcal{S},\mathcal{A},R,\tau)\), where \(\mathcal{S}\) is the set of possible states, \(\mathcal{A}\) is the set of legal actions, a reward function \(R:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{R}\) and the transition function \(\tau:\mathcal{S}\times\mathcal{A}\times\mathcal{R}\rightarrow\mathcal{S}\) that generates a new state in a possibly stochastic or deterministic environment \(\mathcal{E}\). \begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt} p{142.3 pt}} \hline \hline
**Authors (Year)** & **Objectives** & **Findings** \\ \hline [PERSON] et al. (2008) & Discussing evolutionary technologies to optimize trading strategies & Genetic Algorithm approach provides better results in terms of returns than typical parameters and Buy\&Hold strategy. Genetic Algorithm algorithm can be executed in parallel. \\ \hline [PERSON] \& [PERSON] (2012) & Comparing different approaches for neural network optimization Methods: Random Search, Grid Search, Manual Search & Random Search on the same domain in high-dimensional spaces can find better models in less time than Grid Search and Manual Search. \\ \hline [PERSON] et al. (2012) & Presenting methods to perform BO for hyperparameter selection of general machine learning algorithms Methods: BO with different acquisition functions & BO with Gaussian Process as probabilistic regression model and Expected Improvement as acquisition function significantly outperforms Tree Parzen Algorithm. BO surpasses a human expert at selecting hyperparameters and beats the state of the art by over 3\%. \\ \hline [PERSON] et al. (2019) & Proposing a hyperparameter tuning algorithm for machine learning models Methods: BO, Manual Search & BO algorithm based on Gaussian process can achieve high accuracy and less running time than Manual Search. \\ \hline [PERSON] et al. (2019) & Solving hyperparameter optimization problem with RL approach Methods: Random Search, BO, RL & The model based on RL approach does not rely on a heuristic acquisition function like BO. RL method outperforms the Random Search and BO approaches. \\ \hline [PERSON] et al. (2021) & Optimizing technical analysis strategies using machine learning Methods: Grid Search & Linear model and artificial neural network outperform other machine learning models. The hybrid approach shows improved profits and reduced risk of losses. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the previous research findings on parameter optimization The scenario or the state of the environment is defined as the data sets \(\mathcal{D}\) plus the history of evaluated parameter configurations and their corresponding response:  
\[\mathcal{S}=\mathcal{D}\times(\Lambda\times\mathcal{R}). \tag{1}\]  
The agent navigates the parameter response space through a series of actions, which are simply the next parameter configurations to be evaluated, and thus the action space corresponds to the space of all parameter configurations through the function \(g:\mathcal{A}\rightarrow\Lambda\). According to the definition of the action space, the agent executes an action from \(\mathcal{A}=\{1,...,|A|\}\). For example, action \(a=1,a\in\mathcal{A}\), corresponds to parameter set \(\lambda=g(a)=\{\lambda_{1}\}^{dim(\Lambda)}\) and action \(a=|A|\) corresponds to parameter set \(\lambda=\{\lambda_{|A|}\}^{dim(\Lambda)}\). The parameter response surface can be any performance metric which is defined by the function \(f:\mathcal{D}\times\Lambda\rightarrow\mathcal{R}\).