# Reinforcement Learning for Options Trading
## 4 Experiment Result
### Training Result  
In our experiment, we used 50 ETF trading data from 5 January 2009 to 25 January 2017 and 300 ETF trading data from 28 May 2012 to 22 January 2020, in different time intervals (a day, 60 min, 30 min, 15 min, and 5 mins), as training sets. Transaction costs were set as 0.1% (buy and sell were the same). The reinforcement learning framework we used was SLM-Lab ([[https://github.com/kengz/SLM-Lab](https://github.com/kengz/SLM-Lab)]([https://github.com/kengz/SLM-Lab](https://github.com/kengz/SLM-Lab)), accessed on 10 November 2021). We used DQN, PPO, and SAC to train the trading agents in our experiments.  
The agents were initialized with five normalized inputs (open price, highest price, lowest price, close price, and trading volume), two hidden fully-connected layers (512-512), and three outputs (buy, hold, and sell). By using the SLM-Lab, one experiment was a trial in the framework. Additionally, a trial contained four concurrent sessions. Each session included the same algorithm structure and different initialized random parameters. The parameters of the DQN, PPO, and SAC models are shown in Table 3.  
Figure 3: The flowchart of the OTRL framework.  
```
1 Initialize underlying asset trading data in 5 min, 15 min, 30 min, 60 min, and 1 day time intervals as TS.
2 Initialize reinforcement learning algorithms set, including DQN, PPO, and SAC, as RLAS.
3 Initialize underlying assets daily trading data in periods different from TS as VS1.
4 Initialize underlying assets and options daily trading data in periods different from TS as VS2.
5 Initialize target option daily trading data in periods different from TS, VS1, and VS2 as TES.
6 Initialize stop-loss thresholds set 1%, 2%, and 3% as SLT.
7 for each t in TSdo
8 foreach also in RLASdo
9 train algo model on data t.
10 store trained model in M1.
11
12 end for
13
14 end for
15
16 end for
17 for each m in M1 do
18 use model m to trade underlying assets in VS1.
19 find the best two training sets in TS for each algorithm in RLAS.
20
21 end for
22
23 end for
24
25 for each m in M2 do
26 use model m to trade underlying assets and options in VS2.
27 find a proper stop-loss threshold for each algorithm in RLAS.
28
29 end for
30 Use selected models with proper stop-loss thresholds to trade target options in TES.
```  
**Algorithm 2**OTRL Framework.  
Figures 4 and 5 show the training results of 50 ETF, in five different time intervals, using the three RL algorithms. A trial graph takes the mean of sessions and plots an error envelop using standard deviation. As we can see from Figure 4, the models trained on the dataset in larger time intervals (such as 60 min or 1 day) tend to perform better in training results. There is more noise than in the trading data of shorter time intervals (like 5 min or 15 min). It is difficult for models to learn a pattern using training data with too much noise. However, limited training data may lead to overfitting. To select a proper time  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Parameter Description** & **Value** \\ \hline net\_type & Type of the neuron network & fully-connected network \\ batch\_size & Batch size & 32 \\ loss\_func & Loss function & MSE loss \\ optimizer & Optimization method & Adam \\ lr & Learning rate & 0.00025 \\ DQN max\_frame & Training steps of DQN & 6,000,000 \\ PPO max\_frame & Training steps of PPO & 10,000,000 \\ SAC max\_frame & Training steps of SAC & 5,000,000 \\ replay\_buffer & Max size of replay buffer & 1,000,000 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Hyperparameters setting.  
interval for training data, we use models trained on 60-min and 1-day trading data for the validation process.  
Figure 4: Training results on 50 ETF in two different time intervals using PPO, DQN, and SAC.