# Reinforcement Learning for Options Trading
## 1 Introduction  
In recent years, reinforcement learning (RL) emerges as an effective way to trade financial assets, such as stocks, futures, and cryptocurrencies. Reinforcement learning, within the financial market, has been gradually dominated by the use of reinforcement learning, in conjunction with other predictive models, such as neural networks [1]. For stock trading, [PERSON] et al. [2] use historical trading data, features of relevant stocks, and technical analysis indicators as inputs to train a deep reinforcement learning model. The gated recurrent unit (GRU) [3], with a temporal attention mechanism, is utilized to capture long-term dependency. The stacked denoising autoencoders (SDAEs) [4] are employed to denoise the observations, and the asynchronous advantage actor-critic (A3C) model, accompanied with the long short-term memory (LSTM) [5], achieves stable, risk-adjusted returns [6]. [PERSON] et al. [7] put forward a theory of deep reinforcement learning in the stock trading decisions and price predictions. [PERSON] and [PERSON] [8] use multi-objective deep reinforcement learning to get maximum profit by adjusting risk. For future trading, reference [9] uses a time-series momentum and technical indicators to train DQN, policy gradient (PG), and advantage actor-critic (A2C) model. They tested the methods on 50 liquid futures contracts and found that the RL algorithms deliver profits even under high transaction costs. [PERSON] et al. [10] introduced a multi-objective deep reinforcement learning approach for stock index future's intraday trading. For cryptocurrency trading, double and dueling DQN is used for trading bitcoin in reference [11], and the trading system produced positive returns. We can see from the above papers that reinforcement learning has been used widely in algorithmic trading. An option is a contract between two parties in the transaction, regarding the right to buy and sell in the future. The buyer (right party) of an option contract obtains a right by paying a certain fee to the seller (obligation party), that is, the right to buy orsell the agreed amount from the option seller at the agreed time and the agreed price. There are some papers researching options trading or portfolio construction in recent years. [PERSON]'s criterion is used to construct options portfolio [12]. [PERSON] et al. [13] proposed an efficient BSUM-M-based algorithm [14] to solve the portfolio design problem, under the [PERSON] mean-variance framework. Reference [15] incorporated volatility forecasting, utilizing investors' sentiments in the decision-making process, to improve options trading performance. Reference [16] converted index future trading to option trading strategies via LSTM architecture and achieved superior performance. Although the methods mentioned above demonstrate good performance in options trading or portfolio construction, they are not robust for the actual dynamic market and cannot be directly applied to algorithmic trading. Reinforcement learning agents can interact with the environment and learn the decision-making parameters automatically, which is suitable for trading financial assets, including options trading. However, as far as we know, the papers talking about options trading using reinforcement learning are very few. There are still some problems that need to be solved. First of all, there are many different options contracts for one underlying asset, and each contract has its characteristics. Besides, the period from one contract listed to its expiration is relatively short. Usually, the period is less than six months. The period for active trading is even shorter, usually less than two months. The training of reinforcement learning needs plenty of homogeneous data. Training on one or all contracts cannot meet the demand. Secondly, options, as a kind of derivative, have high leverage and volatility in nature. If the trading of options is not appropriately limited, the principal may suffer a considerable drawdown.