# Applying Artificial Intelligence in Cryptocurrency Markets: A Survey
## 3 Artificial Intelligence and Cryptocurrencies
### Reinforcement Learning
#### 3.2.2 Reinforcement Learning Applications in Cryptocurrency Markets  
There are a wide range of applications of RL in different industries from games [85] and robotics [86] to natural language processing [87] and computer vision [88]. RL is also a natural solution for several problems in finance and economics such as option pricing, multi-period portfolio optimisation, and risk management due to their dynamic nature. The focus of this section is on the application of RL in dealing with better investment policies in cryptocurrency markets. There are two main streams of literature for using RL models in cryptocurrency markets. Firstly, there is growing interest in explaining and predicting cryptocurrency price behaviours and trading strategies [89]. This stream describes that cryptocurrencies consist of distinct governing factors which distinguish them from traditional financial assets. However, it focuses on the price volatility of the cryptocurrency assets, and there is no systematic analysis of cryptocurrency trading [90]. The second line of research covers economic applications for developing automated trading systems [89]. Since it is in its early stages of development, most studies may not provide practical solutions and a competitive edge for investors [90; 91]. In addition, creating better market simulations by including transaction fees and applying feature engineering to cryptocurrency trading data are investigated in the literature. To mimic a real cryptocurrency market, [PERSON] [92] designs a new framework, called deep RL applied to market-making, that uses advantage actor-critic and proximal policy optimisation RL algorithms. These two policy-based methods constructed upon gradient are trained by Bitcoin, Ether, and Litecoin data. Positional profit-and-loss and the trade completion ratio are defined as the reward function, and the average daily returns are compared between currencies considering total accumulated rewards. BTC has the highest return on investment among all cryptocurrencies investigated in the study. In this work, limit order books, order flow imbalances, and trade flow imbalances are used as the environment state space, which is the novelty of the study. However, the short time-frame, two days, could be increased to achieve improved generalisation of the proposed approach. In addition, [PERSON] and [PERSON] [89] employ an active trading strategy using deep RL to achieve the greatest risk-adjusted returns. To this end, a gradient ascent algorithm optimises the Sortino ratio as the reward function. Five portfolio performance metrics, including cumulative returns, Sharpe and Sortino ratios, maximum drawdown, and value-at-riskare compared with the buy-and-hold trading strategy as the baseline model. The results indicate that the proposed model outperforms the buy-hold strategy in all tested five cryptocurrencies. On a similar topic of extracting trade ideas, [PERSON] et al. (2019) propose a four-layer architecture deep RL for recommending trading suggestions to maximise short-term profit. The reward function is based on the difference between the selling price and the purchase price. Additionally, as its unique feature, the agent receives negative rewards if the number of sequential purchases is more than a limited number (20 in the study). Thus, it prevents the agent from many open positions to increase the reward. Double-cross strategy, swing trading, and scalping trading strategies are compared with the proposed model based on the number of trading actions and trading quality. Deep RL finds more trading opportunities and hence performs a higher number of trades than other trading strategies. In addition, the net return is 14.4% for Bitcoin within one month of simulated trading, while the scalping strategy has only 6.1% growth as the best performer of traditional strategies. The study by [PERSON] et al. (2019) proposes the integration of inverse RL and agent-based modelling for Bitcoin market movements.