# Applying Artificial Intelligence in Cryptocurrency Markets: A Survey
## 3 Artificial Intelligence and Cryptocurrencies
### Reinforcement Learning
#### 3.2.2 Reinforcement Learning Applications in Cryptocurrency Markets
Inverse RL assumes that the reward of an agent is unknown and tries to infer the reward function in the presence of optimal policy or observed behaviour. The RL model proposed by the study has an environment whose states are defined by price, value possessed, the value realised of Bitcoin, and the difference between its price and a moving average of the price. A main advantage of the proposed model is that it considers interactions between market participants. Another variant of deep RL, called deep double Q-networks, and proximal policy optimisation are utilised to solve the problem of optimising the placement of limit orders by ([PERSON] et al., 2019). Designing a relatively comprehensive state-space RL environment, by including a wide range of financial aspects, is the main advantage of the study. To accomplish the task, market states are divided into four main sections, including transaction imbalances, best-order volumes and imbalances, volatility and current price drift, and current cost of liquidity. The results indicate that the agent guided by the proximal policy optimisation for order placement is more aggressive against unfavourable price moves. The study also concludes that queue imbalances and volumes are the most important features impacting the behaviour of the agents. A multi-agent RL method is proposed by [PERSON] and [PERSON] (2017) to develop a deep Q-learning portfolio management framework. A local agent guided by three learning techniques, namely, deep Q-Networks, double deep Q-Networks, and duelling double deep Q-Networks is evaluated based on two global reward functions. The reward functions are the sum of the local rewards, the weighted sum of the Sharpe ratio of the portfolio, and the net portfolio return. As a result, the framework outperforms weighted portfolio and portfolio selection using a genetic algorithm as the baseline. In another study, a state-augmented RL framework for managing portfolios on two well-known datasets, namely, the Bitcoin market and the HighTech stock market, is proposed by [PERSON] et al. (2018). The proposed portfolio management algorithm performs better in terms of accumulated and risk-adjusted profits in comparison to the standard RL-based portfolio management. [PERSON] and [PERSON] (2018) implement a deterministic deep RL method and convolutional neural network (CNN) for portfolio management. Weight allocation and adjustment are performed by using a Monte Carlo policy gradient method for CNN training. The CNN agent outperforms three benchmark strategies and two out of three portfolio management algorithms. One unrealistic assumption in many trading algorithms is that they neglect trading fees. In particular, in automated systems that perform many trades in a short amount of time, the accumulated fees, even negligible for a single trade, are considerable and can affect the optimal policy over a long time horizon. [PERSON] and [PERSON] (2018) examine seven algorithms, including an altered recurrent RL (RRL), for studying Bitcoin trading strategies. To evaluate the performance of algorithms, they define several metrics including cumulative weighted confidence without a trading fee, profit considering only 0.25% of transaction fees, and finally, incorporating the correct transaction rate for the task of predicting the Bitcoin price. RRL generates the best results for both with or without fee considerations. Itlikely demonstrates the way RL algorithms incorporate fees into the process of optimising trading strategies rather than adding only during trading. In addition, [PERSON] and [PERSON] [100] propose an RL model that takes into account the same assumptions as [99], which are similar to real markets. Their RL model utilises proximal policy optimisation that considers several assets in cryptocurrency markets and adopts the number of assets that may vary over time. In this study, the two agents evaluate the total return and the differential Sharpe ratio metrics.