# Applying Artificial Intelligence in Cryptocurrency Markets: A Survey
## 3 Artificial Intelligence and Cryptocurrencies
### Reinforcement Learning
#### 3.2.1 An Overview on Reinforcement Learning Overview  
The fundamental nature of RL is to learn through a process of trial and error. This means that an RL agent has continuous interactions with its environment, and it learns to modify its behaviour based on rewards received after taking actions [67]. RL can be considered as creating an environment for a learner to learn by interacting with the environment and choosing actions without being told. The learner chooses actions in an environment based on the consequences of prior actions referred to as rewards. The agent modifies its behaviour to choose better actions in order to maximise cumulative rewards. To this end, RL has two distinctive features. One is the trial-and-error search to optimise cumulative rewards. The second is the delayed reward which implies that actions may not only have an immediate reward, but they may affect all subsequent rewards or may have rewards in the later stages of learning [14; 75]. An RL problem can be modeled and described as a Markov decision process (MDP) by a five-tuple \((\mathcal{S},\mathcal{A},\pi,\mathcal{R},\lambda)\). An RL algorithm is composed of the following:  
**States and observations \(\mathcal{S}\)**: A state is a complete description of the circumstances of an environment, and an agent obtains all the information regarding the environment through a state. Additionally, observation is considered in case some information might be omitted due to a partial representation of a state. **Action spaces \(\mathcal{A}\)**: Each environment allows performing several actions. The set of all legitimate actions in a given environment is called an action space. Unlike a continuous action space, an agent has a finite number of available actions in a discrete action space. **Policy \(\pi\)**: A policy \(\pi\) is the way an agent behaves at a given time. It determines the action that has to be chosen by the agent when it is in a particular state. In a probabilistic setting, it maps the current state of the environment into a set of probabilities for taking actions from the action space. **Rewards \(\mathcal{R}\)**: A reward component is an important concept in RL. It is an immediate or instantaneous gain that an agent receives when choosing an action in the current state to move to the next state. **Discount factor \(\lambda\)**: The quantity \(\lambda\in[0,1)\) is the discount factor and generates discounted rewards to prevent infinite cumulative rewards when running for a long period. As a general intuition, discounted rewards mean rewards today are worth more than rewards tomorrow. If it is zero, an agent considers only immediate rewards; while \(\lambda\) closer to one means that the agent evaluates its actions based on cumulative rewards in the future. Figure 6 is one possible taxonomy of RL algorithms produced partially based on work by [PERSON] and [PERSON] [76]. A short description of each component is provided in the following. Based on the proposed taxonomy in Figure 6, two computational strategies constitute reinforcement learning, namely, model-free (MF) and model-based (MB) approaches. In a model-based approach, rewards for taking actions come from internal models of the environment, whereas in a model-free approach a value is associated with actions to maximise the rewards. Additionally, model-free approaches require access to a large number of interactions for the purposes of training, while model-based methods are capable of achieving an optimal policy quickly based on what they have learned about the environment [77]. A model of environment allows an agent to predict future states, and a model-based strategy helps to enhance the efficiency of RL substantially [78]. Additionally, model-free approaches are more suitable for learning complex tasks, such as games. Value-Based Methods  
In an RL setting, an agent receives a reward signal based on choosing an action to move from one state to another. The agent maximises the total received reward (or discounted total reward in an RL setting with an infinite horizon) [75].