## 3 Algorithms
### Reinforcement Learning  
Speaking of reinforcement learning, there are two main approaches: value-based and policy-based. The goal of the former is to maximise a function which evaluates the total reward expected to be accumulated in the future iterations starting at a particular state. Once such a value function has been learned, it is used by the agent at each step to pick such an action which is believed to maximise the value of this function. Contrary to the value-based RL approach, policy-based agents attempt to optimise the policy function directly without learning the value function. The policy function usually outputs a probability for each action the agent can select at the given state. Deep Q-network (DQN) proposed in [53] presents the first deep value-based RL model to learn control policies directly from high-dimensional sensory input. In particular, the original DQN algorithm uses the images shown on the Atari emulator as the input and a convolution neural network - to process the image data. The role of this neural network is to approximate Q-function used to estimate the value of the action taken by an agent at a certain time step. The real reward value observed in the environment is used to calculate the target Q-function value using the Bellman Equation [107]. The neural network weights are then updated to minimise the difference between the target and current Q-function values. The Q-learning algorithm is proven to converge to an optimal value [108]. There are two main issues with using deep Q-networks. First, deep learning assumes data samples to be independent, however, the training data for the Q-network are collected by the sequence correlated states which are led out by actions chosen. Second, the collected data distributions are non-stationary, since the agent keeps learning new strategies at every iteration. To overcome these issues, two following mechanisms can be employed: experience replay and freezing the target. The latter requires constructing two neural networks with the same structure, but different weight values. One of these networks is updated online at each training step, while weights of the other one are kept frozen for a fixed amount of iterations. To calculate the value of the loss function during training, Q-function value is predicted with the first neural network, while the target is evaluated using the second one. Weights of the target network are periodically updated with weight values of the online network. Since the evaluation of the target value uses the old parameters whereas the predicted value uses the current parameters, this can break the data correlation efficiently. Moreover, this mechanism allows one to avoid policy oscillations caused by rapid changes of the Q-function. The second mechanism, experience replay, is the method which aims to break correlations between data samples by accumulating a buffer of experiences from many previous episodes and training the agent by sampling mini-batches of experiences from this buffer uniformly at random. DQN can handle the problems with discrete low-dimensional observations and actions. However, in real world engineering applications, not only the dimensionality of both states and actions can be high, both of them are often continuous. If the number of discrete actions is finite, Q-function maximisation poses no problem, since Q-values can be computed for each action separately and then they can be simply compared to each other. However, in case the action space is continuous, solving the resulting optimization problem is non-trivial. Deep deterministic policy gradient (DDPG) has been developed specifically for dealing with environments that operate in continuous action spaces [109]. Similarly to DQN, DDPG learns the Q-function with the help of the Bellman equation. The Q-function is then used to derive and learn an optimal policy.