# Reinforcement Learning and Physics
## 2 Reinforcement Learning and Physics  
RL is a ML paradigm that optimizes decision making based on stages [17]. While it cannot be considered supervised learning because the desired outputs are not known in advance to train the model, it is not unsupervised or semisupervised learning either because there is not a training limitation due to a lack of labels [2, 18].  
The RL framework is shown in Figure 1; it consists of an agent that takes actions in a given environment; those actions have an associated immediate reward. The goal of thelearning process is to maximize the long-term reward, a function that depends on the sum of all the rewards that are collected over time:  
\[R_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1} \tag{1}\]  
where the immediate reward \(r_{t+k+1}\) is the value returned by the environment depending on the action taken by the agent at time \(t+k+1\) and \(\gamma\in[0,1]\) is the so-called discount-rate, that sets the relative importance of future rewards. The environment is explored by taking different actions, what leads to learning the action-value function or \(Q\)-function, that estimates the expected future reward \(R_{t}\) following the policy \(\pi(s,a)\):  
\[Q^{\pi}(s,a)=E_{\pi}[R_{t}|s_{t}=s,a_{t}=a] \tag{2}\]  
The optimal policy appears as the result of learning the \(Q\)-function:  
\[\pi^{*}(s,a)=\arg\max_{a\in A}Q^{*}(s,a) \tag{3}\]  
Equation (3) corresponds with a deterministic policy, in which given a state, only one action can be taken. Stochastical policies encompass more than one possible action according to probabilities encoded in the policy [17].  
The goal of RL algorithms is to calculate \(Q^{\pi}(s,a)\) in order to obtain the optimal policy using. Many different methods can be considered, basically grouped into three main approaches [17]:  
* Dynamic programming (DP): it makes use of the Bellman equation [19] when a complete model of the environment is available.
* MonteCarlo (MC) methods [17]: They do not need a model of the environment but only sample sequences of states, actions and rewards. \(R_{t}\) is computed when an episode finishes, and \(Q(s,a)\) accordingly updated.
* Temporal difference methods (TD) [20]: They do not require an environment model, either. The values of \(Q_{t+1}(s,a)\) are updated using information from the environment (\(r_{t+1}\) and \(s_{t+1}\)) as well as estimations of \(Q_{t}(s,a)\).  
Sarsa and Q-learning are the most-widely used TD methods. Sarsa is an on-policy algorithm that modifies the starting policy towards the optimal one whereas Q-learning is off-policy and computes the optimal policy while the agent is interacting with the environment by means of another arbitrary policy, Equation (4):  
\[Q_{t+1}(s,a)\gets Q_{t}(s,a)+\pi[r_{t+1}+max(Q_{t}(s^{\prime},a^{\prime}) )-Q_{t}(s,a)] \tag{4}\]  
Figure 1: Reinforcement learning framework: \(a_{t}\) is the action taken by the agent at time \(t\), \(s_{t}\) is the state of the environment at time \(t\), and \(r_{t+1}\) stands for the reward at time \(t+1\).  
where \(\alpha\) is the rate of the update; \(s^{\prime}\) and \(a^{\prime}\) are the state and the action in time \(t+1\), respectively. \(Q_{t}\) stands for the action-value function for a particular state before being visited at time \(t\). \(Q_{t+1}\) is the updated value of that state once it has been visited.  
RL is a natural approach for system control, that has been successfully applied to many different fields, from robotics [21] to marketing [22] and medicine [23], to name a few. In the case of physics, two main approaches arise, namely, the use of standard RL to control different parts of physical systems (either classical or quantum), and the so-called quantum RL, a quantum version of RL that shares objectives with other quantum ML approaches, i.e., to make use of quantum technologies to carry out ML calculations in a more efficient way. Both approaches are described in the next two subsections.