## 2 Theoretical Background
### Reinforcement Learning  
Informally, RL concerns the behavior of a decision-making agent interacting with its unknown environment. In this framework, the goal is to train an agent to take actions that result in preferable states. Figure 1 shows the agent-environment interaction, where at each time step the agent observes the current state of the environment and performs an action based on that state. As a consequence of this action, the environment transitions to a new state. In the next time step, the agent will receive a positive or negative reward from the environment due to the previous action taken [29].  
The mathematical basis of reinforcement learning is the Markov decision process, which is represented by the tuple \((\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)\). \(\mathcal{S}\) and \(\mathcal{A}\) are the state and the action spaces, respectively, \(\mathcal{P}\) contains the state transition probabilities \(p(s^{\prime}|s,a)\) and represents the transition to state \(s^{\prime}\) from \(s\) using action \(a\). \(\mathcal{R}\) contains the rewards, represented by the reward function \(r(s,a,s^{\prime})\), which defines the goal of the problem, and \(0<\gamma\leq 1\) is a discount factor. The mapping from state to action is called the policy, which can be either a deterministic function \(\pi:\mathcal{S}\rightarrow\mathcal{A}\) or a set of conditional distributions \(\pi(a|s)\), depending on the environment the agent is interacting with. The goal of any RL algorithm is to learn an optimal policy \(\pi^{*}\) that maximizes the expected return it receives over time, which is the accumulated reward over time \(G_{t}=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}\), where \(R_{t}=r(s_{t},a_{t},s_{t+1})\). The expected return assuming that the agent starts from the state \(s\) and thereafter follows the policy \(\pi\)  
Figure 1: The reinforcement learning framework.  
is called the value function \(v_{\pi}(s)\). Concretely, the value function specifies the long-term desirability of states, indicating the total amount of reward that is expected by the agent:  
\[v_{\pi}(s)=E_{\pi}[G_{t}|S_{t}=s]=E_{\pi}[\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1} |S_{t}=s].\]  
Similarly, the expected return assuming that the agent starts from the state \(s\), takes action \(a\), and thereafter follows the policy \(\pi\) is called the _action-value function_\(q_{\pi}(s,a)\):  
\[q_{\pi}(s,a) =E_{\pi}[G_{t}|S_{t}=s,A_{t}=a] \tag{1}\] \[=E_{\pi}[\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}|S_{t}=s,A_{t}=a]. \tag{2}\]  
The ultimate goal of RL is to find an optimal policy, a policy that is better than or equal to all other policies based on the values of the states. Realizing this goal in practice has led to two different main branches of RL algorithms, value based algorithms and policy based algorithms, see e.g., [47].  
Value based algorithms aim to estimate the value of each state the agent observes. Decisions are then made such that the agent spends as much time as possible in valuable states. A policy in value based RL is often simply a greedy search over each action in the given state, where the action that gives the highest value is chosen. In the case of a RL agent controlling e.g., an insulin pump in the T1 DM case, such states could be safe blood glucose levels, while states with lower value would be either high or low blood glucose values.  
Policy based algorithms change the viewpoint from looking at how valuable each separate state is, to evaluating how good the policy itself is. Given some parametric policy, a performance measure for the policy is defined--most commonly how much reward the agent can get over a certain amount of time. This measure is then optimized using gradient-based methods. For the T1 DM case, this performance measure could for example be time-in-range.