## 2 Theoretical Background
### Policy Gradient Methods  
Policy gradient algorithms consider a parametric policy, \(\pi(a|s,\theta)=P(a|s,\theta)\), and the goal is to optimize this policy using gradient ascent with a given performance measure \(J(\theta)\) with parameter updates \(\theta_{t+1}=\theta_{t}+\alpha\
abla J(\theta_{t})\)[29]. The most common choice for the performance measure is the expected return of the initial state \(s_{0}\), given as  
\[J(\theta)=v_{\pi}(s_{0})=\mathbb{E}_{\pi}\left[R_{0}+\gamma R_{1}+\gamma^{2}R _{2}+\cdots\right].\]  
This is equivalent to optimize the value of the initial state--a policy is thus considered to be good if it can generate a lot of reward during the course of an episode. There are multiple benefits of using policy gradient algorithms; they can be applied directly on continuous action spaces, the policy gradient theorem, introduced below, shows that any differentiable parametric policy can be used and, in the limit deterministic policies, can be modeled by policy gradients, which is useful if we do not want stochastic actions in an online setting--such as in the diabetes case. One of the key points of policy gradient algorithms is the policy gradient theorem [48]:  
\[\
abla J(\theta)\propto\sum_{s}\mu(s)\sum_{a}q_{\pi}(s,a)\
abla\pi(a|s,\theta). \tag{3}\]  
where the distribution \(\mu\) is the stationary distribution of the states succeeding \(s_{0}\) when following \(\pi\). This theorem states that the gradient of the performance measure is proportional to the gradient of the policy itself. This is of great benefit, as it allows the use of any differentiable policy parameterization. The policy gradient theorem allows, with some simple modifications to Equation (3), the formulation of a simple sample-based algorithm, called REINFORCE. Instead of updating based on summing over all actions, the policy gradient is rewritten using a single sample \(S_{t},A_{t}\), and the gradient update rule becomes  
\[\theta_{t+1}=\theta_{t}+\alpha G_{t}\frac{\
abla\pi(a|s,\theta)}{\pi(a|s,\theta )}. \tag{4}\]  
The complete derivation can be found in [PERSON] and Barto [29] and the entire algorithm is shown in Algorithm 1. ```
1:Input: differentiable policy \(\pi(a|s,\theta)\). 2:Generate episode from environment (See Section 3)
3:while True do\(\triangleright\) Loop until some convergence criteria is met. 4: Generate a sample, \(S_{0}\), \(A_{0},R_{0},\dots,S_{T-1}\), \(A_{T-1},R_{T-1},S_{T}\) from \(\pi(a|s,\theta)\)
5:for\(t=0,1,\dots,T\)do
6:\(G_{t}\leftarrow\sum_{k=t+1}^{T}R_{k}\)
7:\(\theta_{t+1}\leftarrow\theta_{t}+\alpha G_{t}\
abla\ln\pi(A_{t}|S_{t},\theta)\). 8:Return: optimized policy \(\pi(a|s,\theta)\). ```  
**Algorithm 1** REINFORCE  
The REINFORCE algorithm has been well studied and a number of improvements and suggestions have been proposed [45; 46; 49]. The current state-of-the-art in model free policy gradient algorithms is Trust-Region Policy Optimization by [PERSON] et al. [45] and a simplified version of the same algorithm called Proximal Policy Optimization [46]. In this work, we restrict our attention to the former. Trust-region policy optimization (TRPO) is an algorithm that is based on the fact that if the policy gradient update is constrained by the total variation divergence, \(D_{TV}(\pi_{1},\pi_{2})=\max_{s\in\mathcal{S}}|\pi_{1}(\cdot|s)-\pi_{2}(\cdot| s)|\), between the old policy and the new policy, the performance of the policy is guaranteed to increase monotonically [45].