## 5 Conclusions and Future Work  
In this work, we have shown that policy gradient reinforcement learning using TRPO outperforms standard basal-bolus treatment and compares favourably to MPC in our experiments. We consider this work to be a strong proof of concept for the use of policy gradient algorithms in the artificial pancreas framework; the TRPO agent is able to cope with both carbohydrate counting errors and to a certain degree skipped boluses. Furthermore, the control is tighter than using a fixed optimal basal rate and risk indices are, in general, lower than both MPC and basal-bolus insulin therapy.  
The main disadvantage of using RL, which has not been fully explored in this work, is the computational complexity of training. In this work, we fixed the number of policy gradient iterations to 100 for all experiments, but we empirically observed that, in many cases, far fewer iterations were required for convergence. Finally, we observed that a larger action space can lead to better control, but the increase in training data needed for convergence is significant.  
All of the TRPO agents were trained model free, so from the agent's perspective the diabetes simulator is simply a black box that returns a reward when an input is given. Due to the fact that  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Treatment** & **Time-in-Range** & **-Hypo** & **LBGI** & **HBGI** & **RI** & **Std** & **CoV** \\ \hline Basal-bolus & 73.67 & 0.30 & 0.51 & 6.12 & 6.63 & 32.73 & 0.21 \\ TRPO & **88.72** & 0.50 & 0.78 & **3.80** & **4.57** & 32.75 & 0.24 \\ MPC & 79.25 & **0.003** & **0.13** & 5.14 & 5.27 & **30.11** & **0.19** \\ \hline Best and worst cases: & Best TIR & Worst TIR & Worst TH & & & \\ Basal-bolus & 95.59 & 43.80 & 7.11 & & & & \\ TRPO & **97.18** & **63.63** & 5.01 & & & & \\ MPC & 96.02 & 55.27 & **0.15** & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Mean performance measures for all 100 patients with all 100 test meal scenarios. Best results for each category is marked bold. For time-in-range, higher is better, for all other measures, lower is better.  
Figure 8: Kernel density estimate showing the approximate distribution of the same mean maxima and minima shown in Figure 7. The axes have been increased with respect to Figure 7 to fully cover the tails of the distributions.  
T1 DM is a well studied disease and multiple treatment strategies already exist, there is a lot of domain knowledge that gets lost in a model free setting. An obvious direction of research is including domain knowledge into the RL framework for T1 DM, as in e.g., [73; 74].  
Finally, state-of-the-art RL contains a plethora of directions that can be explored, the perhaps most important ones for the artificial pancreas framework are inverse reinforcement learning [75], safe reinforcement learning (safe exploration) [76] and hierarchical reinforcement learning [77].  
Conceptualization, J.N.M., M.T. and I.K.L.; Funding acquisition, F.G.; Investigation, J.N.M. and M.T.; Methodology, J.N.M., M.T., I.K.L. and A.E.F.; Project administration, F.G.; Software, J.N.M., M.T. and A.E.F.; Supervision, F.G.; Validation, M.T. and A.E.F.; Writing-- original draft, J.N.M., M.T. and I.K.L.; Writing--review & editing, J.N.M., M.T., I.K.L., A.E.F. and F.G. All authors have read and agreed to the published version of the manuscript.  
J.N.M. and I.L. were funded by the Tromso Research Foundation during the course of this research.  
The authors declare no conflict of interest.