## 1 Introduction
A bolus can then be provided, either automatically by the system or by the patient itself based on the estimated carbohydrate amount. This setup is highly prone to errors due to the difficulties of carbohydrate counting in everyday situations [14]. This difficulty is well established in the scientific literature, where the true effect of these errors is still a topic of debate. Among others, [PERSON] et al. [15] report that carbohydrate-counting errors are not correlated with meal size, while [PERSON] et al. [16] found that larger meals led to larger estimation errors. On the other hand, [PERSON] et al. [17] found that meals with small amounts of carbohydrate tended to be overestimated. Finally, [PERSON] et al. [14] note that random errors, such as faulty carb-counting, as opposed to systematic bias errors, are more detrimental to glycemic control. Under- and over-bolusing due to these difficulties presents a significant risk of postprandial hyperglycemia and hypoglycemia. The current strategy to compensate for the counting errors is to let the artificial pancreas temporarily change the basal insulin rate. Despite these issues, the artificial pancreas is currently the most promising option for persons struggling with T1 DM with multiple studies showing promising results, both clinical and in-silico [12; 18; 19; 20]. There are currently two dominant artificial pancreas controller algorithm paradigms, proportional-integral-derivative (PID) control, [11; 21], and MPC [22; 23]. Model predictive control, in particular, uses a dynamic model with patient-specific parameters to predict the blood glucose curve into the future, where the prediction window is typically four hours, after which the fast-acting insulin's effect has mostly subsided [24]. Afterwards, if the predicted blood glucose curve and its final value is off the glucose target, MPC calculates an optimized sequence of basal rate actions on the model to correct the prediction towards the target while avoiding hypoglycemia. The first action of this sequence is then picked to change the basal rate momentarily, and the whole process is repeated after a while, usually every five or 10 min. The MPC approaches require a good model of the dynamics. In the artificial pancreas system, MPC algorithms are based on glucose-insulin regulatory models that are not able to capture external perturbations, so these algorithms are limited to compensate for the incomplete model used in the artificial pancreas application [25]. In addition to PID control and MPC, there have been investigations into fuzzy logic [26], and more recently techniques from machine learning and statistics, such as [PERSON] et al. [27], who proposed a blood glucose forecasting approach based on reccurent neural networks. Similarly, [PERSON] et al. [28] created a deep learning based forecasting framework based on convolutional neural networks. The control algorithm used in the artificial pancreas system has to learn models that are rich enough and adapt to the system as a whole [25]. Particularly, reinforcement learning (RL), a branch of machine learning that is based on interactive learning from an unknown environment [29] has, in recent years, gained increased attention in artificial pancreas research [30; 31; 32; 33; 34; 35; 36; 37; 38; 39]. A complete systematic review of reinforcement learning application in diabetes blood glucose control can be found in [40]. Outside of diabetes-related research, it has been particularly successful in achieving performance that exceeds the level of top human players in strategy games. The examples range from Backgammon in the early 1990's and more recently in the game of Go in 2015, where RL was combined with deep neural networks and Monte Carlo tree search [41; 42]. RL allows us to introduce model-free and data driven algorithms that can enable another level of patient individualization [25]. Finally, previous works from the authors have shown promise for the use of RL in the artificial pancreas [32].