Query: Diabetes reinforcement

title: Reinforcement learning application in diabetes blood glucose control: A systematic review
abstract: Reinforcement learning (RL) is a computational approach to understanding and automating goal directed learning and decision making. It is designed for problems which include a learning agent interacting with its environment to achieve a goal. For example, blood glucose (BG) control in diabetes mellitus (DM) where the learning agent and its environment are the controller and the body of the patient respectively. RL algorithms could be used to design a fully closed loop controller, providing a truly personalized insulin dosage regimen based exclusively on the patient's own data.  In this review we aim to evaluate state of the art RL approaches to designing BG control algorithms in DM patients, reporting successfully implemented RL algorithms in closed loop, insulin infusion, decision support and personalized feedback in the context of DM.  An exhaustive literature search was performed using different online databases, analyzing the literature from 1990 to 2019. In a first stage, a set of selection criteria were established in order to select the most relevant papers according to the title, keywords and abstract. Research questions were established and answered in a second stage, using the information extracted from the articles selected during the preliminary selection.  The initial search using title, keywords, and abstracts resulted in a total of 404 articles. After removal of duplicates from the record, 347 articles remained. An independent analysis and screening of the records against our inclusion and exclusion criteria defined in Methods section resulted in removal of 296 articles, leaving 51 relevant articles. A full text assessment was conducted on the remaining relevant articles, which resulted in 29 relevant articles that were critically analyzed. The inter rater agreement was measured using Cohen Kappa test, and disagreements were resolved through discussion.  The advances in health technologies and mobile devices have facilitated the implementation of RL algorithms for optimal glycemic regulation in diabetes. However, there exists few articles in the literature focused on the application of these algorithms to the BG regulation problem. Moreover, such algorithms are designed for control tasks as BG adjustment and their use have increased recently in the diabetes research area, therefore we foresee RL algorithms will be used more frequently for BG control in the coming years. Furthermore, in the literature there is a lack of focus on aspects that influence BG level such as meal intakes and physical activity (PA) which should be included in the control problem. Finally, there exists a need to perform clinical validation of the algorithms.

title: Forecasting the Risk of Type II Diabetes using Reinforcement Learning
abstract: Type II Diabetes (T2D) is one of the most common lifestyle diseases which is characterized by insulin resistance. Lack of insulin's proper working causes uncontrollable blood glucose rise in the body which leads to life taking situations. Therefore, early detection of T2D is imperative to save many lives. Towards this goal, this work presents a machine learning based prediction model to detect T2D. The Q learning algorithm belonging to the Reinforcement Learning (RL) paradigm has been applied to the PIMA Indian Women diabetes dataset in developing the detection model. The model identifies patients with T2D using three factors (such as Body Mass Index, glucose level and age of subject) by generating an off policy based RL and making the learning agent to find an optimal policy for the factors. The information of a subject can be in any of 330 possible states. The proposed RL model's accuracy, Precision, Recall, F measure and AUC values have been compared with the state of the art techniques such as K Nearest Neighbors and Decision Tree. The performance of the proposed RL based T2D prediction outperforms the K Nearest Neighbors and Decision Tree.

title: An Insulin Bolus Advisor for Type 1 Diabetes Using Deep Reinforcement Learning
abstract: People living with type 1 diabetes (T1D) require self management to maintain blood glucose (BG) levels in a therapeutic range through the delivery of exogenous insulin. However, due to the various variability, uncertainty and complex glucose dynamics, optimizing the doses of insulin delivery to minimize the risk of hyperglycemia and hypoglycemia is still an open problem.  In this work, we propose a novel insulin bolus advisor which uses deep reinforcement learning (DRL) and continuous glucose monitoring to optimize insulin dosing at mealtime. In particular, an actor critic model based on deep deterministic policy gradient is designed to compute mealtime insulin doses. The proposed system architecture uses a two step learning framework, in which a population model is first obtained and then personalized by subject specific data. Prioritized memory replay is adopted to accelerate the training process in clinical practice. To validate the algorithm, we employ a customized version of the FDA accepted UVA/Padova T1D simulator to perform in silico trials on 10 adult subjects and 10 adolescent subjects.  Compared to a standard bolus calculator as the baseline, the DRL insulin bolus advisor significantly improved the average percentage time in target range (70 180 mg/dL) from 74.1% 8.4% to 80.9% 6.9% (p<0.01) and 54.9% 12.4% to 61.6% 14.1% (p<0.01) in the the adult and adolescent cohorts, respectively, while reducing hypoglycemia.  The proposed algorithm has the potential to improve mealtime bolus insulin delivery in people with T1D and is a feasible candidate for future clinical validation.

title: Pharmacists report lack of reinforcement and the work environment as the biggest barriers to routine monitoring and follow up for people with diabetes: A survey of community pharmacists.
abstract: Medications with lifestyle are the cornerstone of diabetes management and routine monitoring and follow up are essential to the delivery of quality care. Documented follow up rates by pharmacists for people with diabetes are low despite good uptake of initial medication assessments in medication review programs.  Identify the barriers and facilitators to routine monitoring and follow up for people with diabetes by community pharmacists.  Pharmacists were invited to complete a survey designed using the Theoretical Domains Framework Version 2 TDF (v2) consisting of 39 questions based on the 14 domains of the TDFv2 with quantitative response options using a 7 point Likert scale and 2 open ended questions. Baseline information about the respondents and their practice sites were summarized using descriptive statistics. Mean scores and standard deviations were calculated for each of the Likert scale responses. Responses to open ended questions were analyzed and coded using an inductive thematic approach.  346 pharmacists completed the survey (4.76% response rate) The TDF domains found to be positively influencing the delivery of routine monitoring and follow up activities were beliefs about consequences for people with diabetes (6.08 1.13) pharmacist knowledge (5.93 0.99) pharmacist skills (5.44 1.44) social influences (5.36 1.32) and optimism (5.20 1.58) The domains found to be negatively influencing were reinforcement (3.0 1.89) and environmental context and resources (3.30 1.81) Themes emerging from the thematic analysis included time and competing priorities, reimbursement, patient engagement, workflow and human resources, access to labs and clinical information, information technology and support from the owner/manager.  Our research concludes that pharmacists report that their knowledge, skills, and beliefs about their role and responsibility, social influences and optimism are positive influences on routine monitoring and follow up while reinforcement and the environmental context/resources are the greatest negative influences. Strategies to improve follow up should be focused in these areas.

title: A Group Based Community Reinforcement Approach of Cognitive Behavioral Therapy Program to Improve Self Care Behavior of Patients With Type 2 Diabetes
abstract: This study evaluated a cognitive behavioral based self care intervention program on diabetes management in individuals with type 2 diabetes in Jiangsu Province, China. People with type 2 diabetes were recruited to a 6 month, prospective, intervention study.  The intervention group (n 296) received an intensive cognitive behavioral based self care intervention, including group activities, frequent blood glucose monitoring, nutritional counseling, diabetes specific meal and a weekly progress report. The control group (n 110) received diabetes education, including diet and physical activity instruction only. Assessment data was obtained at baseline, and after 12 and/or 24 weeks of intervention. The intention to treat method was used to assess the effectiveness of the intervention program.  The intervention group showed improved fasting blood glucose, HbA1c, systolic and diastolic blood pressures compared to the control group. The intervention group also had significantly improved knowledge and self care behavior, and general health.  This study demonstrates that significant improvement in glycemic control and markers of cardiovascular health can occur in Chinese people with type 2 diabetes following a CBT based intervention program that includes diabetes education, frequent blood glucose monitoring and daily use of a diabetes specific meal plan, suggesting CBT is beneficial to improve health outcome in patients with type 2 diabetes.

title: A Reinforcement Learning Based Method for Management of Type 1 Diabetes: Exploratory Study
abstract: Type 1 diabetes mellitus (T1DM) is characterized by chronic insulin deficiency and consequent hyperglycemia. Patients with T1DM require long term exogenous insulin therapy to regulate blood glucose levels and prevent the long term complications of the disease. Currently, there are no effective algorithms that consider the unique characteristics of T1DM patients to automatically recommend personalized insulin dosage levels.  The objective of this study was to develop and validate a general reinforcement learning (RL) framework for the personalized treatment of T1DM using clinical data.  This research presents a model free data driven RL algorithm, namely Q learning, that recommends insulin doses to regulate the blood glucose level of a T1DM patient, considering his or her state defined by glycated hemoglobin (HbA1c) levels, body mass index, engagement in physical activity, and alcohol usage. In this approach, the RL agent identifies the different states of the patient by exploring the patient's responses when he or she is subjected to varying insulin doses. On the basis of the result of a treatment action at time step t, the RL agent receives a numeric reward, positive or negative. The reward is calculated as a function of the difference between the actual blood glucose level achieved in response to the insulin dose and the targeted HbA1c level. The RL agent was trained on 10 years of clinical data of patients treated at the Mass General Hospital.  A total of 87 patients were included in the training set. The mean age of these patients was 53 years, 59% (51/87) were male, 86% (75/87) were white, and 47% (41/87) were married. The performance of the RL agent was evaluated on 60 test cases. RL agent recommended insulin dosage interval includes the actual dose prescribed by the physician in 53 out of 60 cases (53/60, 88%  This exploratory study demonstrates that an RL algorithm can be used to recommend personalized insulin doses to achieve adequate glycemic control in patients with T1DM. However, further investigation in a larger sample of patients is needed to confirm these findings.

title: A Dual Hormone Closed Loop Delivery System for Type 1 Diabetes Using Deep Reinforcement Learning
abstract: We propose a dual hormone delivery strategy by exploiting deep reinforcement learning (RL) for people with Type 1 Diabetes (T1D) Specifically, double dilated recurrent neural networks (RNN) are used to learn the hormone delivery strategy, trained by a variant of Q learning, whose inputs are raw data of glucose meal carbohydrate and outputs are dual hormone (insulin and glucagon) delivery. Without prior knowledge of the glucose insulin metabolism, we run the method on the UVA/Padova simulator. Hundreds days of self play are performed to obtain a generalized model, then importance sampling is adopted to customize the model for personal use. \emph{In silico} the proposed strategy achieves glucose time in target range (TIR) $93\ for adults and $83\ for adolescents given standard bolus, outperforming previous approaches significantly. The results indicate that deep RL is effective in building personalized hormone delivery strategy for people with T1D.

title: Reinforcement Learning Based Adaptive Insulin Advisor for Individuals with Type 1 Diabetes Patients under Multiple Daily Injections Therapy*
abstract: The existing adaptive basal bolus advisor (ABBA) was further developed to benefit patients under insulin therapy with multiple daily injections (MDI) Three different in silico experiments were conducted with the DMMS.R simulator to validate the approach of combined use of self monitoring of blood glucose (SMBG) and insulin injection devices, e.g. insulin pen, as are used by the majority of type 1 diabetes patients under insulin therapy. The proposed approach outperforms the conventional method, as it increases the time spent within the target range and simultaneously reduces the risks of hyperglycaemic and hypoglycaemic events.

title: Positive reinforcement by general practitioners is associated with greater physical activity in adults with type 2 diabetes
abstract: In a sample of adults with type 2 diabetes mellitus (T2DM) the aim of this study was to examine whether self reported physical activity level is associated with recall of specific physical activity related interactions used by general practitioners (GP) Research design and methods Adults with T2DM completed an online survey reporting physical activity behaviors and recall of 14 GP patient interactions about physical activity, mapped onto discrete behavior change techniques (BCT) Stepped logistical regression examined associations between recommended physical activity =600 MET min/week) and GP patient interactions, controlling for body mass index, diabetes related comorbidities, depressive symptoms and self efficacy.  In total, 381 respondents (55% men, mean+ SD age: 62+ 10 years and T2DM duration 8+ 8 years) provided complete data. Most (73% reported receiving 'general advice' while interactions related to goal setting, monitoring, and relapse prevention were least commonly reported (all <20% Self reported achievement of the recommended physical activity level was significantly associated with recall of GP interactions involving praise for 'efforts to be active' (OR 2.1; 95% CI 1.24 to 3.53) 'lost weight' (OR 1.81; 95% CI 1.05 to 3.12) or lowering 'glucose levels as a result of being active' (OR 1.75; 95% CI 1.03 to 2.96) Conclusions Findings suggest GPs can be somewhat effective in promoting physical activity with simple, positive, reinforcing messages/interactions. Future research to develop and evaluate very brief primary care BCT based physical activity interventions is needed.

title: Control of Blood Glucose for Type 1 Diabetes by Using Reinforcement Learning with Feedforward Algorithm
abstract: Type 1 diabetes is a condition caused by the lack of insulin hormone, which leads to an excessive increase in blood glucose level. The glucose kinetics process is difficult to control due to its complex and nonlinear nature and with state variables that are difficult to measure.  This paper proposes a method for automatically calculating the basal and bolus insulin doses for patients with type 1 diabetes using reinforcement learning with feedforward controller. The algorithm is designed to keep the blood glucose stable and directly compensate for the external events such as food intake. Its performance was assessed using simulation on a blood glucose model. The usage of the Kalman filter with the controller was demonstrated to estimate unmeasurable state variables.  Comparison simulations between the proposed controller with the optimal reinforcement learning and the proportional integral derivative controller show that the proposed methodology has the best performance in regulating the fluctuation of the blood glucose. The proposed controller also improved the blood glucose responses and prevented hypoglycemia condition. Simulation of the control system in different uncertain conditions provided insights on how the inaccuracies of carbohydrate counting and meal time reporting affect the performance of the control system.  The proposed controller is an effective tool for reducing postmeal blood glucose rise and for countering the effects of external known events such as meal intake and maintaining blood glucose at a healthy level under uncertainties.