# An IoT System for Remote Monitoring of Patients at Home
## 4 Data Classification and Multiclass Q-Learning (MQL) Algorithm  
In this study, PHDs such as activity monitors [12], medication dispensers [13], pulse oximeters [14], and ECG monitors, blood pressure monitors and falling detectors are considered as ADN-AEs in the IoT system for monitoring patients at home. For example, Alarm signals of abnormal SpO\({}_{2}\) belong to the highest priority class (Urgent class), and thus are transmitted before any other data. In addition, the alarm signals of the Urgent class are pushed immediately to medical staffs even when the medical staffs do not request the related data from the patient monitoring IoT server. The multiclass Q-learning scheduling (MQL) algorithm proposed in this study is based on the Q-learning algorithm [31]. As shown in Figure 3, Q-learning is a reinforcement-based learning algorithm that finds an optimal policy by selecting the action with the highest reward in each state. The proposed MQL algorithm evaluates the appropriateness of the previous scheduling decision (action); the evaluation result (reward) is then applied to the next scheduling decision. Based on the evaluation result, the scheduling priority of the subsequent message increases or decreases. If  
Figure 3: Q-learning algorithm. a non-urgent message is scheduled for transmission at time T1, and the message is transmitted successfully at T1 without interfering with the transmission of urgent messages, then the scheduling priority of the non-urgent message increases and subsequent non-urgent messages are likely to be scheduled to transmit at T1. Otherwise, the scheduling priority of the non-urgent message decreases and subsequent non-urgent messages are not likely to be scheduled to transmit at time T1. Through successive scheduling decisions, evaluations and applications, the algorithm determines optimal scheduling decisions. The MQL algorithm is designed to transmit urgent messages immediately and transmit non-urgent messages later, when they will not interfere with urgent message transmissions. When a message arrives for scheduling, the message is inserted into its queue as shown in Algorithm 1 (Insertion of newly arrived message into its queue) discussed below. UrgentQ[*] is a priority queue in which urgent messages wait for scheduling based on their priority basis. NonUrgentQ[\(t\), *] is a priority queue in which non-urgent messages arriving at unit time \(t\) wait for scheduling based on their priority basis (\(1\leq t\leq T\); \(T\) is the number of unit times). If messages are scheduled every hour (every half hour) for 24 h, \(T\) is 24 (48). ```
For a newly arrived urgent message NewM; Insert NewM at the appropriate position in UrgentQ[\(*\)] based on message's priority basis For an existing message OldM whose priority is lower than NewM's priority and therefore is pushed back by one position in UrgentQ[\(*\)]: Increase OldM's priority by AGE\({}^{\text{D}}\) (AGE: an aging constant, 0.9 in this study, D: the distance between the positions of NewM and OldM) For the newly arrived non-urgent message NewM at unit time At: Find QTable[At, \(k\)] for the maximum reward in QTable[At, \(*\)] (\(1\leq k\leq T\)) Insert NewM at the appropriate position in NonUrgentQ[k, \(*\)] based on message's priority basis For an existing message OldM whose priority is lower than NewM's priority and therefore is pushed back by one position in NonUrgentQ[\(k\), *]: Increase OldM's priority by AGE\({}^{\text{D}}\)
```  
**Algorithm 1** Insertion of newly arrived message into its queue  
For example, suppose that a new urgent message UM3 (priority 5) arrives at UrgentQ[\(*\)] and that two urgent messages UM1 (priority 3) and UM2 (priority 1) are waiting in UrgentQ[\(*\)]. Then, UM3, UM1, and UM2 are located at the first, second, and third positions in UrgentQ[\(*\)], respectively, based on their priorities.