# Dynamic Large-Scale Server Scheduling for IVF Queuing Network in Cloud Healthcare System

Authors: Yafei Li; Hongfeng Wang; Li Li; Yaping Fu
Journal: Complexity
 (2021)

---

## Abstract

As one of the most effective medical technologies for the infertile patients, in vitro fertilization (IVF) has been more and more widely developed in recent years. However, prolonged waiting for IVF procedures has become a problem of great concern, since this technology is only mastered by the large general hospitals. To deal with the insufficiency of IVF service capacity, this paper studies an IVF queuing network in an integrated cloud healthcare system, where the two key medical services, that is, egg retrieval and transplantation, are assigned to accomplish in the general hospital, while the routine medical tests are assigned into the community hospital. Based on continuous-time Markov procedure, a dynamic large-scale server scheduling problem in this complicated service network is modeled with consideration of different arrival rates of multiple type of patients and different service capacities of multiple servers that can be defined as doctors of the general hospital. To solve this model, a reinforcement learning (RL) algorithm is proposed, where the reward functions are designed for four conflicting subcosts: setup cost, patient waiting cost, penalty cost for unsatisfied patient personal preferences, and medical cost of patient. The experimental results show that the optimal service rule of each server’s queue obtained by the RL method is significantly superior to the traditional service rule.


---

## Body

## 1. Introduction

According to data released by the Chinese National Health Commission, the infertility rate of couples of child-bearing age in China has risen from 3% to 12%–15% in the past two decades, and infertility has become a younger development trend. It is estimated that about 50 million women are infertile, 66% of whom are below 30 years of age. For giving birth to a baby, most of them try in vitro fertilization (IVF) technology, which places very high requirements for doctors to improve success rate [1]. In China, only doctors in a limited number of general hospitals are qualified for mastering this technology. Due to the limited medical resources of general hospitals, patients have to face a very long wait for IVF procedures. However, the community hospitals are in sharp contrast with the status quo of general hospitals due to backward medical resources and low technical level of medical staff. In very recent years, cloud healthcare has attracted considerable attention of scholars and practitioners, which can fully realize the integration of medical resources of general and community hospitals. Patients can quickly enjoy homogeneous medical services by telemedicine, also reducing the medical costs. Therefore, an integrated problem of meeting the needs of IVF patients and making the medical resources of general hospitals fully utilized is considered in this paper.Due to the importance of improving the efficiency of system, the resource allocation or scheduling problems in a lot of complex systems, such as manufacturing system [2–4], supply chain system [5], and service system [6–8], had gained great concerns by the researchers from OR community in recent years. It is noticeable that the relevant works on medical resource allocation in healthcare system mostly focused on the scheduling of hospital beds and operating rooms [9, 10]. Huang et al. [11] studied a patient scheduling problem in EDs, which was modeled as a multiclass queuing network with service deadlines and feedback paths. The proposed scheduling strategy could effectively alleviate the congestion of emergency departments. Erdogan et al. [12] proposed a new random integer programming model to study the dynamic sequencing and scheduling of patients. In this case, the patient’s request was not in accordance with the first-come first-service (FCFS) rule, and the scheduler would reserve capacity for emergency patients. He et al. [13] applied hybrid robust random method to describe a scheduling problem of emergency room patients. Through the developed dynamic scheduling algorithm, the matching of doctors and patients was realized. Moreover, researchers have begun to focus on the influence of patient preference and selection behavior upon medical resource allocation recently [14–17]. Dogru et al. [18] developed an appointment scheduling model suitable for primary care settings and provided patients with an ideal appointment schedule based on the patient’s order of appointment while meeting patient preferences. Schütz and Kolisch [19] considered a scheduling problem of allocating scarce resources to different customer categories and service types in the case of cancellation, missed appointments, and overbooking. In a patient queuing network with multiple categories, Truong [20] examined a dynamic advance scheduling problem with two patient categories and developed an optimal dynamic scheduling algorithm, which could fully adapt to daily changes in demand and capacity. An unlimited server service system with a limited-service capacity had been considered by Hassin et al. [21], in which servers were sorted by their service rate and arriving clients would join the fastest idle server.In very recent years, regional medical cooperation has shown great advantages in improving the utilization of medical resources [22]. Especially, the cloud medical system fully realizes the sharing of medical resources among general and community hospitals. Saghafian et al. [23] studied a telemedicine system to decide whether to transfer patients to telemedicine doctors based on the knowledge of community hospital triage nurses. Rajan et al. [24] analyzed a trade-off problem between the treatment speed and quality of chronic patients in the telemedicine system and proved that the benefit-maximizing service rate gradually approaches the social optimal service rate. Erdogan et al. [12] proposed a two-stage stochastic linear model. Taking into account the patient’s absentee behavior, they obtained the best planned arrival time for patients in community hospitals using the telemedicine platform and the optimal number of patients that could be arranged for daily telemedicine.It is noticeable that most solution methods in the abovementioned works focused on the metaheuristic approaches [25–29]. The purpose of this paper is to provide an optimal dynamic resource scheduling rule and service order based on the real-time state of the system instead of the FCFS rule. For this optimal scheduling problem of dynamic tasks, many scholars had adopted reinforcement learning (RL) methods. Huang et al. [30] and Noureddine et al. [31] proposed dynamic resource allocation algorithms based on RL method to optimize resource allocation in real time, respectively. Xiao et al. [32] developed a real-time dynamic task allocation algorithm based on Q-learning. The algorithm was not limited to adapt to its own task arrival process but also fully considered the influence of other agents on the task flow. Asghari et al. [33] proposed a RL-based resource allocation method in order to reduce the cost of system and improve the utilization of resource. Wauters et al. [34] developed a learning-based resource scheduling optimization method to minimize the average delay and total completion time of the project. This paper studies the queuing processes of egg retrieval and transplantation of infertile patients with different arrival rates, and then a dynamic resource allocation algorithm based on reinforcement learning is proposed. Considering the personal preference of patients, the algorithm is able to search for the optimal allocation plan of doctors in general hospitals and the optimal service rules of each doctor's cohort in order to minimize the average total cost of patients.The rest of this paper is organized as follows. Section2 presents a detailed description of the IVF process under the telemedicine system. In Section 3, the queuing problem of the IVF in the general hospital is formally defined along with notation, Markov model, objectives, and some preliminary analysis. A resource scheduling algorithm based on reinforcement learning is proposed in Section 4, and the experimental study is presented in Section 5. Finally, concluding remarks are presented in Section 6, followed by some directions for future work.


## 2. IVF Process in Cloud Healthcare System

The procedure of IVF is a complex and multistage process. In order to improve the utilization rate of resources and service quality, cooperation between general hospitals and community hospitals is necessary. The preexamination process of IVF can be carried out in community hospitals because of its easily learned techniques. Then, through the stimulation of drugs to promote women's ovulation process, once the egg is mature, the doctor will schedule the surgery for her to obtain ovum. After the fertilized egg develops healthily into an embryo, the cultivation stage for embryo is carried out in the community hospital, where the doctors in general hospitals use the telemedicine platform to interpret the embryo report and watch the results of fertilization, division, and blastocyst culture for community hospital doctors. After the embryo sac matures, the patient will return to the general hospital for embryo transfer. Then, they returned to the community hospital for pregnancy examination and they do not need to return to the general hospital because of the remote consultation with doctors in the general hospital through the telemedicine platform. Telemedicine can effectively realize the hierarchical medical services and also can be a great approach to save medical costs and reduce waiting time of IVF patients. Patients in community hospitals can enjoy the excellent services of general hospital just through telemedicine system. We give the IVF flow chart, depending on the actual telemedicine IVF process in the cloud healthcare system (see Figure1).Figure 1
The IVF flowchart in the telemedicine system.As one of the top-quality scarce resources, doctors in general hospitals have very close relationship with the health present situation. It is obvious from Figure1 that doctors in general hospitals are the bottleneck that severely restricts the cloud healthcare system for operation, and they only play important roles in the egg retrieval process and the transplant process. Therefore, we just take these two queuing processes into account. The feature of the IVF procedure is time-consuming, so, in the proposed short-term resource scheduling problem, egg retrieval and transplantation are regarded as two different types of patients, and each has its own update arrival process as well as the general service time distribution. After completing the corresponding IVF process in community hospitals, patients line up to enter the queue of general hospitals for egg retrieval or transplant surgery.


## 3. Dynamic Resource Scheduling Model of IVF

The main problems we tackle in this paper are how to allocate the two types of infertility patients to the service queues of different doctors according to the patients’ choice preferences and how to determine the better service rules of each queue. The current optimization and scheduling practice has some obvious shortcomings. First, the service order is based on the rule of FCFS. Second, when considering multiple servers, patients’ choice preferences are often neglected or taken into account exactly. Therefore, we formulate a Markov model to improve the current scheduling process addressing each shortcoming. We can fill the gap in the exciting literature on scheduling problem of IVF queuing network.In this section, we simplify the queuing process of infertile patients in general hospitals to the queuing network shown in Figure2 according to the IVF process of the cloud medical system in Figure 1. In the cloud medical system, egg retrieval patients and transplant patients may be from different community hospitals, and they share the resource pool of general hospitals.Figure 2
The queuing network of infertile patients in general hospital.The notations are given in Table1.Table 1
Notation.
Sets:IA set of specialists in general hospitalJA set of types of infertility patientsHA set of community hospitalsTA set of time periods in the planning cycleIndices:iIndex of specialists in general hospitaljType index of infertile patientshIndex of community hospitalstIndex of time periods in the planning cyclediIndex of patients served by doctoriParameters:E1tThe waiting time cost of the patients in the general hospital during periodtE2The setup cost between disease categories 1 and 2 for specialistsE3Penalty costs of unmet infertility patients’ personal preferencesciUnit medical cost of doctoriπService rules for each doctor queueλh,jtThe arrival rate of typej patient in community hospital h at the beginning of the tth time periodμi,jThe service rate of typei doctor serving type j patient in the general hospitalRandom variables:JitNumber of service settings of typei doctor in the tth time periodDecision variables:zditIn thetth time period, 0 if the personal preference of patient di served by doctor i is met and 1 otherwiseYditIn thetth time period, 1 if doctor i is assigned to patient di and 0 otherwiseWe can get that the arrival rate of type 1 patients and type 2 patients in general hospital are, respectively,λ1t=∑h=1Hλh,1t and λ2t=∑h=1Hλh,2t. The service rates of the system serving type 1 and type 2 patients are μ1=∑i=1Iμi,1 and μ2=∑i=1Iμi,2, respectively. The arrival rate and the service rate are given based on a real data from the actual cloud healthcare system. We model the queuing process of infertile patients in general hospital as the Markov decision process. We presented the state transition diagrams of the two types of patients at different time periods as Figure 3.Figure 3
The state transition diagrams of the two types of patients.In Figure3, the probability that the number of transplanted patients varies from n+1 to n (0≤n≤N) is ∑i=1Iyi2μ2 in the fixed number of egg retrieval patients, because, in different planned time periods, the service queue of each doctor may contain two types of patients or only one type of patients, or even the queue is empty. In this case, the service rate of the system is changed at any moment depending on the number of doctors serving transplant patients. At different moments, the value of ∑i=1Iyi2μ2 is different. For instance, in the tth time period, the general hospital has a total of I available doctors resources, but only a cohort of 4 doctors have 2 types of patients. Then, given the number of patients in one type, the probability that the total number of patients varies from n+1 to n is 4μ2. In the t+1th time period, suppose that there are 2 types of patients in the service queue of 3 doctors; then, given that the number of patients in one type is m, the probability that the number of patients in type 2 varies from n+1 to n is 3μ2.In the steady state, we give the following balance equation:Form=0 and n=0,(1)λ1t+λ2tP0,0=μ1P1,0+μ2P0,1.Form=0 and n=1,2,…,N−1,(2)λ1t+λ2t+∑i=1Iyi2μ2P0,n=λ2tP1,n−1+∑i=1Iyi1μ1P1,n+∑i=1Iyi2μ2P0,n+1.Form=0 and n=N,(3)λ1t+∑i=1Iyi2μ2P0,N=λ2tP0,N−1+∑i=1Iyi1μ1P1,N.Form=1,2,…,M−1 and n=0,(4)λ1t+λ2t+∑i=1Iyi1μ1Pm,0=λ1tPm−1,0+∑i=1Iyi1μ1Pm+1,0+∑i=1Iyi2μ2Pm,1.Form=1,2,…,M−1 and n=1,2,…,N−1,(5)λ1t+λ2t+∑i=1Iyi1μ1+∑i=1Iyi2μ2Pm,n=λ1tPm−1,n+λ2tPm,n−1+∑i=1Iyi1μ1Pm+1,n+∑i=1Iyi2μ2Pm,n+1.Form=1,2,…,M−1 and n=N,(6)λ1t+∑i=1Iyi1μ1+∑i=1Iyi2μ2Pm,N=λ1tPm−1,N+λ2tPm,N−1+∑i=1Iyi1μ1Pm+1,N.Form=M and n=0,(7)λ2t+∑i=1Iyi1μ1PM,0=λ1tPM−1,0+∑i=1Iyi2μ2PM,1.Form=M and n=1,2,…,N−1,(8)λ2t+∑i=1Iyi1μ1+∑i=1Iyi2μ2PM,n=λ1tPM−1,n+λ2tPM,n−1+∑i=1Iyi2μ2PM,n+1.Form=M and n=N,(9)∑i=1Iyi1μ1+∑i=1Iyi2μ2PM,N=λ1tPM−1,N+λ2tPM,N−1.The infertile patients in the cloud healthcare system come from different community hospitals; in that sense, the scale of this problem is a large one. In order to effectively solve the proposed problem, the state transition diagram in Figure3 is simplified. Given the number of patients in type 2, the state transition diagram of patients in type 1 in the tth time period is shown in Figure 4.Figure 4
The state transition diagram for type 1 patients.We assume that the upper limit of the number of patients in type 1 (i.e., egg retrieval patients) isM. Let the steady-state probability of the number (of m) of egg retrieval patients in the system be PQj=m. The state balance equation for the queuing process of egg retrieval patients is shown below.Form=0,(10)λ1tPQ1=0=∑i=1Iyi1μ1PQ1=1.Form=1,2,…,M−1,(11)λ1t+∑i=1Iyi1μ1PQ1=m=λ1tPQ1=m−1+∑i=1Iyi1μ1PQ1=m+1.Form=M,(12)∑i=1Iyi1μ1PQ1=M=λ1tPQ1=M−1.Define the service intensity for egg retrieval patients in the system asρ1=λ1t/∑i=1Iyi1μ1; when m=0, we can have(13)PQ1=0=1∑m=0Σi=1Iyi1−11/m!λ1t/μ1m+1/∑i=1Iyi1!1−ρ1λ1t/μ1Σi=1Iyi1.Whenm≥1,(14)PQ1=m=1∑i=1Iyi1!λ1tμ1mPQ1=0,n≤∑i=1Iyi1,1∑i=1Iyi1m−∑i=1Iyi1∑i=1Iyi1!λ1tμ1mPQ1=0,n>∑i=1Iyi1.In the steady state, from the state balance equation of the egg retrieval patients, the state transition probability matrix of the egg retrieval patients can be obtained:(15)1−λ1tλ1t…0∑i=1Iyi1μ11−∑i=1Iyi1μ1+λ1tλ1t…00∑i=1Iyi1μ11−∑i=1Iyi1μ1+λ1tλ1t…0⋮…⋱⋮0…∑i=1Iyi1μ11−∑i=1Iyi1μ1.According to Little's formula, the queue length of egg retrieval patients in thetth time period is(16)L1t=λ1t/μ1∑i=1Iyi1ρ1PQ1=0∑i=1Iyi1!1−ρ12.The expected number of the egg retrieval patients in thetth time period is(17)Lst=∑m=1MmPQ1=m.The total service intensity of the system is denoted asρ=λ1t+λ2t/∑i=1I∑j=1Jyijμj. The average waiting time for infertile patients in the tth time period is(18)W1t=L1tλ1t=λ1t/μ1∑i=1Iyi1ρρ1Q1=0∑i=1Iyi1!1−ρ12λ1t.The number of patients served in thetth time period is(19)∑i=1IDi1t=λ1t∗t−Lst.The average number of customers of infertile patients in thetth time period is(20)L1′t=W1tλ1t=λ1tμ1+λ1t/μ1∑i=1Iyi1ρ1PQ1=0∑i=1Iyi1!1−ρ12.Furthermore, assume that the upper limit of the number of patients in type 2 (i.e., transplant patients) isN. Let the steady-state probability of the number (of n) of transplant patients in the system be PQj=n. The simplified state transition diagram of transplant patients is shown in Figure 5.Figure 5
The state transition diagram for type 2 patients.In the queuing system for infertile patients, the service intensity of transplant patients is defined asρ2=λ2t/∑i=1Iyi2μ2. Given the number of egg retrieval patients, we can get the number of transplant patients in the tth time period. Hence, we can obtain(21)PQ2=n|Q1=m=1∑i=1Iyi2!λ2tμ2mPQ2=0|Q1=m,m=0,1,…,∑i=1Iyi1,1∑i=1Iyi2m−∑i=1Iyi2∑i=1Iyi2!λ2tμ2nPQ2=0|Q1=m,m=∑i=1Iyi1,…,M,PQ2=0|Q1=m=1∑n=0∑i=1Iyi2−11/n!λ2t/μ2m+1/∑i=1Iyi2−1!1−ρ2λ2t/μ2∑i=1Iyi2.From the conditional probability formula, we can deduce the steady-state probability of the number of transplant patients in the system as(22)PQ2=n=∑m=0MPQ2=n|Q1=mPQ1=m,forn=0,1,…,N.When the number of egg retrieval patients is given, the state transition probability matrix of transplant patients can be obtained as(23)1−λ2tλ2t…0∑i=1Iyi2μ21−∑i=1Iyi2μ2+λ2tλ2t…00∑i=1Iyi2μ21−∑i=1Iyi2μ2+λ2tλ2t…0⋮…⋱⋮0…∑i=1Iyi2μ21−∑i=1Iyi2μ2.The queue length of transplant patients in thetth time period is(24)L2t=λ2t/μ2∑i=1Iyi2ρ2PQ2=0|Q1=m∑i=1Iyi2!1−ρ22.The expected number of transplant patients in thetth time period is(25)Ls′t=∑n=1NnPQ2=n|Q1=m.The average waiting time for transplant patients in thetth time period is(26)W2t=L2tλ2t=λ2t/μ2∑i=1Iyi2ρ2PQ2=0|Q1=m∑i=1Iyi2!1−ρ22λ2t.The number of transplant patients served in thetth time period is(27)∑i=1IDi2t=λ2t∗t−Ls′t.The average number of transplant patients in thetth time period is(28)L2′t=W2tλ2t=λ2tμ2+λ2t/μ2∑i=1Iyi2ρ2PQ2=0|Q1=m∑i=1Iyi2!1−ρ22.We consider the setup cost when the doctor switches from the current type of patients to another; the aim is to gain the largest equilibrium benefits between community hospitals and general hospital. Define the setting times of each doctor int time period as Jit=∑diDixd,d−1i. Let Dit=∑j=1JDijt be the total number of two types of patients served by doctor i in the time period t. The average waiting time and queue length of the two types of infertility patients in the tth time period are, respectively, Wt=W1t+W2t and Lt=L1t+L2t. In a planning period T, the matching problem between doctors and patients can be well solved according to the model established by us. The objective function is established as follows:(29)ℱπt=mint∈T1t∑t=1T∑i=1I∑j=1J∑diDi1μi,jciDitYdit+zditE3+∑t=1TE1tWtLt+∑t=1T∑i=1IJitE2.The objective function minimizes the total costs, the first term of the objective function represents the medical cost of infertile patients, the second term is the penalty cost of unmet infertility patients’ personal preferences, the third term represents the waiting time cost of egg retrieval patients and transplant patients in the system, and the last term is doctors’ setup cost. In general, the above objective reflects the interests of both general hospitals and infertile patients in the cloud medical system. In order to avoid a patient’s too long waiting time in the system, a four-cost reward function based on reinforcement learning is designed.


## 4.Q-Learning-Based Solution Method

Existing scheduling rules seldom solve the problem of doctor resource scheduling with different service rates in cloud medical systems. The doctor scheduling that frequently appeared in the existing related literature is based on some given sequence of the patients. In a dynamic environment, doctors’ service rate and patients’ waiting time are considered as important indices of medical system, especially in cloud medical system with multiple hospitals cooperating with each other. The operating efficiency and the operating costs of the scheduling results greatly varied with the different scheduling rules. Therefore, our purpose is to design an optimal scheduling rule and patient service order for each doctor queue by using reinforcement learning approach. In this section, three-stage dynamic scheduling problems are proposed. At the first step, we divide the planning cycle into hourly dynamic scheduling problems; by different arrival rate and different service rate, we need to decide which patients are assigned to one of the doctors. At the second step, the service rule is presented based on the learning strategies we design. At the third step, we need to make decisions about the number of doctors serving different types of patients from community hospitals in different time periods.Reinforcement learning is an online actor critic method in machine learning (Sutton et al. [35] and Gao et al. [36]), which obtains certain rewards through interaction with the environment and ultimately maximizes long-term returns. The typical learning algorithm is to update the current state-action pair based on the observed reward and the next state-action pair. Combined with the research questions in this article, we give the algorithm framework based on reinforcement learning in Algorithm 1.Algorithm 1:Algorithm framework based on reinforcement learning.
Letter explanation:Qs,a: learning table about s and a;s: the state after each step is executeda: predict the action to be performed nextRequire: initializeQs,a arbitrary value;Repeat (for each episode):InitializesRepeat (for each step of episode):A⟵ action given by strategy π for s in the learning table Qs,a.Take actionA, observe the next state s′ and reward rQs,a⟵Qs,a+αr+γ∗maxa′Qs′,a′−Qs,as⟵s′; a⟵a′;Untils is terminalIn the IVF queuing system based on the cloud medical system studied in this paper, we define that the state space of the system is composed of the number of patients of the two types at different times and the busyness of the respective queues of doctors in general hospitals. Egg retrieval and transplant patients can only make appointments for related operations within the given appointment time period. We give the system the number of appointments allowed on the day and the strict upper limit of doctor resources. Infertility is a special disease, so the operation must be completed in a given time, even if overtime doctors also have to complete all operations. The average cost of IVF operation much surpasses common operation. Therefore, choosing a method is a key step in cost-saving and drives higher operational efficiency.Reinforcement learning is an effective dynamic programming method to solve dynamic scheduling problems, and its basic idea is shown in Algorithm1. It can continuously train based on data to obtain accurate responses to the environment. The main core of reinforcement learning is the design of the reward functions which are guided by the learning system targeted at minimizing the average total cost of the system. However, our total cost consists of four parts: the average service cost, the average waiting time cost, the average setup cost, and the average penalty cost for unmet infertility patients. In order to maximize the long-term total revenue, according to the greedy strategy, we set the reward functions in different states for the four subcosts. After each step is executed, the cumulative reward score of each subcost is treated as the total reward score obtained after the current action is executed, which ultimately maximizes the cumulative reward. The proposed reward functions balance the interests of both doctors and patients. In different planning time periods, different reward and punishment strategies are set according to the state of the system. According to the Markov decision model established in the previous section, the possible state of the system at each moment can be obtained, and, by using the trial-and-error method, we explore all the possible behaviors generated by the current state to find the current maximum return: maxa′Qs′,a′.


## 5. Experimental Study

In this section, we discuss our findings. In particular, we describe the effectiveness of the proposed learning algorithm for reducing the waiting time and the medical cost. The scheduling rules and matching results of patients and doctors can simplify the complex scheduling problem. The IVF queuing network we studied includes one general hospital and 30 community hospitals. The problem raised is to serve hundreds of patients of two different types on six servers (general hospital doctors) with different service rates. As discussed earlier, the patients demand scenarios derived from actual demand for the sake of ensuring the veracity and reliability of this experiment.This paper studies a resource scheduling problem within a day, where the allowable appointment time period is from 8 : 00 to 12 : 00. We divide the one-day scheduling problem into hourly subscheduling problems. We apply the sequence-based scheduling method and the learning algorithm to compare the efficiency and medical costs. Figure6 shows a scheduling Gantt chart based on appointment orders (FCFS) of two types of patients in six parallel service desks. If the service desk is free, the patients will automatically join the queue. We are interested in how much better the overall results can be when patients’ choice preference is not completely considered. In this experiment, idle servers provide services to patients in the order of appointment. However, these patients with personal preferences have to wait until the doctors of their choice are available. Finally, we calculate that the objective function value under this rule is 50816 yuan.Figure 6
Scheduling Gantt diagram based on sequence rules.In Figure7, we conduct 50 rounds of learning based on the Q-learning algorithm, where the abscissa represents the number of learning rounds and the ordinate represents the total reward value corresponding to each round. Obviously, the learning result curve of the first 40 times is very volatile, because Q-learning algorithm tries to find a better result to balance the four subgoals. The curve finally converges to near 45500, which shows that the results of Q-learning algorithm are better than those of FCFS. Figure 8 is the scheduling Gantt chart given by the 50th round of learning results, which not only shows the doctors and patients match relations but also reflects the patients’ personal choice behavior. In order to reduce the long waiting time of patients, the service sequence of patients who arrive at the current time is arranged downstream of the patients who have made an appointment in the previous time period. For example, for patients who make an appointment at 9 : 00–10 : 00, the Q-learning algorithm determines their service order and which doctor they should be served by, but their service order is scheduled after the patients who make an appointment at 8 : 00–9:00.Figure 7
Performance curve ofQ-learning scheduling approach.Figure 8
Scheduling Gantt diagram based onQ-learning.The digits in Figure8 are labeled according to the order of arrival of the patients. Due to the characteristic of large scale, we only give some scheduling results in each time period. Interestingly, the patient service order based on reinforcement learning is almost completely different from that in the FCFS rules, and this is because of the fact that the reinforcement learning algorithm can achieve a better trade-off between the waiting time of patients and the setup cost of doctors, and it also takes into account the service rate of doctors, the different medical costs from doctors, and the personal choice preference of some patients.Our experimental results can provide decision support for managers. In a queuing system with multiple types of patients and multiple service desks, we can reasonably arrange the number of patients and service order of each service desk according to the choice preference of patients. This has certain reference significance for other organizations with scarce resources. According to experiments conducted on the computer using Python, we can obtain satisfactory results within 3 minutes through theQ-learning. Due to the large scale of the proposed problem, the relatively optimal results that can be obtained within a few minutes are enough to prove that our proposed method has high effectiveness for solving the dynamic scheduling problem with multiple types of patients and multiple service queues.


## 6. Conclusions and Future Research

In this paper, a dynamic server scheduling problem in a special in vitro fertilization (IVF) queuing network, which is developed in an integrated cloud healthcare system, is investigated in order to address the prolonged waiting problem for IVF service. Based on continuous-time Markov procedure, a mathematical model is established, in which multiple types of patient selection preferences and multiple doctors with different service rates are considered simultaneously. To solve this model, aQ-learning-based solution method is proposed, where the reward functions are designed according to four conflicting cost functions: setup cost, waiting cost, penalty cost, and medical cost. A series of simulation experiments that are generated according to the actual data from Shenyang cloud hospital are carried out to validate the performance of the proposed reinforcement learning (RL) method for the investigated dynamic server scheduling problem in IVF queuing network.The main contributions in this work lie in three aspects. Firstly, the IVF queueing network developed in cloud healthcare system is helpful to cope with the prolonged waiting problem of IVF medical service. Experimental results show that the waiting costs of patients decrease significantly in this integrated IVF queueing network. Secondly, dynamic server scheduling decision, that is, allocation of doctors in general hospital, is important to improve the efficiency of IVF medical service system. Our developed Markov model can exhibit a very nice flexibility in terms of both patient selection preference and medical resource utilization. Finally, the RL method is effective in solving the investigated problem in this paper. Based on the experimental results, our proposedQ-learning-based solution method significantly outperforms the traditional service rule. In general, the proposed methodology in this paper can not only make good use of the bottleneck medical resources in general hospital but also improve the utilization rate of idle medical resources in community hospital and it enables providing management insights for the cooperation of hospitals in hierarchical medical system.This work can be extended in future researches. Firstly, we can consider the occupation of resources in the telemedicine service process of IVF in community hospitals, as well as the influence of the success rate of surgery on the queuing system. Secondly, how to balance the interests of patients and multiple hospitals is also an interesting research issue. Finally, we can further analyze the impact of telemedicine service process in community hospitals on the service rate of doctors in general hospitals.



---

*Source: 6670288-2021-01-21.xml*