# A Privacy-Preserving Reinforcement Learning Approach for Dynamic Treatment Regimes on Health Data

Authors: Xiaoqiang Sun; Zhiwei Sun; Ting Wang; Jie Feng; Jiakai Wei; Guangwu Hu
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Based on the clinical states of the patient, dynamic treatment regime technology can provide various therapeutic methods, which is helpful for medical treatment policymaking. Reinforcement learning is an important approach for developing this technology. In order to implement the reinforcement learning algorithm efficiently, the computation of health data is usually outsourced to the untrustworthy cloud server. However, it may leak, falsify, or delete private health data. Encryption is a common method for solving this problem. But the cloud server is difficult to calculate encrypted health data. In this paper, based on Cheon et al.’s approximate homomorphic encryption scheme, we first propose secure computation protocols for implementing comparison, maximum, exponentiation, and division. Next, we design a homomorphic reciprocal of square root protocol firstly, which only needs one approximate computation. Based on the proposed secure computation protocols, we design a secure asynchronous advantage actor-critic reinforcement learning algorithm for the first time. Then, it is used to implement a secure treatment decision-making algorithm. Simulation results show that our secure computation protocols and algorithms are feasible.


---

## Body

## 1. Introduction

As a recent healthcare tendency, personalized medicine [1] enables the patient to obtain early diagnoses, risk estimation, optimal treatments with low costs by using molecular and cellular analysis technologies, diagnosis results, genetic information, etc. Personalized medicine is usually implemented by the dynamic treatment regime technology [2, 3], which can provide various therapeutic methods according to the time-varying clinical states of the patient. This technology is particularly suitable for coping with complex chronic illnesses, such as diabetes, mental diseases, alcohol dependence, and human immunodeficiency virus infection, which have various stages.Reinforcement learning [4], which is implemented by trial-and-error and interaction with the dynamic environment, is an important method for developing dynamic treatment regimes, industry automation, vehicular networks [5, 6], and other scenarios [7–12]. Meanwhile, with the developing technologies of internet of things and cloud computing, dynamic treatment regimes that are based on reinforcement learning are becoming increasingly attractive. For example, wearable devices are helpful for monitoring the patient’s health data, which include heart rates and blood sugar levels. Next, collected health data are stored on the cloud. Then, the reinforcement learning algorithm can be implemented on these health data for making treatment decisions.Unfortunately, because of the patient’s limited computation ability, health data are usually outsourced to the cloud server for implementing the reinforcement learning algorithm. Because the cloud server may be untrusted, it is likely that health data will be illegally accessed, forged, tampered, or discarded in the process of transmission and computation. In addition, it may be harmful for personal privacy, economic interests, and even the security of human life. For example, as a billing service company, American Medical Collection Agency was intruded in 2019 [13]. This attack affects the health data of about 12 million patients. Besides, the parent firm of this company has filed for bankruptcy. Furthermore, unlike financial data or other types of human-generated data [14], health data are permanent biological data. They cannot be modified or wiped to avoid the damage, which is caused by health data disclosure.In order to protect health data, we can encrypt health data by using a traditional encryption algorithm. Unfortunately, the reinforcement learning algorithm cannot be executed on the encrypted health data easily and flexibly. Homomorphic encryption [15] supports the operations on the ciphertext. Hence, the cloud server can run the reinforcement learning algorithm on the encrypted health data perfectly by using homomorphic encryption without leaking patient privacy. Finally, the encrypted computation result is returned to the patient. The computation result can be obtained by using the patient’s secret key.In this paper, we endeavor to study the security of health data in the above realistic scenario and focus on the secure implementation of the asynchronous advantage actor-critic (A3C) reinforcement learning algorithm. Taking into account the privacy and computation of health data on the untrusted cloud servers, we adopt homomorphic encryption as the main encryption primitive to carry out our research. Eventually, we make the following three contributions:(1)
Because the efficiency of Cheon et al.’s approximate homomorphic encryption scheme [16] is better than that of fully homomorphic encryption (FHE), we use it to design secure computation protocols, namely, homomorphic comparison protocol, homomorphic maximum protocol, homomorphic exponential protocol, and homomorphic division protocol. Based on these protocols, we first design the homomorphic reciprocal of square root protocol, which needs only one approximate computation(2)
Based on the proposed secure computation protocols, we design the secure A3C reinforcement learning algorithm for the first time. Then, we use it to implement a secure treatment decision-making algorithm(3)
Finally, we simulate the proposed secure computation protocols and algorithms on the personal computer’s virtual machine. Then, we demonstrate the efficiency of our secure computation algorithms according to the thorough analysisThe layout of this paper is as follows. Section2 analyzes related work about homomorphic encryption and secure computation of encrypted health data. Preliminaries are presented in Section 3. Section 4 shows related work about secure dynamic treatment regimes on health data. Building blocks are discussed in Section 5. Section 6 describes the proposed privacy-preserving A3C reinforcement learning algorithm and treatment decision-making algorithm. Performance results are shown and analyzed in Section 7. Finally, this paper is concluded in Section 8.


## 2. Related Work

In this section, we introduce related work about reinforcement learning, homomorphic encryption, and the computation of encrypted health data, which are described as follows.Reinforcement learning can be mainly classified as value-based algorithms, policy-based algorithms, and actor-critic algorithms. Value-based algorithms usually compute the optimum cumulative reward and give a suggested policy. As a typical value-based algorithm, Q-learning is used for estimating the utility of the individual pair that consists of a state and an action. Q-learning has been applied for path planning [17, 18] in vehicular networks. Policy-based algorithms can evaluate the optimum policy directly. Williams [19] proposed a policy-based algorithm REINFORCE. Actor-critic algorithms combine the advantages of value-based algorithms and policy-based algorithms. The A3C reinforcement learning algorithm [20] is an actor-critic algorithm. It can work in discrete action spaces as well as continuous action spaces [21].The concept of homomorphic encryption begins from privacy homomorphism [15]. According to the types of supported homomorphic operations, homomorphic encryption can be divided into partial homomorphic encryption (PHE), somewhat homomorphic encryption (SWHE), and FHE. PHE only supports homomorphic addition or homomorphic multiplication. SWHE is the basis of FHE. SWHE supports finite homomorphic addition and homomorphic multiplication. FHE supports arbitrary homomorphic addition and homomorphic multiplication.In 2009, Gentry [22] designed the first FHE scheme, which is based on ideal lattices. Since then, homomorphic encryption has become a research hotspot. Next, in order to improve the efficiency of homomorphic operations, Gentry et al. [23] first constructed the FHE scheme, which is based on the approximate eigenvector method. In this scheme, the ciphertext noise increases linearly after each homomorphic multiplication. Although homomorphic multiplication of this scheme is efficient, it does not support the technique of single instruction multiple data (SIMD) [24]. Then, based on the learning with errors over rings (RLWE) [25] assumption and relinearization technique [24], Brakerski et al. [24] designed a FHE scheme, which supports the SIMD technique. However, this scheme does not support approximate homomorphic operations. Hence, based on Brakerski et al.’s scheme [24], Cheon et al. [16] proposed an improved homomorphic encryption scheme.In terms of the computation of encrypted health data by using homomorphic encryption, there exist following several schemes. Khedr and Gulak [26] first proposed an optimized homomorphic encryption scheme, which is based on Gentry’s scheme [23]. Then, the proposed scheme is applied for secure medical computations, which include comparison, Pearson goodness-of-fit test, and logistic regression. Sun et al. [27] implemented secure average heart rate, long QT syndrome detection, and chi-square tests by using Dowlin et al.’s FHE scheme [28]. Based on Boneh et al.’s homomorphic encryption scheme [29], Poon et al. [30] implemented the secure Fisher’s exact test algorithm, which is often used to guarantee the statistical stability of genetic analysis. Raisaro et al. [31] used homomorphic encryption to explore genomic cohorts securely in a real scenario. In 2019, based on the distributed two trapdoors public-key algorithm [32] and Q-learning algorithm, Liu et al. [2] constructed a secure reinforcement learning model, which is helpful for making treatment decisions dynamically. Based on Fan’s SWHE scheme [33], Jiang et al. [34] performed secure and efficient feature point detection and image matching for retinal images of diabetic retinopathy.However, most of the above schemes are based on PHE, which only supports homomorphic addition or homomorphic multiplication. PHE may not support homomorphic multiplication. If homomorphic multiplication is required in some schemes, excessive rounds of interactions are needed. FHE can avoid this problem. But FHE confronts the problem of the efficiency of homomorphic operations. Furthermore, the Q-learning algorithm is usually used as the reinforcement learning algorithm in the above schemes. There does not exist an approach that can implement the A3C reinforcement learning algorithm securely.


## 3. Preliminaries

In this section, we begin with basic notations and definition of Cheon et al.’s approximate homomorphic encryption scheme. Then, we give the introduction of the A3C algorithm.


### 3.1. Basic Notations

Letz=z−1/2,z+1/2, where z is a real number. Let zp=z−z/p·p∈−p/2,p/2, where p denotes an integer.LetR=ℤ/Φmx denote the ring modulo Φmx, where λ is the security parameter, m is a positive integer, and Φmx is the mth cyclotomic polynomial. Rq=ℤqx/Φmx represents the ring modulo q and Φmx, where q is the prime modulus, q≥2.As for an integerh>0, the distribution HWTh is selected from 0,±1 randomly with the Hamming weight h. As for a rational number σ>0, the distribution DGσ2 outputs a vector, which coefficients are selected from the discrete Gaussian distribution with the variance σ2. As for a rational number 0≤ρ≤1, the distribution ZOρ is chosen from 0,±1 randomly, where ρ/2 is the probability that ±1 is selected and 1−ρ is the probability that 0 is selected.


### 3.2. Learning with Errors over Rings

In 2010, Lyubashevsky et al. [25] first proposed the RLWE assumption, which is described as follows.Definition 1 (RLWE).
TheRLWEλ,q,χ assumption is to distinguish two distributions, namely, a,a·s+e∈Rq×Rq and a,c∈UnifRq×Rq, where a∈Rq and s∈Rq, e is an error term, and Unif represents uniform random. Lyubashevsky et al. [25] proved that the security of RLWE assumption relies on ideal lattices.


### 3.3. Cheon et al.’s Homomorphic Encryption Scheme

In this subsection, we introduce Cheon et al.’s approximate homomorphic encryption schemeAHE=KeyGen,Enc,Add,Sub,Mul,Dec,ReScale,Ecd,Dcd [16] as follows:
(i)
AHE.KeyGen1λ,p,L: given the security parameter λ, an integer p, and a level L, this algorithm first sets ql=pl·q0, where q0 is a fixed integer, l=L,⋯,1. It selects a power-of-two integer M=Mλ,qL, an integer P=Pλ,qL, and a rational number σ=σλ,qL. Next, it chooses a vector s from HWTh. The secret key sk is set as 1,s. A ring element a is sampled from RqL. An error term e is sampled from DGσ2. The public key pk is set as b,a∈RqL2, where b=−a·s+emodqL. Then, a ring element a′ is sampled from RP·qL. An error term e′ is sampled from DGσ2. The evaluation key evk is set as b′,a′∈RP·qL2, where b′=−a′·s+e′modP·qL(ii)
AHE.Encpk,m: in order to encrypt a plaintext m, this algorithm samples an integer v from ZO0.5. In addition, it chooses two error terms e0 and e1 from DGσ2. m is encrypted as the ciphertext c=v·pk+m+e0,e1modqL(iii)
AHE.Decsk,c: in this algorithm, c=b,a is decrypted as b+a·smodql(iv)
AHE.Addc′,c′′: in this algorithm, input parameters include two ciphertexts c′=c′0ql,c′1ql and c′′=c′′0ql,c′′1ql, which are under the same secret key. Then, the additive ciphertext cadd=c0ql+c′′0ql,c1ql+c′′1ql(v)
AHE.Mulevk,c′,c′′: in this algorithm, input parameters include evk, two ciphertexts c′=c′0ql,c′1ql and c′′=c′′0ql,c′′1ql, where c′ and c′′ are under the same secret key. Then, the ciphertext ctemp=c0,c1,c2=c′0·c′′0ql,c′0·c′′1+c′1·c′′0ql,c′1·c′′1ql. The multiplicative ciphertext cmul=c0,c1+P−1·c2·evkmodql(vi)
AHE.ReScalel⟶l′c: as for a ciphertext c∈Rql2 at the level l, the new ciphertext c′=ql′/qlcmodq′l(vii)
AHE.Ecdz,Δ: as for a vector z=z0,z1,⋯,zN/2−1∈ℂN/2 and a scaled factor Δ>0, this algorithm outputs z′=σ−1Δ·z∈R, where σ−1 is the inverse operation of a canonical embedding map σ·(viii)
AHE.Dcdz′,Δ: as for z′∈R, this algorithm outputs z=Δ−1·σm∈ℂN/2In Cheon et al.’s scheme, the decryption noise should be bounded by82·σ·N+6σ·N+16σ·h·N for the correctness of decryption. In addition, the noise of the rescaling ciphertext is at most N/3·3+8h. Furthermore, the noise of the multiplicative ciphertext should be less than P−1·ql·8σ·N/3+N/3·3+8h. The details about the analysis of Cheon et al.’s scheme can be found in [16].


### 3.4. Asynchronous Advantage Actor-Critic Reinforcement Learning Algorithm

In 2016, Mnih et al. [20] proposed the asynchronous advantage actor-critic reinforcement learning algorithm, which is based on combining the value-based method and the policy-based method. One advantage of the A3C algorithm is that it can work in discrete action spaces as well as continuous action spaces. In addition, in order to improve the learning efficiency of the A3C algorithm, multiple asynchronous actor-learners, which can interact with the environment and acquire various independent exploration policies, are running in parallel. The details of the A3C algorithm are described as follows.In the A3C algorithm, there is a policy functionπat∣st;θ and a value function Vst;θv, where at denotes an action at the time step t, st denotes a state at the time step t, and θ and θv are two parameters. In addition, Vst;θv and πat∣st;θ will be updated tmax times, where tmax denotes the maximum step. Vst;θv and πat∣st;θ are usually approximated by a single convolutional neural network. Specifically, Vst;θv is based on a linear layer. πat∣st;θ is relied on a softmax layer. Namely, Vst;θv=xst·θv, where xst is a function which is related to st. πat∣st;θ=efat∣st·θ/∑j=0tmaxefaj∣sj·θ, aj is an action at the time step j, and faj∣sj is a function which is related to aj and sj.Furthermore, the A3C algorithm uses two loss functions, namely, policy loss function and value loss function, which are described as follows. On the one hand, the policy loss function(1)fπθ=lnπat∣st;θ·R−Vst;θv+β·Hπst;θ,where R is the reward and the parameter k depends on the state. In addition, the upper bound of k is tmax. rt+i is the immediate reward. The discount factor γ∈0,1. The entropy function Hπst;θ can be set as −Σi=0kfat∣st·θ·lnπst;θ. The hyperparameter β can adjust the intensity of the entropy regulation term. Then, we can conclude that
(2)fπθ=fat∣st−Σj=0tmaxfaj∣sj·θ·R−xst·θv+β·−Σi=0kfat∣st·fat∣st−Σj=0tmaxfaj∣sj·θ2.Hence, the differentiation offπθ with respect to θ is
(3)∂fπθ∂θ=fat∣st−Σj=0tmaxfaj∣sjR−xst·θv+2β·θ·fat∣st·fat∣st−Σj=0tmaxfaj∣sj.On the other hand, the value loss function(4)fvθv=R−Vst;θv2=R−xst·θv2.Hence, the differentiation offvθv with respect to θv is
(5)∂fvθv∂θv=2R−xst·θv∂R∂θv−xst.Based on the above two loss functions and corresponding differentiation, the A3C reinforcement learning algorithm is defined in Algorithm1, which is described as follows. Algorithm 1 requires input parameters θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, and α, where the definition of these parameters are shown in Table 1. In order to implement Algorithm 1, we first set T=0, t=1. If T<Tmax and w∈1,W, we implement the iteration, which is shown as follows. Global gradients dθ and dθv are set as 0. θ′ and θ′v are synchronized as θ and θv, respectively. We set t0=t and obtain the system state St∈S, where S is a state set, S=S0,⋯,Sφ−1, and φ is the number of states. Next, we repeat a subalgorithm until t−t0≠tmax. In this subalgorithm, the action At∈A is obtained by using πAt∣St;θ′, where A is an action set, A=A0,⋯,Aχ−1, and χ is the number of actions. We execute At, get the reward Rt, and observe the next state St+1, where Rt is set as Σi=0k−1γi·rt+i+γk·VSt+k;θ′v=Σi=0k−1γi·rt+i+γk·xSt+k·θ′v. In addition, we set t=t+1. After the implementation of the above subalgorithm, we observe whether t%tg equals to 0. If St is terminal, we set R=0. If St is nonterminal R=VSt;θ′v=xSt·θ′v. Then, we repeat a subalgorithm from i=t−1 to i=t0. In this subalgorithm, R is set as Rt+γ·R, namely, R=Σi=0k−1γi·rt+i+γk·xSt·θ′v+γ·R. We compute
(6)∂fπθ′∂θ′=fAt∣St−∑j=0tmaxfAt∣StR−xSt·θv+2β·θ′·fAt∣StfAt∣St−∑j=0tmaxfAt∣St,where ∂fπθ′/∂θ′ is the differentiation of fπθ′ with respect to θ′. dθ is set as dθ+∂fπθ′/∂θ′. We compute
(7)∂fvθ′v∂θ′v=2R−xSt·θ′v·∂R∂θ′v,where ∂fvθ′v/∂θ′v is the differentiation of fvθ′v with respect to θ′v and ∂R/∂θ′v is the differentiation of R with respect to θ′v. Finally, θ and θv can be updated by using equations θ=θ−ηdθ/g+ε and θv=θv−ηdθv/gv+ε, respectively, where g=α·g+1−αdθ2 and gv=α·gv+1−αdθv2.Algorithm 1: A3C reinforcement learning algorithm.
A3Cθ,θv,θ′,θ′v,T,t,tmax,Tmax,tg,η,W,α:Input: θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, α.Output: θ, θv.SetT=0, t=1.WhileT<Tmax do.Forw=1 to W do.Setdθ=0, dθv=0.Synchronizeθ′=θ, θ′v=θv.Sett0=t and get St.Repeat.GetAt according to πAt∣St;θ′.ExecuteAt, get Rt and observe St+1.t=t+1.Untilt−t0=tmaxIfSt is terminal, R=0.IfSt is non-terminal, R=xSt·θ′v.Fori=t−1 to t0 doR=Rt+γ·R.Compute∂fπθ′/∂θ′=fAt∣St−∑j=0tmaxfAt∣StR−xSt·θv+2β·θ′fAt∣St·fAt∣St−∑j=0tmaxfAt∣St.Computedθ=dθ+∂fπθ′/∂θ′.Compute∂fvθ′v/∂θ′v=2R−xSt·θ′v·∂R/∂θ′v.Computedθv=dθv+∂fπθ′v/∂θ′v.End for.Computeg=α·g+1−αdθ2, gv=α·gv+1−αdθv2.Computeθ=θ−ηdθ/g+ε, θv=θv−ηdθv/gv+ε.End forEnd whileTable 1
Notations.
SymbolDescriptionθ, θvShared parameter vectors in the global networkθ′, θ′vThread-specific parameter vectors in the local networkTGlobal countertLocal step countertmax, TmaxUpper boundstgAn integerηLearning rateWNumber of agentsαMomentum


## 3.1. Basic Notations

Letz=z−1/2,z+1/2, where z is a real number. Let zp=z−z/p·p∈−p/2,p/2, where p denotes an integer.LetR=ℤ/Φmx denote the ring modulo Φmx, where λ is the security parameter, m is a positive integer, and Φmx is the mth cyclotomic polynomial. Rq=ℤqx/Φmx represents the ring modulo q and Φmx, where q is the prime modulus, q≥2.As for an integerh>0, the distribution HWTh is selected from 0,±1 randomly with the Hamming weight h. As for a rational number σ>0, the distribution DGσ2 outputs a vector, which coefficients are selected from the discrete Gaussian distribution with the variance σ2. As for a rational number 0≤ρ≤1, the distribution ZOρ is chosen from 0,±1 randomly, where ρ/2 is the probability that ±1 is selected and 1−ρ is the probability that 0 is selected.


## 3.2. Learning with Errors over Rings

In 2010, Lyubashevsky et al. [25] first proposed the RLWE assumption, which is described as follows.Definition 1 (RLWE).
TheRLWEλ,q,χ assumption is to distinguish two distributions, namely, a,a·s+e∈Rq×Rq and a,c∈UnifRq×Rq, where a∈Rq and s∈Rq, e is an error term, and Unif represents uniform random. Lyubashevsky et al. [25] proved that the security of RLWE assumption relies on ideal lattices.


## 3.3. Cheon et al.’s Homomorphic Encryption Scheme

In this subsection, we introduce Cheon et al.’s approximate homomorphic encryption schemeAHE=KeyGen,Enc,Add,Sub,Mul,Dec,ReScale,Ecd,Dcd [16] as follows:
(i)
AHE.KeyGen1λ,p,L: given the security parameter λ, an integer p, and a level L, this algorithm first sets ql=pl·q0, where q0 is a fixed integer, l=L,⋯,1. It selects a power-of-two integer M=Mλ,qL, an integer P=Pλ,qL, and a rational number σ=σλ,qL. Next, it chooses a vector s from HWTh. The secret key sk is set as 1,s. A ring element a is sampled from RqL. An error term e is sampled from DGσ2. The public key pk is set as b,a∈RqL2, where b=−a·s+emodqL. Then, a ring element a′ is sampled from RP·qL. An error term e′ is sampled from DGσ2. The evaluation key evk is set as b′,a′∈RP·qL2, where b′=−a′·s+e′modP·qL(ii)
AHE.Encpk,m: in order to encrypt a plaintext m, this algorithm samples an integer v from ZO0.5. In addition, it chooses two error terms e0 and e1 from DGσ2. m is encrypted as the ciphertext c=v·pk+m+e0,e1modqL(iii)
AHE.Decsk,c: in this algorithm, c=b,a is decrypted as b+a·smodql(iv)
AHE.Addc′,c′′: in this algorithm, input parameters include two ciphertexts c′=c′0ql,c′1ql and c′′=c′′0ql,c′′1ql, which are under the same secret key. Then, the additive ciphertext cadd=c0ql+c′′0ql,c1ql+c′′1ql(v)
AHE.Mulevk,c′,c′′: in this algorithm, input parameters include evk, two ciphertexts c′=c′0ql,c′1ql and c′′=c′′0ql,c′′1ql, where c′ and c′′ are under the same secret key. Then, the ciphertext ctemp=c0,c1,c2=c′0·c′′0ql,c′0·c′′1+c′1·c′′0ql,c′1·c′′1ql. The multiplicative ciphertext cmul=c0,c1+P−1·c2·evkmodql(vi)
AHE.ReScalel⟶l′c: as for a ciphertext c∈Rql2 at the level l, the new ciphertext c′=ql′/qlcmodq′l(vii)
AHE.Ecdz,Δ: as for a vector z=z0,z1,⋯,zN/2−1∈ℂN/2 and a scaled factor Δ>0, this algorithm outputs z′=σ−1Δ·z∈R, where σ−1 is the inverse operation of a canonical embedding map σ·(viii)
AHE.Dcdz′,Δ: as for z′∈R, this algorithm outputs z=Δ−1·σm∈ℂN/2In Cheon et al.’s scheme, the decryption noise should be bounded by82·σ·N+6σ·N+16σ·h·N for the correctness of decryption. In addition, the noise of the rescaling ciphertext is at most N/3·3+8h. Furthermore, the noise of the multiplicative ciphertext should be less than P−1·ql·8σ·N/3+N/3·3+8h. The details about the analysis of Cheon et al.’s scheme can be found in [16].


## 3.4. Asynchronous Advantage Actor-Critic Reinforcement Learning Algorithm

In 2016, Mnih et al. [20] proposed the asynchronous advantage actor-critic reinforcement learning algorithm, which is based on combining the value-based method and the policy-based method. One advantage of the A3C algorithm is that it can work in discrete action spaces as well as continuous action spaces. In addition, in order to improve the learning efficiency of the A3C algorithm, multiple asynchronous actor-learners, which can interact with the environment and acquire various independent exploration policies, are running in parallel. The details of the A3C algorithm are described as follows.In the A3C algorithm, there is a policy functionπat∣st;θ and a value function Vst;θv, where at denotes an action at the time step t, st denotes a state at the time step t, and θ and θv are two parameters. In addition, Vst;θv and πat∣st;θ will be updated tmax times, where tmax denotes the maximum step. Vst;θv and πat∣st;θ are usually approximated by a single convolutional neural network. Specifically, Vst;θv is based on a linear layer. πat∣st;θ is relied on a softmax layer. Namely, Vst;θv=xst·θv, where xst is a function which is related to st. πat∣st;θ=efat∣st·θ/∑j=0tmaxefaj∣sj·θ, aj is an action at the time step j, and faj∣sj is a function which is related to aj and sj.Furthermore, the A3C algorithm uses two loss functions, namely, policy loss function and value loss function, which are described as follows. On the one hand, the policy loss function(1)fπθ=lnπat∣st;θ·R−Vst;θv+β·Hπst;θ,where R is the reward and the parameter k depends on the state. In addition, the upper bound of k is tmax. rt+i is the immediate reward. The discount factor γ∈0,1. The entropy function Hπst;θ can be set as −Σi=0kfat∣st·θ·lnπst;θ. The hyperparameter β can adjust the intensity of the entropy regulation term. Then, we can conclude that
(2)fπθ=fat∣st−Σj=0tmaxfaj∣sj·θ·R−xst·θv+β·−Σi=0kfat∣st·fat∣st−Σj=0tmaxfaj∣sj·θ2.Hence, the differentiation offπθ with respect to θ is
(3)∂fπθ∂θ=fat∣st−Σj=0tmaxfaj∣sjR−xst·θv+2β·θ·fat∣st·fat∣st−Σj=0tmaxfaj∣sj.On the other hand, the value loss function(4)fvθv=R−Vst;θv2=R−xst·θv2.Hence, the differentiation offvθv with respect to θv is
(5)∂fvθv∂θv=2R−xst·θv∂R∂θv−xst.Based on the above two loss functions and corresponding differentiation, the A3C reinforcement learning algorithm is defined in Algorithm1, which is described as follows. Algorithm 1 requires input parameters θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, and α, where the definition of these parameters are shown in Table 1. In order to implement Algorithm 1, we first set T=0, t=1. If T<Tmax and w∈1,W, we implement the iteration, which is shown as follows. Global gradients dθ and dθv are set as 0. θ′ and θ′v are synchronized as θ and θv, respectively. We set t0=t and obtain the system state St∈S, where S is a state set, S=S0,⋯,Sφ−1, and φ is the number of states. Next, we repeat a subalgorithm until t−t0≠tmax. In this subalgorithm, the action At∈A is obtained by using πAt∣St;θ′, where A is an action set, A=A0,⋯,Aχ−1, and χ is the number of actions. We execute At, get the reward Rt, and observe the next state St+1, where Rt is set as Σi=0k−1γi·rt+i+γk·VSt+k;θ′v=Σi=0k−1γi·rt+i+γk·xSt+k·θ′v. In addition, we set t=t+1. After the implementation of the above subalgorithm, we observe whether t%tg equals to 0. If St is terminal, we set R=0. If St is nonterminal R=VSt;θ′v=xSt·θ′v. Then, we repeat a subalgorithm from i=t−1 to i=t0. In this subalgorithm, R is set as Rt+γ·R, namely, R=Σi=0k−1γi·rt+i+γk·xSt·θ′v+γ·R. We compute
(6)∂fπθ′∂θ′=fAt∣St−∑j=0tmaxfAt∣StR−xSt·θv+2β·θ′·fAt∣StfAt∣St−∑j=0tmaxfAt∣St,where ∂fπθ′/∂θ′ is the differentiation of fπθ′ with respect to θ′. dθ is set as dθ+∂fπθ′/∂θ′. We compute
(7)∂fvθ′v∂θ′v=2R−xSt·θ′v·∂R∂θ′v,where ∂fvθ′v/∂θ′v is the differentiation of fvθ′v with respect to θ′v and ∂R/∂θ′v is the differentiation of R with respect to θ′v. Finally, θ and θv can be updated by using equations θ=θ−ηdθ/g+ε and θv=θv−ηdθv/gv+ε, respectively, where g=α·g+1−αdθ2 and gv=α·gv+1−αdθv2.Algorithm 1: A3C reinforcement learning algorithm.
A3Cθ,θv,θ′,θ′v,T,t,tmax,Tmax,tg,η,W,α:Input: θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, α.Output: θ, θv.SetT=0, t=1.WhileT<Tmax do.Forw=1 to W do.Setdθ=0, dθv=0.Synchronizeθ′=θ, θ′v=θv.Sett0=t and get St.Repeat.GetAt according to πAt∣St;θ′.ExecuteAt, get Rt and observe St+1.t=t+1.Untilt−t0=tmaxIfSt is terminal, R=0.IfSt is non-terminal, R=xSt·θ′v.Fori=t−1 to t0 doR=Rt+γ·R.Compute∂fπθ′/∂θ′=fAt∣St−∑j=0tmaxfAt∣StR−xSt·θv+2β·θ′fAt∣St·fAt∣St−∑j=0tmaxfAt∣St.Computedθ=dθ+∂fπθ′/∂θ′.Compute∂fvθ′v/∂θ′v=2R−xSt·θ′v·∂R/∂θ′v.Computedθv=dθv+∂fπθ′v/∂θ′v.End for.Computeg=α·g+1−αdθ2, gv=α·gv+1−αdθv2.Computeθ=θ−ηdθ/g+ε, θv=θv−ηdθv/gv+ε.End forEnd whileTable 1
Notations.
SymbolDescriptionθ, θvShared parameter vectors in the global networkθ′, θ′vThread-specific parameter vectors in the local networkTGlobal countertLocal step countertmax, TmaxUpper boundstgAn integerηLearning rateWNumber of agentsαMomentum


## 4. Secure Dynamic Treatment Regimes on Health Data

### 4.1. System Model

As shown in Figure1, the system model of secure dynamic treatment regimes on health data consists of four parts, namely, undiagnosed patient, key generation center, cloud servers, and historical data owners, which are described as follows:
(i)
The undiagnosed patient’s current state is collected by using wearable devices, which integrate modules of physiological sensors, weak computation, and communication. Wearable devices include smart bracelet, smart glasses, sleep monitoring sensors, and smart watch. They can collect a variety of health data, such as body temperature, heart rate, blood sugar, and blood volume index. Then, these health data are transmitted to cloud servers for computation. Based on the returned computation result, the patient can obtain the diagnosis(ii)
Key generation center is an indispensable and independent entity, which is trusted by the other entities in this system model. It is responsible for distributing and managing all the public keys and private keys of Cheon et al.’s homomorphic encryption scheme for wearable devices and undiagnosed patients via a secure channel(iii)
Cloud servers have the powerful data storage space. Hence, they store and manage ciphertexts, which come from undiagnosed patients and wearable devices. Additionally, they can perform some computations on these ciphertexts(iv)
Historical data owners have a sequence of medical data and their corresponding decision results. These encrypted data are transmitted and stored on cloud servers. Cloud servers can compute these ciphertexts for training the reinforcement learning modelFigure 1
The model of secure dynamic treatment regimes on health data.


### 4.2. Attack Model

In this paper, we suppose that the entities in the system model are honest-but-curious. Namely, the entities strictly follow the designed protocols. But they are interested in acquiring medical data of other entities. We suppose that there is an adversaryA1∗ in the attack model. The goal of A1∗ is to guess the plaintexts of the challenge historical data owners’ ciphertexts or the challenge wearable devices’ ciphertexts.In order to acquire the ciphertexts of historical data owners and wearable devices, middle ciphertext results during the execution of privacy-preserving A3C reinforcement learning algorithm and treatment decision-making algorithm (Section6), A1∗ eavesdrops on the communication links among the entities in the system model. However, these ciphertexts are based on Cheon et al.’s approximate homomorphic encryption scheme [16]. Hence, A1∗ cannot decrypt these ciphertexts without knowing their secret keys. It can be guaranteed by using the semantic security of Cheon et al.’s scheme. In addition, the key generation center distributes key pairs to historical data owners and wearable devices in a secure way. Furthermore, due to the lack of private keys of these ciphertexts, A1∗ cannot generate evaluation keys. Hence, A1∗ cannot transform these ciphertexts into some domains that A1∗ can decrypt. Besides, A1∗ cannot get useful information by adding or multiplying a plaintext with these ciphertexts. In a conclusion, the proposed model is secure.


### 4.3. System Setup and Overview

Our secure model of dynamic treatment regimes consists of two phases, which are described as follows.(i)
Training dataset outsourcing and initialization: historical data owners initialize input parametersθ, θv, learning rate η, and discount factor γ. The state set S=S0,⋯,Sφ−1 and action set A=A0,⋯,Aχ−1 are encrypted as cS=cS0,⋯,cSφ−1 and cA=cA0,⋯,cAχ−1. Then, historical data owners send cS, cA, η, γ, and other parameters to cloud servers for storage and computation, where j=0,⋯,n−1, and n is the number of historical data owners(ii)
Outsourced sequential treatment decision making: in order to achieve sequential treatment decision making, the undiagnosed patient’s current statex, which comes from wearable devices, is encrypted as cx. Then, cx is transmitted to cloud servers for treatment decision making. Based on our privacy-preserving A3C reinforcement learning algorithm and treatment decision-making algorithm, cloud servers output the encrypted treatment decision ca. The undiagnosed patient decrypts ca to obtain the treatment decision a by using his own secret key


## 4.1. System Model

As shown in Figure1, the system model of secure dynamic treatment regimes on health data consists of four parts, namely, undiagnosed patient, key generation center, cloud servers, and historical data owners, which are described as follows:
(i)
The undiagnosed patient’s current state is collected by using wearable devices, which integrate modules of physiological sensors, weak computation, and communication. Wearable devices include smart bracelet, smart glasses, sleep monitoring sensors, and smart watch. They can collect a variety of health data, such as body temperature, heart rate, blood sugar, and blood volume index. Then, these health data are transmitted to cloud servers for computation. Based on the returned computation result, the patient can obtain the diagnosis(ii)
Key generation center is an indispensable and independent entity, which is trusted by the other entities in this system model. It is responsible for distributing and managing all the public keys and private keys of Cheon et al.’s homomorphic encryption scheme for wearable devices and undiagnosed patients via a secure channel(iii)
Cloud servers have the powerful data storage space. Hence, they store and manage ciphertexts, which come from undiagnosed patients and wearable devices. Additionally, they can perform some computations on these ciphertexts(iv)
Historical data owners have a sequence of medical data and their corresponding decision results. These encrypted data are transmitted and stored on cloud servers. Cloud servers can compute these ciphertexts for training the reinforcement learning modelFigure 1
The model of secure dynamic treatment regimes on health data.


## 4.2. Attack Model

In this paper, we suppose that the entities in the system model are honest-but-curious. Namely, the entities strictly follow the designed protocols. But they are interested in acquiring medical data of other entities. We suppose that there is an adversaryA1∗ in the attack model. The goal of A1∗ is to guess the plaintexts of the challenge historical data owners’ ciphertexts or the challenge wearable devices’ ciphertexts.In order to acquire the ciphertexts of historical data owners and wearable devices, middle ciphertext results during the execution of privacy-preserving A3C reinforcement learning algorithm and treatment decision-making algorithm (Section6), A1∗ eavesdrops on the communication links among the entities in the system model. However, these ciphertexts are based on Cheon et al.’s approximate homomorphic encryption scheme [16]. Hence, A1∗ cannot decrypt these ciphertexts without knowing their secret keys. It can be guaranteed by using the semantic security of Cheon et al.’s scheme. In addition, the key generation center distributes key pairs to historical data owners and wearable devices in a secure way. Furthermore, due to the lack of private keys of these ciphertexts, A1∗ cannot generate evaluation keys. Hence, A1∗ cannot transform these ciphertexts into some domains that A1∗ can decrypt. Besides, A1∗ cannot get useful information by adding or multiplying a plaintext with these ciphertexts. In a conclusion, the proposed model is secure.


## 4.3. System Setup and Overview

Our secure model of dynamic treatment regimes consists of two phases, which are described as follows.(i)
Training dataset outsourcing and initialization: historical data owners initialize input parametersθ, θv, learning rate η, and discount factor γ. The state set S=S0,⋯,Sφ−1 and action set A=A0,⋯,Aχ−1 are encrypted as cS=cS0,⋯,cSφ−1 and cA=cA0,⋯,cAχ−1. Then, historical data owners send cS, cA, η, γ, and other parameters to cloud servers for storage and computation, where j=0,⋯,n−1, and n is the number of historical data owners(ii)
Outsourced sequential treatment decision making: in order to achieve sequential treatment decision making, the undiagnosed patient’s current statex, which comes from wearable devices, is encrypted as cx. Then, cx is transmitted to cloud servers for treatment decision making. Based on our privacy-preserving A3C reinforcement learning algorithm and treatment decision-making algorithm, cloud servers output the encrypted treatment decision ca. The undiagnosed patient decrypts ca to obtain the treatment decision a by using his own secret key


## 5. Building Blocks

### 5.1. Encoding Rational Number

In order to implement the privacy-preserving A3C reinforcement learning algorithm, we need to encrypt health data. Health data are usually rational numbers. However, most of the homomorphic encryption schemes only support homomorphic operations over integers. They cannot cope with rational numbers. Hence, in this paper, we take Cheon et al.’s encoding technique [16], which can encode a rational number. Then, the rational number can be converted to a ring element just like using the integer encoding technique. We can use Cheon et al.’s scheme [16] to encrypt the converted result.


### 5.2. Homomorphic Comparison Protocol

In order to implement comparison for our secure computation algorithms, we design a homomorphic comparison protocol by using Cheon et al.’s scheme [16] and Sun et al.’s method [35]. We suppose that the user owns plaintexts m0 and m1. Then, the user uses Cheon et al.’s scheme to encrypt these plaintexts. The ciphertexts are c0 and c1, respectively. The user owns the secret key sk. The cloud server is responsible for storing the ciphertexts. As shown in Algorithm 2, the cloud server first computes the ciphertext cb=t+c0−c1, where t is the plaintext modulus. Next, the cloud server transmits cb to the user. The user uses sk to decrypt cb. The decryption result is b. If b>t, m0>m1. If b=t, m0=m1. If b<t, m0<m1. For example, we suppose that t=2, m0=0, m1=1, and then cb=2+c0−c1, where c0 and c1 are ciphertexts of 0 and 1, respectively. The decryption result of cb equals to 1. Hence, m0<m1.Algorithm 2: Homomorphic comparison protocol.
compc0,c1:Input:c0 and c1.Output:cb.1. Computecb=t+c0−c1.2. Returncb to the user.3. Ifb>t, m0>m1.Ifb=t, m0=m1.Ifb<t, m0<m1.


### 5.3. Homomorphic Maximum Protocol

In order to compute the encrypted index of the largest plaintext, we design a homomorphic maximum protocol by using the above protocol and Sun et al.’s method [35]. We suppose that the user owns m0, ⋯, mk−1, where k is the number of plaintexts. The user uses Cheon et al.’s scheme [16] to encrypt these plaintexts. The ciphertexts are c0, ⋯, ck−1, respectively. The user owns the secret key sk. As shown in Algorithm 3, the cloud server computes cb=compcmax,ci, where cmax is initialized as c0. If the decryption result b<t, cmax=ci, i=i+1. The cloud server continues to compare cmax and ci until i>k−1. Finally, the user can obtain the index of the largest plaintext by decrypting cmax. For example, there exist ciphertexts c2, c3, and c4, whose plaintexts are 2, 3, and 4, respectively. We set cmax=c2. Next, we first compare cmax and c3. cmax is updated as c3. Then, we compare cmax and c4. cmax is updated as c4. After the decryption of cmax, the user gets the maximum result 4.Algorithm 3: Homomorphic maximum protocol.
argmaxc0,c1,⋯,ck−1:Input:c0,c1,⋯,ck−1.Output:cmax.1. Setcmax=c0.2. Fori=1 to k−1 do.3.cb=compcmax,ci.4. Decryptcb to get b.5. Ifb=1, cmax=ci.6.i=i+1.End for


### 5.4. Homomorphic Exponential Protocol

In this section, based on the Taylor series, we begin to describe the homomorphic exponential protocol. We suppose that the user owns the plaintextm. Next, it is encrypted as cm by using Cheon et al.’s homomorphic encryption scheme. Only the user has the secret key. Then, cm is stored on the cloud server. In the homomorphic exponential protocol (Algorithm 4), the cloud server first computes the ciphertext cem=1+cm+cm2/2!+cm3/3!+⋯+cmn/n! without decryption, where n denotes an integer. The precision of em increases with the increasing of n. Then, cem is returned to the user. The user gets the exponential result em by using his secret key. For example, we can set m=4, n=3, and then ce2=1+c4+c42/2!+c43/3!, where ce2 and c4 are ciphertexts of e2 and 4, respectively. After the decryption of ce2, the user gets the exponential result e2.Algorithm 4: Homomorphic exponential protocol.
expcm,n:Input:cm, n.Output:cem.1. Computecem=1+cm+⋯+cmn/n!.2. Returncem to the user.


### 5.5. Homomorphic Division Protocol

In this section, we begin to describe the homomorphic division protocol. We suppose that the user owns plaintextsm0, m1, and m2. Then, they are encrypted as cm0, cm1, and cm2 by using Cheon et al.’s homomorphic encryption scheme. Only the user has the secret key. Then, cm0, cm1, and cm2 are transmitted to the cloud server. In order to output the ciphertext cdiv of the plaintext m2/m0+m1, we design the homomorphic division protocol (Algorithm 5), which is described as follows. The cloud server first computes the ciphertext cadd=cm0+cm1 without decryption, where the plaintext of cadd is add=m0+m1. Then, cadd is returned to the user. The user gets the plaintext add by using his secret key. The user calculates rev=1/add. rev is encrypted as crev by using Cheon et al.’s scheme. crev is transmitted to the cloud server. The cloud server calculates the ciphertext cdiv=cm2·crev. Finally, cdiv is returned to the user. After the decryption of cdiv, the user gets the division result div=m2/m0+m1. For example, we set m0=6, m1=1, and m2=2. Then, the cloud server calculates cadd=cm0+cm1=c3, where the plaintext of c3 is 3. c3 is returned to the user. After the decryption of c3, the user calculates rev=1/3=0.33. The ciphertext crev is sent to the cloud server. The cloud server calculates cdiv=cm2·crev=c0.33×6=c1.98, where the plaintext of c1.98 is 1.98.Algorithm 5: Homomorphic division protocol.
divcm0,cm1,cm2:Input:cm0, cm1 and cm2.Output: The ciphertext cdiv.1. Computecadd=cm0+cm1.2. Returncadd to the user.3. Decryptcadd to obtain add.4. Calculaterev=1/add.5.rev is encrypted as crev.6.crev is transmitted to the cloud server.7. Calculatecdiv=cm2·crev.8.cdiv is returned to the user.


### 5.6. Homomorphic Reciprocal of Square Root Protocol

In this section, we begin to describe the homomorphic reciprocal of square root protocol. The traditional method is to compute the ciphertext of the approximate square root firstly. Then, it computes the approximate reciprocal of square root homomorphically. However, two approximate computations will affect the precision of the final result. Hence, based on Lomont’s fast inverse square root algorithm [36], we design a new homomorphic reciprocal of square root protocol (Algorithm 6), which only needs one approximate computation. In our protocol, we suppose that the user owns the floating number m=1+m02m1−127, where 0<m0<1 and 0<m1<255. It is encrypted as cm by using Cheon et al.’s homomorphic encryption scheme. Only the user has the secret key. cm is stored on the cloud server. cm is first transmitted to the user. The user decrypts cm by his secret key. The decryption result m is converted to an integer m′=m1·223+m0·223. m′ is encrypted as cm′ by using Cheon et al.’s scheme. Next, cm′ is transmitted to the cloud server. The cloud server computes the intermediate ciphertext
(8)ctemp=32127−0.045223−0.5cm′,where the plaintext of ctemp is the floating number temp. Then, the cloud server sends ctemp to the user. The user decrypts ctemp to obtain temp=temp1·223+temp0·223, where 0<temp0<1 and 0<temp1<255. temp is converted to an integer temp′=1+temp02temp1−127. temp′ is encrypted as ctemp′. ctemp′ is transmitted to the cloud server. The cloud server computes the ciphertext
(9)c1/m=32ctemp′−12cm·ctemp′3.Then,c1/m is returned to the user. The user gets the reciprocal of square root 1/m by using his secret key. For example, we set m=1+0.25124−127=0.156. Then, m is converted to m′=62·223+0.125·223. The ciphertext cm′ of m′ is sent to the cloud server. The cloud server computes
(10)ctemp=32127−0.045223−0.5cm′.The user decryptsctemp to obtain temp=128·223+0.3075·223. temp is converted to temp′=1+0.30752128−127=2.615. The ciphertext ctemp′ of temp′ is sent to the cloud server. The cloud server computes the ciphertext
(11)c1/0.156=32ctemp′−12cm·ctemp′3.The user decryptsc1/m to obtain 1/0.156=3/2·2.615−1/2·0.156·2.6153≈2.528.Algorithm 6: Homomorphic reciprocal of square root protocol.
1/cm:Input: The ciphertext cm.Output:c1/m.1. Transmitcm to the user.2. Decryptcm to obtain m=1+m02m1−127.3. Convertm to obtain m′=m1·223+m0·223.4. Encryptm′ as cm′.5. Transmitcm′ to the cloud server.6. Computectemp=3/2127−0.045223−0.5cm′.7. Transmitctemp to the user.8. Decryptctemp to obtain temp=temp1+temp0·223.9. Converttemp to temp′=1+temp02temp1−127.10. Encrypttemp′ as ctemp′.11. Transmitctemp′ to the cloud server.12. Computec1/m=3/2ctemp′−1/2cm·ctemp′3.13. Returnc1/m to the user.


## 5.1. Encoding Rational Number

In order to implement the privacy-preserving A3C reinforcement learning algorithm, we need to encrypt health data. Health data are usually rational numbers. However, most of the homomorphic encryption schemes only support homomorphic operations over integers. They cannot cope with rational numbers. Hence, in this paper, we take Cheon et al.’s encoding technique [16], which can encode a rational number. Then, the rational number can be converted to a ring element just like using the integer encoding technique. We can use Cheon et al.’s scheme [16] to encrypt the converted result.


## 5.2. Homomorphic Comparison Protocol

In order to implement comparison for our secure computation algorithms, we design a homomorphic comparison protocol by using Cheon et al.’s scheme [16] and Sun et al.’s method [35]. We suppose that the user owns plaintexts m0 and m1. Then, the user uses Cheon et al.’s scheme to encrypt these plaintexts. The ciphertexts are c0 and c1, respectively. The user owns the secret key sk. The cloud server is responsible for storing the ciphertexts. As shown in Algorithm 2, the cloud server first computes the ciphertext cb=t+c0−c1, where t is the plaintext modulus. Next, the cloud server transmits cb to the user. The user uses sk to decrypt cb. The decryption result is b. If b>t, m0>m1. If b=t, m0=m1. If b<t, m0<m1. For example, we suppose that t=2, m0=0, m1=1, and then cb=2+c0−c1, where c0 and c1 are ciphertexts of 0 and 1, respectively. The decryption result of cb equals to 1. Hence, m0<m1.Algorithm 2: Homomorphic comparison protocol.
compc0,c1:Input:c0 and c1.Output:cb.1. Computecb=t+c0−c1.2. Returncb to the user.3. Ifb>t, m0>m1.Ifb=t, m0=m1.Ifb<t, m0<m1.


## 5.3. Homomorphic Maximum Protocol

In order to compute the encrypted index of the largest plaintext, we design a homomorphic maximum protocol by using the above protocol and Sun et al.’s method [35]. We suppose that the user owns m0, ⋯, mk−1, where k is the number of plaintexts. The user uses Cheon et al.’s scheme [16] to encrypt these plaintexts. The ciphertexts are c0, ⋯, ck−1, respectively. The user owns the secret key sk. As shown in Algorithm 3, the cloud server computes cb=compcmax,ci, where cmax is initialized as c0. If the decryption result b<t, cmax=ci, i=i+1. The cloud server continues to compare cmax and ci until i>k−1. Finally, the user can obtain the index of the largest plaintext by decrypting cmax. For example, there exist ciphertexts c2, c3, and c4, whose plaintexts are 2, 3, and 4, respectively. We set cmax=c2. Next, we first compare cmax and c3. cmax is updated as c3. Then, we compare cmax and c4. cmax is updated as c4. After the decryption of cmax, the user gets the maximum result 4.Algorithm 3: Homomorphic maximum protocol.
argmaxc0,c1,⋯,ck−1:Input:c0,c1,⋯,ck−1.Output:cmax.1. Setcmax=c0.2. Fori=1 to k−1 do.3.cb=compcmax,ci.4. Decryptcb to get b.5. Ifb=1, cmax=ci.6.i=i+1.End for


## 5.4. Homomorphic Exponential Protocol

In this section, based on the Taylor series, we begin to describe the homomorphic exponential protocol. We suppose that the user owns the plaintextm. Next, it is encrypted as cm by using Cheon et al.’s homomorphic encryption scheme. Only the user has the secret key. Then, cm is stored on the cloud server. In the homomorphic exponential protocol (Algorithm 4), the cloud server first computes the ciphertext cem=1+cm+cm2/2!+cm3/3!+⋯+cmn/n! without decryption, where n denotes an integer. The precision of em increases with the increasing of n. Then, cem is returned to the user. The user gets the exponential result em by using his secret key. For example, we can set m=4, n=3, and then ce2=1+c4+c42/2!+c43/3!, where ce2 and c4 are ciphertexts of e2 and 4, respectively. After the decryption of ce2, the user gets the exponential result e2.Algorithm 4: Homomorphic exponential protocol.
expcm,n:Input:cm, n.Output:cem.1. Computecem=1+cm+⋯+cmn/n!.2. Returncem to the user.


## 5.5. Homomorphic Division Protocol

In this section, we begin to describe the homomorphic division protocol. We suppose that the user owns plaintextsm0, m1, and m2. Then, they are encrypted as cm0, cm1, and cm2 by using Cheon et al.’s homomorphic encryption scheme. Only the user has the secret key. Then, cm0, cm1, and cm2 are transmitted to the cloud server. In order to output the ciphertext cdiv of the plaintext m2/m0+m1, we design the homomorphic division protocol (Algorithm 5), which is described as follows. The cloud server first computes the ciphertext cadd=cm0+cm1 without decryption, where the plaintext of cadd is add=m0+m1. Then, cadd is returned to the user. The user gets the plaintext add by using his secret key. The user calculates rev=1/add. rev is encrypted as crev by using Cheon et al.’s scheme. crev is transmitted to the cloud server. The cloud server calculates the ciphertext cdiv=cm2·crev. Finally, cdiv is returned to the user. After the decryption of cdiv, the user gets the division result div=m2/m0+m1. For example, we set m0=6, m1=1, and m2=2. Then, the cloud server calculates cadd=cm0+cm1=c3, where the plaintext of c3 is 3. c3 is returned to the user. After the decryption of c3, the user calculates rev=1/3=0.33. The ciphertext crev is sent to the cloud server. The cloud server calculates cdiv=cm2·crev=c0.33×6=c1.98, where the plaintext of c1.98 is 1.98.Algorithm 5: Homomorphic division protocol.
divcm0,cm1,cm2:Input:cm0, cm1 and cm2.Output: The ciphertext cdiv.1. Computecadd=cm0+cm1.2. Returncadd to the user.3. Decryptcadd to obtain add.4. Calculaterev=1/add.5.rev is encrypted as crev.6.crev is transmitted to the cloud server.7. Calculatecdiv=cm2·crev.8.cdiv is returned to the user.


## 5.6. Homomorphic Reciprocal of Square Root Protocol

In this section, we begin to describe the homomorphic reciprocal of square root protocol. The traditional method is to compute the ciphertext of the approximate square root firstly. Then, it computes the approximate reciprocal of square root homomorphically. However, two approximate computations will affect the precision of the final result. Hence, based on Lomont’s fast inverse square root algorithm [36], we design a new homomorphic reciprocal of square root protocol (Algorithm 6), which only needs one approximate computation. In our protocol, we suppose that the user owns the floating number m=1+m02m1−127, where 0<m0<1 and 0<m1<255. It is encrypted as cm by using Cheon et al.’s homomorphic encryption scheme. Only the user has the secret key. cm is stored on the cloud server. cm is first transmitted to the user. The user decrypts cm by his secret key. The decryption result m is converted to an integer m′=m1·223+m0·223. m′ is encrypted as cm′ by using Cheon et al.’s scheme. Next, cm′ is transmitted to the cloud server. The cloud server computes the intermediate ciphertext
(8)ctemp=32127−0.045223−0.5cm′,where the plaintext of ctemp is the floating number temp. Then, the cloud server sends ctemp to the user. The user decrypts ctemp to obtain temp=temp1·223+temp0·223, where 0<temp0<1 and 0<temp1<255. temp is converted to an integer temp′=1+temp02temp1−127. temp′ is encrypted as ctemp′. ctemp′ is transmitted to the cloud server. The cloud server computes the ciphertext
(9)c1/m=32ctemp′−12cm·ctemp′3.Then,c1/m is returned to the user. The user gets the reciprocal of square root 1/m by using his secret key. For example, we set m=1+0.25124−127=0.156. Then, m is converted to m′=62·223+0.125·223. The ciphertext cm′ of m′ is sent to the cloud server. The cloud server computes
(10)ctemp=32127−0.045223−0.5cm′.The user decryptsctemp to obtain temp=128·223+0.3075·223. temp is converted to temp′=1+0.30752128−127=2.615. The ciphertext ctemp′ of temp′ is sent to the cloud server. The cloud server computes the ciphertext
(11)c1/0.156=32ctemp′−12cm·ctemp′3.The user decryptsc1/m to obtain 1/0.156=3/2·2.615−1/2·0.156·2.6153≈2.528.Algorithm 6: Homomorphic reciprocal of square root protocol.
1/cm:Input: The ciphertext cm.Output:c1/m.1. Transmitcm to the user.2. Decryptcm to obtain m=1+m02m1−127.3. Convertm to obtain m′=m1·223+m0·223.4. Encryptm′ as cm′.5. Transmitcm′ to the cloud server.6. Computectemp=3/2127−0.045223−0.5cm′.7. Transmitctemp to the user.8. Decryptctemp to obtain temp=temp1+temp0·223.9. Converttemp to temp′=1+temp02temp1−127.10. Encrypttemp′ as ctemp′.11. Transmitctemp′ to the cloud server.12. Computec1/m=3/2ctemp′−1/2cm·ctemp′3.13. Returnc1/m to the user.


## 6. Privacy-Preserving Computation Algorithms

### 6.1. Privacy-Preserving A3C Reinforcement Learning Algorithm

Algorithm 7: Privacy-preserving A3C reinforcement learning algorithm.
PA3Cθ,θv,θ′,θ′v,T,t,tmax,Tmax,tg,η,W,α,cS,cA:Input: θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, α, cS, cA.Output: θ, θv.SetT=0, t=1.WhileT<Tmax doForw=1 to W doSetdθ=0, dθv=0. Synchronize θ′=θ, θ′v=θv.Sett0=t, get cSt.RepeatGetcAt according to argmaxcπ0,⋯,cπtmax.ExecutecAt, get cRt.ObservecSt+1, t=t+1.Untilt−t0=tmaxIfcSt is terminal, cR=0.IfcSt is non-terminal, cR=xcSt·θ′v.Fori=t−1 to t0 doIfcSt is terminalcR=Σi=0k−1γi·rt+i+γk·xcSt·θ′v.Computec∂fπθ′/∂θ′, c∂fvθ′v/∂θ′v.End ifIfcSt is non-terminalcR=Σi=0k−1γi·rt+i+γk·xcSt·θ′v+γ·xcSt·θ′v.Computec∂fπθ′/∂θ′, c∂fvθ′v/∂θ′v.End ifComputecdθ=cdθ+c∂fπθ′/∂θ′.Computecdθv=cdθv+c∂fπθ′v/∂θ′v.End forc′dθ and c′dθv are returned to the cloud server.Computecg=αcg+1−αcdθ2, cgv=αcgv+1−αcdθv2.c′g and c′gv are sent to the cloud server.Setcg=c′g, cgv=c′gv.Computecθ=θ−ηc′dθ/cg+ε, cθv=θv−ηc′dθv/cgv+ε.cθ and cθv are returned to the user.Decryptcθ and cθv to obtain θ and θv.End forEnd whileIn this section, we begin to describe how to implement a privacy-preserving A3C reinforcement learning algorithm by using Cheon et al.’s approximate homomorphic encryption scheme [16]. As shown in Algorithm 7, we describe the privacy-preserving A3C reinforcement learning algorithm as follows. Algorithm 7 requires input parameters θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, α, cS, and cA. Set T=0, t=1. If T<Tmax and w∈1,W, we implement the iteration, which is shown as follows. dθ and dθv are set as 0. θ′ and θ′v are synchronized as θ and θv, respectively. We set t0=t and obtain the encrypted system state cSt∈cS. Next, we repeat a subalgorithm until t−t0≠tmax. In this subalgorithm, the encrypted action cAt∈cA is obtained based on argmaxcπ0,cπ1,⋯,cπtmax, where cπj=efcAj∣cSj·θ′, j=0,1,⋯,tmax. We execute cAt and get the encrypted reward
(12)cRt=Σi=0k−1γi·rt+i+γk·xcSt+k·θv.We observe the next encrypted statecSt+1∈cS. In addition, we set t=t+1. After the implementation of the above subalgorithm, we observe whether cSt is terminal. Then, we repeat a subalgorithm from i=t−1 to i=t0. In this subalgorithm, if cSt is terminal, cR is set as
(13)cR=Σi=0k−1γi·rt+i+γk·cxSt·θ′v.The cloud server computes(14)c∂fπθ′∂θ′=fcAt∣cSt−Σj=0tmaxfcAt∣cStΣi=0k−1γi·rt+i+γk·xcSt·θ′v−xcSt·cθv+2β·cθ′·fcAt∣cStfcAt∣cSt−Σj=0tmaxfcAj∣cSj.The cloud server computes(15)c∂fvθ′v∂θ′v=2Σi=0k−1γi·rt+i+γk·xcSt·cθ′v−xcStcθ′v·γk·xcSt−xcSt.IfcSt is nonterminal, cR is set as
(16)cRt+γcR=Σi=0k−1γi·rt+i+γk·xcSt·cθv′+γ·xcSt·θv′.The cloud server computes(17)c∂fπθ′∂θ′=fcAt∣cSt−Σj=0tmaxfcAt∣cSjΣi=0k−1γi·rt+i+γk·xcSt·θ′v+γ·xcSt·θ′v−xcSt·θv+2β·θ′fcAt∣cSt·fcAt∣cSt−Σj=0tmaxfcAt∣cSj,where c∂fπθ′/∂θ′ is the ciphertext of ∂fπθ′/∂θ′. The cloud server computes
(18)c∂fvθ′v∂θ′v=2Σi=0k−1γi·rt+i+γk·xcSt·θ′v+γ·xcSt·θ′v−xcStθ′v·γk·xcSt+γ·xcSt−xcSt,where c∂fvθ′v/∂θ′v is the ciphertext of ∂fvθ′v/∂θ′v. Then, cdθ is set as cdθ+c∂fπθ′/∂θ′. cdθv is set as cdθv+c∂fπθ′v/∂θ′v. The cloud server computes cg=αcg+1−αcdθ2 and cgv=αcgv+1−αcdθv2. The cloud server sends cdθ and cdθv to the user. The user decrypts cdθ and cdθv to obtain dθ and dθv. dθ and dθv are encrypted as c′dθ and c′dθv. c′dθ and c′dθv are returned to the cloud server. In order to reduce the depth of homomorphic multiplication for the calculation of cg and cgv, the cloud server sends cg and cgv to the user. The user decrypts cg and cgv to obtain g and gv. g and gv are encrypted as c′g and c′gv. The user sends c′g and c′gv to the cloud server. The cloud server sets cg=c′g and cgv=c′gv. Finally, based on the above homomorphic reciprocal of square root protocol, θ and θv can be updated by using equations cθ=θ−ηc′dθ/cg+ε and c′θv=θv−ηcdθv/cgv+ε, respectively. After the execution of Algorithm 7, we can get the encrypted optimized parameters cθ and cθv, which are returned to the user. The user decrypts cθ and cθv to obtain θ and θv, which can be used the implementation of secure treatment decision-making algorithm.In order to better understand Algorithm7, we give an example, which is described as follows. In this example, as shown in Table 2, we set the initial values of related parameters. We suppose that cS0,⋯,cS4 are ciphertexts of S=S0,⋯,S4=0.1,0.1,0.15,0.3,0.35, respectively. cA0,⋯,cA4 are ciphertexts of A=A0,⋯,A4=0.1,0.1,0.15,0.3,0.35, respectively. For the convenience of computation, we let xcSj=cSj, fcAj∣cSj=cAjcSj, πAj∣Sj=efAj∣Sj, i=0,1,2,3,4, j=0,1,2,3,4. The cloud server first computes cπ0=efcA0∣cS0·θ′ and cπ1=efcA1∣cS1·θ′, where the ciphetext of cπ0 is e0.1×0.1×0.5=e0.005 and the cyphertext of cπ1 is e0.1×0.1×0.5=e0.005. Based on the implementation of the protocol argmaxcπ0,cπ1, A1 is executed. The cloud server computes cRt=rt+γxcStθv, where
(19)Rt=rt+γxStθv=0.5+0.6×0.1×0.5=0.505.Table 2
Symbols and initial values.
SymbolInitial valueθ, θv0.5θ′, θ′v0.5Tmax, tmax1t01k1γ0.6r1, r20.5β0.1W1g, gv0α0.5ε0.01η0.1Sett=2. Because S2 is nonterminal, the cloud server computes cR=xcS2·θ′v, where
(20)R=xS2·θv′=0.15×0.5=0.075.Then, the cloud server computes(21)cR=r2+γ·xcS2·θv′+γ·xcS2·θv′,where
(22)R=r2+γ·xS2·θ′v+γ·xS2·θ′v=0.5+0.6×0.15×0.5+0.6×0.15×0.5=0.59.The cloud server computes(23)c∂fπθ′/∂θ′=fcA2∣cS2−Σj=01fcAj∣cSjr2+γ·xcS2·θv′+γ·xcS2·θv′−xcS2θv+2β·θ′·fcA2∣cS2·fcA2∣cS2−Σj=01fcAj∣cSj,where
(24)∂fπθ′∂θ′=fA2∣S2−Σj=01fAj∣Sjr2+γ·xS2·θ′v+γ·xS2·θ′v−xS2θv+2β·θ′·fA2∣S2·fA2∣S2−Σj=01fAj∣Sj=0.15×0.15−0.1×0.1+0.1×0.10.5+0.6×0.15×0.5+0.6×0.15×0.5−0.15×0.5+2×0.1×0.5×0.15×0.150.15×0.15−0.1×0.1+0.1×0.1=0.0013.The cloud server computes(25)c∂fvθ′v/∂θ′v=2r2+γ·xcS2·cθ′v+γ·xcS2·θ′v−xcS2θ′v·γ·xcS2+γ·xcS2−xcS2,where
(26)∂fvθ′v∂θ′v=2r2+γ·xS2·θ′v+γ·xS2·θ′v−xS2θ′v·γ·xS2+γ·xS2−xS2=20.5+0.6×0.15×0.5+0.6×0.15×0.5−0.15×0.50.6×0.15+0.6×0.15−0.15=0.0309.Setcdθ=0, cdθv=0. The cloud server computes cdθ=cdθ+c∂fπθ′/∂θ′, where
(27)dθ=dθ+∂fπθ′∂θ′=0.0013.We computecdθv=cdθv+c∂fπθ′v/∂θ′v, where
(28)dθv=dθv+∂fπθ′v∂θ′v=0.0309.With the help of the user, the cloud server gets refreshed ciphertextscdθv and cdθv. The cloud server computes cg=αcg+1−αcdθ2, where
(29)g=αg+1−αdθ2=1−0.5×0.00132=0.000000845.The cloud server computescgv=αcgv+1−αcdθv2, where
(30)gv=αgv+1−αdθv2=1−0.5×0.03092=0.000477405.After the cloud server obtains ciphertextsc′g and c′gv, we set cg=c′g and cgv=c′gv. The cloud server computes cθ=θ−ηc′dθ/cg+ε, where
(31)θ=θ−ηdθg+ε=0.5−0.10.00130.000000845+0.01≈0.4987.The cloud server computescθv=θv−ηc′dθv/cgv+ε, where
(32)θv=θv−ηdθvgv+ε=0.5−0.10.03090.000477405+0.01≈0.4698.The user can obtain refreshedθ≈0.4987 and θv≈0.4698 by using his secret key.


### 6.2. Secure Treatment Decision-Making Algorithm

In this subsection, based on the above privacy-preserving A3C reinforcement learning algorithm, secure treatment decision-making algorithmTDMθ,θv,cx,cS,cA is implemented in Algorithm 8, which is described as follows. In this algorithm, input parameters include θ, θv, and the undiagnosed patient’s encrypted current state cx, cS, and cA. Set the index col=0. The ciphertext ci,j is initiated, where i=0,⋯,χ−1 and j=0,⋯,φ−1. Firstly, cS’s element cSj is compared with cx by using the above homomorphic comparison protocol compcSj,cx. cb is the encrypted comparison result. After the decryption of cb, if the comparison result b=t, it means that Sj=x, set col=j, where j=0,⋯,φ−1. Next, the cloud server computes the value function’s encrypted value cv=VcScol;θv=xcSj·θv, where the plaintext of cv is VScol;θv. Set the ciphertext chor=c0. The cloud server computes the policy function’s encrypted value
(33)cπ,i=πcAi∣cScol=efcAi∣cScolθ∑j=0tmaxefcAj∣cSjθ,where the plaintext of cπ,i is πAi∣Scol. The cloud server computes homomorphic multiplication between cv and cπ,i, namely, ci,col=cv·cπ,i, where ci,col is the ciphertext of VScol;θvπAi∣Scol, i=0,⋯,χ−1. Set index=0. Finally, the cloud server computes the ciphertext chor,col by using the homomorphic maximum protocol argmaxc0,⋯,cχ−1. Set index=hor. Hence, the treatment decision is cdr=cAindex. Then, cdr will be transmitted to the undiagnosed patient. In order to obtain the treatment decision dr, cdr can be decrypted by using his own secret key.Algorithm 8: Secure treatment decision-making algorithm.
TDMθ,θv,cx,cS,cA:Input: θ, θv, cx, cS, cA.Output: cdr.Fori=0 to χ−1.Forj=0 to φ−1.Initiateci,j.End forEnd forInitiatecb.Forj=0 to φ−1.cb=compcSj,cx.End forDecryptcb to obtain b.Ifb=t.Setcol=j.cv=VcScol;θv=xcSjθv.Setchor=c0.Fori=0 to χ−1.cπ,i=πcAi∣cScol=efcAi∣cScolθ/∑j=0tmaxefcAj∣cSjθ.ci,col=cv·cπ,i.End forSetindex=0.chor,col=argmaxc0,col,⋯,cχ−1,col.Setindex=hor.cdr=cAindex.In order to better understand Algorithm8, we give an example, which is described as follows. In this example, we set χ=4, φ=4, θ=0.5, θv=0.5, and tmax=3. We suppose that cS0,⋯,cS3 are ciphertexts of 0.1,0.2,0.3,0.4, respectively. cA0,⋯,cA3 are ciphertexts of 0.1,0.2,0.3,0.4, respectively. For the convenience of computation, we let xcSj=cSj, fcAi∣cSj=cAicSj, πAi∣Sj=efAi∣Sj, i=0,1,2,3, j=0,1,2,3. The calculation of ci,j=VcSj;θvπcAi∣cSj is the key component for the execution of Algorithm 8, where the corresponding plaintext of ci,j is VSj;θvπAi∣Sj. In this example, the distribution of ci,j can be shown in Table 3.Table 3
The distribution of the encrypted probability.
Encrypted probabilityEncrypted actionscA0cA1cA2cA3Encrypted statescS0c0,0c1,0c2,0c3,0cS1c0,1c1,1c2,1c3,1cS2c0,2c1,2c2,2c3,2cS3c0,3c1,3c2,3c3,3If the patient requires a diagnostic service, the encrypted current statecx is input for the implementation of Algorithm 8. Then, cSbe is compared with cx by the execution of the protocol cb=compcSbe,cx, where be=0. If the comparison result b=t, set col=be. If the comparison result b≠t, the next element cSbe+1 will be compared with cx until be+1>3. We suppose that x=S1, namely, col=1. Hence, cv=xcS1θv=xcS10.5, where the plaintext of cv is 0.2×0.5=0.1. We compute ciphertexts cπ,0, cπ,1, cπ,2, and cπ,3, which corresponding plaintexts are e0.01/temp, e0.02/temp, e0.03/temp, and e0.04/temp, respectively, where temp=e0.01+e0.02+e0.03+e0.04. Then, we compute ciphertexts c0,1, c1,1, c2,1, and c3,1, which corresponding plaintexts are 0.1e0.01/temp, 0.1e0.02/temp, 0.1e0.03/temp, and 0.1e0.04/temp, respectively. Based on the execution of the protocol argmaxc0,1,c1,1,c2,1,c3,1, we can obtain the output ciphertext is c3,1. The encrypted treatment decision is cA3, which corresponding plaintext is A3.


## 6.1. Privacy-Preserving A3C Reinforcement Learning Algorithm

Algorithm 7: Privacy-preserving A3C reinforcement learning algorithm.
PA3Cθ,θv,θ′,θ′v,T,t,tmax,Tmax,tg,η,W,α,cS,cA:Input: θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, α, cS, cA.Output: θ, θv.SetT=0, t=1.WhileT<Tmax doForw=1 to W doSetdθ=0, dθv=0. Synchronize θ′=θ, θ′v=θv.Sett0=t, get cSt.RepeatGetcAt according to argmaxcπ0,⋯,cπtmax.ExecutecAt, get cRt.ObservecSt+1, t=t+1.Untilt−t0=tmaxIfcSt is terminal, cR=0.IfcSt is non-terminal, cR=xcSt·θ′v.Fori=t−1 to t0 doIfcSt is terminalcR=Σi=0k−1γi·rt+i+γk·xcSt·θ′v.Computec∂fπθ′/∂θ′, c∂fvθ′v/∂θ′v.End ifIfcSt is non-terminalcR=Σi=0k−1γi·rt+i+γk·xcSt·θ′v+γ·xcSt·θ′v.Computec∂fπθ′/∂θ′, c∂fvθ′v/∂θ′v.End ifComputecdθ=cdθ+c∂fπθ′/∂θ′.Computecdθv=cdθv+c∂fπθ′v/∂θ′v.End forc′dθ and c′dθv are returned to the cloud server.Computecg=αcg+1−αcdθ2, cgv=αcgv+1−αcdθv2.c′g and c′gv are sent to the cloud server.Setcg=c′g, cgv=c′gv.Computecθ=θ−ηc′dθ/cg+ε, cθv=θv−ηc′dθv/cgv+ε.cθ and cθv are returned to the user.Decryptcθ and cθv to obtain θ and θv.End forEnd whileIn this section, we begin to describe how to implement a privacy-preserving A3C reinforcement learning algorithm by using Cheon et al.’s approximate homomorphic encryption scheme [16]. As shown in Algorithm 7, we describe the privacy-preserving A3C reinforcement learning algorithm as follows. Algorithm 7 requires input parameters θ, θv, θ′, θ′v, T, t, tmax, Tmax, tg, η, W, α, cS, and cA. Set T=0, t=1. If T<Tmax and w∈1,W, we implement the iteration, which is shown as follows. dθ and dθv are set as 0. θ′ and θ′v are synchronized as θ and θv, respectively. We set t0=t and obtain the encrypted system state cSt∈cS. Next, we repeat a subalgorithm until t−t0≠tmax. In this subalgorithm, the encrypted action cAt∈cA is obtained based on argmaxcπ0,cπ1,⋯,cπtmax, where cπj=efcAj∣cSj·θ′, j=0,1,⋯,tmax. We execute cAt and get the encrypted reward
(12)cRt=Σi=0k−1γi·rt+i+γk·xcSt+k·θv.We observe the next encrypted statecSt+1∈cS. In addition, we set t=t+1. After the implementation of the above subalgorithm, we observe whether cSt is terminal. Then, we repeat a subalgorithm from i=t−1 to i=t0. In this subalgorithm, if cSt is terminal, cR is set as
(13)cR=Σi=0k−1γi·rt+i+γk·cxSt·θ′v.The cloud server computes(14)c∂fπθ′∂θ′=fcAt∣cSt−Σj=0tmaxfcAt∣cStΣi=0k−1γi·rt+i+γk·xcSt·θ′v−xcSt·cθv+2β·cθ′·fcAt∣cStfcAt∣cSt−Σj=0tmaxfcAj∣cSj.The cloud server computes(15)c∂fvθ′v∂θ′v=2Σi=0k−1γi·rt+i+γk·xcSt·cθ′v−xcStcθ′v·γk·xcSt−xcSt.IfcSt is nonterminal, cR is set as
(16)cRt+γcR=Σi=0k−1γi·rt+i+γk·xcSt·cθv′+γ·xcSt·θv′.The cloud server computes(17)c∂fπθ′∂θ′=fcAt∣cSt−Σj=0tmaxfcAt∣cSjΣi=0k−1γi·rt+i+γk·xcSt·θ′v+γ·xcSt·θ′v−xcSt·θv+2β·θ′fcAt∣cSt·fcAt∣cSt−Σj=0tmaxfcAt∣cSj,where c∂fπθ′/∂θ′ is the ciphertext of ∂fπθ′/∂θ′. The cloud server computes
(18)c∂fvθ′v∂θ′v=2Σi=0k−1γi·rt+i+γk·xcSt·θ′v+γ·xcSt·θ′v−xcStθ′v·γk·xcSt+γ·xcSt−xcSt,where c∂fvθ′v/∂θ′v is the ciphertext of ∂fvθ′v/∂θ′v. Then, cdθ is set as cdθ+c∂fπθ′/∂θ′. cdθv is set as cdθv+c∂fπθ′v/∂θ′v. The cloud server computes cg=αcg+1−αcdθ2 and cgv=αcgv+1−αcdθv2. The cloud server sends cdθ and cdθv to the user. The user decrypts cdθ and cdθv to obtain dθ and dθv. dθ and dθv are encrypted as c′dθ and c′dθv. c′dθ and c′dθv are returned to the cloud server. In order to reduce the depth of homomorphic multiplication for the calculation of cg and cgv, the cloud server sends cg and cgv to the user. The user decrypts cg and cgv to obtain g and gv. g and gv are encrypted as c′g and c′gv. The user sends c′g and c′gv to the cloud server. The cloud server sets cg=c′g and cgv=c′gv. Finally, based on the above homomorphic reciprocal of square root protocol, θ and θv can be updated by using equations cθ=θ−ηc′dθ/cg+ε and c′θv=θv−ηcdθv/cgv+ε, respectively. After the execution of Algorithm 7, we can get the encrypted optimized parameters cθ and cθv, which are returned to the user. The user decrypts cθ and cθv to obtain θ and θv, which can be used the implementation of secure treatment decision-making algorithm.In order to better understand Algorithm7, we give an example, which is described as follows. In this example, as shown in Table 2, we set the initial values of related parameters. We suppose that cS0,⋯,cS4 are ciphertexts of S=S0,⋯,S4=0.1,0.1,0.15,0.3,0.35, respectively. cA0,⋯,cA4 are ciphertexts of A=A0,⋯,A4=0.1,0.1,0.15,0.3,0.35, respectively. For the convenience of computation, we let xcSj=cSj, fcAj∣cSj=cAjcSj, πAj∣Sj=efAj∣Sj, i=0,1,2,3,4, j=0,1,2,3,4. The cloud server first computes cπ0=efcA0∣cS0·θ′ and cπ1=efcA1∣cS1·θ′, where the ciphetext of cπ0 is e0.1×0.1×0.5=e0.005 and the cyphertext of cπ1 is e0.1×0.1×0.5=e0.005. Based on the implementation of the protocol argmaxcπ0,cπ1, A1 is executed. The cloud server computes cRt=rt+γxcStθv, where
(19)Rt=rt+γxStθv=0.5+0.6×0.1×0.5=0.505.Table 2
Symbols and initial values.
SymbolInitial valueθ, θv0.5θ′, θ′v0.5Tmax, tmax1t01k1γ0.6r1, r20.5β0.1W1g, gv0α0.5ε0.01η0.1Sett=2. Because S2 is nonterminal, the cloud server computes cR=xcS2·θ′v, where
(20)R=xS2·θv′=0.15×0.5=0.075.Then, the cloud server computes(21)cR=r2+γ·xcS2·θv′+γ·xcS2·θv′,where
(22)R=r2+γ·xS2·θ′v+γ·xS2·θ′v=0.5+0.6×0.15×0.5+0.6×0.15×0.5=0.59.The cloud server computes(23)c∂fπθ′/∂θ′=fcA2∣cS2−Σj=01fcAj∣cSjr2+γ·xcS2·θv′+γ·xcS2·θv′−xcS2θv+2β·θ′·fcA2∣cS2·fcA2∣cS2−Σj=01fcAj∣cSj,where
(24)∂fπθ′∂θ′=fA2∣S2−Σj=01fAj∣Sjr2+γ·xS2·θ′v+γ·xS2·θ′v−xS2θv+2β·θ′·fA2∣S2·fA2∣S2−Σj=01fAj∣Sj=0.15×0.15−0.1×0.1+0.1×0.10.5+0.6×0.15×0.5+0.6×0.15×0.5−0.15×0.5+2×0.1×0.5×0.15×0.150.15×0.15−0.1×0.1+0.1×0.1=0.0013.The cloud server computes(25)c∂fvθ′v/∂θ′v=2r2+γ·xcS2·cθ′v+γ·xcS2·θ′v−xcS2θ′v·γ·xcS2+γ·xcS2−xcS2,where
(26)∂fvθ′v∂θ′v=2r2+γ·xS2·θ′v+γ·xS2·θ′v−xS2θ′v·γ·xS2+γ·xS2−xS2=20.5+0.6×0.15×0.5+0.6×0.15×0.5−0.15×0.50.6×0.15+0.6×0.15−0.15=0.0309.Setcdθ=0, cdθv=0. The cloud server computes cdθ=cdθ+c∂fπθ′/∂θ′, where
(27)dθ=dθ+∂fπθ′∂θ′=0.0013.We computecdθv=cdθv+c∂fπθ′v/∂θ′v, where
(28)dθv=dθv+∂fπθ′v∂θ′v=0.0309.With the help of the user, the cloud server gets refreshed ciphertextscdθv and cdθv. The cloud server computes cg=αcg+1−αcdθ2, where
(29)g=αg+1−αdθ2=1−0.5×0.00132=0.000000845.The cloud server computescgv=αcgv+1−αcdθv2, where
(30)gv=αgv+1−αdθv2=1−0.5×0.03092=0.000477405.After the cloud server obtains ciphertextsc′g and c′gv, we set cg=c′g and cgv=c′gv. The cloud server computes cθ=θ−ηc′dθ/cg+ε, where
(31)θ=θ−ηdθg+ε=0.5−0.10.00130.000000845+0.01≈0.4987.The cloud server computescθv=θv−ηc′dθv/cgv+ε, where
(32)θv=θv−ηdθvgv+ε=0.5−0.10.03090.000477405+0.01≈0.4698.The user can obtain refreshedθ≈0.4987 and θv≈0.4698 by using his secret key.


## 6.2. Secure Treatment Decision-Making Algorithm

In this subsection, based on the above privacy-preserving A3C reinforcement learning algorithm, secure treatment decision-making algorithmTDMθ,θv,cx,cS,cA is implemented in Algorithm 8, which is described as follows. In this algorithm, input parameters include θ, θv, and the undiagnosed patient’s encrypted current state cx, cS, and cA. Set the index col=0. The ciphertext ci,j is initiated, where i=0,⋯,χ−1 and j=0,⋯,φ−1. Firstly, cS’s element cSj is compared with cx by using the above homomorphic comparison protocol compcSj,cx. cb is the encrypted comparison result. After the decryption of cb, if the comparison result b=t, it means that Sj=x, set col=j, where j=0,⋯,φ−1. Next, the cloud server computes the value function’s encrypted value cv=VcScol;θv=xcSj·θv, where the plaintext of cv is VScol;θv. Set the ciphertext chor=c0. The cloud server computes the policy function’s encrypted value
(33)cπ,i=πcAi∣cScol=efcAi∣cScolθ∑j=0tmaxefcAj∣cSjθ,where the plaintext of cπ,i is πAi∣Scol. The cloud server computes homomorphic multiplication between cv and cπ,i, namely, ci,col=cv·cπ,i, where ci,col is the ciphertext of VScol;θvπAi∣Scol, i=0,⋯,χ−1. Set index=0. Finally, the cloud server computes the ciphertext chor,col by using the homomorphic maximum protocol argmaxc0,⋯,cχ−1. Set index=hor. Hence, the treatment decision is cdr=cAindex. Then, cdr will be transmitted to the undiagnosed patient. In order to obtain the treatment decision dr, cdr can be decrypted by using his own secret key.Algorithm 8: Secure treatment decision-making algorithm.
TDMθ,θv,cx,cS,cA:Input: θ, θv, cx, cS, cA.Output: cdr.Fori=0 to χ−1.Forj=0 to φ−1.Initiateci,j.End forEnd forInitiatecb.Forj=0 to φ−1.cb=compcSj,cx.End forDecryptcb to obtain b.Ifb=t.Setcol=j.cv=VcScol;θv=xcSjθv.Setchor=c0.Fori=0 to χ−1.cπ,i=πcAi∣cScol=efcAi∣cScolθ/∑j=0tmaxefcAj∣cSjθ.ci,col=cv·cπ,i.End forSetindex=0.chor,col=argmaxc0,col,⋯,cχ−1,col.Setindex=hor.cdr=cAindex.In order to better understand Algorithm8, we give an example, which is described as follows. In this example, we set χ=4, φ=4, θ=0.5, θv=0.5, and tmax=3. We suppose that cS0,⋯,cS3 are ciphertexts of 0.1,0.2,0.3,0.4, respectively. cA0,⋯,cA3 are ciphertexts of 0.1,0.2,0.3,0.4, respectively. For the convenience of computation, we let xcSj=cSj, fcAi∣cSj=cAicSj, πAi∣Sj=efAi∣Sj, i=0,1,2,3, j=0,1,2,3. The calculation of ci,j=VcSj;θvπcAi∣cSj is the key component for the execution of Algorithm 8, where the corresponding plaintext of ci,j is VSj;θvπAi∣Sj. In this example, the distribution of ci,j can be shown in Table 3.Table 3
The distribution of the encrypted probability.
Encrypted probabilityEncrypted actionscA0cA1cA2cA3Encrypted statescS0c0,0c1,0c2,0c3,0cS1c0,1c1,1c2,1c3,1cS2c0,2c1,2c2,2c3,2cS3c0,3c1,3c2,3c3,3If the patient requires a diagnostic service, the encrypted current statecx is input for the implementation of Algorithm 8. Then, cSbe is compared with cx by the execution of the protocol cb=compcSbe,cx, where be=0. If the comparison result b=t, set col=be. If the comparison result b≠t, the next element cSbe+1 will be compared with cx until be+1>3. We suppose that x=S1, namely, col=1. Hence, cv=xcS1θv=xcS10.5, where the plaintext of cv is 0.2×0.5=0.1. We compute ciphertexts cπ,0, cπ,1, cπ,2, and cπ,3, which corresponding plaintexts are e0.01/temp, e0.02/temp, e0.03/temp, and e0.04/temp, respectively, where temp=e0.01+e0.02+e0.03+e0.04. Then, we compute ciphertexts c0,1, c1,1, c2,1, and c3,1, which corresponding plaintexts are 0.1e0.01/temp, 0.1e0.02/temp, 0.1e0.03/temp, and 0.1e0.04/temp, respectively. Based on the execution of the protocol argmaxc0,1,c1,1,c2,1,c3,1, we can obtain the output ciphertext is c3,1. The encrypted treatment decision is cA3, which corresponding plaintext is A3.


## 7. Performance Results

In this section, based on Cheon et al.’s homomorphic encryption scheme, we analyze the efficiency of our secure computation protocols, secure A3C reinforcement learning algorithm, and secure treatment decision-making algorithm. We use the virtual machine to implement experiments without the GPU hardware platform. In our experimental environment, the operating system is macOS 10.14.6. Our personal computer has two Intel (R) Core (TM) i5 CPU processors, which runs at 2.3 GHz with 8.00 GB RAM. The operation system of a virtual machine is ubuntu 16.04. The virtual machine is allocated single Intel (R) Core (TM) i5 CPU processor with 1.0 GB RAM. In order to implement high-level numeric algorithms, we choose the NTL library. We use the GCC platform to compile our C++ codes. We adopt the UC Irvine Machine Learning Repository (http://archive.ics.uci.edu/ml/index.php) for implementing the experiments. For convenience, we set logq ranging from 400 to 700, the scaling factor logp=30.Figure2 shows the efficiency of our homomorphic comparison protocol, where the number of comparison oc ranges from 1 to 4. As shown in Figure 2, the running time of our homomorphic comparison protocol increases significantly with the increasing of oc. Figures 3–5 show the efficiency of our homomorphic maximum protocol, where the number of maximum om ranges from 1 to 4 and the number of plaintexts k ranges from 5 to 7. It can be easily observed that the running time of our homomorphic maximum protocol increases significantly with the increasing of k and om. Figures 6–8 show the efficiency of our homomorphic exponential protocol, where the number of exponential operation oe ranges from 1 to 4 and the integer n ranges from 2 to 4. We can observe that the running time of our homomorphic exponential protocol increases rapidly with the increasing of oe and n.Figure 2
The efficiency of our homomorphic comparison protocol.Figure 3
The efficiency of our homomorphic maximum protocol (k=5).Figure 4
The efficiency of our homomorphic maximum protocol (k=6).Figure 5
The efficiency of our homomorphic maximum protocol (k=7).Figure 6
The efficiency of our homomorphic exponential protocol (n=2).Figure 7
The efficiency of our homomorphic exponential protocol (n=3).Figure 8
The efficiency of our homomorphic exponential protocol (n=4).Then, Figure9 shows the efficiency of our homomorphic division protocol, where the number of division od ranges from 1 to 4. We can observe the changing trend of the running time of our homomorphic division protocol. This protocol has an obvious growth of running time with the increasing of od and logq. Figure 10 shows the efficiency of our homomorphic reciprocal of square root protocol, where os ranges from 1 to 4; os denotes the number of operations of reciprocal of square root. With the increasing of os and logq, more running time is needed for implementing our homomorphic reciprocal of square root protocol. It can be observed that its running time is longer than the above homomorphic comparison, maximum, exponential, and division protocols. Figure 11 shows the efficiency of our secure A3C reinforcement learning algorithm, where ot ranges from 1 to 4; ot denotes the number of operations of A3C training algorithm. With the increasing of ot and logq, our A3C reinforcement learning algorithm requires more running time. This algorithm is responsible for training the parameters θ and θv. Hence, this algorithm is complicated. We can observe too much running time is needed for this algorithm, which can demonstrate the above viewpoint. Figure 12 shows the efficiency of our secure treatment decision-making algorithm, where dm ranges from 1 to 4; dm denotes the number of operations of treatment decision-making algorithm. The running time of this algorithm grows with the increasing of dm. This algorithm uses the optimized θ and θv. Hence, this algorithm is less complicated than the secure A3C algorithm. The running time of this algorithm is shorter than the secure A3C algorithm, which can verify the above viewpoint. In a conclusion, the above efficiency analysis shows the feasibility of our secure computation protocols and algorithms.Figure 9
The efficiency of our homomorphic division protocol.Figure 10
The efficiency of our homomorphic reciprocal of square root protocol.Figure 11
The efficiency of our secure A3C reinforcement learning algorithm.Figure 12
The efficiency of our secure treatment decision-making algorithm.


## 8. Conclusion

Reinforcement learning is helpful for implementing dynamic treatment regimes on health data. However, private health data may be illegally leaked, falsified, or deleted in the execution of the reinforcement learning algorithm. Hence, we study secure dynamic treatment regimes on health data. In this paper, we have designed homomorphic comparison protocol, homomorphic maximum protocol, homomorphic exponential protocol, homomorphic division protocol, and homomorphic reciprocal of square root protocol. Based on these secure computation protocols, we have proposed a privacy-preserving A3C reinforcement learning algorithm for the first time. Then, it is used for implementing the secure treatment decision-making algorithm. Finally, we simulate the proposed secure computation protocols and algorithms. Simulation results show that our secure computation protocols and algorithms are feasible.In the future research, we will use homomorphic encryption to implement other machine learning algorithms, such as distributed learning [37] and federated reinforcement learning [38], which can successfully dominate multiple real devices that have the same type and slightly different dynamics. In addition, we plan to evaluate the performance of the secure A3C algorithm in other real-world scenarios, for example, vehicular ad hoc network.



---

*Source: 8952219-2021-11-23.xml*