## 2 Related Works  
To protect user privacy, there are a variety of tools available, such as homomorphic encryption, secret sharing, oblivious transfer, and garbled circuits. In general, homomorphic encryption requires more computing power, while secret sharing incurs a lot of communication overhead. Oblivious transfer and garbled circuit rely on specific hardware implementations, so they are not universal. In this privacy-preserving outsourced policy iteration method, both statistics and policy updates should be run on the server side. The specific values of states and behaviors should be encrypted by the user, which raises the need to perform calculations in a secret state. In order to minimize unnecessary communication and user-side computing overhead, we consider using homomorphic encryption to protect user privacy. Homomorphic encryption is the algorithms supporting calculations on encrypted data. While the operations supported by homomorphic encryption can usually not include all operations available on plaintext, homomorphic encryption is divided into three categories: Fully Homomorphic Encryption (FHE), Somewhat Homomorphic Encryption (SWHE) and Partially Homomorphic Encryption (PHE). The first proposed FHE scheme introduced by [PERSON] et al. in [12], with which any polynomial functions could be calculated under encrypted state but the calculation cost it needed is too much to become usable in actual scene. Therefore, BGN algorithm [13] is more used to keep its efficiency with the supported operation is reduced into a limited times of addition and multiplication and become SWHE. Moreover, PHE is another way to keep efficiency which only supports a single type of operations. This supported operation is usually addition or multiplication. If a PHE support addition, we have homomorphic add operation \(\hat{+}\) on ciphertext \(E(m_{1})\) and ciphertext \(E(m_{2})\) so that \(E(m_{1}+m_{2})=E(m_{1})\hat{+}E(m_{2})\). Similarly, we have homomorphic multiple operation \(\hat{\times}\) on ciphertext \(E(m_{1})\) and ciphertext \(E(m_{2})\) so that \(E(m_{1}\times m_{2})=E(m_{1})\hat{\times}E(m_{2})\), if multiplication is supported. The detail operation in \(\hat{+}\) or \(\hat{\times}\) is defined individually in each PHE algorithms. The most used is Paillier brought out in [14]. The basic way to calculate encrypted data with different key is to make it able to be compute by keep their homomorphism under different keys like what the [15; 16] have done. This way is called multi-key homomorphic encryption (MKHE), but will not be adopted by us for their heavy computing cost. Therefore a way to switch the key of a ciphertext is needed, as is introduced by [PERSON] et al. in [17]. They uses a key-switching matrices to change keys into a unified one, which is efficient but, in essence, this method reduces the security of multiply keys to a single key. That is to say the matrix weakened the security of the original scheme, which is not good. To keep the security the same while considering the efficiency, many methods like [18; 19] introduce two servers. In [18], the ciphertexts are decrypted and than encrypted again by the Second server, which is a brief way but the Second server keeps all keys and its royalty and invincibility must be ensured. In [19; 20], they divide the proxy re-encrypt and calculate into different server, but they still need interactions between them to accomplish the conversion and calculation process, which is complicated and significantly increased the consumption on communication. Since the communication cost is huge, refs. [21; 22] tried remove the process of ciphertext conversion, but they can not reduce that in process of calculation. To canceling the Second server, the scheme proposed by [PERSON] et al. in [23] perform the re-encryption with server and users, but the communicate cost is still high. In summary, they can not solve the problems they aim to by avoid the problem itself. Concentrate to the accrual scene similar to us, ref.