# Research on Biological Population Evolutionary Algorithm and Individual Adaptive Method Based on Quantum Computing

Authors: Xuandiyang Lu
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

On the basis of classical computer, quantum computer has been developed. In dealing with some large-scale parallel problems, quantum computer is simpler and faster than traditional computer. Nowadays, physical qubit computers have many limitations. Classical computers have many ways to simulate quantum computing, the most effective of which are quantum superiority and quantum algorithm. Ensuring computational efficiency, accuracy, and precision is of great significance to the study of large-scale quantum computing. Compared with other algorithms, genetic algorithm has more advantages, so it can be more widely used. For example, strong adaptability and global optimization ability are the advantages of genetic algorithm. Through the research in Chapter 4, we can conclude that the variance of A2C is obviously smaller than that of PPO. Furthermore, it can be concluded that A2C has better robustness.


---

## Body

## 1. Introduction

The purpose of this paper is to summarize all the important aspects and results affecting the field of quantum computation and quantum information. First, we are familiar with the characteristics and principles of quantum mechanics and provide more effective and secure information processing. Then, their applications in general information theory, cryptography, algorithms, computational complexity, and error correction are discussed [1]. If one day the bits of a computer are reduced to the size of a single atom, the quantum mechanical effect may profoundly change the nature of computing itself [2]. This special issue on mathematical structure in computer science contains several contributions related to modern quantum information and quantum computing [3]. Computer is usually regarded as a general-purpose computing device. In other words, it is believed that it can simulate any physical computing device at a computational time cost of up to one polynomial factor [4]. Quantum information is stored in a state containing multiple quasiparticles, which have topological degeneracy. The unitary gate operation required by quantum computation is realized by weaving quasiparticles and then measuring the states of multiple quasiparticles. The fault tolerance of topological quantum computer comes from the nonlocal coding of quasiparticle state, which makes quasiparticle immune to errors caused by local disturbance [5]. Decoherence in quantum computer is expressed by semigroup method. It is proved that these subspaces are stable to perturbation, and it is possible to carry out universal quantum computation in these subspaces [6]. It has been proved that the combination of genetic algorithm and K-nearest neighbor classifier can extract protein-water binding information from X-ray crystal protein structure data well [7]. Combining the advantages of traditional real genetic algorithm and structured genetic algorithm, this paper proposes a real structured genetic algorithm to deal with biological logic circuit design problems [8]. The interference cancellation system is designed by using LM algorithm of artificial neural network with adaptive learning ability, and the number of hidden layer neurons, learning rate, and momentum factor of neural network is optimized by using swarm intelligence algorithm [9]. We use the main biological learning methods instead of machine learning to adjust synaptic weights. We use evolutionary algorithm to optimize the learning meta-parameters. These algorithms are implemented in parallel and use the island model approach to achieve sufficient speed [10]. In this paper, the biological evolutionary algorithm is combined with other artificial intelligence algorithms, and the adaptive and self-learning ability of improved support vector machine optimized by genetic computing is used to predict the early failure of gears [11]. The algorithm proposed in this paper is based on the concept of variable step-size control. We prove that this weakness of variable step size control can be avoided by making relatively small changes to the algorithm, and the derandomization scheme of variable step size control is helpful to reliably adapt to a single step size [12]. We attempted to analyse the interaction networks leading to individual differences in perceived odor quality by assessing the effects of self-adaptation and cross-adaptation on odor intensity and quality reporting in 18 human subjects [13]. In this paper, we will study a generalized evolutionary minority game model, in which agents are divided into several groups, and the performance of each agent is averaged in each group. We find that there are three different effects in the generalized model: (1) group average effect, (2) left-right asymmetry effect, and (3) self-interaction effect [14]. The statistical results show that the positive coping style of night shift nurses significantly positively predicts the positive self-suggestion of sleep and negatively predicts the importance of sleep. Negative coping style can negatively predict the positive self-suggestion of sleep, while coping style can well predict the cognitive aspect of individual sleep adaptation [15].


## 2. Genetic Algorithm

### 2.1. Standard Genetic Algorithm

The standard genetic algorithm has two important characteristics for simulating the whole learning process of the population, which are the population search strategy and the information exchange among individuals in the population. First, create a specific function to test the adaptability of individuals. The algorithm steps are as follows:(a)
Randomly selectN starting points to form a group. The total number N is the size of the whole population. In order to express it conveniently, it is necessary to use a specific coding form to represent each individual in the population. An encoded content may represent a specific feature of a chromosome, and the content it represents varies according to the type of problem to be solved(b)
Choose suitable individuals according to certain selection requirements. We need to select theM individuals with the best adaptability within the standard to form a new population to breed the next generation(c)
Among the selectedM individuals, pairwise grouping is carried out, and then each group is hybridized and recombined. Hybridization is an important way for two chromosomes to exchange information(d)
If the situation requires, we can select some suitable individuals fromM individuals by the given mutation probability, and then carry out mutation operation on the suitable individuals according to certain requirements. Mutation operation plays a very important role in the ability of genetic algorithm to find the optimal solution(e)
It is judged whether the bundle condition of the junction is met, if not, it returns to processB, continues the process, and if the condition is met, it ends the process. Its basic workflow is shown in Figure 1Figure 1
Basic workflow chart.


### 2.2. Adaptive Genetic Algorithm

Its most important idea is to adjust the size ofPc,Pm at any time according to the actual situation of the population in the evolution process. That is, when a population converges, reduce Pc and increase Pm, that is, increase the probability of mutation and reduce the probability of hybridization, thus ensuring the diversity of the population and avoiding “premature maturity;” when the population deviates, Pc is increased and Pm is decreased, that is, the probability of hybridization is increased and the probability of mutation is reduced, so that the population tends to converge and the concentration rate of the algorithm is increased.The index that defines the degree of “premature maturity” of population isΔ. Ftmax Is the fitness of the optimal individual, and the difference Δ between Ftmax and Ftmax is defined as follows
(1)Δ=Ftmax−Ftmax.The “premature maturity” degree of the population can be obtained by the size of formula (1). According to the index Δ, a new strategy is proposed, so that Δ can influence the values of hybridization probability Pc and mutation probability Pm. Its mathematical description is shown in the following formulas
(2)Pc=11+exp−k1⋅Δ,(3)Pm=1.0−11+exp−k2⋅Δ,where k1,k2>0. Since Δ is always greater than or equal to 0, the value range of Pc is between [0.5, 1] and the value range of Pm is between [0, 0.5]. It can be seen from the above formula that Δ can affect the value of Pc,Pm in the evolution process: when Δ increases, Pc increases, Pm decreases, and the ability to develop excellent individuals increases.


## 2.1. Standard Genetic Algorithm

The standard genetic algorithm has two important characteristics for simulating the whole learning process of the population, which are the population search strategy and the information exchange among individuals in the population. First, create a specific function to test the adaptability of individuals. The algorithm steps are as follows:(a)
Randomly selectN starting points to form a group. The total number N is the size of the whole population. In order to express it conveniently, it is necessary to use a specific coding form to represent each individual in the population. An encoded content may represent a specific feature of a chromosome, and the content it represents varies according to the type of problem to be solved(b)
Choose suitable individuals according to certain selection requirements. We need to select theM individuals with the best adaptability within the standard to form a new population to breed the next generation(c)
Among the selectedM individuals, pairwise grouping is carried out, and then each group is hybridized and recombined. Hybridization is an important way for two chromosomes to exchange information(d)
If the situation requires, we can select some suitable individuals fromM individuals by the given mutation probability, and then carry out mutation operation on the suitable individuals according to certain requirements. Mutation operation plays a very important role in the ability of genetic algorithm to find the optimal solution(e)
It is judged whether the bundle condition of the junction is met, if not, it returns to processB, continues the process, and if the condition is met, it ends the process. Its basic workflow is shown in Figure 1Figure 1
Basic workflow chart.


## 2.2. Adaptive Genetic Algorithm

Its most important idea is to adjust the size ofPc,Pm at any time according to the actual situation of the population in the evolution process. That is, when a population converges, reduce Pc and increase Pm, that is, increase the probability of mutation and reduce the probability of hybridization, thus ensuring the diversity of the population and avoiding “premature maturity;” when the population deviates, Pc is increased and Pm is decreased, that is, the probability of hybridization is increased and the probability of mutation is reduced, so that the population tends to converge and the concentration rate of the algorithm is increased.The index that defines the degree of “premature maturity” of population isΔ. Ftmax Is the fitness of the optimal individual, and the difference Δ between Ftmax and Ftmax is defined as follows
(1)Δ=Ftmax−Ftmax.The “premature maturity” degree of the population can be obtained by the size of formula (1). According to the index Δ, a new strategy is proposed, so that Δ can influence the values of hybridization probability Pc and mutation probability Pm. Its mathematical description is shown in the following formulas
(2)Pc=11+exp−k1⋅Δ,(3)Pm=1.0−11+exp−k2⋅Δ,where k1,k2>0. Since Δ is always greater than or equal to 0, the value range of Pc is between [0.5, 1] and the value range of Pm is between [0, 0.5]. It can be seen from the above formula that Δ can affect the value of Pc,Pm in the evolution process: when Δ increases, Pc increases, Pm decreases, and the ability to develop excellent individuals increases.


## 3. Quantum Machine Algorithm

### 3.1. Quantum Nearest Neighbor Algorithm

QNN algorithm firstly loads the training sample setvjj=1,2,⋯,M and the sample to be classified v0≔u onto the quantum state by probability amplitude coding, as shown in the following formulas.
(4)v=d−1/2∑i:vi≠0i1−rji2rmax2e−iϕi0+vjirmax11,(5)u=d−1/2∑i:v0i≠0i11−r0i2rmax2e−iϕi0+v0irmax1,where vji=rjieiφji; rmax is the upper bound of eigenvalue; rji is a number greater than 0. You can get the square of the inner product of v and u as shown in the following formula
(6)u∣v2=2P0−1d2rmax4.Although Euclidean distance is widely used to calculate the Euclidean distance of quantum states, the experimental results show that the method using Euclidean distance is more complicated than the inner product method.


### 3.2. Quantum Support Vector Machine Algorithm

The combination of quantum computing and support vector machine is to use the second acceleration effect of Grover search algorithm to deal with the optimization calculation of SVM model at the earliest. QSVM algorithm first encodesj eigenvectors of each data sample point xi=xi1,xi2,⋯,xij,j=1,2,⋯m into quantum states by probability amplitude coding, as shown in the following formula.
(7)xi=xi−1⋅∑j=1mxijj,where xi−1 is the normalized vector and m is the feature dimension. The quantum state preparation of the training set is shown in the following formula.
(8)χ=Nχ−1⋅∑i=1nxiixi,where Nχ=∑i=1nxi2 and xi are the i training samples. The inner product operation Kij=xi⋅xj of the training data set can be obtained by solving the offset trace of the density matrix χχ, as shown in the following formula.
(9)tr2χχ=1Nχ∑i,j=1Mxixjxi∣xjij=KtrK,where xi⋅xj=xixjxi∣xj. The above formula can be transformed into solving a linear equation, as shown in the following formula.
(10)01T1K+γ−1Ibα=0y,where Kij=xi⋅xj is the kernel matrix, b,αT can be solved according to HHL algorithm to obtain the following formula.
(11)b,α=1Nb,αb0+∑k=1Mαii,where Nb,α=b2+∑k=1Mαk2. Finally, the quantum state u~ and the quantity x~ to be classified are constructed, and the class of x~ can be obtained by calculating the similarity between the two quantum states by using the control exchange gate, as shown in the following formulas
(12)u~=1Nub00+∑k=1Mαkxkkxk,(13)x~=1Nx~00+∑k=1Mxkx,where Nu~=b2+∑k=1Mak2xk2, Nx~=Mx2+1.


### 3.3. Quantum Linear Regression

The purpose of quantum linear regression algorithm is to use quantum algorithm to solve the optimal fitting parameterw, as shown in the following formula
(14)w=X+y=X+X−1X+y,where y=y1,y2,⋯,ynT; X=x1,x2,⋯,xnT is the data matrix. In order to calculate w more conveniently, formula (14) is decomposed into two formulas, and the two formulas are solved, respectively, namely, y′=X+y and w=X+X−1y′. For y′=W+y, the matrix y is first loaded on the quantum state by probability amplitude coding, as shown in the following formula
(15)y=∑P=M+1M+Nypyp.Since matrixX is not a Hermite matrix, we will transform it into a Hermite matrix, as shown in the following formula
(16)IX=0XX+0.At this time,y′=X+y can be transformed into y′=IXy to solve. For w=X+X−1y′, the matrix X+X−1 needs to be transformed into the Hermite matrix IX+X−1, as shown in the following equation
(17)IX+X−1=X+X00XX+−1=X+X−100XX+−1.At this time, forw=X+X−1y′, it can be transformed into solving w=IX+X−1y′. y′=IXy and w=IX+X−1y′ solve the transformed formula by quantum algorithm and then get the quantum state w of fitting parameters.


### 3.4. QuantumK-Means Algorithm

In 2013, the quantum version of the nearest center algorithm was proposed, which can be regarded as a subprocess of the quantumK-means algorithm and can accelerate the classical algorithm exponentially, as shown in the following equations.
(18)ψ=120u+1M∑j=1Mjvj,(19)φ=1Zu0−1M∑jvjj,where Z=u2+1/M∑jvj2. Quantum K-means algorithm includes minimum quantum algorithm and phase estimation algorithm. Its application can greatly reduce the overall computational complexity.


### 3.5. Quantum Principal Component Analysis

On the feature space,ρ can be expressed as the following formula.
(20)ρ=∑jλjujuj,where λj is the eigenvalue; uj is its corresponding eigenvector. The most critical part of QPCA is the construction of controlled e−iρt. The method of constructing e−iρt can be expressed by the following equation:
(21)trpe−iSΔtρ⊗δeiSΔt=cos2Δtδ+sin2Δtρ−isinΔtρ,δ=δ−iΔtρ,δ+OΔt2,where trp is the bias trace for the first variable; matrix S is a commutative operator, δ=uu is a feature vector. Thus, e−iρt can be realized by partial tracing operation with the aid of N replica density operator ρ and a sparse commutation matrix S.Controllede−iρt can be realized by applying unitary operator ∑nnΔtnΔt⊗∏j=1ne−iSjΔt to nΔtnΔt⊗δ⊗ρ⊗⋯⊗ρ, and then performing partial trace operation on each subsystem, as shown in the following equation:
(22)trp∑nnΔtnΔt⊗∏j=1ne−iSjΔtnΔtnΔt⊗δ⊗ρ⊗⋯⊗ρ.


## 3.1. Quantum Nearest Neighbor Algorithm

QNN algorithm firstly loads the training sample setvjj=1,2,⋯,M and the sample to be classified v0≔u onto the quantum state by probability amplitude coding, as shown in the following formulas.
(4)v=d−1/2∑i:vi≠0i1−rji2rmax2e−iϕi0+vjirmax11,(5)u=d−1/2∑i:v0i≠0i11−r0i2rmax2e−iϕi0+v0irmax1,where vji=rjieiφji; rmax is the upper bound of eigenvalue; rji is a number greater than 0. You can get the square of the inner product of v and u as shown in the following formula
(6)u∣v2=2P0−1d2rmax4.Although Euclidean distance is widely used to calculate the Euclidean distance of quantum states, the experimental results show that the method using Euclidean distance is more complicated than the inner product method.


## 3.2. Quantum Support Vector Machine Algorithm

The combination of quantum computing and support vector machine is to use the second acceleration effect of Grover search algorithm to deal with the optimization calculation of SVM model at the earliest. QSVM algorithm first encodesj eigenvectors of each data sample point xi=xi1,xi2,⋯,xij,j=1,2,⋯m into quantum states by probability amplitude coding, as shown in the following formula.
(7)xi=xi−1⋅∑j=1mxijj,where xi−1 is the normalized vector and m is the feature dimension. The quantum state preparation of the training set is shown in the following formula.
(8)χ=Nχ−1⋅∑i=1nxiixi,where Nχ=∑i=1nxi2 and xi are the i training samples. The inner product operation Kij=xi⋅xj of the training data set can be obtained by solving the offset trace of the density matrix χχ, as shown in the following formula.
(9)tr2χχ=1Nχ∑i,j=1Mxixjxi∣xjij=KtrK,where xi⋅xj=xixjxi∣xj. The above formula can be transformed into solving a linear equation, as shown in the following formula.
(10)01T1K+γ−1Ibα=0y,where Kij=xi⋅xj is the kernel matrix, b,αT can be solved according to HHL algorithm to obtain the following formula.
(11)b,α=1Nb,αb0+∑k=1Mαii,where Nb,α=b2+∑k=1Mαk2. Finally, the quantum state u~ and the quantity x~ to be classified are constructed, and the class of x~ can be obtained by calculating the similarity between the two quantum states by using the control exchange gate, as shown in the following formulas
(12)u~=1Nub00+∑k=1Mαkxkkxk,(13)x~=1Nx~00+∑k=1Mxkx,where Nu~=b2+∑k=1Mak2xk2, Nx~=Mx2+1.


## 3.3. Quantum Linear Regression

The purpose of quantum linear regression algorithm is to use quantum algorithm to solve the optimal fitting parameterw, as shown in the following formula
(14)w=X+y=X+X−1X+y,where y=y1,y2,⋯,ynT; X=x1,x2,⋯,xnT is the data matrix. In order to calculate w more conveniently, formula (14) is decomposed into two formulas, and the two formulas are solved, respectively, namely, y′=X+y and w=X+X−1y′. For y′=W+y, the matrix y is first loaded on the quantum state by probability amplitude coding, as shown in the following formula
(15)y=∑P=M+1M+Nypyp.Since matrixX is not a Hermite matrix, we will transform it into a Hermite matrix, as shown in the following formula
(16)IX=0XX+0.At this time,y′=X+y can be transformed into y′=IXy to solve. For w=X+X−1y′, the matrix X+X−1 needs to be transformed into the Hermite matrix IX+X−1, as shown in the following equation
(17)IX+X−1=X+X00XX+−1=X+X−100XX+−1.At this time, forw=X+X−1y′, it can be transformed into solving w=IX+X−1y′. y′=IXy and w=IX+X−1y′ solve the transformed formula by quantum algorithm and then get the quantum state w of fitting parameters.


## 3.4. QuantumK-Means Algorithm

In 2013, the quantum version of the nearest center algorithm was proposed, which can be regarded as a subprocess of the quantumK-means algorithm and can accelerate the classical algorithm exponentially, as shown in the following equations.
(18)ψ=120u+1M∑j=1Mjvj,(19)φ=1Zu0−1M∑jvjj,where Z=u2+1/M∑jvj2. Quantum K-means algorithm includes minimum quantum algorithm and phase estimation algorithm. Its application can greatly reduce the overall computational complexity.


## 3.5. Quantum Principal Component Analysis

On the feature space,ρ can be expressed as the following formula.
(20)ρ=∑jλjujuj,where λj is the eigenvalue; uj is its corresponding eigenvector. The most critical part of QPCA is the construction of controlled e−iρt. The method of constructing e−iρt can be expressed by the following equation:
(21)trpe−iSΔtρ⊗δeiSΔt=cos2Δtδ+sin2Δtρ−isinΔtρ,δ=δ−iΔtρ,δ+OΔt2,where trp is the bias trace for the first variable; matrix S is a commutative operator, δ=uu is a feature vector. Thus, e−iρt can be realized by partial tracing operation with the aid of N replica density operator ρ and a sparse commutation matrix S.Controllede−iρt can be realized by applying unitary operator ∑nnΔtnΔt⊗∏j=1ne−iSjΔt to nΔtnΔt⊗δ⊗ρ⊗⋯⊗ρ, and then performing partial trace operation on each subsystem, as shown in the following equation:
(22)trp∑nnΔtnΔt⊗∏j=1ne−iSjΔtnΔtnΔt⊗δ⊗ρ⊗⋯⊗ρ.


## 4. Experimental Research on Deep Reinforcement Learning

Deep reinforcement learning has the characteristics of various methods, extreme sensitivity to super parameters and great randomness. Even the same algorithm will bring about performance differences due to different implementation methods or different test environment platforms. The best way to integrate these problems is to hope that the domain can unify the model and test environment, so that the domain can replicate and improve and try new ideas. Therefore, many research institutions have begun to develop their own algorithm models and test environments.According to the algorithm model, DeepMind, OpenAI, and Baidu have all opened up some self-implemented deep reinforcement learning algorithms and applications, among which OpenAI is the most abundant, including the high-quality implementation of nine algorithms: TRPO, PPO, HER, GAIL, DQN, DDPG, AVKTR, ACER, and A2C, which is also the preferred benchmark code for researchers in the field at present. For the test environment, there are mainly OpenAIGym, MuJoCo, rllab, DeepMindLab, TORCS, PySC2, and so on. Among them, MuJoCo is mainly a robot control problem, Gym, rllab, and DeepMindLab are similar, mainly focusing on video games, TORCS is a racing car control problem, and PySC2 is the environment of StarCraft II. At present, Atari video game environment is widely used in the field of machine game, and Gym interface is usually used.Deep learning of machine games is divided into many parts, the most important of which is to interact with some simulators. If the algorithm supports multiprocess mode, the training period for obtaining state information will be shortened. Therefore, A2C, DDPG, and PPO, which support multiprocess mode, are currently the most used algorithms in the field of machine game. In this section, the A2C, DDPG, and PPO algorithm for some related experimental research and analysis, to provide reference for scholars in the field.


### 4.1. Test Environment

Related experiments will be conducted in Atari 2600 video game environment. There are many test environments for deep reinforcement learning, but Atar video game environment is generally used most. Gym environment is usually needed for interaction in reinforcement learning. The specific step is to enter the action produced by the agent into the function step, and then you will get four return values: done, reward, observation, and info, as shown in Table1.Table 1
Gym excuses return data list.
Return valueDescriptionObservationReturns the pixel picture of the video gameRewardThe scale of the reward for the previous action varies according to the environmentDoneUsed to determine whether a reset environment is required for the surface. When the value is true, the surface game endsInfoContains diagnostic information for debuggingIt is used by agent in a very simple way. In order to ensure the reproducibility of the experimental results, all the video game experiments involved in this paper are completed on Gym platform. In this paper, six popular classic video games: BeamRider, Breakout, MsPacman, Qbert, Seaques, and Spacelnvaders are selected as specific test environments, and the task description of each game is shown in Table2.Table 2
Video game description.
GamesDescribeBeamRiderInitialize 3 lives, need to avoid bombs, and destroy enemy planes. There are 15 enemy planes in each level, and the reward for destroying enemy planes is40+4∗i (level I). After level 4, there are special rewards with high scores.BreakoutInitialization has 5 lives. Move the flat plate to make the ball rebound and break the color block to score. The ball falls into the bottom line and loses a life. The more the color block is hit, the faster the rebound speed is.MsPacmanInitialize three lives, avoid the enemy, hit the enemy, and lose one. Swallow the pink square on the blue road to get a reward. The enemy will become weak periodically. At this time, you can swallow the enemy to get a higher reward.QbertInitialize 3 lives, jump on the pyramid to light up each square to get a reward, all light up to enter the next level to give a large reward, jump outside the pyramid, or hit the enemy will lose a life.SeaquestInitialize 3 lives, destroy sharks and enemy submarines to get rewards, avoid sharks and enemy submarines and launch torpedoes, hit and lose one life, submarine runs out of oxygen and loses one life, and returns to the sea without saving people to lose one lifeSpaceInvadersInitialize 3 lives, destroy the enemy to get rewards, and lose one life when hit by the enemy. The enemy will invade at a faster and faster speed. If the enemy reaches the bottom line, no matter how many lives are left, the game will end.


### 4.2. Experimental Setup and Configuration

In order to ensure the reproducibility of the results, this paper uses A2C and PPO provided by baselines of OpenAI as benchmark experimental codes, Gym version is 0.10.5, baselines version is 0.1.5. All the experiments were completed on a PC with the following configuration: CPU: Inter (R) Core (TM) i9-7900X @ 3.3 GHz, GPU: NVIDIAGTX1080Ti, and memory: 32 G. All the experiments were repeated five times under its related settings.


### 4.3. Performance Analysis of Reinforcement Learning for Machine Game

As OpenAI updated the V4 version of Gym and announced that the∗NoFrameSkip-v4 environment is considered the standard Atari environment, it was reminded that the V4 version will not provide the same experimental results as the previous V3 version. The above problems lead to the fact that when researchers need to do comparative experiments, they often need to reexperiment in the new version environment, which is very time-consuming. At the same time, our research found that there is no brand-new baseline result for industry reference. In addition, different researchers often need the performance results of benchmark algorithms under different computational budgets, but there is no complete and detailed experimental results under different computational budgets at present. In this paper, six mainstream classic video games are used as the test environment, and the latest Gym version is used to do standard experimental tests on A2C, PPO, and DDPG under 10 M, 20 M, 30 M, and 40 M calculation budgets, respectively, in which the default number of processes is 16. The final performance data of its model are shown in Tables 3–5.Table 3
A2C game scores under 10 M, 20 M, 30 M, and 40 M calculation budget.
Game (A2C)10 M20 M30 M40 MBeamRider1894497260397162Breakout189416471517MsPacman1564215621522283Qbert346897631451415576Seaquest1249175417521740SpaceInvaders6697428991746Table 4
PPO game scores under 10 M, 20 M, 30 M, and 40 M calculation budget.
Game (PPO)10 M20 M30 M40 MBeamRider2711430254086495Breakout227290376413MsPacman2022235730513762Qbert10737150611821121695Seaquest1072164218501881SpaceInvaders782153620212137Table 5
DDPG game scores under 10 M, 20 M, 30 M, and 40 M calculation budget.
Game (DDPG)10 M20 M30 M40 MBeamRider2304426452016207Breakout179259329436MsPacman1978220529883652Qbert11454149871789220194Seaquest1104156917601798SpaceInvaders773143520182065Sampling data tables of A2C, PPO, and GGPG algorithms with different thread numbers are shown in Table6.Table 6
Sampling data under different thread numbers.
Thread quantity124816A2C139222404679976PPO153198353580840DDPG147216388634893In addition, we also give the performance comparison diagrams of A2C, PPO, and DDPG under different environments and different computing budgets, as shown in Figures2–5.Figure 2
Performance comparison in 10 M environment.Figure 3
Performance comparison chart in 20 M environment.Figure 4
Performance comparison in 30 M environment.Figure 5
Performance comparison in 40 M environment.Figure 6
Schematic diagram of sampling efficiency of A2C, PPO, and DDPG algorithms under different thread numbers.By observing graph 2, graph 3, graph 4, and graph 5, we can conclude that PPO is the largest in any environment, DDPG is the middle, and A2C is the smallest, which is particularly easy to observe in Qbert.We use the sampled data of A2C and PPO algorithms with different thread numbers to draw an efficiency diagram as shown in Figure6.By observing the data table and the schematic diagram of the performance curve obtained from the experiment, some new discoveries are also obtained in this paper. For example, in some test tasks, the performance of A2C will be lower than PPO and DDPG in the early stage, but with the increase of computing budget, the performance will exceed PPO and DDPG. However, in PPO’s paper, only the experiment was carried out at the calculation cost of 10 M, and it was concluded that its effect was better than A2C, which is not rigorous and correct at present. Usually, due to the long training time of video games and the limitation of computing resources, researchers often only conduct experiments under a small computing budget, so it is difficult to find this phenomenon, which also provides a new focus for the comparison of algorithms in the future. In addition, it can be found that the variance of A2C is significantly smaller than that of PPO algorithm in almost all test environments, especially in Seaquest and Spaceliwaders games. Therefore, it can be considered that A2C has better robustness.In this chapter, we first briefly introduce the background of reinforcement learning and deep learning, which leads to the deep reinforcement learning method. On this basis, the deep reinforcement learning methods for machine game are systematically investigated, including reinforcement learning methods based on value function and reinforcement learning methods based on strategy gradient. In addition, hierarchical reinforcement learning and multiagent reinforcement learning are introduced. In the experimental research part, we briefly introduce the current mainstream code base and standard test platform for deep reinforcement learning. Finally, the mainstream reinforcement learning algorithms are studied experimentally, the performance of A2C, DDPG, and PPO under different computational costs and the specific training curves is given, and the experimental results are analyzed.


## 4.1. Test Environment

Related experiments will be conducted in Atari 2600 video game environment. There are many test environments for deep reinforcement learning, but Atar video game environment is generally used most. Gym environment is usually needed for interaction in reinforcement learning. The specific step is to enter the action produced by the agent into the function step, and then you will get four return values: done, reward, observation, and info, as shown in Table1.Table 1
Gym excuses return data list.
Return valueDescriptionObservationReturns the pixel picture of the video gameRewardThe scale of the reward for the previous action varies according to the environmentDoneUsed to determine whether a reset environment is required for the surface. When the value is true, the surface game endsInfoContains diagnostic information for debuggingIt is used by agent in a very simple way. In order to ensure the reproducibility of the experimental results, all the video game experiments involved in this paper are completed on Gym platform. In this paper, six popular classic video games: BeamRider, Breakout, MsPacman, Qbert, Seaques, and Spacelnvaders are selected as specific test environments, and the task description of each game is shown in Table2.Table 2
Video game description.
GamesDescribeBeamRiderInitialize 3 lives, need to avoid bombs, and destroy enemy planes. There are 15 enemy planes in each level, and the reward for destroying enemy planes is40+4∗i (level I). After level 4, there are special rewards with high scores.BreakoutInitialization has 5 lives. Move the flat plate to make the ball rebound and break the color block to score. The ball falls into the bottom line and loses a life. The more the color block is hit, the faster the rebound speed is.MsPacmanInitialize three lives, avoid the enemy, hit the enemy, and lose one. Swallow the pink square on the blue road to get a reward. The enemy will become weak periodically. At this time, you can swallow the enemy to get a higher reward.QbertInitialize 3 lives, jump on the pyramid to light up each square to get a reward, all light up to enter the next level to give a large reward, jump outside the pyramid, or hit the enemy will lose a life.SeaquestInitialize 3 lives, destroy sharks and enemy submarines to get rewards, avoid sharks and enemy submarines and launch torpedoes, hit and lose one life, submarine runs out of oxygen and loses one life, and returns to the sea without saving people to lose one lifeSpaceInvadersInitialize 3 lives, destroy the enemy to get rewards, and lose one life when hit by the enemy. The enemy will invade at a faster and faster speed. If the enemy reaches the bottom line, no matter how many lives are left, the game will end.


## 4.2. Experimental Setup and Configuration

In order to ensure the reproducibility of the results, this paper uses A2C and PPO provided by baselines of OpenAI as benchmark experimental codes, Gym version is 0.10.5, baselines version is 0.1.5. All the experiments were completed on a PC with the following configuration: CPU: Inter (R) Core (TM) i9-7900X @ 3.3 GHz, GPU: NVIDIAGTX1080Ti, and memory: 32 G. All the experiments were repeated five times under its related settings.


## 4.3. Performance Analysis of Reinforcement Learning for Machine Game

As OpenAI updated the V4 version of Gym and announced that the∗NoFrameSkip-v4 environment is considered the standard Atari environment, it was reminded that the V4 version will not provide the same experimental results as the previous V3 version. The above problems lead to the fact that when researchers need to do comparative experiments, they often need to reexperiment in the new version environment, which is very time-consuming. At the same time, our research found that there is no brand-new baseline result for industry reference. In addition, different researchers often need the performance results of benchmark algorithms under different computational budgets, but there is no complete and detailed experimental results under different computational budgets at present. In this paper, six mainstream classic video games are used as the test environment, and the latest Gym version is used to do standard experimental tests on A2C, PPO, and DDPG under 10 M, 20 M, 30 M, and 40 M calculation budgets, respectively, in which the default number of processes is 16. The final performance data of its model are shown in Tables 3–5.Table 3
A2C game scores under 10 M, 20 M, 30 M, and 40 M calculation budget.
Game (A2C)10 M20 M30 M40 MBeamRider1894497260397162Breakout189416471517MsPacman1564215621522283Qbert346897631451415576Seaquest1249175417521740SpaceInvaders6697428991746Table 4
PPO game scores under 10 M, 20 M, 30 M, and 40 M calculation budget.
Game (PPO)10 M20 M30 M40 MBeamRider2711430254086495Breakout227290376413MsPacman2022235730513762Qbert10737150611821121695Seaquest1072164218501881SpaceInvaders782153620212137Table 5
DDPG game scores under 10 M, 20 M, 30 M, and 40 M calculation budget.
Game (DDPG)10 M20 M30 M40 MBeamRider2304426452016207Breakout179259329436MsPacman1978220529883652Qbert11454149871789220194Seaquest1104156917601798SpaceInvaders773143520182065Sampling data tables of A2C, PPO, and GGPG algorithms with different thread numbers are shown in Table6.Table 6
Sampling data under different thread numbers.
Thread quantity124816A2C139222404679976PPO153198353580840DDPG147216388634893In addition, we also give the performance comparison diagrams of A2C, PPO, and DDPG under different environments and different computing budgets, as shown in Figures2–5.Figure 2
Performance comparison in 10 M environment.Figure 3
Performance comparison chart in 20 M environment.Figure 4
Performance comparison in 30 M environment.Figure 5
Performance comparison in 40 M environment.Figure 6
Schematic diagram of sampling efficiency of A2C, PPO, and DDPG algorithms under different thread numbers.By observing graph 2, graph 3, graph 4, and graph 5, we can conclude that PPO is the largest in any environment, DDPG is the middle, and A2C is the smallest, which is particularly easy to observe in Qbert.We use the sampled data of A2C and PPO algorithms with different thread numbers to draw an efficiency diagram as shown in Figure6.By observing the data table and the schematic diagram of the performance curve obtained from the experiment, some new discoveries are also obtained in this paper. For example, in some test tasks, the performance of A2C will be lower than PPO and DDPG in the early stage, but with the increase of computing budget, the performance will exceed PPO and DDPG. However, in PPO’s paper, only the experiment was carried out at the calculation cost of 10 M, and it was concluded that its effect was better than A2C, which is not rigorous and correct at present. Usually, due to the long training time of video games and the limitation of computing resources, researchers often only conduct experiments under a small computing budget, so it is difficult to find this phenomenon, which also provides a new focus for the comparison of algorithms in the future. In addition, it can be found that the variance of A2C is significantly smaller than that of PPO algorithm in almost all test environments, especially in Seaquest and Spaceliwaders games. Therefore, it can be considered that A2C has better robustness.In this chapter, we first briefly introduce the background of reinforcement learning and deep learning, which leads to the deep reinforcement learning method. On this basis, the deep reinforcement learning methods for machine game are systematically investigated, including reinforcement learning methods based on value function and reinforcement learning methods based on strategy gradient. In addition, hierarchical reinforcement learning and multiagent reinforcement learning are introduced. In the experimental research part, we briefly introduce the current mainstream code base and standard test platform for deep reinforcement learning. Finally, the mainstream reinforcement learning algorithms are studied experimentally, the performance of A2C, DDPG, and PPO under different computational costs and the specific training curves is given, and the experimental results are analyzed.


## 5. Conclusion

In this paper, the current situation of biological population evolutionary algorithm, quantum machine learning algorithm, and deep reinforcement learning is analyzed and studied. This paper analyzes that in the upcoming era of “quantum computing,” we are facing many unprecedented opportunities and challenges: how to cross-integrate“biologicalpopulationevolutionaryalgorithm+individualadaptation+quantumcomputing” to quantitatively and accurately describe the uncertainties of big data; in the coming era of “quantum computing,” how to make use of the advantages of parallel and efficient processing of quantum computing to efficiently deal with the complexity problems of biological population evolutionary algorithm and individual adaptive method. These opportunities and challenges will become important problems to be solved urgently in the era of “quantum computing” and will also become research hotspots in the field of big data in the era of intelligence.



---

*Source: 5188335-2022-03-22.xml*