ON FUSION ALGORITHM OF MULTI- ATTRIBUTE DECISION MAKING AND REINFORCEMENT LEARNING BASED ON INTUITIONISTIC FUZZY NUMBER IN WARGAME ENVIRONMENT
Year: 2021

Abstract:
Intelligent games have seen an increasing interest within the research community on artificial intelligence . The article proposes an algorithm that combines the multi-attribute management and reinforcement learning methods, and that joined their effect on wargaming, it solves the problem of the agent’s low rate of winning against specific rules and its inability to quickly converge during intelligent wargame training. At the same time, this paper studied a multi-attribute decision making and reinforcement learning algorithm in a wargame simulation environment, yielding data on the conflict between red and blue sides. We calculate the weight of each attribute based on the intuitionistic fuzzy number weight calculations. And then we determine the threat posed by each opponent’s game agents . Using the red side reinforcement learning reward function, the AC framework is trained on the reward function, and an algorithm combining multi-attribute decision making with reinforcement learning is obtained. A simulation experiment confirms that the algorithm of multi-attribute decision making combined with reinforcement learning presented in this paper is significantly more intelligent than the pure reinforcement learning algorithm. By resolving the shortcomings of the agent’s neural network, coupled with sparse rewards in large-map combat games, this robust algorithm effectively reduces the difficulties of convergence. It is also the first time in this field that an algorithm design for intelligent wargaming combines multi-attribute decision making with reinforcement learning. Finally, another novelty of this research is the interdisciplinary, like designing intelligent wargames and improving reinforcement learning algorithms. ABSTRACT must be centered, in small caps, and in point size 12. Two line spaces precede the abstract. The abstract must be limited to one paragraph.

heading: 1 INTRODUCTION
text: Artificial intelligence (AI) and machine learning (ML) are becoming increasingly popular in realworld applications. For example, AlphaGo has attracted huge attention in the research community and society by showing the capability of AI defeating professional human players in the board game Go. Yet Alphastar, another strong AI program, has achieved great success in the human-machine combating game ’StarCraft’ Pang et al. (2019); Silver et al. (2016). In RTS games, AI-driven methods are widely studied and integrated into the game AI design to increase the intelligence of computer opponent and generate more realistic confrontation gaming experience. In the King Glory Game, Ye D used an improved PPO algorithm to train the game AI, with positive results Ye et al. (2020). By using reinforcement learning techniques, Silver D et al. developed a training framework that requires no human knowledge other than the rules of the game, allowing AlphaGo to train itself, and achieving high levels of intelligence in the process Silver et al. (2017). Using deep reinforcement learning and supervised strategy learning, Barrigan el al. improved the AI performance of RTS games, and defeats the built-in game AI Barriga et al. (2019). AI has become a hot research topic in recent years, showing a wide variety of applications such as deduction and analysis Schrittwieser et al. (2020); Barriga et al. (2017); O’Hanlon (2021). However, there are still limited research to
address the problem of slow convergence during AI training process under a variety of conditions, especially when it comes to human-AI confrontation games.
Indexes measure the value of things or the parameter of an evaluation system. It is the scale of the effectiveness of things to the subject. As an attribute value, it provides the subjective consciousness or the objective facts expressed in numbers or words. It is important to select a scientifically valid target threat assessment (TA) index and evaluate that index scientifically. Target threat assessment contributes to intelligence wargame decision-making as part of current intelligent wargames. It is mainly based on rules, decision trees, reinforcement learning, and other technologies in the current mainstream game intelligent decision-making field, but rarely incorporates multi-attribute decisionmaking theory and methods into the intelligent decision-making field. The actual wargame data obtained through wargame environments are presented in this paper, as well as the multi-attribute threat assessment indicators that are effectively transformed and presented as a unified expression. Using three expression forms of real number, interval number, and intuitionistic fuzzy number, the multi-attribute decision-making theory and methods are used to analyse the target threat degree. Then , an enhanced reward function based on the generated threat degree is established to train more effective intelligent decision making model. To the best of our knowledge, this is the first work that combines the multi-attribute decision making with reinforcement learning to produce high performance for game AI in a wargame experiment.

heading: 2 WARGAMING MULTIPLE ATTRIBUTE INDEX THREAT QUANTIFICATION
text: Obtaining scientific evaluation results requires a reasonable quantification of indicators. An important aspect of decision-making assistance in wargames is target threat assessment, and the evaluation result directly affects the effectiveness of wargame AI. The aim of this section is to introduce threat quantification methods for different types of indicators. By combining the target type, this section divides the target into target distance threat, target attack threat, target speed threat, terrain visibility threat, environmental indicator threat, and target defense value. The acquired confrontation data are incorporated into different indicator types, and then the corresponding comprehensive threat value is calculated. In Table 1 are the attributes and meanings of specific indicators.

heading: 3 ESTABLISHMENT OF A MULTI-ATTRIBUTE QUANTITATIVE THREAT MODEL BASED ON INTUITIONISTIC FUZZY NUMBERS
text: By using the interval number method, our framework indicates whether visibility is possible, and different threats are generated. Nevertheless, the quantified values of other threat targets are real numbers. To unify the problem-solving method, our algorithm converts all interval numbers and real numbers to intuitionistic fuzzy numbers, and calculates the size of the threat by calculating the intuitionistic fuzzy numbers.
(1) This intuitionistic fuzzy entropy describes the degree of fuzzy judgment information provided by an intuitionistic fuzzy set. The larger the intuitionistic fuzzy entropy of an evaluation criterion, the smaller the weight it is; otherwise, the larger needs to be. Based on formulas from the literature Vlachos & Sergiadis (2007), we calculated the entropy weights for each intuitionistic fuzzy. Among them, ideal solution S+i is a conceived optimal solution (scheme), and its attribute values hit the best value among the alternatives; and the negative ideal solution S−i is the worst conceived solution (scheme), and its attribute values hit the worst value among the alternatives. pi is generated by comparing each alternative scheme with the ideal solution and negative ideal solution. If one of the solutions is closest to the ideal solution, but at the same time far from the negative ideal solution,
then it is the best solution among the alternatives.
Hj = − 1
n ln 2 m∑ i=1 [µij lnµij + vij ln vij− (µij + vij) ln (µij + vij)− (1− µij − vij) ln 2] (1)
If µij = 0, vij = 0, then µij lnµij = 0, vij ln vij = 0, (µij + vij) ln (µij + vij) = 0.
The entropy weight of the j attribute is defined as:
wj = 1−Hj n− n∑
j=1
Hj
(2)
Among wj ≥ 0, j = 1, 2, · · · , n, n∑
j=1
wj = 1
(2) Determine the optimal solution A+ and the worst solution A- using the following formula:{ A+ = {〈 µ+1 , ν + 1 〉 , 〈 µ+2 , ν + 2 〉 , · · · , 〈µ+n , ν+n 〉 } A− = {〈 µ−1 , ν − 1 〉 , 〈 µ−2 , ν − 2 〉 , · · · , 〈µ−n , ν−n 〉
} (3) Where
µ+i = max j=12......m {µij} , ν+i = min j=1,2,...,m {νij} (4)
µ−i = min j=1,2,··· ,m {µij} , ν−i = max j=1,2,··· ,m {νij} (5)
(3) Calculate the similarity between the fuzzy intuitionistic A and B as follows:
s (〈µ1, ν1〉 , 〈µ2, ν2〉) = 1− |2 (µ1 − µ2)− (ν1 − ν2)| 3 × ( 1− π1 + π2 2 ) −|2 (ν1 − ν2)− (µ1 − µ2)| 3 × ( π1 + π2 2 ) (6)

heading: In which, π1 = 1− µ1 − ν1, π2 = 1− µ2 − ν2
text: (4) Calculate the similarity S+i and S − i between each solution and the optimal solution and the worst solution based on the following formula: S+i = n∑ k=1 wk · s (〈 µ+k , ν + k 〉 , 〈µik, νik〉 ) S−i = n∑ k=1 wk · s (〈 µ−k , ν − k 〉 , 〈µik, νik〉
) (7) (5) Then calculate the relative closeness
pi = S − i / ( S+i + S − i ) (8)
Comparing threat levels of opponents based on their closeness to the target depends on the level of threat assessment performed.

heading: 4 MULTI-ATTRIBUTE THREAT QUANTITATIVE SIMULATION
text: The threat assessment problem is transformed into a multi-attribute decision making problem, while the combat intention of the target is incorporated into the evaluation system to make the evaluation more realistic and the results more reliable. A simulation scene includes ten tanks on each side, i.e. red and blue, fighting each other, and ten opposite are found as game agents in the wargame.
A unified intuitiveistic fuzzy number representation has been created for all multi-attribute indicators. An example of an intuitionistic fuzzy number representation of threat assessment indicators is illustrated in Table 2.
By obtaining data represented by the intuitionistic vagueness of the threat assessment indicators shown in the Table 2, formulae in (7) and (8) may be used to obtain the intuitionistic vague target threat assessment based on multi-attribute decision making approaches. Table 3 shows the assessment scores to determine the target threat level.
In Table 4, the opposite target at T1 is shown as a threat.
Based on the evaluation results, it can be concluded that the blue T6 tank is the most harmful and the T7 tank is the second most harmful, this is shown in figure 1. This paper does not limit evaluation to subjective analysis of experts, but also introduces reinforcement learning, associates the reinforcement learning algorithm through a reward function and analyses the actual wargame AI’s winning rate.

heading: 5 A FUSION MODEL OF REINFORCEMENT LEARNING AND MULTI-ATTRIBUTE THREAT ANALYSIS

heading: 5.1 REINFORCEMENT LEARNING ALGORITHM AND MULTI-ATTRIBUTE MODEL FORMULATION
text: Previous sections described the quantified value of multi-attribute analysis of threat levels based on the entropy weight method. The section integrate this method with with reinforcement learning. Its essence is to establish a multi-attribute decision-making mechanism that is based on reinforcement learning, and then select the entity with the highest threat level to establish the return value and threat level. The higher the threat level, the greater the return value, this is shown in figure 2.
A reinforcement learning algorithm is built using the AC framework to achieve intelligent decisionmaking. It includes a reinforcement learning pre-training module that integrates multi-attribute decision-making, critic evaluation network update module and a new and old strategy network update module. In the intensive pre-training module, multi-attribute decision making mainly uses state data obtained from the wargame environment, such as elevation, distance, armour thickness, etc., to make multi-attribute decisions. By normalizing the data, calculating the threat of each piece of the opponent by using the entropy method, and then setting the reward function and storing it in the experience, further actions in the environment will be taken to obtain the next state and action rewards.
The critic network calculates the value from the reward value determined during the last step of the action. combines the experience store data with the value calculated by the critic network, slashes it from the reward value determined during the last action, then returns to update the critic network parameters. As the advantage value guides the calculation of the actor network value, the network outputs the action value according to the old and new networks, and the distribution probability overall, and outputs the action from the network. As a result, the advantage value is corrected, the actor loss is calculated, and the actor network is updated in the reverse direction.

heading: 5.2 SETTING REWARD FUNCTION VALUE
text: As a core challenge of deep reinforcement learning in solving practical tasks, the sparse reward problem relates to the fact that the training environment cannot supervise the updating of agent parameters in the process of reinforcement learning Kaelbling et al. (1996). When supervised learning is used, the training process is supervised by humans, while in reinforcement learning, rewards are used to supervise the training process, and the agent optimizes strategies based on rewards ?. The specific additional rewards is showed in Table 5.
When the above additional rewards are added to the training process, the convergence speed can be significantly accelerated, and the likelihood that the agent falls into the local optimum is significantly reduced.

heading: 6 WARGAMES AI SIMULATIONS AND EVALUATIONS

heading: 6.1 EXPERIMENT SETTING
text: Figure 3 shows the starting interface of our simulation which generates the initial states of red and blue tanks Sun et al. (2021) Sun et al. (2020). There are two tank pawns on each side, and the centre is the point of contention. In a confrontation, both sides compete for control points, and the party that reaches the middle red flag first wins. At the same time, both red and blue parties can shoot at each other, while they can hide in urban residential areas. By concealing, it is difficult for our opponents to find our targets. Each hexagon has its own number and elevation. The higher the elevation, the darker the hexagon. On the highway, the tanks move faster than on the secondary roads. The red straight line represents the secondary road and the black straight line represents the primary road. As the cross symbol represents aiming and shooting, the destroyed target disappears from the map.

heading: 6.2 RESULTS AND ANALYSIS OF THE EXPERIMENT
text: In this article, the PPO algorithm Schulman et al. (2017) and the PPO algorithm combined with multi-attribute decision-making are used to compare and analyse the winning rate. MADM-PPO and PPO are trained for 24 hours, and this article uses the MADM-PPO algorithm as the red side and the rule-based blue side algorithm to fight. At the same time, the second round uses the PPO algorithm as the red side, and the blue side fights according to rules. Next, this article observes the winning percentage of both algorithms in 100 games. Experiments have shown that the agents using the PPO reinforcement learning algorithm combined with the multi-attribute decision-making method performed better than the agents using the PPO algorithm based on the threat of the opponent. As can be seen in the Figure 4 and Figure 5, our proposed multi-attribute decision-making method, combined with PPO algorithm of reinforcement learning, proves to effectively improve the effectiveness of intelligent wargame decision-making. A winning rate chart is presented in the Table 6, and Table 7.
The experimental results show that the MADM-PPO model can reduce the number of times to explore during training, and improve the problem that the PPO algorithm takes too long to train. It shows that the introduction of prior knowledge improves the performance of the PPO algorithm, and has a certain theoretical significance for improving the efficiency of the algorithm, the detail score is shown in Figure 6.

heading: 7 CONCLUSION
text: We have designed an intelligent wargaming AI that To design intelligent wargaming AI that combines multi-attribute decision making and reinforcement learning to improve both the convergence speed of the online training process and the winning rate of wargaming AI. As part of this study, this paper conducts experiments on the multi-attribute decision making and reinforcement learning algorithms in a wargame simulation environment, and obtains red and blue confrontation data from the wargame environment. Calculate the weight of each attribute based on the intuitionistic fuzzy number weight calculations. Then determine the threat posed by each opponent’s game agents . On the basis of the degree of threat, the red side reinforcement learning reward function is constructed and the AC framework is trained with the reward function, and the algorithm combines multi-attribute decision making with reinforcement learning. A study demonstrated that the algorithm can gradually increase the reward value of the agent when exploring an environment over a short training period, while the final victory rate of the agent against specific rules and strategies reached 78%, which is significantly higher than that of a pure reinforcement learning algorithm, which is 62%. Solved the convergence difficulties of the state-space wargame’s sparse rewards caused by the randomization of an agent’s neural network. For the algorithm design of intelligent wargaming, this is the first research in this field to combine the multi-attribute decision making method in management with the reinforcement learning algorithm in cybernetics. An interdisciplinary approach to crossinnovation in academia could lead to improvements in the design of intelligent wargames and even improvements in reinforcement learning algorithms. The future research direction can be based on this paper to carry out a series of research, including the introduction of new methods in management multi-attribute decision-making and the fusion and intersection of a series of algorithms such as reinforcement learning SAC, MADDPG and DDQN etc, which can develop more, better and more stable fusion innovative algorithms.

Review summaries:
This paper explores training a deep reinforcement learning agent to play a wargame. To my understanding, a hand-crafted feature extraction, “multi-attribute decision making”, and a hand-crafted reward shaping module were used to create a dataset of pre-training data for the agent. The resulting PPO agent was shown to outperform a baseline rules-based agent, and also train faster and reach a higher asymptotic win-rate against a baseline opponent than a comparable PPO agent.  The contribution of the paper is the use of multi-attribute decision making to learn weights for attributes of opponent game pieces, to guide the creation of a pre-training dataset and shaping rewards for the RL agent.