$$Research on fusion algorithm of multi-attribute decision making and reinforcement learning based on intuitionistic fuzzy number in wargame environment$$

1 INTRODUCTION . Artificial intelligence ( AI ) and machine learning ( ML ) are becoming increasingly popular in realworld applications . For example , AlphaGo has attracted huge attention in the research community and society by showing the capability of AI defeating professional human players in the board game Go . Yet Alphastar , another strong AI program , has achieved great success in the human-machine combating game ’ StarCraft ’ Pang et al . ( 2019 ) ; Silver et al . ( 2016 ) . In RTS games , AI-driven methods are widely studied and integrated into the game AI design to increase the intelligence of computer opponent and generate more realistic confrontation gaming experience . In the King Glory Game , Ye D used an improved PPO algorithm to train the game AI , with positive results Ye et al . ( 2020 ) . By using reinforcement learning techniques , Silver D et al . developed a training framework that requires no human knowledge other than the rules of the game , allowing AlphaGo to train itself , and achieving high levels of intelligence in the process Silver et al . ( 2017 ) . Using deep reinforcement learning and supervised strategy learning , Barrigan el al . improved the AI performance of RTS games , and defeats the built-in game AI Barriga et al . ( 2019 ) . AI has become a hot research topic in recent years , showing a wide variety of applications such as deduction and analysis Schrittwieser et al . ( 2020 ) ; Barriga et al . ( 2017 ) ; O ’ Hanlon ( 2021 ) . However , there are still limited research to address the problem of slow convergence during AI training process under a variety of conditions , especially when it comes to human-AI confrontation games . Indexes measure the value of things or the parameter of an evaluation system . It is the scale of the effectiveness of things to the subject . As an attribute value , it provides the subjective consciousness or the objective facts expressed in numbers or words . It is important to select a scientifically valid target threat assessment ( TA ) index and evaluate that index scientifically . Target threat assessment contributes to intelligence wargame decision-making as part of current intelligent wargames . It is mainly based on rules , decision trees , reinforcement learning , and other technologies in the current mainstream game intelligent decision-making field , but rarely incorporates multi-attribute decisionmaking theory and methods into the intelligent decision-making field . The actual wargame data obtained through wargame environments are presented in this paper , as well as the multi-attribute threat assessment indicators that are effectively transformed and presented as a unified expression . Using three expression forms of real number , interval number , and intuitionistic fuzzy number , the multi-attribute decision-making theory and methods are used to analyse the target threat degree . Then , an enhanced reward function based on the generated threat degree is established to train more effective intelligent decision making model . To the best of our knowledge , this is the first work that combines the multi-attribute decision making with reinforcement learning to produce high performance for game AI in a wargame experiment . 2 WARGAMING MULTIPLE ATTRIBUTE INDEX THREAT QUANTIFICATION . Obtaining scientific evaluation results requires a reasonable quantification of indicators . An important aspect of decision-making assistance in wargames is target threat assessment , and the evaluation result directly affects the effectiveness of wargame AI . The aim of this section is to introduce threat quantification methods for different types of indicators . By combining the target type , this section divides the target into target distance threat , target attack threat , target speed threat , terrain visibility threat , environmental indicator threat , and target defense value . The acquired confrontation data are incorporated into different indicator types , and then the corresponding comprehensive threat value is calculated . In Table 1 are the attributes and meanings of specific indicators . 3 ESTABLISHMENT OF A MULTI-ATTRIBUTE QUANTITATIVE THREAT MODEL BASED ON INTUITIONISTIC FUZZY NUMBERS . By using the interval number method , our framework indicates whether visibility is possible , and different threats are generated . Nevertheless , the quantified values of other threat targets are real numbers . To unify the problem-solving method , our algorithm converts all interval numbers and real numbers to intuitionistic fuzzy numbers , and calculates the size of the threat by calculating the intuitionistic fuzzy numbers . ( 1 ) This intuitionistic fuzzy entropy describes the degree of fuzzy judgment information provided by an intuitionistic fuzzy set . The larger the intuitionistic fuzzy entropy of an evaluation criterion , the smaller the weight it is ; otherwise , the larger needs to be . Based on formulas from the literature Vlachos & Sergiadis ( 2007 ) , we calculated the entropy weights for each intuitionistic fuzzy . Among them , ideal solution S+i is a conceived optimal solution ( scheme ) , and its attribute values hit the best value among the alternatives ; and the negative ideal solution S−i is the worst conceived solution ( scheme ) , and its attribute values hit the worst value among the alternatives . pi is generated by comparing each alternative scheme with the ideal solution and negative ideal solution . If one of the solutions is closest to the ideal solution , but at the same time far from the negative ideal solution , then it is the best solution among the alternatives . Hj = − 1 n ln 2 m∑ i=1 [ µij lnµij + vij ln vij− ( µij + vij ) ln ( µij + vij ) − ( 1− µij − vij ) ln 2 ] ( 1 ) If µij = 0 , vij = 0 , then µij lnµij = 0 , vij ln vij = 0 , ( µij + vij ) ln ( µij + vij ) = 0 . The entropy weight of the j attribute is defined as : wj = 1−Hj n− n∑ j=1 Hj ( 2 ) Among wj ≥ 0 , j = 1 , 2 , · · · , n , n∑ j=1 wj = 1 ( 2 ) Determine the optimal solution A+ and the worst solution A- using the following formula : { A+ = { 〈 µ+1 , ν + 1 〉 , 〈 µ+2 , ν + 2 〉 , · · · , 〈µ+n , ν+n 〉 } A− = { 〈 µ−1 , ν − 1 〉 , 〈 µ−2 , ν − 2 〉 , · · · , 〈µ−n , ν−n 〉 } ( 3 ) Where µ+i = max j=12 ...... m { µij } , ν+i = min j=1,2 , ... , m { νij } ( 4 ) µ−i = min j=1,2 , ··· , m { µij } , ν−i = max j=1,2 , ··· , m { νij } ( 5 ) ( 3 ) Calculate the similarity between the fuzzy intuitionistic A and B as follows : s ( 〈µ1 , ν1〉 , 〈µ2 , ν2〉 ) = 1− |2 ( µ1 − µ2 ) − ( ν1 − ν2 ) | 3 × ( 1− π1 + π2 2 ) −|2 ( ν1 − ν2 ) − ( µ1 − µ2 ) | 3 × ( π1 + π2 2 ) ( 6 ) In which , π1 = 1− µ1 − ν1 , π2 = 1− µ2 − ν2 . ( 4 ) Calculate the similarity S+i and S − i between each solution and the optimal solution and the worst solution based on the following formula : S+i = n∑ k=1 wk · s ( 〈 µ+k , ν + k 〉 , 〈µik , νik〉 ) S−i = n∑ k=1 wk · s ( 〈 µ−k , ν − k 〉 , 〈µik , νik〉 ) ( 7 ) ( 5 ) Then calculate the relative closeness pi = S − i / ( S+i + S − i ) ( 8 ) Comparing threat levels of opponents based on their closeness to the target depends on the level of threat assessment performed . 4 MULTI-ATTRIBUTE THREAT QUANTITATIVE SIMULATION . The threat assessment problem is transformed into a multi-attribute decision making problem , while the combat intention of the target is incorporated into the evaluation system to make the evaluation more realistic and the results more reliable . A simulation scene includes ten tanks on each side , i.e . red and blue , fighting each other , and ten opposite are found as game agents in the wargame . A unified intuitiveistic fuzzy number representation has been created for all multi-attribute indicators . An example of an intuitionistic fuzzy number representation of threat assessment indicators is illustrated in Table 2 . By obtaining data represented by the intuitionistic vagueness of the threat assessment indicators shown in the Table 2 , formulae in ( 7 ) and ( 8 ) may be used to obtain the intuitionistic vague target threat assessment based on multi-attribute decision making approaches . Table 3 shows the assessment scores to determine the target threat level . In Table 4 , the opposite target at T1 is shown as a threat . Based on the evaluation results , it can be concluded that the blue T6 tank is the most harmful and the T7 tank is the second most harmful , this is shown in figure 1 . This paper does not limit evaluation to subjective analysis of experts , but also introduces reinforcement learning , associates the reinforcement learning algorithm through a reward function and analyses the actual wargame AI ’ s winning rate . 5 A FUSION MODEL OF REINFORCEMENT LEARNING AND MULTI-ATTRIBUTE THREAT ANALYSIS . 5.1 REINFORCEMENT LEARNING ALGORITHM AND MULTI-ATTRIBUTE MODEL FORMULATION . Previous sections described the quantified value of multi-attribute analysis of threat levels based on the entropy weight method . The section integrate this method with with reinforcement learning . Its essence is to establish a multi-attribute decision-making mechanism that is based on reinforcement learning , and then select the entity with the highest threat level to establish the return value and threat level . The higher the threat level , the greater the return value , this is shown in figure 2 . A reinforcement learning algorithm is built using the AC framework to achieve intelligent decisionmaking . It includes a reinforcement learning pre-training module that integrates multi-attribute decision-making , critic evaluation network update module and a new and old strategy network update module . In the intensive pre-training module , multi-attribute decision making mainly uses state data obtained from the wargame environment , such as elevation , distance , armour thickness , etc. , to make multi-attribute decisions . By normalizing the data , calculating the threat of each piece of the opponent by using the entropy method , and then setting the reward function and storing it in the experience , further actions in the environment will be taken to obtain the next state and action rewards . The critic network calculates the value from the reward value determined during the last step of the action . combines the experience store data with the value calculated by the critic network , slashes it from the reward value determined during the last action , then returns to update the critic network parameters . As the advantage value guides the calculation of the actor network value , the network outputs the action value according to the old and new networks , and the distribution probability overall , and outputs the action from the network . As a result , the advantage value is corrected , the actor loss is calculated , and the actor network is updated in the reverse direction .

Summary: This paper explores training a deep reinforcement learning agent to play a wargame. To my understanding, a hand-crafted feature extraction, “multi-attribute decision making”, and a hand-crafted reward shaping module were used to create a dataset of pre-training data for the agent. The resulting PPO agent was shown to outperform a baseline rules-based agent, and also train faster and reach a higher asymptotic win-rate against a baseline opponent than a comparable PPO agent.  The contribution of the paper is the use of multi-attribute decision making to learn weights for attributes of opponent game pieces, to guide the creation of a pre-training dataset and shaping rewards for the RL agent.