$$Research on fusion algorithm of multi-attribute decision making and reinforcement learning based on intuitionistic fuzzy number in wargame environment$$

Abstract: Intelligent games have seen an increasing interest within the research community on artificial intelligence. The article proposes an algorithm that combines the multi-attribute management and reinforcement learning methods, and that joined their effect on wargaming, it solves the problem of the agent’s low rate of winning against specific rules and its inability to quickly converge during intelligent wargame training. At the same time, this paper studied a multi-attribute decision making and reinforcement learning algorithm in a wargame simulation environment, yielding data on the conflict between red and blue sides. We calculate the weight of each attribute based on the intuitionistic fuzzy number weight calculations. And then we determine the threat posed by each opponent’s game agents . Using the red side reinforcement learning reward function, the AC framework is trained on the reward function, and an algorithm combining multi-attribute decision making with reinforcement learning is obtained. A simulation experiment confirms that the algorithm of multi-attribute decision making combined with reinforcement learning presented in this paper is significantly more intelligent than the pure reinforcement learning algorithm. By resolving the shortcomings of the agent’s neural network, coupled with sparse rewards in large-map combat games, this robust algorithm effectively reduces the difficulties of convergence. It is also the first time in this field that an algorithm design for intelligent wargaming combines multi-attribute decision making with reinforcement learning. Finally, another novelty of this research is the interdisciplinary, like designing intelligent wargames and improving reinforcement learning algorithms.

Reviews and author responses:

The paper presents a method for using fuzzy logic and domain knowledge to generate the reward signal for RL algorithms. The method is evaluated on a war-game domain.  The paper isn't easy to read and I am not sure I fully understand the method that is proposed. The abstract and introduction are somewhat hard to read but I managed to understand the general idea of the method the paper presents. However, I got lost in Section 3 when the intuitionistic fuzzy numbers are introduced. I am not familiar with this concept and I couldn't learn it from the paper. For example, the paper doesn't define what are the $\mu$s and $v$s in Equation 1. 

The tables and plots are also hard to read. Tables 2 and 3 use a tiny font and I am not sure what those numbers represent. Figure 2 is only mentioned in the text, but never explained. And I found it hard to understand Figure 2 without guidance from the text. The plots with the learning curves are misleading. Instead of labelling the curves with the name of the agents, they are labelled "blue" and "red". The range of the y-axis is different for different plots, which makes it hard to compare lines across different plots. 

The method seems to outperform PPO in the war-game domain though. From what I understand, the method uses domain knowledge from the game to guide the learning process of the RL algorithm by providing helpful reward signals. This makes me wonder if the method is general enough to be applicable to other domains. If my understanding of the method is correct, after rewriting the paper, I would recommend the authors to submit their work to a Game AI conference such as CoG and AIIDE. These communities are interested in solutions that rely on domain knowledge to solve the problem (maybe I should say that these communities wouldn't mind using domain knowledge to solve the problem). The paper presents a method that seems to outperform PPO on a war-game domain. The paper does not explain the method and results clearly and I am not sure I understand the method that is being proposed. The method seems to rely on domain knowledge to speed up the learning process, which cast doubt on the generality of the approach. 

This paper uses multi-attribute decision-making in order to construct a more rich reward function for a Reinforcement Learning algorithm. The algorithm used is PPO and the authors demonstrate that multi-attribute decision making outperforms vanilla PPO on a wargame environment. I have many major concerns with this submission:

- Equations (1) to (7) contain terms that are never defined in the paper, so it is not possible to analyze them.

- Tables 2 and 3 are extremely tiny. Is it really important to provide the 16thdecimal number for these parameters? There is no caption in the tables and they are difficult to interpret.

- It is not clear to me how formulas 7 and 8 are incorporated into the RL algorithm. No pseudo-algorithm is provided and the reader is left with just figure 2, which is confusing since most of the components in the figure were never really explained. In particular, there are no formulas or background provided for Reinforcement Learning.

- The abstract still includes the text "ABSTRACT must be centered, in small caps, and in point size 12. Two line spaces precede the abstract. The abstract must be limited to one paragraph.", which is disappointing, if not offensive for this kind of conference.

-  “Indexes measure the value of things or the parameter of an evaluation system. It is the scale of the effectiveness of things to the subject. As an attribute value, it provides the subjective consciousness or the objective facts expressed in numbers or words” I do not understand this sentence. What is the subjective consciousness? What is “the scale of the effectiveness of things to the subject”?

- No hyperparameters are provided, no multiple runs, so the results can be arbitrary and almost impossible to reproduce.

- The captions in figures 4, 5, and 6 and the comment after figure 5 are copied and pasted from two articles that the authors cite. The authors only changed the name of the algorithm. Compare captions in Figures 4,5 of the current paper with captions in Figures 11 and 12 in [1] and Figures 12 and 13 in [2]. Compare captions in Figure 6 of the current paper with those in Figure 13 in [1] and 14 in [2]. Compare the small paragraph before section "Conclusions" with the small paragraph before section "Conclusions in [2]. I think it is a bad practice to copy even small sections from other, already published, papers.

- The authors claim that Figures 4 and 5 show that multi-attribute PPO vs ruled-based AI outperforms PPO vs ruled-based AI. However, we can see from Figure 4 (a) that multi-attribute PPO is already winning all the games at beginning of training, and then its performance degrades over time, while from Figure 5 PPO is losing at beginning of training (as expected) and then improve. Why is multi-attribute PPO winning all the games without being trained? Why is the algorithm not improving over time? Note that this issue is present also in the cited papers that have the exact same caption [1,2]

- How are the rewards in Table 5 chosen? How are the states on the left side of Table 5 computed? Are these depending on the multi-attribute analysis? If not, can the authors provide an ablation for the reward component and the multi-attribute component?

Minor concerns:

- The title in OpenReview is in the wrong format

- The format for references used in the paper is not correct. See many examples in the Introduction.

- The environment used is not properly explained, while the authors use often technical terms that are specific to the environment. Is there a reference for what environment, in particular, has been used?

- the abstract is confusing. For instance, the authors start talking about “red and blue sides” where these were never defined.

- There is a reference missing in section 5.2

[1] Yuxiang Sun, Bo Yuan, Yongliang Zhang, Wanwen Zheng, Qingfeng Xia, Bojian Tang, and Xianzhong Zhou. Research on action strategies and simulations of drl and mcts-based intelligent round game. International Journal of Control, Automation and Systems, pp. 1–15, 2021.

[2] Yuxiang Sun, Bo Yuan, Tao Zhang, Bojian Tang, Wanwen Zheng, and Xianzhong Zhou. Research and implementation of intelligent decision based on a priori knowledge and dqn algorithms in wargame environment. Electronics, 9(10):1668, 2020.
 This paper is far from being ready for publication. The theory introduced is incomplete, with many terms undefined. The writing is sometimes confusing: the authors did not explain how multi-attribute decision-making is used in this setting and did not give proper background on Reinforcement Learning. Finally, the experiments seem wrong because the algorithm proposed achieves optimality before the beginning of training and then degrades in performance.

This paper explores training a deep reinforcement learning agent to play a wargame. To my understanding, a hand-crafted feature extraction, “multi-attribute decision making”, and a hand-crafted reward shaping module were used to create a dataset of pre-training data for the agent. The resulting PPO agent was shown to outperform a baseline rules-based agent, and also train faster and reach a higher asymptotic win-rate against a baseline opponent than a comparable PPO agent.

The contribution of the paper is the use of multi-attribute decision making to learn weights for attributes of opponent game pieces, to guide the creation of a pre-training dataset and shaping rewards for the RL agent.
 I found this paper difficult to read, as its presentation often either does not describe foundational material, or does not adequately describe how the components of the technique and agent fit together. I will describe these issues as I go, but at times I had to guess at how the resulting agent works, and have likely misunderstood the authors’ intent at times.

A key weakness of the paper is its presentation. The technique relies on foundational techniques: multi-attribute decision making, intuitionistic fuzzy numbers, reinforcement learning, and so on. The paper needs a background section right after the introduction to describe these concepts, and none is given. Section 2 moves directly into a game-specific decomposition of state features into attributes, and Section 3, which I cannot follow, starts using these concepts. The single wargame used throughout the paper, which much of the first few pages rely on, is only described briefly on page 6, and is not even named. The details provided are not sufficient for other researchers to replicate the work or to find the environment online to learn more.

The largest presentation issue, perhaps, is that it is unclear to me how the pieces of the technique fit together. For example, in my summary I described the technique as creating additional features and rewards for creating pre-training data for an RL agent. That is admittedly a guess, guided by Fig 2, Table 5, and Section 5.1: I’m unsure about exactly how the opponent unit threat features that are produced in Sections 2-4 are actually integrated with the RL agent (which should be the topic of Section 5.1), but it seems like the only options are to produce additional features observed by the RL agent, or to produce shaping rewards, or just to produce pre-training data, or some combination of these. But this is not adequately explained.

To clarify this, the paper really needs a clear presentation of the flow of data through the system. For example, if I’ve understood correctly, a description of how environment observations come in, additional features are created and weighted by the multi-attribute decision making system, shaping rewards are added, and then presented to the agent. If the MADM-PPO and PPO agents differ after pre-training (e.g., if their online learning process is different) then that needs to be explained. As it is, I do not understand exactly how MADM-PPO and PPO differ, how the pre-training described in Sec 5.1 works, whether the agent is only trained on pre-training data or if it trains online, and so on.

The difference between MADM-PPO and PPO is particularly important. How exactly do these agents differ? Does PPO not get the pre-training from MADM-PPO, or additional state features, or the reward shaping, or any of these? To properly credit which part of the technique provides the advantages claimed in Sec 6.2, if the technique is separable into parts, the paper needs an ablation study which attempts to separate these changes. For example, every shaping reward listed in Table 5 sounds like it does not depend on the opponent threat values described earlier in the paper, and could be provided in isolation: is this alone the source of the advantage?

The empirical study in Section 6.2 and Figs 4 and 5 is also a weakness. These figures appear to present the results of a single run of each of the MADM-PPO and PPO algorithms, and show performance against a baseline rule-based agent. With only a single run, we cannot determine anything about whether MADM-PPO consistently outperforms PPO. Further, while (from this one run) PPO appears to take slightly longer to start learning (25 out of 100 episodes), the slope of its “win times of both sides”, Fig 5b, appears very similar to that of Fig 4b, suggesting similar asymptotic performance, which is hidden by how Fig4a and Fig5a are presented. Figs 4a and 5a show the running average winrate over time for the entire match, and not separated into bins, which is misleading: the PPO agent’s final performance appears very similar to MADM-PPO but is shown as ~15% lower because of the persistent weight on the earliest episodes while it is still training. A better presentation would 1) replicate many independent runs of each experiment, and 2) bin the data into blocks of episodes, and show the within-bin mean win rate and confidence interval. That way, we could actually see how each agent improves over time, hopefully with statistical confidence, and then compare whether MADM-PPO has an actual advantage in win-rate, or only in initial training speed, or in both.

The empirical study has another possible flaw. In Fig 4 and Fig 5, the AI agent (either MADM-PPO or PPO) is always presented as the red player, and the rules-based agent is always presented as the blue player. Is there any strategic advantage to being one player instead of the other, and is the map always the same, or is it randomly sampled in each episode? For example, does one color always go first, as in chess or go? Or perhaps elevation: Sec 6.1 describes map hexagons as having elevation, with darker hexagons being at higher elevations, and Fig 3, which presents the game map, shows the red side at a consistently higher elevation than the blue side. If there is any strategic advantage to being at a higher elevation (common in wargames, and if not, why describe it as a game element?), then the empirical results could be unfairly biased in favour of the AI agents over the rule-based agents. Why are the agents not evaluated by averaging their performance across both teams to even this out, perhaps using duplicate games (playing once as red and once as blue on the same map) to cancel out variance due to luck. Put another way: the paper does not demonstrate that the AI agents would still beat the rules-based agent, or have similar learning trajectories, if they played as blue instead of red. Similarly, why do MADM-PPO and PPO only play against a rules-based agent, and not against each other? It seems quite likely that if MADM-PPO gets the benefit of pre-training data that PPO does not, then it could overfit to it, and lose to PPO in a head-to-head match.

Finally, even if the presentation and empirical issues were addressed, the technique as described appears very game specific. It relies on domain knowledge to distill enemy unit state features into attributes (Table 1) and hand-crafted shaping rewards (Table 5). Even if the technique could be demonstrated to have a statistically meaningful performance advantage over the PPO baseline, the paper still needs a discussion section to describe how easily this approach could be moved to other environments, to make it relevant to other AI researchers.


Smaller issues:
 - The final sentences of the abstract include the submission instructions for ICLR.
 - What is an “intelligent game” in the first words of the abstract?
 - The paper describes algorithms/agents as being “more intelligent” (in the abstract) or attributes as providing “subjective consciousness” (sec 1). Please use more concrete terms, to describe learning rate, asymptotic performance, and so on.
 - The final paragraph of Sec 1 describes “Indexes measure the value of things”... Do the authors mean ‘Metrics’ here? Or features?
 - Throughout the paper, citations appear to be given with the wrong latex command (\cite vs \citep, perhaps?).  For example, “...defeats the built-in game AI Barrigan et al (2019).” instead of “...defeats the built-in game AI (Barrigan et al., 2019).”
 - Just above Fig 4, the sentence “A winning rate chart is presented in the Table 6, and Table 7”, but these tables to not appear in the paper, and no supplementary material was given.
 - Throughout the paper, “opposite” is used where “opponent” would be the better term.
 - Sec 2: Is this the best spot for this section? The text really needs a background section here, and this section is game-specific far before the game itself is described.
 - Sec 3: I found this section very hard to follow without a background section to describe what multi-attribute decision making, intuitionistic fuzzy entropy, etc are.
 - Sec 3: $S^+_i$ is described as an optimal solution. But above Equation 7, $S^+_i$ is defined as a similarity between a solution and the optimal solution. Which is it?
 - Sec 3: Why does each paragraph start with (1), (2), (3), etc? This is confusing as they are interleaved with the Equations (1), (2), (3), etc.
Equation 1: What do mu_ij and v_ij refer to? Presumably attributes from Table 1, but these symbols are used without any description of what they are, what they index, and so on.
 - Sec 4: Is “intuitiveistic” a typo for “intuitionistic”?
 - Sec 4 was also hard to follow. How exactly are these threat ratings and rankings used?
 - Table 2: Text is far too small to be legible. Is Table 2 just an example of state features in one particular state, or are these constants or weights used across all states?
 - Sec 5.1: First paragraph describes a multi-attribute decision-making mechanism based on RL. Is that true, or is it used to produce features/shaping rewards *for* a downstream RL component? If MADM is based *on* RL then I don’t understand how, and it needs to be described in much more detail.
 - Sec 5.1: How exactly does MADM *make decisions* for pre-training the RL agent? Earlier in the text it appears only to be generating features. MADM just needs to be explained to a much greater degree right after the introduction.
 - Sec 5.1: “...calculated by the critic network, slashes it from the reward value…” What does ‘slashes’ mean? Subtracts, divides, or something else?
 - Fig 2. ‘Train Data’ on the left is part of the red box labeled ‘Storage of RL pre-training experiences’. This is unclear: are the actor and critic agents only trained on stored (i.e., offline) experience, or are they (also?) trained on online experience? If they are trained online, maybe this diagram can be expanded to show how the actor selects actions, those are sent to the environment, and the resulting transitions are either used for training or fed into the ‘Storing experience’ data store.
 - Fig 2. In the actor network section on the right, several labels have their words strangely (unnecessarily?) reordered. E.g., “action probability of 1” and “probability of action 2”, “normative distribution 1” and “distribution normative 2”, etc. What’s going on here?
 - Section 5.2 has a broken reference: “agent optimizes strategies based on rewards ?”.
 - Sec 5.2: The final sentence describes “...likelihood that the agent falls into the local optimum is significantly reduced”. Is there any evidence for this claim? It seems baseless, given the results presented in the paper, and I cannot see any theoretical support for it either.
 - Sec 6.2: The final sentence says that the introduction of prior knowledge “...has a certain theoretical significance for improving the efficiency of the algorithm”. Where is the theoretical justification for this claim? I cannot find any supporting evidence for it.
 - Section 7: First sentence seems to be an editing error where two sentences were joined (wargaming AI that To design intelligent wargaming)
 - References: Maybe just push the references to a new page, instead of dangling the first reference on page 8, then being spliced by a figure on page 9, then continuing the figures on page 10. It’s still well within the page limit.
 The paper is deficient in several aspects. It has clarity and presentation issues throughout, and does not at all describe the techniques it depends on. There is no theoretical contribution, and the empirical contribution is both insufficient and appears to present the results of a single run of each algorithm, so we cannot conclude that the proposed technique improves on the baseline. The paper is also not reproducible from the details provided (i.e., the environment is not named or adequately described, the neural net details and parameters are not given, etc). Even if these issues were addressed, the proposed technique involves (to my understanding) augmenting the agent’s observation with game-specific hand-crafted shaping rewards and state features, and there is no discussion of how easily this can be ported to other environments, or generalized.

If the technique was more clearly described and evaluated, it could support the use of multi-attribute decision making as a way to augment a deep RL agent’s state representation. But with this presentation of the technique, the paper should be rejected.


The article proposes an algorithm that combines the multi-attribute management and reinforcement learning methods, and that joined their effect on wargaming, it solves the problem of the agent’s low rate of winning against specific rules and its inability to quickly converge during intelligent wargame training.   Strengths: The author believes that this is the first work that combines multi-attribute decision making with reinforcement learning to produce a high performance for game AI in a wargame experiment. 

Weaknesses:  

1. Tables 2 and 3 are too small. 

2. The Last Line of the Abstract should be removed: “ABSTRACT must be centered, in small caps, and in point size 12. Two line spaces precede the abstract. The abstract must be limited to one paragraph.”

I think the writing of this paper could be better  Leaning to Reject. I tend to vote for rejecting this submission, but accepting it would not be that bad.

Meta-review: All reviewers agreed on rejection. Unfortunately, there was no author response so there was nothing to drive further discussion on the paper. The reviewers gave very detailed advice on improving the work.

Decision: Reject