Feint in Multi-Player Games
Authors: Junyu Liu
Wangkai Jin
Xiangjun Peng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Feint
Multi-Player Games

Abstract: This paper introduces the first formalization, implementation and quantitative evaluation of \feint in Multi-Player Games. Our work first formalizes \feint from the perspective of Multi-Player Games, in terms of the temporal, spatial and their collective impacts. The formalization is built upon \textit{Non-transitive Active Markov Game Model}, where \feint can have a considerable amount of impacts. Then, our work considers practical implementation details of \feint in Multi-Player Games, under the state-of-the-art progress of multi-agent modeling to date (namely Multi-Agent Reinforcement Learning). Finally, our work quantitatively examines the effectiveness of our design, and the results show that our design of Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption. We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting.

Under review as a conference paper at ICLR 2023
FEINT IN MULTI-PLAYER GAMES
Anonymous authors Paper under double-blind review
ABSTRACT
This paper introduces the first formalization, implementation and quantitative evaluation of Feint in Multi-Player Games. Our work first formalizes Feint from the perspective of Multi- Games, in terms of the temporal, spatial and their collective impacts. The formalization built upon uon-transitive Active Markov Game Model, where Feint can have considerable amount of impacts. Then, our work considers practical implementation details of Feint Multi-Player Games, under the state-of-the the- progress of multi-agent modeling to date (namely Multi- Agent Reinforcement Learning). Finally, our work quantitatively examines the effectiveness of our design, and the results show that our design of Feint can (I) greatly improve the reward gains from the game; (2) significantly improve the di- versity of Multi- Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption. We conclude that our design of Feint is effective and practical, make Multi-Player Games more interesting.
INTRODUCTION
Game simulations, which only use Markov Game Model (Filar (1976)) or its variants (Wampler et al. (2010); Kim et al. (2022), ) breed the needs for the diversity and the randomness to improve the game experiences. The trends of evolving more details into simulated games demand: O the need for non-transitivity (i.e. there
dominant gaming strategies), which allow players dynamically change game strategies. In this way, the newly-incorporated strategies can maintain a high level of the diversity, which guarantee high extent of unexploitability (Liu al. (202I)); I); and @ the strict requirements on temporal impacts (and its implications on spatial and collective impacts), since modern game simulations are highly timessensitive (Nota & Thomas (2020). Therefore, new optimizations on these game models are expected to be elegant and casy-to-implement, to preserve the original spirits of these games.
Our work first builds upon representative examples from the above two trends, by unifying two state- of-the-art progress of Multi Paaltir Games: 1 we use Unified Behavioral and Response Diversity (described in Liu et al. (2021). which exploits non-transitivity (i.e. single dominant strategy many complex games), to highlight the importance the diversity in game policies. Moreover, we address the issue from their work, which fails to consider the intensity and future impacts from complex interactions among agents; and we incorporate Long- Term Behavior Learning (described in Kim et al. (202)), ) which proposes Active Markov Game Model to emphasize the convoluted future impacts from complex interactions among agents. Based on the above two results, we unify them new model called Non-transitive Active Markov Game Model (NTAMGM), and use it throughout this work. This unification satisfies the need for a game model where (A) agents have intense and time-critical interactions; and (B) the design space of game policies is highly diverse. The definition of NTAMGM is described below.
Non-transitive Active Markov Game Model: We define a K -agent Non-transitive Active Markov Game Model as tuple (K, S, A, P. R, θ, U): K = [L...., k} is the set of k agents; S is the state space; = { ] is the set of action space for each agent, where there are no dominant actions; P performs state transitions of current state by agents' actions: P S x A x A2 x x Ag - P(S), where P(S) denotes the set of probability distribution over state space S: R {R
K is the set of reward functions for each agent; θ {O,) is the set of policy parameters for each agent; and U (U.) is the set of policy update functions for each agent.

---

Under review as a conference paper at ICLR 2023
Based on the above assumption of Multi-Player Games, goal is to incorporate Feint, set of actions to mislead opponents, for strategic advantages in Multi-Player Games. Prior works simply incorporate Feint in the context of Two-Player Games (e.g. Wampler al. (2010); Won et al. (202Ta)) and our works begins by addressing the limitations of the derived version (denoted as the basic formalization of Feint) from these works. We find that: the basic formalization of Feint overlooks the complexity of potential impacts in Multi-Player Games, and therefore can not be gen- eralized for Multi-Player Games. To this end, we deliver the first comprehensive formalization of Feint, by separating the complex impacts into ① the temporal dimension; @ the spatial dimension; and @ the collective impacts from these two dimensions. We also show that how the above compo- nents of our formalization can be synergistically put together. Based on the proposed formalization, we clear the implementation roadmap, under both Inference Learning and Reinforcement Learning models, to justify the applicability of our proposed formalization.
To properly examine the benefits of our method, we first extensively build two complex scenar- ios, using Multi-Agent Deep Deterministic Policy Gradient (MADDPG Lowe et al. (2017)) and Multi-Agent Actor-attention Critic (MAAC Iqbal & Sha (2019). with six agents in total. Then, implement our formalization upon these two eengiieeeeeeee scenarios. Our quantitative evaluations show that our formalization and implementations have great potential in practice. We first show that our work can make the game more interesting, via the following two metries: for the Diversity Gain, our method can increase the exploitation of the search space by 1.98X, measured by the Exploitability metric: and for Gaming Reward Gain, our method can achieve 1.90X and 2.86X gains, when using MADDPG and MAAC respectively. We then show that our method only incur negligible overheads, by using per-episode execution time the metric: our method only introduces less than 5% more for the time consumption. We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting.
BACKGROUND AND MOTIVATION
2.1 EXISTING MARL MODELS
Multi-Agent Reinforcement Learning (MARL) aims to learn optimal policies for agents in a multi- agent environment, which consists of various agent-agent and agent-environment interactions. Many single-agent Reinforcement Learning methods (e.g. DDPG [Lllicrap et al. (2016), SAC Haarnoja ct al. (2018), Schulmmulman et al. (2017) and Fujimooo et al.] (2018)) can not be directly used in multi-agent scenarios, since the rapidly-
multi-agent environment can cause highly unsta- ble learning results (evidenced by Lowe et (2017)) Thus, recent efforts on MARL model designs aim to address such an issue. |oorsterr et al. (2018) proposes Counterfactual Multi-Agent (COMA) policy gradients, which uses centralised critic to estimate the Q-function and decentralised actors to optimize agents' policies. Lowe et al. (2017) proposes Multi-Agent Deep Deterministic Policy Gradient (MADDPG), which decreases the variance in policy gradient and instability of Q-function of DDPG in multi-agent scenarios. [Iqall & Sha (2019) proposes Aulti-Agent Actor-attention Critic (MAAC), which applies attention entropy mechanism to enable effective and scalable policy learn- ing. These models can have varied impacts within diverse set of scenarios.
2.2 FEINT IN NUTSHELL
Feint is common for human players, as a set of active actions to obtain strategic advantages in real-world games. Examples can include sports games such as boxing, basketball and car racing Güldenpenning et al. (2017; 2018); Hyman (1989), J, and electronic games such as King of Fighters and Starcraft Team (2021): p Critch & Churchill (202I). Though Feint is undoubtedly important in game simulations, there still lacks a comprehensive formalization of Feint for Non- Non-Player Characters (NPCs) in Multi-Player Games. Only a limited amount of works tackle this issue. |Wampler et al. (2010) is an early example to incorporate Feinf as a proof-of-concept, which focuses on constructing animations for nuanced game strategies for more unpredictability from NPCs. More recently, Wonn ct al. (202 uses a set of pre-defined Feint actions for the animation, which further serves under an optimized version of control strategy based on Online Reinforcement Learning (i.e. in animating combat scenes). However, these prior works (1) solely focus Two-Player Games, which can not be effectively generalized to multi multi-player scenarios; and (2) lack an comprehensive exploration of potential implications from Feint actions in game strategies.
2

---

Under review as a conference paper at ICLR 2023
2.3 NOVELTY OF OUR WORK
The novelty of our work is three-folded. First, our work introduces the first formalization of Feint which can be generalized to Multi-Player Games. Prior works solely focuses on Two-Player Games, which have the flexibility and scalability issue from the basic formalization. Second, our work provides effective implementations of Feint in Multi-Player Games, by exploiting our formalization appropriately on common parts of MARL models (i.e. the reward function). Our formalization can applicable in future MARL models.
be applied existing MARL, models, and expected Third, our work identifies the unique characteristies of Feint, by differentiating Feint with other regular actions. Hence, our work is expected to be applicable in different scenarios, with only limited amount of refinements.
FEINT FORMALIZATION
3.1 THE BASIC FORMALIZATION, DERIVATION AND LIMITATIONS
We summarize two major limitations of existing works to justify that they cannot deliver a sufficient formalization of Feint in Player Games. Since there are no prior formalization, we discuss relevant works and derive the key features to discuss them in detail.
The basic formalization on temporal impacts is insufficient for Multi-Playe Player Games. Multi-Player Games require agents to account for future planning for decision-making, which is critical for de- ceptive actions like Feint MMih et al.] (2013); Naik et al.] (2019); NNta & Thomas (2020). Sev- eral works simplify the temporal impacts of deceptive game strategies in different gaming scenar- ios. Mnnhh et al. (2013) uses a discount factor to calculate the reward for following actions as Ei=n x (8t,,, aj. Or for agent i. However, such a method suffers from the "short-sight" issue Naik et al. (2019), since the weights for future actions' rewards shrink exponentially with time, which are not suitable for all gaming situations (discussed in Nota & Thomas (2020). More re- cently, Kim et al. (2022) applies long-term average reward, equalize the rewards of all future actions as T Ei= R' ("(s,,a) a) (i.e. for agent i). However, such method is restricted by the "far-sight" issue, since there are no differentiation between near-future and far-future planning. The mismatch between abstraction granularity heavily saddles with the design of Feint because they use relatively static representations (e.g. static y and 7).. Therefore, they cannot be aware of any potential changes of strategies in different phases of a game. Hence, the temporal dimension is simplified for the basic Feint formalization.
The basic formalization cannot be effectively generalized to Multi-Player Game scenarios. Prior works, which attempt fuse Feint into complete game scenarios, only consider two-player scenar- ios WWon et al. (202a)); (So et al. (2022). However, in Multi-Player (more then two player) Games, gaming strategies (especially deceptive strategies) yield spatial impacts on other agents. Such im- pacts have been overlooked by all prior works. This is because an agent, who launches the Feint actions, can impact not only the target agent but also other agents in the scenario. Therefore, the influences of such an action needs to account for spatial impacts Liu et al. (202T). Moreover, with new dimension accounted, the interactions between these two dimensions also raise potential issue for their mutual collective impacts.
3.2 OUR FORMALIZATION GENERALIZED FOR MULTI-PLAYER GAMES
Therefore, to deliver an effective formalization of Feint in Multi-Player Games, it's essential to consider the temporal, spatial and their collective impacts comprehensively. We first discuss the Temporal Dimension, then elaborate our considerations on Spatial Dimension, and finally summa- rize the design for the collective impacts from both temporal and spatial dimensions.
3.2.1 TEMPORAL DIMENSION; INFLUENCE TIME
Different from prior works, our work consider the temporal dimension of Feint impacts by emulating them in Dynamic Short-Long-Tens Term manner. The rationale behind such design choice is that: the purpose of Feint is to obtain strategic advantages against the opponent in temporal dimension, aiming to benefit following attacks. Hence, the Dynamic Short-Long- Term temporal impacts of
3

---

Under review as a conference paper at ICLR 2023
Feint shall be (I) the actions that follow Feint actions (e.g. actual attacks) in short-term period of time should have strong correlation to Feint (2) the actions in the long-term periods explicitly or implicitly depend the effect of the Feint and its following actions; and (3) for different Feint actions in different gamingg scenarios, the threshold that divides short-term and long-term should be dynamically adjusted to enable sufficient flexibility in strategy making.
For Dynamic Short- Shoort-onng-eraee we first set up a short-term planning threshold st to select the follow- up actions, which are decided by Feint policy n. at 4a- Note that actions { ..., afg+st) are strongly related to the Feint action di.- For the actions bounded by the short-term threshold, a set of large weights faio. fot } are used to calculate the reward:
I-ta+et
ReWshort-term to st, a) at
os
since these actions are expected to deliver higher reward (i.e. the purpose of Feint is to obtain strategic advantages) via the Feint action. We then consider long-term planning after the short-term planning threshold st: we use a set of discount factor B = (Bto-st+1:....B Br) on the long-term
average reward calculation (proposed by Kim et al. (2022) ). distinguish these reward from short- term rewards:
T
Rewlong -term (TT, to: st, T, = T.
Σ
t=ta+st+1
where T denotes the end time of the game.
Finally, we put them together to formalize the Short-Long- Term reward calculation mechanism, when an agent plans to perform Feint action at time to with short-term planning threshold st and the end time of game T as:
ReWtemporal (T. to, st, T. = Ashort Rewshort term (to, st, a++ Atong ReWlong term (to, st, T, T)
(3)
where Ashorz and Atong are weights for dynamically balancing the weight of short-term and long- term rewards for different gaming scenarios. Aghort and Avong are initially set as 0.67 and 0.33 and are adjusted to achieve better performance with the iterations of training.
3.2.2 SPATIAL DIMENSION: INFLUENCE RANGE
In a Multi- Game (i.c. usually more than two players), the strict one-to-one relationship be- tween two agents is not realistic, since an agent can impact both the target agent and other agents. Therefore, the influences to all other agents shall maintain different levels Liu et al. (202T). There- fore, our work includes the spatial dimension of Feint impacts by fusing spatial distributions. The key idea of this design is to combine spatial distribution with the influence range during the game. More specifically, we incorporate Behavioral Diversity from Liu et al. (202T), to mathematically calculate and maximize the diversity gain of Feint actions in terms of the influence range.
We formalize the influence range of an action policy on S X A X ... Ak. which follows a distribution of multi-to-one relationships T + (a.Ti..)) a2T (T))k(s))) The influence distribution can have different factors in different gaming scenarios. We demonstrate set of commonly used factors in boxing games Won et al.] (20216) where agent plays against opponent -i V (Af, A posilions(i, 1), orientationssi. -i), linear velocities(i, -i), angular velocities(i, i)), in which the factors represent the chosen action k of agent i, the chosen action of opponent -i, the relative positions, the relative moving orientations, the linear velocities and angular velocities of agent i and opponent -i. When a Feint policy πÝ added, we aim to maximize the effective influence range under the influence distribution of Feint Assuming the row agent i maintains a policy pool P. [T/ xMM,, }, such influence distribution can be fused into Behavior Diversity measurement of the effective influence range by maximizing the discrepancy between the old influence effectiveness of policy occupancy measure Prg (T) and the influence effectiveness when adding Feint policy of new policy occupancy Pxi (V'):
agent based on
4

---

Under review as a conference paper at ICLR 2023
maz, ReWssptial V') = =Dy(P,, (V') ll Pn (V))
(4)
where the general f-divergence is use to measure the discrepancy of two distributions.
3.2.3 COLLECTIVE IMPACTS: INFLUENCE DEGREE
Solely relying on Temporal Dimension and Spatial Dimension overlooks the interactions between them, and these two dimensions are expected have mutual influences for realistic modeling Liu al. (2021). Therefore, we consider the influence degree, so the collective impacts of these two dimensions can be aggregated in a proper manner.
We formulate the collective impacts for a Feint policy म; in a Multi-Playe Game that starts at to and end at 7 as:
k
ReWeollective (T,) fi
ReWtemporal (1, T2 to. st, a, P22 Σ max ReW (Tyy V', m²
(5) where temporal impacts Rewttm (Section 3.2.1 are aggregated on spatial domain and spa- tial impacts ReWssatial (Section 3.2.2) are aggregated on temporal domain. μ4 and μ2 denote the weights of aggregated temporal impacts and spatial impacts respectively, enabling flexible adap- tion to different gaming scenarios. They are initially set as 0.5 and are adjusted to achieve better performance with the iterations of training.
In addition to the collective impact of Feint itself terms of temporal domain and spatial domain, our formalized Feint impacts can also result in response diversity of opponents, since different lated opponents (spatial domain) at different time steps (temporal domain) can have diverse re- sponse. Such diversity be used reward factor that make the final reward calculation more comprehensive Nieves et al. (202T): Liu et al. (202T). Thus, to incorporate such diversity together with final reward calculation model, refer Liu al. (2021) characterize the diversity gain incurred by our collective impact formalization. When the impact ReWcollective of Feint pol- icy π +11 in M x N payoff matrix AP, at when opponents choose policy x² is collectively calculated, the derived diversity gain can be measured as follows:
ReWeollective -aniteeed-vvvveee diversity (TT
(T. (TM++)) D(aM+1 |l Ap, AP.)..)
(6)
T (Rewealeettve (T. M+1 ==)))
(7)
where D(a || Ap, xb represents the diversity gain of the Feint action on current policy space. We follow the method in Liu et al. (021) for the quantification of diversity gain.
4 FEINT IMPLEMENTATIONS
4.1 IMPLEMENTING THE FORMALIZATION AS A COLLECTIVE REWARD CALCULATION
To provide comprehensive reward calculation model for Feint in Multi-Player Games, synthe- size the above collective impacts and collective diversity gain into the overall Reward Calculation Model. The robustness of similar design idea is proved in Liu et al. (20TT) that the synthesised direct impacts and diversity gain can provide a more comprehensive reward calculation model for each player. Thus, we synthesize the collective impacts (Equation ( and collective diversity gain (Equation 6) for Feint policy т; into the overall Collective Reward Calculation Model by applying weighted sum \ of collective impact and Xa of collective diversity gain:
Rew ( A1 ReW (x)) + X2 divertieeeddieeitteee (T))
(8)
4.2 FOR INFERENCE LEARNING MODELS
Inference Learning module used to predict whether an observed action is Feint or not and policy parameters of 0-1 and policy dynamics U-t Model-based approaches explicit model to

---

Under review as a conference paper at ICLR 2023
fit agent with the learning strategies of other agents from the observation Kim al. (2021) but often suffer from the infinite recursion problem when an agent the model models the agent it self Tesauro (2003). Blei al. (2016) proposes model-free approach using approximate variational inference and Kim et al. (2022) optimizes a tractable evidence lower bound to infer accurate latent strategies of others. We add random weight onto the ELBO fit the randomness and uncertainty incurred by Feint actions, Specifically, the random-weighted ELBO is defined together with an encoder p(2 | To::: Penci Tenc) and a decoder p(a,"' St, 2, deei Idee) parameterized by a set of encoder and corresponding random weighted decision parameters (d'nci Venci and set of decoder and corresponding random weighted decision parameters (6'eci Tdec):
Ja pp(r,.) Idd. io
log p(a 186 a Odeci Ydee)
(9)
k=
D DKz(p(ETt Tous Penci Yene) p(2x
where s are latent strategies that represent inferred policy parameters of other agents 0,* and The {so, ap: Og ,r' ,....., 8 denotes i's trajectories up to timestep t. The random weight param- eters Venc and "dee enable agents to randomly guess the probability whether an action a Feint action not from the observations, since no inferred policy can properly select a Feint action.
4.3 FOR REINFORCEMENT LEARNING MODELS
Reinforcement Learning module updates the policy using various gradient ascending mechanisms Lowe et al. (2017); Iqbal & Sha (209)); Yu et al. (202T). Although these designs are different, one of the key components in policy updating is the reward function Q, which directly evaluates the reward of the policy and guides the policy updating process. Our proposed reward calculation mechanism (Section 33 can thus be fused into the MARL model by replacing the Q functions. Such replacement directly provide temporal, spatial and their collective considerations in policy making and keep the main structures of current MARL models. Thus, Feint actions can be effectively fused into policy learning process current MARL models based on our reward calculation mechanism. We demonstrate the feasibility of fusion using two state-of-the-re MARL models, MADDPG Lowe al. 2017) and MAAC Iqbal Sha (2019) below.
In a K K-agent game, agents {aI...., } maintain policies π { T parameterized by 0 {0).... ..., }x). where the policies for other agents are inferred from the Inference Learning module 4.2 Each agent evaluates the expected reward Q (s, a1...., an) at state s using reward term Rew (s)
Equation g
In MADDPG Lowe et al. (2017) model, centralized critic (s, ai......., an) is used on decentralized execution to provide global information that can stabilize training. The centralized critic can be replaced by our reward calculation mechanism and the gradient descent of the policy learning is:
Jo. E, pi n, Vo, log (a: 0.) Rew
(10)
where Rew (s) is naturally fused into the gradient update function and provide influence range, influence degree and influence time length for adjusting learning outcomes.
In MAAC model Iqbal & Shal (2019), although the policy update mechanisms vary, the key part of policy updates is still the Q (8, (5,,),,,,, an)) function, which can also be replaced by our reward calculation term. The slight difference that MAAC uses temperature parameter a to balance between the entropy and rewards. Therefore, the gradient ascent for updating policy is:
Ve, = E. pn xy V log (-) ,q(a; o;) + Rew' (s))
(1I)
where Rew'(9 (s) replace the original QT(s di,..., a,) function and the multi-agent advantage func- tion b(o, a.), since the expected return multi-action interactions are already comprehensively cal- culated in Rew (s). Thus, proposed reward calculation be seamlessly fused into of the-art MARL models with simple replacement of the original Q-function while guaranteeing the feasibility.

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTAL METHODOLOGY
Testbed Implementations. We implement two complex Multi-player Games to examine the ef- fectivenss of our design. We first re-implement and extend strategic real-world game, AlphaS- tar Arulkumaran et al. (2019), which is widely used as the experimental testbed in recent studies of Reinforcement Learning studies Risi & Preuss (2020); Liu et al. (2021). We make extra ef- forts to emulate a six-player game, where players are free to have convoluted interactions with each other. And we implement Feint as dynamically generated policies, based on the 888 regular gaming policies. Then, we create a complex multi-player tagging game, based on Multi-Agent Particle En- vironment Mordatch & Abbeel (2017). open-source environment from OpenAl. We handcraft tagging game scenario, where six agents can freely fight with each other with 30 nuanced and flexi- ble actions. Such an implementation requires an extensive amount of efforts since current available codebases only have a limited set of actions, which are insufficient to demonstrate the impacts of Feint We follow the methodology from Wampler et al. (2010); Won et al. (2021a) to form Feinr actions, based on 30 hand-craften actions. Our tagging game resembles intense free fight scenar- ios ancient Roman free fight scenarios Matz 2019), where interactions are intense and Feint is expected to be effective.
Experiment Procedure. We choose MADDPG[Lowe al. (2017) and MAAC & Sha (2019) model in our experiments. We first train all six agents without Fein! from our formalization on the state-of-the-art MARL models. Then we randomly select 3 agents (always labeled as Agent 1, 2, and 3), who incorporate formalization of Feint, and keep the other 3 agents regular. All experiments are done with 4,000 training iterations on each model and 150 gaming iterations.
Evaluation Metrics. We examine the effects of Feint using O gaming rewards of training, di- versity gain of policy space and 2 overhead of computation load. We first examine the learning outcomes (i.e. rewards) trained using both MADDPG and MAAC MARL. model, by comparing the rewards of agents across all scenarios. We then examine the effects of Feins actions on how Feint can improve the diversity of gaming policies (Section 3}. Finally, we perform overhead analysis, incurred by fusing Feint formalization in strategy learning.
EXPERIMENTAL RESULTS
6.1 GAMING REWARD GAIN
Figure I shows the rewards for each agent in two scenarios using MADDPG model. We make three observations. First, in O in Figure when no Feint is enabled, all agents' rewards tend to progress to a similar level when after enough training iterations. However, in 1 in Figure I when Feint actions are enabled on agent 1, 2, and 3, these agents gain significantly higher rewards than the agents who are not enabled to perform Feint actions (agent 4, 5, and 6). Second, when comparing O with @, the average rewards for agents who perform Feint actions (agent 1, 2, and in 2) is around 9.5, which is higher than the average rewards (around 5.0) for agents who do not perform Feint actions (all agents in 1). These two observations demonstrate that our formalized Feint can provide effective improvement for agents' rewards. Third, the training results before 1000 iterations are not stable for both scenarios. This is mainly because the natural characteristics of MADDPG training process in which the training stables generally after 2000 iterations Lowe et al. (0177).
Figure 2 shows the rewards for each agent in two scenarios using MAAC model. When comparing to the results trained with MADDPG model, though the rewards differ in specific numbers, similar trends and observations are shown. First, when no Feint are enabled, all agents' rewards tend to progress to a similar level when after enough training training iterations (0 in Figure while when Feint actions are enabled on agent 1, 2, and 3, these agents gain significantly higher rewards than the agents who are not enabled to perform Feint actions (agent 4, 5, and 6) (@ in Figure Second, when comparing O with @, the average rewards for agents who perform Feint actions (agent 1, 2, and 3 in @ is around 10.0, which is higher than the average rewards (around 3.5) for agents who do not perform Feint actions (all agents in 0). And third, training in MAAC also suffers unstable results before the first 1000 iterations. With the comparison of the results for MADDPG and MAAC model, we have an addition observation that our formalized reward caleulation for Feint actions and the MARL fusion can be effectively adapted on current ofate-of-the-an art MARL models, providing promising feasibility and scalability for extension studies.

---

Under review as a conference paper at ICLR 2023
KADOPG wihhout Feint
NADORS with Feint
Agenz3
Agenz3
15.0
Agenb4
Aganbi
22.5
Agents
Agenes
Howardd for Eacth
Agends
Agen60
1500
2000
2003
4000
5008
1000
2000 Thainng Berations
3000
4000
5009
Thaining Beration
Figure 1: Reward for each agent in two scenarios trained by MADDPG. shows rewards for the first scenario where agents are enabled Feint actions. shows rewards for the second scenario where agent 1, 2, and are enabled Feint actions while agent 4, 5, and 6 are not enabled Feint actions.
MAAE without
MAAC with twint
17.
15.0
Agertt
15.0
Resond for Esch lagent
Agentd Acentt
Agwn1
20.0
Apurdd
Aqundé
25
1000
3000
Traniig Beration
Traniing torations
Figure 2: Reward for each agent in two scenarios trained by MAAC. 1 shows rewards for the first scenario where agents are not enabled Feint actions. @ shows rewards for the second scenario where agent 1, 2, and 3 are enabled Feint actions while agent 4, 5, and 6 are not enabled Feint actions.
6.2 DIVERSITY GAIN
To examine the impacts the policy diversity in games, we perform comparative study between MARL training with and without Feint Specifically, We use Exploitability and Population Efficacy (PE) to measure the diversity gain in the policy space. Exploitability Lanctot et al. (2017) measures the distance of a joint policy chosen by the multiple agents to the Nash equilibrium, indicating the gains of players compared to their best response. The mathematical expression of Exploitability is expressed as:
N Expl(T)= = E (mao Rew;(T, n;)) - Rew,(T, i-)))
(12)
where т; stands for the policy of agent and т-; stands for the joint policy of other agents. Rew; denotes formalized Reward Calculation Model (Section 4.1). Thus, small Exploitability values show that the joint policy is close to Nash Equilibrium, showing higher diversity. In addition, we also Population Efficacy (PE) Liu al. (202T) to measure the diversity of the whole policy space. PE is a generalized opponent- free concept of Exploitability by looking for the optimal aggregation in the worst cases, which expressed as:
N
PE([= = min= max a=1 a:>=0
ae Rew; T-))
(13)
where π. stands for the policy of agent and .- stands for the joint policy of other agents. a denotes an optimal aggregation where agents owning the population optimizes towards. Rew, denotes formalized Reward Calculation Model (Section[4.1) and opponents can search over the entire policy space. PE gives more generalized measurement of diversity gain from the whole policy space.
8

---

Under review as a conference paper at ICLR 2023
Figure shows the experimental results for evaluating diversity gains. From the figure, obtain two observations. First, agents that can dynamically perform Feint actions (Agent 1, 2, and 3) achieve lower Exploitability (around 4.9 10 3) compared agents who perform regular actions (around 9.7 x 10-2) and have higher PE (lower negative PE around 5.3 x 10-2) than those who only perform regular actions (around 1.2 10-2). This result shows that formalized Feint can effectively increase the the diversity and effectiveness of policy space. Second, agents with Feint have slightly higher variations in both metrics. This is because Feint naturally incurs more randomness (e.g. succeed or not) in games, resulting in higher variations in metrics.
Apert
10
Ager3
Sagddee mopuater Elfecitty
Agantt
121
20
60
190
120
20
Comming
Figure 3: The difference for each agent, in terms of 1 the exploitablity; and @ the negative popula-
tion efficacy.
6.3 OVERHEAD ANALYSIS
Figure shows the results of our overhead analysis. We make two observations. First, fusing Feint in MARL training do incur some overhead increment in terms of running time. This is because the formalization and fusion of Feint in MARL incur additional calculation load. Secondly, in both MADDPG models and MAAC models, the increased overhead is generally lower than 5%, which still indicates that our proposed formalization of Feint actions have enough feasibility and scalability on fusing with MARL models.
Cvarhoad Incrument
Overhcad Incemmntt
Doerheed Increment
4.0%%
3.85
3.5%
8.0%
3.8%
3.2%
500
2000
fur
Marations
Figure 4: Overhead of Feint in MADDPG; and @ MAAC models.
7 CONCLUSION
We present the first formalization, implementation and quantitative evaluations of Feint in Multi- Player Games, Our work formalizes. implements and quantitatively examines Feint in Multi-Player Games, on the temporal, spatial and their collective impacts. The results show that our design of Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi- Games: and (3) only incur negligible overheads in terms of the time consumption. We conclude that our design of Feint is effective and practical, to make Multi- Multi-Player Games more interesting. Our design is also expected to be applicable for future models of Multi-Pi Player Games.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: an evolutionary computation perspective. In Manuel López-lbáñez, Anne Auger, and Thomas Stützle (eds.), Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO 2019, Prague, Czech Republic, July 13-17, 2019, pp. 314-315, ACM, 2019. doi[0.11465/33196112222333 URL
https: /ooi org.
3321894
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe, Variational inference: A review for statisti- 601 00670
cians. CoRR, abs/160) 600600070, 2016.
Lucas Critch and David Churchill. Sneak-attacks in starcraft using influence maps with heuristic search. In 202/ IEEE Conference on Games (CoG), Copenhagen, Denmurk, August 17-20, 2021, URI. /ooi http:/doo.oogg////
pp. 1-8. IEFE, 2021. doi 21199/C0G5622222211.00
1109/CoG52621 2021 9619156
Jerzy A Filar. Estimation of strategies in markov game. Naval Research Logistics Quarterly, 23 (3):469-448) 1976.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.). Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2974-2982 AAAI Press, 2018. URL https: http//wwww..m aaai............................ orgg/
fess/index.oo hwtpppppppa/paaaaap TAAAAAAAAAAAAA pwwppaapppppeaaa wttpa//////aappee.eea/aaaaaaaa// vi1./11193
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause ), Proceedings of the 35th Inter- national Conference
Machine Learning, ICMI. 2018, Stockholmsmäissan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018. URL http: http:://pooceeinggg mir. wtt.e/eeee///////eeeeeeee
html
Iris Güldenpenning, Wilfried Kunde, and Matthias Weigelt. How to trick your opponent: A review article on deceptive actions in interactive sports. Frontiers psychology, 8:917, 2017.
Iris Güldenpenning, Mustafa Alhaj Ahmad Alaboud, Wilfried Kunde, and Matthias Weigelt. The impact of global and local context information on the processing of deceptive actions in game sports. German Journal of Exercise and Sport Research, 48(3):366 305))5,66775,,00 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Reseurch, pp. 1856-1865. PMLR, 2018. URI. http::ttp///. ptttt::/prrceeel/ ngs. m1 wtwppesss/v0//eeeee... ja Bb. html
Ray Hyman. The psychology of deception. Annual review of psychology, 40(1):133-15: 1989.
Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2961-2970. PMLR, 2019. URL http:// proceedings. www..............mm..m..mmm.ee.e /vt////vv////// /tt///gallll1 19...9...... html
Dong-Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5541-5550. PMLR, 2021. URL httphtt:://prreeedd ngs ml wrw.s v9 m21 g html
10

---

Under review as a conference paper at ICLR 2023
Dong- Ki Kim, Matthew Riemer, Miao Liu, Jakob N. Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonatham P. How. Influencing long- term behavior in multiagent reinforcement learning. CoRR, wwwp/220.../3//3322 2022. doi 10.485500mmx..22233...5
org 10. 40.48550/axX0 2203. 03535.
Mare Lanctot, Vinícius Flores Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4190-4203, 2017. URL /BR.httpss///rrccedinggg.go neurips CC/ httppaper//211/heee 332 33235 Abstrsaa-bbbrrrttt html
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico May 2-4, 2016, Conference Track Proceedings, 2016. AtLhttp: //axxiv. org. abs 1509. 02971
Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang. Yujing Hu, Yingfeng Chen, Changjie Fan, and Zhipeng Hu. Unifying behavioral and response diversity for open-ended learning in zero-sum games. CoRR, 0es/2106.04958. 2021. http:/tsss//axii.orr/////// /bwp///////////// 21 06. 04958
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperatieecooppettie environments. In Isabelle Guyon, Ulike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6379-6390, 2017. 2aw000333aa1111866eeeeeeee63e3
neurips cc/papprr/211/hasttee Abstract html
David Matz. Ancient Roman Sports, AZ: Athletes, Venues, Events and Terms. McFarland, 2019.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.55022, 2013.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, arXiv preprint /rX3v: 04908, 2017.
Abhishek Naik, Roshan Shariff, Niko Yasui, and Richard S. Sutton. Discounted reinforcement learning is not an optimization problem. CoRR, abs/1910.021140, 2019, URL http::/arxxix htg/abs/1910 02140
Nicolas Perez Nieves, Yaodong Yang, Oliver Slumbers, David Henry Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 1021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8514 8524. PMLR, 2021. URL httt::::///ocoeedggggg.... mlr. hti..ppees//v/////// perez- sineesnnieeeslla..ie
Chris Nota and Philip S. Thomas. Is the policy gradient a gradient? In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke-Smith (eds.), Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '20, Auckland, New Zealand, May 9-13, 2020, pp. 939-947. International Foundation for Autonomous Agents and Multiagent Systems, 2020. URL / //l acm. org/ http://o//aassaae/ac/oo 05151..5555511
/ttp://///./////..55533333331111 03115/33388111133877777 3398871
Sebastian Risi and Mike Preuss. Behind deepmind's 's alphastar ai that reached grandmaster level in starcraft ii. KI- Ki-iitttle Intelligenz, 34(1):85-88 2020.

---

Under review as a conference paper at ICLR 2023
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, fos/1707.06337,, 2017. URL httt:://arxi.oore abs/
Oswin So, Kyle Stachowicz, and Evangelos A. Theodorou. Multimodal maximum entropy dynamic /2001. 12925
games. CoRR, abs/2201.1292.. 2022. URL
ToalhaNerd Team. Toalha nerd-king of fighter xv: Trailer de ryo sakazaki e robert garcia! 2021.
Gerald Tesauro. Extending q-learning to general adaptive multi-agent systems. In Sebas- tian Thrun, Lawrence K. Saul, and Bernhard Schölkopf (eds.), Advances Neural Infor- mation Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, De- cember 8-13, 2003, Vancouver and Whistler, British Columbia, Canada), pp. 871-878. MIT Press, 2003. URL htpp:ptppppprccce..gggsssss neurips.........p.e............ cc/papprr22033/aa... e e71e5cd11 o4 Abstract html
Kevin Wampler, Erik Andersen, Evan Herbst, Yongjoon Lee, and Zoran Popovic. Character animation in two-player adversarial games. ACM Trans. Graph., 29(3):26:1 2010. wtt..//.....5111052080.111 000.8855554111199790 1805970
18059700 URL[tttss:
Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. Control strategies for physically simulated characters performing two-player competitive sports. ACM Trans. Graph., 40(4):
146:1-146:1, 2021a. doi 10.114534500266.459771 [RLlhttps://cc.. http://tt..org/1.1111111 0150626.335977
Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. Control strategies for physically simulated characters performing two-player competitive sports. ACM Trans. Graph., 40(4):
146: 146:1-146:11 11, 2021b. (or:10.1145/350066..49999 URL https://dcci otg/10.11100 3450626. 3459761
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
12


Reviews:
[{"review_id": "r1g64gSi4Q", "reviewer": "Reviewer_b6NC", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method \"Feint\" to enable agents in multi-agent reinforcement learning to exhibit behaviors analagous to feints in human games; i.e. deceptive actions that are intended to look like an attack to draw a response from other agents.", "summary_of_the_review": "\nBased upon the above comments concerning originality and clarity, I will recommend to reject the paper.", "strength_and_weaknesses": "# Strengths\n- The notion of feinting is interesting to consider in a multi-agent reinforcement learning environment.\n\n# Weaknesses\n\n- The notion of feint is never formally defined. So it is unclear exactly what behavior is sought from the agents, and how it differs from standard behavior. For example, if agents are learning optimal policies, wouldn't such policies include feinting if indeed it was beneficial?\n- The proposed method to induce feinting is a simple modification of the reward function to emphasize near-term rewards over longer-term ones. Again, it is unclear how this is meant to encourage feints.\n- The experimental results show that agents with the feinting behavior receive higher rewards. But, that is because their reward functions were directly modified to give different (larger values) to near-term rewards. So this is expected and an artifact of the definition -- it does not show that these agents actually have a competitive advantage, as far as I can tell.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "eytL4tO19K", "reviewer": "Reviewer_ebuk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "If I understand this paper correctly, the authors introduce a way how to optimize for integrating “feint” actions into the solutions of stochastic-game-like interactions to promote diversity. They do so by gradually building up the criterion function from three aspects they consider the most important, and refer to them as “temporal”, “spacial”, and “collective” impacts. The first half of the paper is dedicated to formal mathematical formulation of each of these aspects. The authors then show how to integrate their build up reward function into reinforcement-learning algorithms, or feint recognition models. The last part of the paper presents empirical results achieved with agents using “feints”, examining gains in utilities, exploitability and diversity, and computational overheads incurred when optimizing for “feint” inclusion. ", "summary_of_the_review": "This paper presents a very interesting idea of including feints into agents’ strategies in multiplayer games, yet I feel the exposition need to be substantially improved for this paper to be considered at a conference like ICLR.", "strength_and_weaknesses": "The idea of examining how bluffing in multi-player games may influence the game-play seems indeed compelling to me, yet I have to admit I find this work rather confusing. First, I believe the motivation could be greatly improved. Are the authors attempting to integrate feints, how they say, to “improve game experiences”? So do they imagine the applications of their work to be in the video-game industry, for example? Or is the inclusion of feints targeted more towards strategic robustness when multiple agents stray too far from the equilibrial paths, as examining reward gains in the experimental section seem to hint? The authors mention some “need for diversity” yet I do not entirely understand where this need comes from.\n\nSecond, I believe my confusions stems partly also from the language the authors are using. At times, I found myself going through some parts of the text multiple times because I felt like I understood the individual words, yet the sentences did not make much sense as they were worded together. One of the examples is “... [Naik et al.] … simply shrink smaller weights and deliver more future actions,” which I am not sure how to interpret. Overall, the text seem maybe a bit too ambiguous, and some more concrete or detailed explanation of, e.g., what temporal or spacial impact are could greatly help the readers to familiarize with these concepts and why the authors find them so important.\n\nThis initial exposition seems especially important in the light of the seemingly trivial integration of the authors’ concept into the feint detection or multiagent reinforcement-learning models. \n\nI also lack some intuition why feints would help gaining more utility or achieve lower exploitability agains agents who are not using feints, as the experimental section seems to suggest. If I understanding the feints correctly, as explained in section 3.2.1, the bluffing agents sacrifice longterm full rationality for the possibility of obtaining short-term high rewards. Are the higher gains reported in the empirical evaluation the result of the agents not using feints being at this points still too far from some equilibrium? Or is it because there are multiple feinting agents in the environment, so the deviations are not unilateral?\n\nOverall, I think the idea this paper presents is very interesting, but I feel the presentation needs to be further improved before I can support this paper's acceptance. \n\nNits:\nMethods in 3.1 do not seem to relate to feint at all.\nWhat are “Non-People Characters”? Do you mean “non-playable characters”?\n“Multi-to-on”, maybe you mean “many-to-one”?\n“we synthesis” to “we syntesize” \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UXk9RqWD4W", "reviewer": "Reviewer_r9R4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed work intends to improve the diversity of multi-agent games by introducing a comprehensive Feint formalization in the combination of ‘temporal’, ‘spatial’, and ‘collective impact’; Despite some minor concerns about the experiments, the proposed method Feint’s effectiveness was proven through the overhead evaluation, which cost a relative low time-consumption to realize the diversity boosting and reward increase. ", "summary_of_the_review": "The work proposed an effective way to encourage the depiction of comprehensive feint, from spatial, temporal, and most innovative part, collective impact sides. According to the author’s explanation, the overall logic is clear and rational, however, as per the weakness, if the author could address the above concerns, the work will be more convincing.", "strength_and_weaknesses": "Strength:\n\n- The proposed work’s motivation is explicit, the statement clearly proposed the problems to be solved and how to support the influence.\nThe authors pay effort into the multi-agent gaming’s feint’s formalization and tried to solve the problem in a more realistic scenario inspired by the previous two-agent feint work.\n\n - It is proven to be low timing cost while increasing the agent players’ diversity, encouraging it’s the possibility of deployment to other multi-agent tasks.\n\n\nWeakness: \n\n- For the results shown in Figure, are the second sub-figure guaranteed to be converged? It seems that the continuous tendency might result in a worse reward. Maybe the comparison of the results after a longer training iteration would be more convincing.\n\n- The paper didn’t explicitly explain whether the two params “μ1 and μ2” are manually set weights or learned weights in Section 3.2.3. The same confusion also exists in part 4.1: ​​λ1 and λ2. If the parameters are empirical values, it’s better to be provided in the paper for better instruction on the method’s implementation.\n\n- It’s stated in the experiments part that the environment Mordatch & Abbeel (2017), alphaStar, etc. were realized and tested to justify the Feint’s effectiveness. If the real scene of gaming can be provided in a case study or presentation video, the performance of diversity would be better verified.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject